http://blog.gmane.org/gmane.linux.gentoo.portage.devel hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3468 <pre>Sometimes people wrap long lines in their ebuilds to make it easier to read, but this causes us issues when doing line-by-line checking. So automatically unroll those lines before passing the full content down to our checkers. This seems to work, but maybe someone can suggest something simpler. Signed-off-by: Mike Frysinger &lt;vapier&lt; at &gt;gentoo.org&gt; --- pym/repoman/checks.py | 35 +++++++++++++++++++++++++++++++++++ 1 files changed, 35 insertions(+), 0 deletions(-) diff --git a/pym/repoman/checks.py b/pym/repoman/checks.py index 65e7136..67f2b0a 100644 --- a/pym/repoman/checks.py +++ b/pym/repoman/checks.py &lt; at &gt;&lt; at &gt; -750,11 +750,46 &lt; at &gt;&lt; at &gt; _ignore_comment_re = re.compile(r'^\s*#') def run_checks(contents, pkg): checks = _constant_checks here_doc_delim = None +multiline = None for lc in checks: lc.new(pkg) for num, line in enumerate(contents): +# Unroll multiline escaped strings so that we can check things: +#inherit foo bar \ +#moo \ +#cow +# This will merge these lines like so: +#inherit foo bar moo cow +try: +# A normal line will end in the two bytes: &lt;\&gt; &lt;\n&gt;. So decoding +# that will result in python thinking the &lt;\n&gt; is being escaped +# and eat the single &lt;\&gt; which makes it hard for us to detect. +# Instead, strip the newline (which we know all lines have), and +# append a &lt;0&gt;. Then when python escapes it, if the line ended +# in a &lt;\&gt;, we'll end up with a &lt;\0&gt; marker to key off of. This +# shouldn't be a problem with any valid ebuild ... +line_escaped = (line.rstrip('\n') + '0').decode('string_escape') +except: +# Who knows what kind of crazy crap an ebuild will have +# in it -- don't allow it to kill us. +line_escaped = line +if multiline: +# Chop off the \ and \n bytes from the previous line. +multiline = multiline[:-2] + line +if not line_escaped.endswith('\0'): +line = multiline +num = multinum +multiline = None +else: +continue +else: +if line_escaped.endswith('\0'): +multinum = num +multiline = line +continue + # Check if we're inside a here-document. if here_doc_delim is not None: if here_doc_delim.match(line): </pre> Mike Frysinger 2012-05-24T04:06:39 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3464 <pre>Rather than copying &amp; pasting the same behavior for the different eclass checks, add a common class for them to extend. This makes adding more eclass checks trivial, and keeps down bitrot. This does abuse the checking interface slightly -- the eclass will change its category between unused and missing based on the checks. URL: https://bugs.gentoo.org/417159 URL: https://bugs.gentoo.org/417231 Signed-off-by: Mike Frysinger &lt;vapier&lt; at &gt;gentoo.org&gt; --- bin/repoman | 6 +- pym/repoman/checks.py | 143 ++++++++++++++++++++++++++++++++----------------- pym/repoman/errors.py | 1 - 3 files changed, 97 insertions(+), 53 deletions(-) diff --git a/bin/repoman b/bin/repoman index 3697403..d7ffcdd 100755 --- a/bin/repoman +++ b/bin/repoman &lt; at &gt;&lt; at &gt; -315,8 +315,9 &lt; at &gt;&lt; at &gt; qahelp={ "file.size.fatal":"Files in the files directory must be under 60 KiB", "file.name":"File/dir name must be composed of only the following chars: %s " % allowed_filename_chars, "file.UTF8":"File is not UTF8 compliant", -"inherit.autotools":"Ebuild inherits autotools but does not call eautomake, eautoconf or eautoreconf", "inherit.deprecated":"Ebuild inherits a deprecated eclass", +"inherit.missing":"Ebuild uses functions from an eclass but does not inherit it", +"inherit.unused":"Ebuild inherits an eclass but does not use it", "java.eclassesnotused":"With virtual/jdk in DEPEND you must inherit a java eclass", "wxwidgets.eclassnotused":"Ebuild DEPENDs on x11-libs/wxGTK without inheriting wxwidgets.eclass", "KEYWORDS.dropped":"Ebuilds that appear to have dropped KEYWORDS for some arch", &lt; at &gt;&lt; at &gt; -382,7 +383,6 &lt; at &gt;&lt; at &gt; qahelp={ "ebuild.majorsyn":"This ebuild has a major syntax error that may cause the ebuild to fail partially or fully", "ebuild.minorsyn":"This ebuild has a minor syntax error that contravenes gentoo coding style", "ebuild.badheader":"This ebuild has a malformed header", -"eprefixify.defined":"The ebuild uses eprefixify, but does not inherit the prefix eclass", "manifest.bad":"Manifest has missing or incorrect digests", "metadata.missing":"Missing metadata.xml files", "metadata.bad":"Bad metadata.xml files", &lt; at &gt;&lt; at &gt; -425,7 +425,7 &lt; at &gt;&lt; at &gt; qawarnings = set(( "ebuild.badheader", "ebuild.patches", "file.size", -"inherit.autotools", +"inherit.unused", "inherit.deprecated", "java.eclassesnotused", "wxwidgets.eclassnotused", diff --git a/pym/repoman/checks.py b/pym/repoman/checks.py index 77df603..088a916 100644 --- a/pym/repoman/checks.py +++ b/pym/repoman/checks.py &lt; at &gt;&lt; at &gt; -331,24 +331,6 &lt; at &gt;&lt; at &gt; class EbuildQuotedA(LineCheck): if match: return "Quoted \"${A}\" on line: %d" -class EprefixifyDefined(LineCheck): -""" Check that prefix.eclass is inherited if needed""" - -repoman_check_name = 'eprefixify.defined' - -_eprefixify_re = re.compile(r'\beprefixify\b') -_inherit_prefix_re = re.compile(r'^\s*inherit\s(.*\s)?prefix\b') - -def new(self, pkg): -self._prefix_inherited = False - -def check(self, num, line): -if self._eprefixify_re.search(line) is not None: -if not self._prefix_inherited: -return errors.EPREFIXIFY_MISSING_INHERIT -elif self._inherit_prefix_re.search(line) is not None: -self._prefix_inherited = True - class NoOffsetWithHelpers(LineCheck): """ Check that the image location, the alternate root offset, and the offset prefix (D, ROOT, ED, EROOT and EPREFIX) are not used with &lt; at &gt;&lt; at &gt; -464,43 +446,104 &lt; at &gt;&lt; at &gt; class InheritDeprecated(LineCheck): (eclass, replacement) del self._indirect_deprecated -class InheritAutotools(LineCheck): -""" -Make sure appropriate functions are called in -ebuilds that inherit autotools.eclass. +class InheritEclass(LineCheck): """ +Base class for checking for missing inherits, as well as excess inherits. -repoman_check_name = 'inherit.autotools' -_inherit_autotools_re = re.compile(r'^\s*inherit\s(.*\s)?autotools(\s|$)') -_autotools_funcs = ( -"eaclocal", "eautoconf", "eautoheader", -"eautomake", "eautoreconf", "_elibtoolize") -_autotools_func_re = re.compile(r'\b(' + \ -"|".join(_autotools_funcs) + r')\b') -# Exempt eclasses: -# git - An EGIT_BOOTSTRAP variable may be used to call one of -# the autotools functions. -# subversion - An ESVN_BOOTSTRAP variable may be used to call one of -# the autotools functions. -_exempt_eclasses = frozenset(["git", "subversion"]) +Args: +_eclass: Set to the name of your eclass. +_funcs: A tuple of functions that this eclass provides. +_comprehensive: Is the list of functions complete? +_exempt_eclasses: If these eclasses are inherited, disable the missing + inherit check. +""" def new(self, pkg): -self._inherit_autotools = None -self._autotools_func_call = None -self._disabled = self._exempt_eclasses.intersection(pkg.inherited) +self.repoman_check_name = 'inherit.missing' +self._inherit_re = re.compile(r'^\s*inherit\s(.*\s)?%s(\s|$)' % self._eclass) +self._func_re = re.compile(r'\b(' + '|'.join(self._funcs) + r')\b') +# We can't use pkg.inherited because that tells us all the eclass that +# have been inherited and not just the ones we inherit directly. +self._inherit = False +self._func_call = False +if '_exempt_eclasses' in dir(self): +self._disabled = self._exempt_eclasses.intersection(pkg.inherited) +else: +self._disabled = False def check(self, num, line): -if self._disabled: -return -if self._inherit_autotools is None: -self._inherit_autotools = self._inherit_autotools_re.match(line) -if self._inherit_autotools is not None and \ -self._autotools_func_call is None: -self._autotools_func_call = self._autotools_func_re.search(line) +if not self._inherit: +self._inherit = self._inherit_re.match(line) +if not self._inherit: +if self._disabled: +return +s = self._func_re.search(line) +if s: +self._func_call = True +return '%s.eclass is not inherited, but "%s" found at line: %s' % \ +(self._eclass, s.group(0), '%d') +elif not self._func_call: +self._func_call = self._func_re.search(line) def end(self): -if self._inherit_autotools and self._autotools_func_call is None: -yield 'no eauto* function called' +if self._comprehensive and self._inherit and not self._func_call: +self.repoman_check_name = 'inherit.unused' +yield 'no function called from %s.eclass; please drop' % self._eclass + +class InheritAutotools(InheritEclass): +_eclass = 'autotools' +_funcs = ( +'eaclocal', 'eautoconf', 'eautoheader', +'eautomake', 'eautoreconf', '_elibtoolize', +'eautopoint' +) +_comprehensive = True + +# Exempt eclasses: +# git - An EGIT_BOOTSTRAP variable may be used to call one of +# the autotools functions. +# subversion - An ESVN_BOOTSTRAP variable may be used to call one of +# the autotools functions. +_exempt_eclasses = frozenset(['git', 'subversion']) + +class InheritEutils(InheritEclass): +_eclass = 'eutils' +_funcs = ( +'estack_push', 'estack_pop', 'eshopts_push', 'eshopts_pop', +'eumask_push', 'eumask_pop', 'epatch', 'epatch_user', +'emktemp', 'edos2unix', 'in_iuse', 'use_if_iuse', 'usex', +'makeopts_jobs' +) +_comprehensive = False + +class InheritLibtool(InheritEclass): +_eclass = 'libtool' +_funcs = ( +'elibtoolize', +) +_comprehensive = True + +class InheritPrefix(InheritEclass): +_eclass = 'prefix' +_funcs = ( +'eprefixify', +) +_comprehensive = True + +class InheritToolchainFuncs(InheritEclass): +_eclass = 'toolchain-funcs' +_funcs = ( +'gen_usr_ldscript', +) +_comprehensive = True + +class InheritUser(InheritEclass): +_eclass = 'user' +_funcs = ( +'enewuser', 'enewgroup', +'egetent', 'egethome', 'egetshell' +) +_comprehensive = True class IUseUndefined(LineCheck): """ &lt; at &gt;&lt; at &gt; -679,8 +722,10 &lt; at &gt;&lt; at &gt; _constant_checks = tuple((c() for c in ( EbuildHeader, EbuildWhitespace, EbuildBlankLine, EbuildQuote, EbuildAssignment, Eapi3EbuildAssignment, EbuildUselessDodoc, EbuildUselessCdS, EbuildNestedDie, -EbuildPatches, EbuildQuotedA, EapiDefinition, EprefixifyDefined, -ImplicitRuntimeDeps, InheritAutotools, InheritDeprecated, IUseUndefined, +EbuildPatches, EbuildQuotedA, EapiDefinition, +ImplicitRuntimeDeps, InheritAutotools, InheritDeprecated, InheritEutils, +InheritLibtool, InheritPrefix, InheritToolchainFuncs, InheritUser, +IUseUndefined, EMakeParallelDisabled, EMakeParallelDisabledViaMAKEOPTS, NoAsNeeded, DeprecatedBindnowFlags, SrcUnpackPatches, WantAutoDefaultValue, SrcCompileEconf, Eapi3DeprecatedFuncs, NoOffsetWithHelpers, diff --git a/pym/repoman/errors.py b/pym/repoman/errors.py index 3209243..c515502 100644 --- a/pym/repoman/errors.py +++ b/pym/repoman/errors.py &lt; at &gt;&lt; at &gt; -19,7 +19,6 &lt; at &gt;&lt; at &gt; EAPI_DEFINED_AFTER_INHERIT = 'EAPI defined after inherit on line: %d' NO_AS_NEEDED = 'Upstream asneeded linking bug (no-as-needed on line: %d)' PRESERVE_OLD_LIB = 'Upstream ABI change workaround on line: %d' BUILT_WITH_USE = 'built_with_use on line: %d' -EPREFIXIFY_MISSING_INHERIT = "prefix.eclass is not inherited, but eprefixify is used on line: %d" NO_OFFSET_WITH_HELPERS = "Helper function is used with D, ROOT, ED, EROOT or EPREFIX on line :%d" SANDBOX_ADDPREDICT = 'Ebuild calls addpredict on line: %d' USEQ_ERROR = 'Ebuild calls deprecated useq function on line: %d' </pre> Mike Frysinger 2012-05-23T19:21:51 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3460 <pre>Hello I recently installed Gentoo on my uncle's laptop and he was a bit annoyed about needing to run "dispatch-conf" and merge a lot of changes on files nobody ever touched. Looking to /etc/dispatch-conf.conf I noticed options to improve this situation exist, but they are disabled by default. I would want to confirm if they are safe enough or could cause problems. Options are: # Automerge files comprising only whitespace and/or comments # (yes or no) replace-wscomments=no # Automerge files that the user hasn't modified # (yes or no) replace-unmodified=no Looks really surprising to me that "replace-wscomments" is not enabled by default as merging that changes shouldn't hurt at all. About "replace-unmodified", if it works as intended, it should also be safer to get it enabled by default as would prevent breakage if people forgets to run dispatch-conf, reboot and, for example, sees some init.d script fail to start. Thanks a lot for the info </pre> Pacho Ramos 2012-05-15T11:15:52 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3446 <pre>Stealing some ideas from ferringb, add a new API for doing parallel processing in bash, and then deploy this with the stripping and compressing stages. For stripping coreutils which has about 100 ELFs, this brings time to strip down from ~7 seconds to ~0.7 seconds on my system. Signed-off-by: Mike Frysinger &lt;vapier&lt; at &gt;gentoo.org&gt; --- note: i'm not terribly happy with the name "helper-functions.sh", so any better suggestions would be good. i didn't want to use "ebuild-helpers.sh" as that messes up tab completion ;). bin/ebuild-helpers/ecompressdir | 30 ++++++++++++++++--- bin/ebuild-helpers/prepstrip | 20 ++++++++++--- bin/helper-functions.sh | 62 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 104 insertions(+), 8 deletions(-) create mode 100644 bin/helper-functions.sh diff --git a/bin/ebuild-helpers/ecompressdir b/bin/ebuild-helpers/ecompressdir index 17ecd80..a2c9e52 100755 --- a/bin/ebuild-helpers/ecompressdir +++ b/bin/ebuild-helpers/ecompressdir &lt; at &gt;&lt; at &gt; -2,7 +2,7 &lt; at &gt;&lt; at &gt; # Copyright 1999-2011 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 -source "${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}"/isolated-functions.sh +source "${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}"/helper-functions.sh if [[ -z $1 ]] ; then helpers_die "${0##*/}: at least one argument needed" &lt; at &gt;&lt; at &gt; -116,6 +116,16 &lt; at &gt;&lt; at &gt; ret=0 rm -rf "${T}"/ecompress-skip +decompressors=( +".Z" "gunzip -f" +".gz" "gunzip -f" +".bz2" "bunzip2 -f" +".xz" "unxz -f" +".lzma" "unxz -f" +) + +multijob_init + for dir in "$&lt; at &gt;" ; do dir=${dir#/} dir="${ED}${dir}" &lt; at &gt;&lt; at &gt; -136,14 +146,26 &lt; at &gt;&lt; at &gt; for dir in "$&lt; at &gt;" ; do find "${dir}" -type f -name '*.ecompress.file' -print0 | ${XARGS} -0 rm -f # not uncommon for packages to compress doc files themselves -funk_up_dir "decompress" ".Z" "gunzip -f" -funk_up_dir "decompress" ".gz" "gunzip -f" -funk_up_dir "decompress" ".bz2" "bunzip2 -f" +for (( d = 0; d &lt; ${#decompressors[&lt; at &gt;]}; d += 2 )) ; do +# It's faster to parallelize at this stage than to try to +# parallelize the compressors. This is because the find|xargs +# ends up launching less compressors overall, so the overhead +# of forking children ends up dominating. +( +multijob_child_init +funk_up_dir "decompress" "${decompressors[i]}" "${decompressors[i+1]}" +) &amp; +multijob_post_fork +: $(( ret |= $? )) +done # forcibly break all hard links as some compressors whine about it find "${dir}" -type f -links +1 -exec env file="{}" sh -c \ 'cp -p "${file}" "${file}.ecompress.break" ; mv -f "${file}.ecompress.break" "${file}"' \; +multijob_finish +: $(( ret |= $? )) + # now lets do our work if [[ -n ${suffix} ]] ; then vecho "${0##*/}: $(ecompress --bin) /${actual_dir#${ED}}" diff --git a/bin/ebuild-helpers/prepstrip b/bin/ebuild-helpers/prepstrip index daaa252..09b0333 100755 --- a/bin/ebuild-helpers/prepstrip +++ b/bin/ebuild-helpers/prepstrip &lt; at &gt;&lt; at &gt; -1,8 +1,8 &lt; at &gt;&lt; at &gt; #!/bin/bash -# Copyright 1999-2011 Gentoo Foundation +# Copyright 1999-2012 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 -source "${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}"/isolated-functions.sh +source "${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}"/helper-functions.sh # avoid multiple calls to `has`. this creates things like: # FEATURES_foo=false &lt; at &gt;&lt; at &gt; -62,6 +62,8 &lt; at &gt;&lt; at &gt; prepstrip_sources_dir=${EPREFIX}/usr/src/debug/${CATEGORY}/${PF} type -P debugedit &gt;/dev/null &amp;&amp; debugedit_found=true || debugedit_found=false debugedit_warned=false +multijob_init + unset ${!INODE_*} inode_var_name() { &lt; at &gt;&lt; at &gt; -171,6 +173,8 &lt; at &gt;&lt; at &gt; process_elf() { # We want to log already stripped binaries, as this may be a QA violation. # They prevent us from getting the splitdebug data. if ! ${RESTRICT_binchecks} &amp;&amp; ! ${RESTRICT_strip} ; then +( +multijob_child_init log=$T/scanelf-already-stripped.log qa_var="QA_PRESTRIPPED_${ARCH/-/_}" [[ -n ${!qa_var} ]] &amp;&amp; QA_PRESTRIPPED="${!qa_var}" &lt; at &gt;&lt; at &gt; -193,6 +197,8 &lt; at &gt;&lt; at &gt; if ! ${RESTRICT_binchecks} &amp;&amp; ! ${RESTRICT_strip} ; then else rm -f "$log" fi +) &amp; +multijob_post_fork fi # Now we look for unstripped binaries. &lt; at &gt;&lt; at &gt; -205,8 +211,10 &lt; at &gt;&lt; at &gt; do banner=true fi -f=$(file "${x}") || continue -[[ -z ${f} ]] &amp;&amp; continue +( +multijob_child_init +f=$(file "${x}") || exit 0 +[[ -z ${f} ]] &amp;&amp; exit 0 if ! ${SKIP_STRIP} ; then # The noglob funk is to support STRIP_MASK="/*/booga" and to keep &lt; at &gt;&lt; at &gt; -253,6 +261,8 &lt; at &gt;&lt; at &gt; do if ${was_not_writable} ; then chmod u-w "${x}" fi +) &amp; +multijob_post_fork done if [[ -s ${T}/debug.sources ]] &amp;&amp; \ &lt; at &gt;&lt; at &gt; -274,3 +284,5 &lt; at &gt;&lt; at &gt; then &gt;&gt; "$emptydir"/.keepdir done &lt; &lt;(find "${D}${prepstrip_sources_dir}/" -type d -empty -print0) fi + +multijob_finish diff --git a/bin/helper-functions.sh b/bin/helper-functions.sh new file mode 100644 index 0000000..c69a41a --- /dev/null +++ b/bin/helper-functions.sh &lt; at &gt;&lt; at &gt; -0,0 +1,62 &lt; at &gt;&lt; at &gt; +#!/bin/bash +# Copyright 1999-2012 Gentoo Foundation +# Distributed under the terms of the GNU General Public License v2 + +# For routines we want to use in ebuild-helpers/ but don't want to +# expose to the general ebuild environment. + +source "${PORTAGE_BIN_PATH:-/usr/lib/portage/bin}"/isolated-functions.sh + +# +# API functions for doing parallel processing +# +numjobs() { +# Copied from eutils.eclass:makeopts_jobs() +local jobs=$(echo " ${MAKEOPTS} " | \ +sed -r -n 's:.*[[:space:]](-j|--jobs[=[:space:]])[[:space:]]*([0-9]+).*:\2:p') +echo ${jobs:-1} +} + +multijob_init() { +# Setup a pipe for children to write their pids to when they finish. +mj_control_pipe=$(mktemp -t multijob.XXXXXX) +rm "${mj_control_pipe}" +mkfifo "${mj_control_pipe}" +exec {mj_control_fd}&lt;&gt;${mj_control_pipe} +rm -f "${mj_control_pipe}" + +# See how many children we can fork based on the user's settings. +mj_max_jobs=$(numjobs) +mj_num_jobs=0 +} + +multijob_child_init() { +trap 'echo ${BASHPID} $? &gt;&amp;'${mj_control_fd} EXIT +trap 'exit 1' INT TERM +} + +multijob_finish_one() { +local pid ret +read -r -u ${mj_control_fd} pid ret +: $(( --mj_num_jobs )) +return ${ret} +} + +multijob_finish() { +local ret=0 +while [[ ${mj_num_jobs} -gt 0 ]] ; do +multijob_finish_one +: $(( ret += $? )) +done +# Let bash clean up its internal child tracking state. +wait +return ${ret} +} + +multijob_post_fork() { +: $(( ++mj_num_jobs )) +if [[ ${mj_num_jobs} -ge ${mj_max_jobs} ]] ; then +multijob_finish_one +fi +return 0 +} </pre> Mike Frysinger 2012-05-11T16:39:59 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3444 <pre>Hello, I've added a new variable to portage named COLLISION_UNOWNED_IGNORE_PATTERNS. It works similar to COLLISION_IGNORE, it specifies shell patterns for unowned files that can be overwritten (will not cause collision-protect or protect-owned to go off). I'm not convinced about the name - if anyone has better idea I'd be more than happy to change it :) Attached patch adds code to vartree, documents new variable in manual page and adds a default to make.globals. </pre> Krzysztof Pawlik 2012-05-03T09:03:07 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3442 <pre>Hi all, I am working on a proposal for GSoC 2012 and I'd appreciate any input that you can have towards my proposal, which I'm enclosing below: Project Name: Package statistics reporting tool Abstract I am proposing to work on improving the Package statistics reporting tool that was started last as part of GCoC 2011 for Gentoo, Gentoostats (http://wiki.gentoo.org/wiki/Gentoostats). 
The plan is to continue working on the user end program to upload anonymous information about installed packages on users’ machines to a database that package maintainers and developers have access to, as well as offline support. Additional package information statistics will be aggregated and a more robust web application will be written to allow for third party data exports, as well as graphical analysis graphs and tools will be incorporated. Objective The objective of this project will improve both the data being collected, stored and presented by last year’s iteration of this project, Gentoostats (http://wiki.gentoo.org/wiki/Gentoostats). The anonymous data being uploaded being analyzed and we will improve the data submission model to support additional fields for future updates via a more general package build description file metadata that is more easily extensible. We will also extend the application and the web site itself to allow for offline submission of data statistics. At the core of the project will be creating a rich web 2.0 application with aggregated package information statistics, third party export to CSV and other formats, data manipulation via API calls, as well as graphical analysis graphs and tools will be incorporated. Deliverables The deliverables of this project proposal are as follows: extend data collection schema supported create offline data export support mechanism for the web application: create a feature-rich, web 2.0 web application with user authentication, data submission, data export, user comments, user permissions and roles create feature-rich data import and export functions create an API for data import and export create statistical analysis tools, such as graphs, plots, and package version heat-maps for package version stability analysis and other QA-centric analytics Timeline Between April 24 and May 22, a complete breakdown of all the features, functions, and 80% of the web UI and API functions should be wire-framed and completely documented. When coding begins, on May 22, start working on the command line improvements. Specifically: May 22 - May 30, finish Deliverable 1. May 30 - June 10, finish offline data export utility June 10 - July 13, complete 3.a and 3.b. A fully functional web 2.0 web site written in PHP/MySQL or Rails have the following features fully functional: user authentication data submission data export user comments user permissions user roles July 13 - August 10, finish 3.c and 3.d August 10 - August 20, create full project documentation and review comments for all functions and the API August 20 (pencils down date) - submit project to Google Code repository Throughout the entire project, engage the community and mentor for feedback, ideas, and direction. What do you guys think? Any comments and ideas are welcome! Thanks, Andrei Oprisan </pre> Andrei Oprisan 2012-04-06T18:54:52 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3438 <pre>Signed-off-by: Mike Frysinger &lt;vapier&lt; at &gt;gentoo.org&gt; --- bin/isolated-functions.sh | 2 +- bin/portageq | 8 ++++++++ pym/portage/output.py | 6 ++++++ pym/portage/package/ebuild/doebuild.py | 8 ++------ 4 files changed, 17 insertions(+), 7 deletions(-) diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh index 9321ad5..98be41e 100644 --- a/bin/isolated-functions.sh +++ b/bin/isolated-functions.sh &lt; at &gt;&lt; at &gt; -431,8 +431,8 &lt; at &gt;&lt; at &gt; set_colors() { BAD=$'\e[31;01m' HILITE=$'\e[36;01m' BRACKET=$'\e[34;01m' +NORMAL=$'\e[0m' fi -NORMAL=$'\e[0m' } RC_ENDCOL="yes" diff --git a/bin/portageq b/bin/portageq index 5ecbb21..fcdb9d9 100755 --- a/bin/portageq +++ b/bin/portageq &lt; at &gt;&lt; at &gt; -44,6 +44,7 &lt; at &gt;&lt; at &gt; del pym_path from portage import os from portage.eapi import eapi_has_repo_deps from portage.util import writemsg, writemsg_stdout +from portage.output import colormap portage.proxy.lazyimport.lazyimport(globals(), 'subprocess', '_emerge.Package:Package', &lt; at &gt;&lt; at &gt; -685,6 +686,13 &lt; at &gt;&lt; at &gt; def distdir(argv): print(portage.settings["DISTDIR"]) +def colormap(argv): +""" +Display the color.map as environment variables. +""" +print(portage.output.colormap()) + + def envvar(argv): """&lt;variable&gt;+ Returns a specific environment variable as exists prior to ebuild.sh. diff --git a/pym/portage/output.py b/pym/portage/output.py index 43d7503..98bec81 100644 --- a/pym/portage/output.py +++ b/pym/portage/output.py &lt; at &gt;&lt; at &gt; -325,6 +325,12 &lt; at &gt;&lt; at &gt; def style_to_ansi_code(style): ret += codes.get(attr_name, attr_name) return ret +def colormap(): +mycolors = [] +for c in ("GOOD", "WARN", "BAD", "HILITE", "BRACKET", "NORMAL"): +mycolors.append("%s=$'%s'" % (c, style_to_ansi_code(c))) +return "\n".join(mycolors) + def colorize(color_key, text): global havecolor if havecolor: diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py index c45aa03..4ff3eea 100644 --- a/pym/portage/package/ebuild/doebuild.py +++ b/pym/portage/package/ebuild/doebuild.py &lt; at &gt;&lt; at &gt; -50,7 +50,7 &lt; at &gt;&lt; at &gt; from portage.exception import DigestException, FileNotFound, \ IncorrectParameter, InvalidDependString, PermissionDenied, \ UnsupportedAPIException from portage.localization import _ -from portage.output import style_to_ansi_code +from portage.output import colormap from portage.package.ebuild.prepare_build_dirs import prepare_build_dirs from portage.util import apply_recursive_permissions, \ apply_secpass_permissions, noiselimit, normalize_path, \ &lt; at &gt;&lt; at &gt; -300,11 +300,7 &lt; at &gt;&lt; at &gt; def doebuild_environment(myebuild, mydo, myroot=None, settings=None, mysettings["PORTAGE_CONFIGROOT"], EBUILD_SH_ENV_DIR) # Allow color.map to control colors associated with einfo, ewarn, etc... -mycolors = [] -for c in ("GOOD", "WARN", "BAD", "HILITE", "BRACKET"): -mycolors.append("%s=$'%s'" % \ -(c, style_to_ansi_code(c))) -mysettings["PORTAGE_COLORMAP"] = "\n".join(mycolors) +mysettings["PORTAGE_COLORMAP"] = colormap() if "COLUMNS" not in mysettings: # Set COLUMNS, in order to prevent unnecessary stty calls </pre> Mike Frysinger 2012-03-11T04:15:41 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3436 <pre>This avoids having to pipe through multiple greps, as well as running diff multiple times on the same set of files. Signed-off-by: Mike Frysinger &lt;vapier&lt; at &gt;gentoo.org&gt; --- bin/dispatch-conf | 37 ++++++++++++++++++++++++------------- pym/portage/dispatch_conf.py | 8 ++++---- 2 files changed, 28 insertions(+), 17 deletions(-) diff --git a/bin/dispatch-conf b/bin/dispatch-conf index a778118..f6d7499 100755 --- a/bin/dispatch-conf +++ b/bin/dispatch-conf &lt; at &gt;&lt; at &gt; -27,13 +27,11 &lt; at &gt;&lt; at &gt; except ImportError: from portage import os from portage import dispatch_conf from portage import _unicode_decode -from portage.dispatch_conf import diffstatusoutput_len +from portage.dispatch_conf import diffstatusoutput from portage.process import find_binary FIND_EXTANT_CONFIGS = "find '%s' %s -name '._cfg????_%s' ! -name '.*~' ! -iname '.*.bak' -print" DIFF_CONTENTS = "diff -Nu '%s' '%s'" -DIFF_CVS_INTERP = "diff -Nu '%s' '%s' | grep '^[+-][^+-]' | grep -v '# .Header:.*'" -DIFF_WSCOMMENTS = "diff -Nu '%s' '%s' | grep '^[+-][^+-]' | grep -v '^[-+]#' | grep -v '^[-+][[:space:]]*$'" # We need a secure scratch dir and python does silly verbose errors on the use of tempnam oldmask = os.umask(0o077) &lt; at &gt;&lt; at &gt; -62,7 +60,7 &lt; at &gt;&lt; at &gt; def cleanup(mydir=SCRATCH_DIR): shutil.rmtree(mydir) atexit.register(cleanup) -MANDATORY_OPTS = [ 'archive-dir', 'diff', 'replace-cvs', 'replace-wscomments', 'merge' ] +MANDATORY_OPTS = [ 'archive-dir', 'diff', 'replace-cvs', 'replace-wscomments', 'merge' ] def cmd_var_is_valid(cmd): """ &lt; at &gt;&lt; at &gt; -152,6 +150,9 &lt; at &gt;&lt; at &gt; class dispatch: portage.util.shlex_split( portage.settings.get('CONFIG_PROTECT_MASK', ''))) + def diff(file1, file2): + return diffstatusoutput(DIFF_CONTENTS, file1, file2) + # # Remove new configs identical to current # and &lt; at &gt;&lt; at &gt; -168,11 +169,11 &lt; at &gt;&lt; at &gt; class dispatch: else: mrgfail = portage.dispatch_conf.file_archive(archive, conf['current'], conf['new'], mrgconf) if os.path.exists(archive + '.dist'): - unmodified = diffstatusoutput_len(DIFF_CONTENTS % (conf['current'], archive + '.dist'))[1] == 0 + unmodified = len(diff(conf['current'], archive + '.dist')[1]) == 0 else: unmodified = 0 if os.path.exists(mrgconf): - if mrgfail or diffstatusoutput_len(DIFF_CONTENTS % (conf['new'], mrgconf))[1] == 0: + if mrgfail or len(diff(conf['new'], mrgconf)[1]) == 0: os.unlink(mrgconf) newconf = conf['new'] else: &lt; at &gt;&lt; at &gt; -183,24 +184,34 &lt; at &gt;&lt; at &gt; class dispatch: if newconf == mrgconf and \ self.options.get('ignore-previously-merged') != 'yes' and \ os.path.exists(archive+'.dist') and \ - diffstatusoutput_len(DIFF_CONTENTS % (archive+'.dist', conf['new']))[1] == 0: + len(diff(archive+'.dist', conf['new'])[1]) == 0: # The current update is identical to the archived .dist # version that has previously been merged. os.unlink(mrgconf) newconf = conf['new'] - mystatus, myoutput_len = diffstatusoutput_len( - DIFF_CONTENTS % (conf ['current'], newconf)) + mystatus, myoutput = diff(conf['current'], newconf) + myoutput_len = len(myoutput) same_file = 0 == myoutput_len if mystatus &gt;&gt; 8 == 2: # Binary files differ same_cvs = False same_wsc = False else: - same_cvs = 0 == diffstatusoutput_len( - DIFF_CVS_INTERP % (conf ['current'], newconf))[1] - same_wsc = 0 == diffstatusoutput_len( - DIFF_WSCOMMENTS % (conf ['current'], newconf))[1] + # Extract all the normal diff lines (ignore the headers). + mylines = re.findall('^[+-][^\n+-].*$', myoutput, re.MULTILINE) + + # Filter out all the cvs headers + cvs_header = re.compile('# [$]Header:') + cvs_lines = filter(cvs_header.search, mylines) + same_cvs = len(mylines) == len(cvs_lines) + + # Filter out comments and whitespace-only changes. + # Note: be nice to also ignore lines that only differ in whitespace... + wsc_lines = [] + for x in ['^[-+]\s*#', '^[-+]\s*$']: + wsc_lines += filter(re.compile(x).match, mylines) + same_wsc = len(mylines) == len(wsc_lines) # Do options permit? same_cvs = same_cvs and self.options['replace-cvs'] == 'yes' diff --git a/pym/portage/dispatch_conf.py b/pym/portage/dispatch_conf.py index 7f407ff..6e8de0f 100644 --- a/pym/portage/dispatch_conf.py +++ b/pym/portage/dispatch_conf.py &lt; at &gt;&lt; at &gt; -23,18 +23,18 &lt; at &gt;&lt; at &gt; RCS_MERGE = "rcsmerge -p -r" + RCS_BRANCH + " '%s' &gt; '%s'" DIFF3_MERGE = "diff3 -mE '%s' '%s' '%s' &gt; '%s'" -def diffstatusoutput_len(cmd): +def diffstatusoutput(cmd, file1, file2): """ Execute the string cmd in a shell with getstatusoutput() and return a - 2-tuple (status, output_length). If getstatusoutput() raises + 2-tuple (status, output). If getstatusoutput() raises UnicodeDecodeError (known to happen with python3.1), return a 2-tuple (1, 1). This provides a simple way to check for non-zero output length of diff commands, while providing simple handling of UnicodeDecodeError when necessary. """ try: - status, output = portage.subprocess_getstatusoutput(cmd) - return (status, len(output)) + status, output = portage.subprocess_getstatusoutput(cmd % (file1, file2)) + return (status, output) except UnicodeDecodeError: return (1, 1) </pre> Mike Frysinger 2012-03-08T21:48:28 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3433 <pre>When I run into test failures in www-client/chromium, the normal procedure is to file a bug and filter the failing tests until the problem can be fixed (usually upstream). Ideally, I would like to be able to do the following: 1. Start building chromium using emerge or ebuild. 2. If src_test dies, modify the ebuild to filter the failing tests. 3. Re-run the test phase without rebuilding the source code. If I do not run ebuild clean, portage re-uses the environment from the previous run, which does not include my changes to the ebuild. Is there an easy way to have portage source the modified ebuild instead of just using the previous environment? Wiping out the environment file isn't really an option because we set some global variables in pkg_setup. </pre> Mike Gilbert 2012-03-05T16:50:26 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3426 <pre>Since binutils/gdb can compress the .debug* sections with zlib into .zdebug*, we should be able to save quite a bit of space -- on my system, I saw /usr/lib/debug/ go from ~20GB to ~7.5GB. This requires binutils/gdb to be built with USE=zlib which is why this defaults to off. Signed-off-by: Mike Frysinger &lt;vapier&lt; at &gt;gentoo.org&gt; --- bin/ebuild-helpers/prepstrip | 6 ++++-- man/make.conf.5 | 10 +++++++++- pym/_emerge/EbuildPhase.py | 6 ++++-- pym/portage/const.py | 2 +- 4 files changed, 18 insertions(+), 6 deletions(-) diff --git a/bin/ebuild-helpers/prepstrip b/bin/ebuild-helpers/prepstrip index ee547f2..daaa252 100755 --- a/bin/ebuild-helpers/prepstrip +++ b/bin/ebuild-helpers/prepstrip &lt; at &gt;&lt; at &gt; -15,7 +15,7 &lt; at &gt;&lt; at &gt; exp_tf() { eval ${var}_${flag}=$(tf has ${flag} ${!var}) done } -exp_tf FEATURES installsources nostrip splitdebug +exp_tf FEATURES compressdebug installsources nostrip splitdebug exp_tf RESTRICT binchecks installsources strip [[ " ${FEATURES} " == *" force-prefix "* ]] || \ &lt; at &gt;&lt; at &gt; -121,7 +121,9 &lt; at &gt;&lt; at &gt; save_elf_debug() { if [[ -e ${T}/prepstrip.split.debug ]] ; then mv "${T}"/prepstrip.split.debug "${y}" else -${OBJCOPY} --only-keep-debug "${x}" "${y}" +local objcopy_flags="--only-keep-debug" +${FEATURES_compressdebug} &amp;&amp; objcopy_flags+=" --compress-debug-sections" +${OBJCOPY} ${objcopy_flags} "${x}" "${y}" ${OBJCOPY} --add-gnu-debuglink="${y}" "${x}" fi local args="a-x,o-w" diff --git a/man/make.conf.5 b/man/make.conf.5 index f7491a9..e5a9ae1 100644 --- a/man/make.conf.5 +++ b/man/make.conf.5 &lt; at &gt;&lt; at &gt; -260,6 +260,12 &lt; at &gt;&lt; at &gt; Log file names have an extension that is appropriate for the compression type. Currently, only \fBgzip\fR(1) compression is supported, so build logs will have a '.gz' extension when this feature is enabled. .TP +.B compressdebug +Compress the debug sections in the split debug files with zlib to save +space. Make sure you have built both binutils and gdb with USE=zlib +support for this to work. See \fBsplitdebug\fR for general split debug +information (upon which this feature depends). +.TP .B config\-protect\-if\-modified This causes the \fBCONFIG_PROTECT\fR behavior to be skipped for files that have not been modified since they were installed. &lt; at &gt;&lt; at &gt; -476,7 +482,9 &lt; at &gt;&lt; at &gt; instead of using \fBPORT_LOGDIR\fR directly. .B splitdebug Prior to stripping ELF etdyn and etexec files, the debugging info is stored for later use by various debuggers. This feature is disabled by -\fBnostrip\fR. For installation of source code, see \fBinstallsources\fR. +\fBnostrip\fR. You should also consider setting \fBcompressdebug\fR so +the files don't suck up a lot of space. For installation of source code, +see \fBinstallsources\fR. .TP .B strict Have portage react strongly to conditions that have the potential to be diff --git a/pym/_emerge/EbuildPhase.py b/pym/_emerge/EbuildPhase.py index ed0be0b..27944f4 100644 --- a/pym/_emerge/EbuildPhase.py +++ b/pym/_emerge/EbuildPhase.py &lt; at &gt;&lt; at &gt; -33,11 +33,13 &lt; at &gt;&lt; at &gt; class EbuildPhase(CompositeTask): ("_ebuild_lock",) # FEATURES displayed prior to setup phase -_features_display = ("ccache", "distcc", "distcc-pump", "fakeroot", +_features_display = ( +"ccache", "compressdebug", "distcc", "distcc-pump", "fakeroot", "installsources", "keeptemp", "keepwork", "nostrip", "preserve-libs", "sandbox", "selinux", "sesandbox", "splitdebug", "suidctl", "test", "userpriv", -"usersandbox") +"usersandbox" +) # Locked phases _locked_phases = ("setup", "preinst", "postinst", "prerm", "postrm") diff --git a/pym/portage/const.py b/pym/portage/const.py index 5fcb24f..bebb922 100644 --- a/pym/portage/const.py +++ b/pym/portage/const.py &lt; at &gt;&lt; at &gt; -88,7 +88,7 &lt; at &gt;&lt; at &gt; EBUILD_PHASES = ("pretend", "setup", "unpack", "prepare", "configure" SUPPORTED_FEATURES = frozenset([ "assume-digests", "binpkg-logs", "buildpkg", "buildsyspkg", "candy", "ccache", "chflags", "clean-logs", - "collision-protect", "compress-build-logs", + "collision-protect", "compress-build-logs", "compressdebug", "config-protect-if-modified", "digest", "distcc", "distcc-pump", "distlocks", "ebuild-locks", "fakeroot", "fail-clean", "force-mirror", "force-prefix", "getbinpkg", </pre> Mike Frysinger 2012-01-13T20:29:57 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3425 <pre>The writable issue shows up when using `ebuild` as non-root users in non-prefix setups. So always do it. Signed-off-by: Mike Frysinger &lt;vapier&lt; at &gt;gentoo.org&gt; --- bin/ebuild-helpers/prepstrip | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/bin/ebuild-helpers/prepstrip b/bin/ebuild-helpers/prepstrip index 1d7b3d2..ee547f2 100755 --- a/bin/ebuild-helpers/prepstrip +++ b/bin/ebuild-helpers/prepstrip &lt; at &gt;&lt; at &gt; -224,7 +224,7 &lt; at &gt;&lt; at &gt; do # unwritable objects. Make them temporarily writable for the # stripping. was_not_writable=false -if [[ -n ${EPREFIX} &amp;&amp; ! -w ${x} ]] ; then +if [[ ! -w ${x} ]] ; then was_not_writable=true chmod u+w "${x}" fi </pre> Mike Frysinger 2012-01-13T19:52:47 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3421 <pre>I am referring in this case to abiword, it has a "plugins" USE flag that enables some minimal set of plugins and, then, a lot of USE flags for building extra plugins (with extra dependencies). All of this extra plugins need "plugins" USE flag to be enabled. Is there any way to write a REQUIRED_USE flag variable without needing to list all USE flags depending on "plugins" to be set? Thanks a lot for the info :) </pre> Pacho Ramos 2011-12-17T10:24:37 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3420 <pre>I have just rebased the emaint re-write that I did some time ago. I have also updated the modules for changes since I originally did the rewrite into a plug-in modules system. The changes have been condensed down into several key commits. The emaint branch is available on github: https://github.com/dol-sen/portage It is now very easy to add or remove modules, by only adding/removing the modules directory to/from the emaint/modules directory. All the help and menu options are picked up and added automatically. The modules.py code also only imports the individual module's __init__.py which contains the module_spec. The target module is only imported when it is requested, keeping speed and overhead to a minimum. The overall benefit to portage is that emaint should be easier to maintain, the modules are also now available to portage/emerge or other tools for internal script use, not just the bin/emaint script. The plug-in code in modules.py is also universal in nature and can easily be moved elsewhere in portage's namespace. It can be used for any number of independant plug-in systems. The plug-in modules can also be more complex in nature as well as export more than one specific functionality as well as more than just ["check", "fix"] options. I have also just created a cleanconfig module inspired from this forum topic: http://forums.gentoo.org/viewtopic-t-905022.html Please test it out to ensure everything is working 100%. To test just how easy it is to add/remove modules, simply move one or more modules in/out of the emaint/modules directory and run emaint --help, etc... I have some other dummy example modules I had prepared and will add them to the branch in an example-modules directory. </pre> Brian Dolbec 2011-12-11T04:20:48 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3412 <pre>Reading: http://en.gentoo-wiki.com/wiki/Portage_tips#Portage_and_python3 Looks like some people think portage is faster with python3, is that true? I have python3 set as main interpreter (for catching packages failing to build/work with it), but my portage is merged without any python* USE flag, I guess, it's still using python2, no? If it's faster really, is python3 safe to use with portage or it's only offered for testing purposes? Thanks for the info :) </pre> Pacho Ramos 2011-11-09T00:38:10 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3407 <pre>I have user "pacho" under "portage" group to be able to make some tasks without becoming root, but, sadly, I am unable to make portage create /var/log/portage files/dirs with portage:portage owners instead of portage:root, preventing me from removing old files without becoming root. Is there anyway to configure this? Thanks a lot </pre> Pacho Ramos 2011-11-01T12:28:42 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3394 <pre>The full context of this message is from a thread on gentoo-alt ml: http://archives.gentoo.org/gentoo-alt/msg_db73b1a140fd958efb88f2437170646d.xml Long story short, this person has to deal with a fileserver that sets the rw world bits on for all files that are created. While I don't think the suggested patch is fair (this is obviously a very badly configured fileserver), I do wonder if the "sleep 1" that is in the code is actually desired. In case we want it, I would suggest we move down the "sleep 1" so it's only done once for the entire list with files. Can we use eqawarn in bin/misc-functions.sh:177 instead and avoid the sleep? </pre> Fabian Groffen 2011-10-20T11:20:14 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3391 <pre>The following three patches change repoman's echangelog feature slightly: 1) default for updating the changelog is retrieved from layout.conf 2) skel.ChangeLog is used for constructing the header of new ChangeLogs 3) modified files are checked for copyright years to be updated Please review before I push them. </pre> Fabian Groffen 2011-10-19T19:55:43 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3378 <pre>The readelf utility is much more common than debugedit. Signed-off-by: Mike Frysinger &lt;vapier&lt; at &gt;gentoo.org&gt; --- bin/ebuild-helpers/prepstrip | 32 +++++++++++++++++++++++--------- 1 files changed, 23 insertions(+), 9 deletions(-) diff --git a/bin/ebuild-helpers/prepstrip b/bin/ebuild-helpers/prepstrip index 67ceead..7a08aba 100755 --- a/bin/ebuild-helpers/prepstrip +++ b/bin/ebuild-helpers/prepstrip &lt; at &gt;&lt; at &gt; -26,10 +26,13 &lt; at &gt;&lt; at &gt; if ${RESTRICT_strip} || ${FEATURES_nostrip} ; then ${FEATURES_installsources} || exit 0 fi -STRIP=${STRIP:-${CHOST}-strip} -type -P -- ${STRIP} &gt; /dev/null || STRIP=strip -OBJCOPY=${OBJCOPY:-${CHOST}-objcopy} -type -P -- ${OBJCOPY} &gt; /dev/null || OBJCOPY=objcopy +# look up the tools we might be using +for t in STRIP:strip OBJCOPY:objcopy READELF:readelf ; do +v=${t%:*} # STRIP +t=${t#*:} # strip +eval ${v}=\"${!v:-${CHOST}-${t}}\" +type -P -- ${!v} &gt;/dev/null || eval ${v}=${t} +done # We'll leave out -R .note for now until we can check out the relevance # of the section when it has the ALLOC flag set on it ... &lt; at &gt;&lt; at &gt; -65,8 +68,15 &lt; at &gt;&lt; at &gt; save_elf_sources() { local x=$1 local inode=$(inode_var_name "$x") [[ -n ${!inode} ]] &amp;&amp; return 0 -debugedit -b "${WORKDIR}" -d "${prepstrip_sources_dir}" \ --l "${T}"/debug.sources "${x}" + +# since we're editing the ELF here, we should recompute the build-id +# (the -i flag below). save that output so we don't need to recompute +# it later on in the save_elf_debug step. +buildid=$(debugedit -i \ +-b "${WORKDIR}" \ +-d "${prepstrip_sources_dir}" \ +-l "${T}"/debug.sources \ +"${x}") } save_elf_debug() { &lt; at &gt;&lt; at &gt; -78,9 +88,6 &lt; at &gt;&lt; at &gt; save_elf_debug() { # dont save debug info twice [[ ${x} == *".debug" ]] &amp;&amp; return 0 -# this will recompute the build-id, but for now that's ok -local buildid="$( ${debugedit_found} &amp;&amp; debugedit -i "${x}" )" - mkdir -p "${y%/*}" local inode=$(inode_var_name "$x") &lt; at &gt;&lt; at &gt; -95,6 +102,12 &lt; at &gt;&lt; at &gt; save_elf_debug() { chmod ${args} "${y}" fi +# if we don't already have build-id from debugedit, look it up +if [[ -z ${buildid} ]] ; then +# convert the readelf output to something useful +buildid=$(${READELF} -x .note.gnu.build-id "${x}" 2&gt;/dev/null \ +| awk '$NF ~ /GNU/ { getline; printf $2$3$4$5; getline; print $2 }') +fi if [[ -n ${buildid} ]] ; then local buildid_dir="${D}usr/lib/debug/.build-id/${buildid:0:2}" local buildid_file="${buildid_dir}/${buildid:2}" &lt; at &gt;&lt; at &gt; -165,6 +178,7 &lt; at &gt;&lt; at &gt; do # actually causes problems. install sources for all # elf types though cause that stuff is good. +buildid= if [[ ${f} == *"current ar archive"* ]] ; then vecho " ${x:${#D}}" if ${strip_this} ; then </pre> Mike Frysinger 2011-10-11T04:50:53 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3376 <pre>I'm not an expert at python, so trying to make sure I'm doing anything too stupid. I tried to keep existing portage style, but feel free to point out whatever. I can also push these myself, so if people are OK with them, just let me know ... you don't have to merge+push yourself. Mike Frysinger (4): runtests: make sure we are in the right dir tests: split up getTests into helper funcs to avoid duplication tests: add a --help option to make runtest more friendly tests: add --list flag to show available tests pym/portage/tests/__init__.py | 93 ++++++++++++++++++++++++----------------- runtests.sh | 3 + 2 files changed, 57 insertions(+), 39 deletions(-) </pre> Mike Frysinger 2011-10-09T18:53:24 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3351 <pre>Respun now with the help of ferringb. Cleans up the implementation and catches a few bug and improvements: - mhash priority moved lower than pycrypto/hashlib because mhash holds GIL while the other implementations don't. - hashlib does offer whirlpool if it was built against openssl 1.0. 1/5: Refactor RMD160 hashlib code for less-hardcoding 2/5: Manifest2 hash: Whirlpool 3/5: Manifest2 hash: SHA512 4/5: Manifest2 hash backend provider: mhash 5/5: GLEP59: Change live Manifest2 hashes to SHA256, </pre> Robin H. Johnson 2011-10-01T07:40:50 http://comments.gmane.org/gmane.linux.gentoo.portage.devel/3340 <pre>For an overview of tree-signing, please see informational GLEP57. This patch series implements all parts needed for GLEP59. Upcoming patch series will include support for the rest of the tree-signing, GLEP59-61. [1/4] Manifest2 hash: Whirlpool [2/4] Manifest2 hash: SHA512 New hashes. [3/4] Manifest2 hash backend provider: mhash New hash provider. Please note that neither pyopenssl nor python-gnutls provide suitable hashing functionality (pyopenssl misses hashing, gnutls misses WHIRLPOOL). [4/4] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL Per subject. Change the actual defaults. </pre> Robin H. Johnson 2011-09-30T01:27:38 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.linux.gentoo.portage.devel