gmane.linux.gentoo.portage.devel

[RFC/PATCH] repoman: unroll escaped lines so we can check the entirety of it

Mike Frysinger — 2012-05-24T04:06:39

Sometimes people wrap long lines in their ebuilds to make it easier to
read, but this causes us issues when doing line-by-line checking.  So
automatically unroll those lines before passing the full content down
to our checkers.

This seems to work, but maybe someone can suggest something simpler.

Signed-off-by: Mike Frysinger <vapier< at >gentoo.org>
---
 pym/repoman/checks.py |   35 +++++++++++++++++++++++++++++++++++
 1 files changed, 35 insertions(+), 0 deletions(-)

diff --git a/pym/repoman/checks.py b/pym/repoman/checks.py
index 65e7136..67f2b0a 100644
--- a/pym/repoman/checks.py
+++ b/pym/repoman/checks.py
< at >< at > -750,11 +750,46 < at >< at > _ignore_comment_re = re.compile(r'^\s*#')
 def run_checks(contents, pkg):
 checks = _constant_checks
 here_doc_delim = None
+multiline = None
 
 for lc in checks:
 lc.new(pkg)
 for num, line in enumerate(contents):
 
+# Unroll multiline escaped strings so that we can check things:
+#inherit foo bar \
+#moo \
+#cow
+# This will merge these lines like so:
+#

[PATCH] repoman: add a mini framework for checking eclasses, and fill it out

Mike Frysinger — 2012-05-23T19:21:51

Rather than copying & pasting the same behavior for the different eclass
checks, add a common class for them to extend.  This makes adding more
eclass checks trivial, and keeps down bitrot.

This does abuse the checking interface slightly -- the eclass will change
its category between unused and missing based on the checks.

URL: https://bugs.gentoo.org/417159
URL: https://bugs.gentoo.org/417231
Signed-off-by: Mike Frysinger <vapier< at >gentoo.org>
---
 bin/repoman           |    6 +-
 pym/repoman/checks.py |  143 ++++++++++++++++++++++++++++++++-----------------
 pym/repoman/errors.py |    1 -
 3 files changed, 97 insertions(+), 53 deletions(-)

diff --git a/bin/repoman b/bin/repoman
index 3697403..d7ffcdd 100755
--- a/bin/repoman
+++ b/bin/repoman
< at >< at > -315,8 +315,9 < at >< at > qahelp={
 "file.size.fatal":"Files in the files directory must be under 60 KiB",
 "file.name":"File/dir name must be composed of only the following chars: %s " % allowed_filename_chars,
 "file.UTF8":"File is not UTF8 compliant",
-"inherit.auto

About some settings to auto-replace with dispatch-conf

Pacho Ramos — 2012-05-15T11:15:52

Hello

I recently installed Gentoo on my uncle's laptop and he was a bit
annoyed about needing to run "dispatch-conf" and merge a lot of changes
on files nobody ever touched.

Looking to /etc/dispatch-conf.conf I noticed options to improve this
situation exist, but they are disabled by default. I would want to
confirm if they are safe enough or could cause problems. Options are:

# Automerge files comprising only whitespace and/or comments
# (yes or no)
replace-wscomments=no

# Automerge files that the user hasn't modified
# (yes or no)
replace-unmodified=no

Looks really surprising to me that "replace-wscomments" is not enabled
by default as merging that changes shouldn't hurt at all. About
"replace-unmodified", if it works as intended, it should also be safer
to get it enabled by default as would prevent breakage if people forgets
to run dispatch-conf, reboot and, for example, sees some init.d script
fail to start.

Thanks a lot for the info

[RFC/PATCH] prepstrip/ecompressdir: parallelize operations

Mike Frysinger — 2012-05-11T16:39:59

Stealing some ideas from ferringb, add a new API for doing parallel
processing in bash, and then deploy this with the stripping and
compressing stages.

For stripping coreutils which has about 100 ELFs, this brings time
to strip down from ~7 seconds to ~0.7 seconds on my system.

Signed-off-by: Mike Frysinger <vapier< at >gentoo.org>
---
note: i'm not terribly happy with the name "helper-functions.sh", so any
better suggestions would be good.  i didn't want to use "ebuild-helpers.sh"
as that messes up tab completion ;).

 bin/ebuild-helpers/ecompressdir |   30 ++++++++++++++++---
 bin/ebuild-helpers/prepstrip    |   20 ++++++++++---
 bin/helper-functions.sh         |   62 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 104 insertions(+), 8 deletions(-)
 create mode 100644 bin/helper-functions.sh

diff --git a/bin/ebuild-helpers/ecompressdir b/bin/ebuild-helpers/ecompressdir
index 17ecd80..a2c9e52 100755
--- a/bin/ebuild-helpers/ecompressdir
+++ b/bin/ebuild-helpers/ecompressdir
< at >< at > -2,7 +2,7 < at >< at >
 # Copy

[PATCH] Ignore some unowned files during collision check

Krzysztof Pawlik — 2012-05-03T09:03:07

Hello,

I've added a new variable to portage named
COLLISION_UNOWNED_IGNORE_PATTERNS. It works similar to COLLISION_IGNORE, it
specifies shell patterns for unowned files that can be overwritten (will not
cause collision-protect or protect-owned to go off).

I'm not convinced about the name - if anyone has better idea I'd be more than
happy to change it :)

Attached patch adds code to vartree, documents new variable in manual page and
adds a default to make.globals.

GSoC project proposal

Andrei Oprisan — 2012-04-06T18:54:52

Hi all,

I am working on a proposal for GSoC 2012 and I'd appreciate any input that you can have towards my proposal, which I'm enclosing below:

Project Name: Package statistics reporting tool

Abstract

I am proposing to work on improving the Package statistics reporting tool that was started last as part of GCoC 2011 for Gentoo, Gentoostats (http://wiki.gentoo.org/wiki/Gentoostats).  The plan is to continue working on the user end program to upload anonymous information about installed packages on users’ machines to a database that package maintainers and developers have access to, as well as offline support.
Additional package information statistics will be aggregated and a more robust web application will be written to allow for third party data exports, as well as graphical analysis graphs and tools will be incorporated.


Objective

The objective of this project will improve both the data being collected, stored and presented by last year’s iteration of this project, Gentoostats (http://wiki.gen

[PATCH] portageq: add "colormap" helper

Mike Frysinger — 2012-03-11T04:15:41

Signed-off-by: Mike Frysinger <vapier< at >gentoo.org>
---
 bin/isolated-functions.sh              |    2 +-
 bin/portageq                           |    8 ++++++++
 pym/portage/output.py                  |    6 ++++++
 pym/portage/package/ebuild/doebuild.py |    8 ++------
 4 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
index 9321ad5..98be41e 100644
--- a/bin/isolated-functions.sh
+++ b/bin/isolated-functions.sh
< at >< at > -431,8 +431,8 < at >< at > set_colors() {
 BAD=$'\e[31;01m'
 HILITE=$'\e[36;01m'
 BRACKET=$'\e[34;01m'
+NORMAL=$'\e[0m'
 fi
-NORMAL=$'\e[0m'
 }
 
 RC_ENDCOL="yes"
diff --git a/bin/portageq b/bin/portageq
index 5ecbb21..fcdb9d9 100755
--- a/bin/portageq
+++ b/bin/portageq
< at >< at > -44,6 +44,7 < at >< at > del pym_path
 from portage import os
 from portage.eapi import eapi_has_repo_deps
 from portage.util import writemsg, writemsg_stdout
+from portage.output import colormap
 portage.proxy.lazyimport.lazyimport(globals(),
 'subprocess',
 '_emerge.P

[PATCH] dispatch-conf: do regex matching ourselves

Mike Frysinger — 2012-03-08T21:48:28

This avoids having to pipe through multiple greps, as well as running
diff multiple times on the same set of files.

Signed-off-by: Mike Frysinger <vapier< at >gentoo.org>
---
 bin/dispatch-conf            |   37 ++++++++++++++++++++++++-------------
 pym/portage/dispatch_conf.py |    8 ++++----
 2 files changed, 28 insertions(+), 17 deletions(-)

diff --git a/bin/dispatch-conf b/bin/dispatch-conf
index a778118..f6d7499 100755
--- a/bin/dispatch-conf
+++ b/bin/dispatch-conf
< at >< at > -27,13 +27,11 < at >< at > except ImportError:
 from portage import os
 from portage import dispatch_conf
 from portage import _unicode_decode
-from portage.dispatch_conf import diffstatusoutput_len
+from portage.dispatch_conf import diffstatusoutput
 from portage.process import find_binary
 
 FIND_EXTANT_CONFIGS  = "find '%s' %s -name '._cfg????_%s' ! -name '.*~' ! -iname '.*.bak' -print"
 DIFF_CONTENTS        = "diff -Nu '%s' '%s'"
-DIFF_CVS_INTERP      = "diff -Nu '%s' '%s' | grep '^[+-][^+-]' | grep -v '# .Header:.*'"
-DIFF_WSCOMMENTS      = "dif

Rerunning src_test with modifications

Mike Gilbert — 2012-03-05T16:50:26

When I run into test failures in www-client/chromium, the normal
procedure is to file a bug and filter the failing tests until the
problem can be fixed (usually upstream).

Ideally, I would like to be able to do the following:

1. Start building chromium using emerge or ebuild.

2. If src_test dies, modify the ebuild to filter the failing tests.

3. Re-run the test phase without rebuilding the source code.

If I do not run ebuild clean, portage re-uses the environment from the
previous run, which does not include my changes to the ebuild.

Is there an easy way to have portage source the modified ebuild
instead of just using the previous environment? Wiping out the
environment file isn't really an option because we set some global
variables in pkg_setup.

[PATCH] add support for compressing debug sections to save space

Mike Frysinger — 2012-01-13T20:29:57

Since binutils/gdb can compress the .debug* sections with zlib into
.zdebug*, we should be able to save quite a bit of space -- on my
system, I saw /usr/lib/debug/ go from ~20GB to ~7.5GB.

This requires binutils/gdb to be built with USE=zlib which is why
this defaults to off.

Signed-off-by: Mike Frysinger <vapier< at >gentoo.org>
---
 bin/ebuild-helpers/prepstrip |    6 ++++--
 man/make.conf.5              |   10 +++++++++-
 pym/_emerge/EbuildPhase.py   |    6 ++++--
 pym/portage/const.py         |    2 +-
 4 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/bin/ebuild-helpers/prepstrip b/bin/ebuild-helpers/prepstrip
index ee547f2..daaa252 100755
--- a/bin/ebuild-helpers/prepstrip
+++ b/bin/ebuild-helpers/prepstrip
< at >< at > -15,7 +15,7 < at >< at > exp_tf() {
 eval ${var}_${flag}=$(tf has ${flag} ${!var})
 done
 }
-exp_tf FEATURES installsources nostrip splitdebug
+exp_tf FEATURES compressdebug installsources nostrip splitdebug
 exp_tf RESTRICT binchecks installsources strip
 
 [[ " ${FEATURES} " == *" force-pre

[PATCH] prepstrip: add writable workaround for everyone

Mike Frysinger — 2012-01-13T19:52:47

The writable issue shows up when using `ebuild` as non-root users
in non-prefix setups.  So always do it.

Signed-off-by: Mike Frysinger <vapier< at >gentoo.org>
---
 bin/ebuild-helpers/prepstrip |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/bin/ebuild-helpers/prepstrip b/bin/ebuild-helpers/prepstrip
index 1d7b3d2..ee547f2 100755
--- a/bin/ebuild-helpers/prepstrip
+++ b/bin/ebuild-helpers/prepstrip
< at >< at > -224,7 +224,7 < at >< at > do
 # unwritable objects.  Make them temporarily writable for the
 # stripping.
 was_not_writable=false
-if [[ -n ${EPREFIX} && ! -w ${x} ]] ; then
+if [[ ! -w ${x} ]] ; then
 was_not_writable=true
 chmod u+w "${x}"
 fi

Is there any short syntax for REQUIRED_USE when a lot of USE flags need another one enabled?

Pacho Ramos — 2011-12-17T10:24:37

I am referring in this case to abiword, it has a "plugins" USE flag that
enables some minimal set of plugins and, then, a lot of USE flags for
building extra plugins (with extra dependencies). All of this extra
plugins need "plugins" USE flag to be enabled. Is there any way to write
a REQUIRED_USE flag variable without needing to list all USE flags
depending on "plugins" to be set?

Thanks a lot for the info :)

emaint rewrite

Brian Dolbec — 2011-12-11T04:20:48

I have just rebased the emaint re-write that I did some time ago.  I
have also updated the modules for changes since I originally did the
rewrite into a plug-in modules system.

The changes have been condensed down into several key commits.

The emaint branch is available on github:

https://github.com/dol-sen/portage

It is now very easy to add or remove modules, by only adding/removing
the modules directory to/from the emaint/modules directory.

All the help and menu options are picked up and added automatically.
The modules.py code also only imports the individual module's
__init__.py which contains the module_spec.  The target module is only
imported when it is requested, keeping speed and overhead to a minimum.

The overall benefit to portage is that emaint should be easier to
maintain, the modules are also now available to portage/emerge or other
tools for internal script use, not just the bin/emaint script.  The
plug-in code in modules.py is also universal in nature and can easily be
moved elsewhere i

Is portage faster when using python3?

Pacho Ramos — 2011-11-09T00:38:10

Reading:
http://en.gentoo-wiki.com/wiki/Portage_tips#Portage_and_python3

Looks like some people think portage is faster with python3, is that
true? I have python3 set as main interpreter (for catching packages
failing to build/work with it), but my portage is merged without any
python* USE flag, I guess, it's still using python2, no?

If it's faster really, is python3 safe to use with portage or it's only
offered for testing purposes?

Thanks for the info :)

How to make portage create /var/log/portage/ with portage:portage owners?

Pacho Ramos — 2011-11-01T12:28:42

I have user "pacho" under "portage" group to be able to make some tasks
without becoming root, but, sadly, I am unable to make portage
create /var/log/portage files/dirs with portage:portage owners instead
of portage:root, preventing me from removing old files without becoming
root.
Is there anyway to configure this?

Thanks a lot

sleep 1 in misc-functions.sh

Fabian Groffen — 2011-10-20T11:20:14

The full context of this message is from a thread on gentoo-alt ml:
http://archives.gentoo.org/gentoo-alt/msg_db73b1a140fd958efb88f2437170646d.xml

Long story short, this person has to deal with a fileserver that sets
the rw world bits on for all files that are created.
While I don't think the suggested patch is fair (this is obviously a
very badly configured fileserver), I do wonder if the "sleep 1" that is
in the code is actually desired.  In case we want it, I would suggest we
move down the "sleep 1" so it's only done once for the entire list with
files.

Can we use eqawarn in bin/misc-functions.sh:177 instead and avoid the sleep?

[PATCH 0 of 3] repoman changelog patches

Fabian Groffen — 2011-10-19T19:55:43

The following three patches change repoman's echangelog feature
slightly:

1) default for updating the changelog is retrieved from layout.conf
2) skel.ChangeLog is used for constructing the header of new ChangeLogs
3) modified files are checked for copyright years to be updated

Please review before I push them.

[PATCH 1/2] prepstrip: extract buildid with readelf to avoid debugedit when possible

Mike Frysinger — 2011-10-11T04:50:53

The readelf utility is much more common than debugedit.

Signed-off-by: Mike Frysinger <vapier< at >gentoo.org>
---
 bin/ebuild-helpers/prepstrip |   32 +++++++++++++++++++++++---------
 1 files changed, 23 insertions(+), 9 deletions(-)

diff --git a/bin/ebuild-helpers/prepstrip b/bin/ebuild-helpers/prepstrip
index 67ceead..7a08aba 100755
--- a/bin/ebuild-helpers/prepstrip
+++ b/bin/ebuild-helpers/prepstrip
< at >< at > -26,10 +26,13 < at >< at > if ${RESTRICT_strip} || ${FEATURES_nostrip} ; then
 ${FEATURES_installsources} || exit 0
 fi
 
-STRIP=${STRIP:-${CHOST}-strip}
-type -P -- ${STRIP} > /dev/null || STRIP=strip
-OBJCOPY=${OBJCOPY:-${CHOST}-objcopy}
-type -P -- ${OBJCOPY} > /dev/null || OBJCOPY=objcopy
+# look up the tools we might be using
+for t in STRIP:strip OBJCOPY:objcopy READELF:readelf ; do
+v=${t%:*} # STRIP
+t=${t#*:} # strip
+eval ${v}=\"${!v:-${CHOST}-${t}}\"
+type -P -- ${!v} >/dev/null || eval ${v}=${t}
+done
 
 # We'll leave out -R .note for now until we can check out the relevance
 # of the section when it

[PATCH 0/4] make testing more user friendly

Mike Frysinger — 2011-10-09T18:53:24

I'm not an expert at python, so trying to make sure I'm doing anything
too stupid.  I tried to keep existing portage style, but feel free to
point out whatever.

I can also push these myself, so if people are OK with them, just let
me know ... you don't have to merge+push yourself.

Mike Frysinger (4):
  runtests: make sure we are in the right dir
  tests: split up getTests into helper funcs to avoid duplication
  tests: add a --help option to make runtest more friendly
  tests: add --list flag to show available tests

 pym/portage/tests/__init__.py |   93 ++++++++++++++++++++++++-----------------
 runtests.sh                   |    3 +
 2 files changed, 57 insertions(+), 39 deletions(-)

[GLEP59v2 0/5] GLEP59: Manifest2 hash types

Robin H. Johnson — 2011-10-01T07:40:50

Respun now with the help of ferringb. Cleans up the implementation and catches
a few bug and improvements:
- mhash priority moved lower than pycrypto/hashlib because mhash holds GIL
  while the other implementations don't.
- hashlib does offer whirlpool if it was built against openssl 1.0.

1/5: Refactor RMD160 hashlib code for less-hardcoding
2/5: Manifest2 hash: Whirlpool
3/5: Manifest2 hash: SHA512
4/5: Manifest2 hash backend provider: mhash
5/5: GLEP59: Change live Manifest2 hashes to SHA256,

GLEP59: Manifest2 hash types implementation

Robin H. Johnson — 2011-09-30T01:27:38

For an overview of tree-signing, please see informational GLEP57.

This patch series implements all parts needed for GLEP59. Upcoming patch series
will include support for the rest of the tree-signing, GLEP59-61. 

[1/4] Manifest2 hash: Whirlpool
[2/4] Manifest2 hash: SHA512
New hashes.

[3/4] Manifest2 hash backend provider: mhash
New hash provider. 
Please note that neither pyopenssl nor python-gnutls provide suitable hashing
functionality (pyopenssl misses hashing, gnutls misses WHIRLPOOL).

[4/4] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL
Per subject. Change the actual defaults.