gmane.comp.python.distutils.devel

entry_points.txt survey

Daniel Holth — 2012-05-25T00:48:40

It looks like you can call pkg_resources.register_finder() with a
function that also yields distributions with a .dist-info directory
(though you would have to override the existing find_on_path). From
there I think pkg_resources.load_entry_point() is likely to work.

package surveyed), 5878 define entry_points.txt but 1399 of those are
empty. Some of the most popular sections are:

 (10, 'babel.extractors'),
 (10, 'lava_server.extensions'),
 (10, 'paste.composite_factory'),
 (10, 'turbogears.extensions'),
 (10, 'yafowil.plugin'),
 (10, 'zest.releaser.prereleaser.middle'),
 (11, 'chandler.parcels'),
 (11, 'rsl.register'),
 (12, 'hurry.resource.libraries'),
 (15, 'paste.global_paster_command'),
 (16, 'pyramid.scaffold'),
 (17, 'zc.buildout.uninstall'),
 (18, 'paste.server_runner'),
 (18, 'setuptools.installation'),
 (19, 'python.templating.engines'),
 (20, 'redsolutioncms'),
 (20, 'zope2.initialize'),
 (22, 'pytest11'),
 (23, 'setuptools.file_finders'),
 (26, 'paste.paster_command'),
 (30, 'nose.plugins'),
 (31, 'paste.filter_factory'),
 (31, 'zc.buildout.extension'),
 (35, 'toscawidgets.widgets'),
 (35, 'turbogears.widgets'),
 (37, 'tw2.widgets'),
 (47, 'paste.app_install'),
 (69, 'nose.plugins.0.10'),
 (73, 'distutils.commands'),
 (81, 'trytond.modules'),
 (82, 'gui_scripts'),
 (83, 'paste.filter_app_factory'),
 (94, 'fanstatic.libraries'),
 (96, 'trac.plugins'),
 (110, 'egg_info.writers'),
 (120, 'distutils.setup_keywords'),
 (144, 'paste.paster_create_template'),
 (190, 'paste.app_factory'),
 (354, 'zc.buildout'),
 (1004, 'z3c.autoinclude.plugin'),
 (1754, 'console_scripts')
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: command hooks...

PJ Eby — 2012-05-25T00:27:39


What plugin mechanism are you talking about here, specifically?

For that matter, which "existing entry points code" are you talking about,
too?  ;-)
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: command hooks...

Daniel Holth — 2012-05-24T22:56:12

'packaging' will be powerful enough to copy entry_points.txt into the
.dist-info directory as a resource. Python bug #11880.

The existing entry points code has a plugin mechanism for itself that
will make it easy to iterate over entry_points.txt in dist-info as
well as egg-info directories. Just write that plugin and you should be
good to go. Though you might have to install the plugin in an egg-info
directory at first...
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: command hooks...

Erik Bray — 2012-05-24T21:37:14

Sorry to hear about that.


For something like Pyramid I don't actually think it would be too
difficult to get it *mostly* working with packaging/distutils2 in its
present state.  The word is "mostly".  And that's to say nothing of
all its dependencies, though they don't all necessarily need to work
with the same installer.

For something like Pyramid the main piece that's missing is support
for an entry_points like system.  As Tarek and other have pointed out
the past, such a system could still be supported through a third-party
plugin system that works via setup hooks and custom metadata.  I think
there was even a prototype for something like that at one point...

But that raises another issue:  If packaging is going to be like a
"framework" that additional features like plugin systems or console
script generators can be tacked on to, there needs to be a way to
specify build dependencies in setup.cfg, and to have those
dependencies automatically downloaded and added to sys.path during
setup, a la setup_requires in setuptools.

I know there have been discussions and proposals in the past to add
something like this to packaging, and to amend PEP 345 to include it.
Metadata for test dependencies and docs dependencies have also been
discussed in the past.  But I think a way of adding build/setup
dependencies is critical if packaging is going to be useful as a
framework.

I have a few packages that use d2to1, which supports setup_requires
through setuptools.  Thanks to that, I'm able to make those packages
have an additional setup_requirement of a package that contains
several pre-canned setup_hooks that I use in all my projects.  I would
point out specific examples, but they're a bit obscure...

Anyways, I'm working right now to finish a software release.  But then
I want to spend some time working on this issue of adding build
requirements support to packaging and to PEP 345.  I might also work
on a plugin system that can work in packaging as a replacement for
entry_points, if no one else does.  Without these features I don't see
packaging being very successful as an installation framework, IMO.

Erik
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: command hooks...

Chris McDonough — 2012-05-24T03:59:48

It sounds like a reasonable task for me to try out.  In the meantime 
we've had a bit of a family emergency which will take some time to 
overcome and I won't be able to dedicate much energy to this.  As a 
result, I have to declare bankruptcy here, so I'll have to live with 
whatever I get.

I'm hoping though that someone else will step up here and do an 
evaluation and try to get things like Pyramid and popular Zope packages 
installed in a way that makes sense for straddling Python 2 and Python 
3.  I suspect if no one does this, it's going to be rough going.

- C
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

pypm binary package format

Daniel Holth — 2012-05-23T17:12:56

I was curious about the pypm binary package format. They are in the
.tar.gz format (but with .pypm extension). They contain two files:

info.json:

{u'author': u'Zope Foundation and Contributors',
 u'author_email': u'zope-dev< at >zope.org',
 u'description': '...',
 u'home_page': u'http://pypi.python.org/pypi/zope.interface',
 u'install_requires': {u'': [u'distribute'],
                       u'docs': [u'z3c.recipe.sphinxdoc'],
                       u'test': [u'zope.event']},
 u'keywords': None,
 u'license': u'ZPL 2.1',
 u'maintainer': None,
 u'maintainer_email': None,
 u'name': u'zope.interface',
 u'osarch': u'win32-x86',
 u'pkg_version': 1,
 u'pyver': u'2.7',
 u'summary': u'Interfaces for Python',
 u'version': u'3.8.0'}

data.tar.gz:

Lib/site-packages/zope.interface/... (including .pyd or .so)
Lib/site-packages/zope.interface-3.8.0-py27.egg-info/(normal egg-info stuff)

When installed, Lib/ is unpacked into %APPDATA%\Python\Python27\site-packages\
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: command hooks...

Robert Park — 2012-05-23T16:27:02

Yeah, the [files] definition in setup.cfg looks fine, the space syntax
in the source definition I would consider highly important in order to
avoid installing things into {datadir}/doc/foo/doc/ as given in the
example.


I didn't see any mention of this aspect in the documentation you
linked. What does the API look like?

command hooks...

Oscar Benjamin — 2012-05-23T11:50:54



How do gettext translation files fit into this scheme? Should they go under
appdata?


_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: command hooks...

Tarek Ziadé — 2012-05-23T06:48:27

Please read this section and let us know what you think:

http://docs.python.org/dev/packaging/setupcfg.html#resources

This works in conjunction with the new sysconfig module, which can be 
configured system-wide by the linux distribution, or locally per projet

Then you can use an API to get the file from your code.

Gosh the documentation is a mess ... we need to fix this - it has bits 
from the previous version that should be removed :(

Cheers
Tarek
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: command hooks...

Robert Park — 2012-05-23T06:16:01

This right here is the killer feature for me. By my limited
observations, most people solve the data files problem either by
dumping their data files inside their python modules (which is an ugly
abuse of the filesystem), or are simply using the cumbersome autotools
in order to record the installation prefix for their data files. It
blows my mind that it is standard practice for many GNOME apps written
in Python to use a C compilation preprocessor in order to set a python
variable so their python scripts can find their data files.

Currently I am hacking distutils in order to accomplish this, like so:

https://github.com/robru/gottengeography/blob/master/setup.py

But this is ugly because it modifies the file in-place, so I always
have to be careful not to accidentally commit the munged file into my
git repo.

It's absolutely critical that any replacement for distutils have
built-in functionality for installed python code being able to query
at run-time the location that data files were placed at install time.

Re: command hooks...

Erik Bray — 2012-05-22T12:58:48

I don't usually do this, and I think most people on this list are
already aware of it, but this seems like a decent place to insert a
shameless plug for d2to1 [1].  d2to1 allows Python developers to take
advantage of many of the features and advantages of packaging
today--in particular static metadata and hook functions--while still
supporting existing installers like easy_install and pip.  It's a
total hack, and only meant to aid with transition, but it actually
works quite well in practice.

Unfortunately I haven't found the time to work on it in a little while
so I need to find that time.  In particular it would be useful for it
to add resources support.

Erik

[1] http://pypi.python.org/pypi/d2to1
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: command hooks...

Tarek Ziadé — 2012-05-21T20:59:20

yes.

on a side note: The PyPI stats numbers are biased for many libraries 
because they are artificially increased by all the buildouts and Jenkins 
and Travis-CI calls out there.
IOW: if a server downloads 100 times a day "PasteDeploy" because someone 
did not set a cache, does it make it 100 times more popular ?
Same goes for any library that's a core part of the non-monolithic 
frameworks out there.


we want to make the dist-info  (see PEP 376) structure a directory where 
you can drop anything, so you can have arbitrary metadata. Right now, 
entry_points.txt is copied over in that directory by pysetup.

the new setup.cfg file allows you to define extra metadata as well, and 
what's awesome: we want to publish that static file to PyPI so tools can 
browse it !

=> http://docs.python.org/dev/packaging/setupcfg.html#extensibility


So what about this suggestion:

let's make this goal : pysetup should be able to install Pyramid 
(without [extras] sorry, we can live without it) - with a few 
adaptations in a branch if needed.

if we can make it work before 3.3rc1, great. If we fail we remove the 
installer part of packaging and move it to an external project

That's just a suggestion, I'll defer the decision to Eric because while 
I can help around, I don't have the time, neither the motivation to do 
packaging work these days.

But we have to hurry up - and check with Guido if he's ok with those 
decisions.

Cheers
Tarek


_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: command hooks...

Tarek Ziadé — 2012-05-21T20:42:56

Who are you ?

If you are a user that just installs software, you might want to use it 
because you can *uninstall* things. This is now less relevant now that 
pip has this feature, but easy_install does not. And with the PEP we 
wrote, any tool should be able to install/uninstall any project. And 
well, have this feature bundled into Python makes sense.

If you are a packager for a project, you can describe in details your 
data files, and add more metadata that are understood by PyPI.

If you are a debian packager, you will be able to define where the data 
files of a python project should be installed without having to patch 
some python code.

I have much more but it's a bit hard to summarize this in a mail - I 
have a packaging fatigue anyways.

It really depends what you put behind the word "better". And no, My task 
is not to convince you that packaging is better.

My tasks were :

1 - try to define standards which every packaging tool can implement, 
for the sake of interoperability, have a consensus on them, have them 
"accepted"
2 - allow more "static" definitions of metadata
3 - add in the standard library
   a/ a reference implementation for all the standards
   b/ a minimal installer.

3-b is not ready for prime time obviously - but this should not take too 
long.


Cheers
Tarek




_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: packaging in Python-3.3.0 ???

Daniel Holth — 2012-05-19T11:01:01

They are the same. In python 3.3 distutils2 is called packaging.

http://stackoverflow.com/questions/6344076/differences-between-distribute-distutils-and-setuptools

Daniel Holth

On May 19, 2012, at 12:49 AM, Rob Healey <robhealey1< at >gmail.com> wrote:

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

packaging in Python-3.3.0 ???

Rob Healey — 2012-05-19T04:49:17

Hello once agai:

Is packaging in Python-3.3.0 a more complete software than DU2???  If it
is, then I will start using it instead...?

Re: Distutils2 in Python-3.3.0

Daniel Holth — 2012-05-19T04:05:04

I think the process may be to checkout Python from
hg.python.org/cpython, edit the 'packaging' module, and then those
fixes should wind up in the next distutils2 backport. I've been using
bitbucket to publish my changes, and the Python bug tracker can
generate patches from external Mercurial repositories.

I've already implemented two things that are important to me in
http://bitbucket.org/dholth/cpython . A {dist-info} category in
setup.cfg to add arbitrary files to the .dist-info directory (intended
for entry_points.txt) Python bug #11880.

And nearly-PEP-376-compliant relative paths in RECORD, because I
really want to be able to move site-packages around (or use a chroot)
without rewriting RECORD.

Does PEP-317 really mean to say "use / instead of os.path.sep for
relative paths on any platform, but the platform path separator on
absolute paths"? I notice pip uses paths relative to the .egg-info
directory instead of its parent directory which seems a bit simpler to
implement, but not every path can be relative on Windows.

Daniel
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Distutils2 in Python-3.3.0

Rob Healey — 2012-05-19T03:14:29

Good evening to all!

I am also willing to help out too...

What is the process for submitting patches?  But I must, in all respects,
patches are USELESS without someone to commit them!!!

There needs to be more than one person who can commit the patches...

Re: Odd problem with distribute 0.6.26 and 2to3

Arfrever Frehtes Taifersar Arahesis — 2012-05-18T20:30:17

2012-05-18 17:24:59 Vinay Sajip napisał(a):

Try with Distribute 0.6.27, which fixes support for current snapshots of CPython 3.3.

Odd problem with distribute 0.6.26 and 2to3

Vinay Sajip — 2012-05-18T15:24:59

I'm having an odd problem with distribute and 2to3 when working with the venv 
branch of Python 3.3 (PEP 405 - http://bugs.python.org/issue14712), and I'm 
hoping one of the Fellowship of the Packaging can help.

The problem: I created a venv with no Distribute installed. Then I downloaded 
and extracted distribute-0.6.26.tar.gz in a scratch folder, and ran setup.py 
install using the venv's python. This gave errors because some of the setuptools 
sources still have 2.x syntax, despite having been processed by 2to3.

To investigate further, I added logging to distutils refactoring code to print 
what was being refactored. I won't list the entire log output [1], but when 
looking at just one file which caused a SyntaxError - 
setuptools/tests/test_packageindex.py - I see the following output from the 
"setup.py install" operation:

creating build
creating build/src
...
copying setuptools/tests/test_packageindex.py -> build/src/setuptools/tests
...
copying launcher.c -> build/src
Skipping implicit fixer: buffer
Skipping implicit fixer: idioms
Skipping implicit fixer: set_literal
Skipping implicit fixer: ws_comma
refactored build/src/setuptools/script template (dev).py
...
refactored build/src/setuptools/tests/test_packageindex.py
...
refactored build/src/release.py
Before install bootstrap.
Scanning installed packages
No setuptools distribution found
running install
running build
running build_py
creating build/lib
...
copying setuptools/tests/test_packageindex.py -> build/lib/setuptools/tests
...
creating /tmp/venv/lib/python3.3/site-packages/setuptools/tests
copying build/lib/setuptools/tests/test_packageindex.py -> 
/tmp/venv/lib/python3.3/site-packages/setuptools/tests
...
running install_lib
...
creating /tmp/venv/lib/python3.3/site-packages/setuptools
creating /tmp/venv/lib/python3.3/site-packages/setuptools/tests
copying build/lib/setuptools/tests/test_packageindex.py -> 
/tmp/venv/lib/python3.3/site-packages/setuptools/tests
...
byte-compiling /tmp/venv/lib/python3.3/site-
packages/setuptools/tests/test_packageindex.py to test_packageindex.cpython-
33.pyc
...
running install_egg_info
Writing /tmp/venv/lib/python3.3/site-packages/distribute-0.6.26-py3.3.egg-info
setup.py:139: ResourceWarning: unclosed file <_io.TextIOWrapper 
name='README.txt' mode='r' encoding='UTF-8'>
  long_description = open('README.txt').read() + open('CHANGES.txt').read(),
setup.py:139: ResourceWarning: unclosed file <_io.TextIOWrapper 
name='CHANGES.txt' mode='r' encoding='UTF-8'>
  long_description = open('README.txt').read() + open('CHANGES.txt').read(),
/home/vinay/projects/python/sandbox/Lib/distutils/dist.py:257: UserWarning: 
Unknown distribution option: 'test_suite'
  warnings.warn(msg)
/home/vinay/projects/python/sandbox/Lib/distutils/dist.py:257: UserWarning: 
Unknown distribution option: 'entry_points'
  warnings.warn(msg)
/home/vinay/projects/python/sandbox/Lib/distutils/dist.py:257: UserWarning: 
Unknown distribution option: 'zip_safe'
  warnings.warn(msg)
  File "/tmp/venv/lib/python3.3/site-
packages/setuptools/tests/test_packageindex.py", line 17
    except Exception, v:
                    ^
SyntaxError: invalid syntax

Now the first part seems straightfoward: files are copied to build/src and then 
2to3 is run over them. But then, following the "No setuptools distribution 
found", files are copied to build/lib not from the 2to3-processed output at 
build/src, but rather the actual 2.x sources shipped with distribute. That looks 
wrong; at the point where setup is called, src_root is correctly set to 
build/src, and moreover, that's the first entry on sys.path. I'm not sure why 
there's no error immediately following the "byte-compiling ..." line when 
running install_lib, but the error seems to occur a little later, when running 
install_egg_info ...  

Can anyone shed any light on what's happening here?

Regards,


Vinay Sajip

[1] https://gist.github.com/2725747

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: two setup scripts

Chris Jerdonek — 2012-05-18T13:09:22

I asked the question below about a week ago.  Does anyone have any
thoughts on it?

I'll rephrase it to be simpler.  Does anyone know which arguments to
setup() it is safe to leave out when invoking run "setup.py install"
from a source distribution?  I'm assuming the source distribution
directory is one created using "setup.py sdist" (i.e. one that
contains a PKG-INFO file, a *.egg-info directory, etc).  For example,
it doesn't seem like long_description would be needed.

Since the setup() function was created to be a universal function
usable by both module developers and installers, it seems like it
would be useful to know which subset of arguments need to be provided
in the latter, simpler case.  It would also help to know where the
install command looks for and gets its information from (from the
arguments to setup(), or from the PKG-INFO file, etc).

It seems like a possible improvement to distutils would be for it to
be able to say which arguments to setup() are needed or used by which
setup commands, so that things can be more explicit.  Does that seem
like a worthwhile improvement?  It might be an intermediate step to
making the function less God-like (if that itself is a worthy goal).

Thanks,
--Chris


On Fri, May 11, 2012 at 11:36 AM, Chris Jerdonek
<chris.jerdonek< at >gmail.com> wrote:
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig

{dist-info} category in resources = ...

Daniel Holth — 2012-05-17T04:03:20

My take on installing entry_points.txt is to add a {dist-info}
category in [files] resources = to copy extra files into the
.dist-info dir...

How do I write entry_points.txt when it is given as an argument to
setup(entry_points=...)? Should I write a build command hook to write
the file and a setup_hook to append the new file to [files] resources
= so it will finally be copied?
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG< at >python.org
http://mail.python.org/mailman/listinfo/distutils-sig