http://blog.gmane.org/gmane.comp.python.devel hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://comments.gmane.org/gmane.comp.python.devel/98639 About a month ago, I submitted two patches that address Pdb and doctest inability to load source code from modules with custom loaders such as modules loaded from zip files: http://bugs.python.org/issue4201 http://bugs.python.org/issue4197 The patches are very simple, basically calls to linecache.getline() need to be provided with the module's dict to enable linecache to find the module's __loader__. Is there a chance that these patches could make it to 2.6.1? _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Alexander Belopolsky 2008-12-03T03:44:43 http://comments.gmane.org/gmane.comp.python.devel/98634 Hi all, Currently, Parser/parsetok.c has a dependency on graminit.h. This can cause headaches when rebuilding after adding new syntax to Grammar/Grammar because parsetok.c is part of pgen, which is responsible for *generating* graminit.h. This circular dependency can result in parsetok.c using a different value for encoding_decl to what is used in ast.c, which causes PyAST_FromNode to fall over at runtime. It effectively looks something like this: * Grammar/Grammar is modified * build begins -- pgen compiles, parsetok.c uses encoding_decl=X * graminit.h is rebuilt with encoding_decl=Y * ast.c is compiled using encoding_decl=Y * when python runs, parsetok() emits encoding_decl nodes that PyAST_FromNode can't recognize: SystemError: invalid node XXX for PyAST_FromNode A nice, easy short term solution that doesn't require unwinding this dependency would be to simply move encoding_decl to the top of Grammar/Grammar and add a big warning noting that it needs to come before everything else. This will help to ensure its value never changes when syntax is added/removed. I'm happy to provide a patch for this (including some additional dependency info for files dependent upon graminit.h and Python-ast.h), but was wondering if there were any opinions about how this should be resolved. Cheers, Tom _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Thomas Lee 2008-12-02T06:57:11 http://comments.gmane.org/gmane.comp.python.devel/98621 Tracing support shows up fairly heavily an a Python profile, even though it's nearly always turned off. The attached patch against the trunk speeds up PyBench by 2% for me. All tests pass. I have 2 questions: 1) Can other people corroborate this speedup on their machines? I'm running on a Macbook Pro (Intel Core2 processor, probably Merom) with a 32-bit build from Apple's gcc-4.0.1. (Apple's gcc consistently produces a faster python than gcc-4.3.) 2) Assuming this speeds things up for most people, should I check it in anywhere besides the trunk? I assume it's out for 3.0; is it in for 2.6.1 or 3.0.1? Pybench output: ------------------------------------------------------------------------------- PYBENCH 2.0 ------------------------------------------------------------------------------- * using CPython 2.7a0 (trunk:67458M, Nov 30 2008, 17:14:10) [GCC 4.0.1 (Apple Inc. build 5488)] * disabled garbage collection * system check interval set to maximum: 2147483647 * using timer: time.time ------------------------------------------------------------------------------- Benchmark: pybench.out ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.time Machine Details: Platform ID: Darwin-9.5.0-i386-32bit Processor: i386 Python: Implementation: CPython Executable: /Users/jyasskin/src/python/trunk-fast-tracing/build/python.exe Version: 2.7.0 Compiler: GCC 4.0.1 (Apple Inc. build 5488) Bits: 32bit Build: Nov 30 2008 17:14:10 (#trunk:67458M) Unicode: UCS2 ------------------------------------------------------------------------------- Comparing with: ../build_orig/pybench.out ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.time Machine Details: Platform ID: Darwin-9.5.0-i386-32bit Processor: i386 Python: Implementation: CPython Executable: /Users/jyasskin/src/python/trunk-fast-tracing/build_orig/python.exe Version: 2.7.0 Compiler: GCC 4.0.1 (Apple Inc. build 5488) Bits: 32bit Build: Nov 30 2008 13:51:09 (#trunk:67458) Unicode: UCS2 Test minimum run-time average run-time this other diff this other diff ------------------------------------------------------------------------------- BuiltinFunctionCalls: 127ms 130ms -2.4% 129ms 132ms -2.1% BuiltinMethodLookup: 90ms 93ms -3.2% 91ms 94ms -3.1% CompareFloats: 88ms 91ms -3.3% 89ms 93ms -4.3% CompareFloatsIntegers: 97ms 99ms -2.1% 97ms 100ms -2.4% CompareIntegers: 79ms 82ms -4.2% 79ms 85ms -6.1% CompareInternedStrings: 90ms 92ms -2.4% 94ms 94ms -0.9% CompareLongs: 86ms 83ms +3.6% 87ms 84ms +3.5% CompareStrings: 80ms 82ms -3.1% 81ms 83ms -2.3% CompareUnicode: 103ms 105ms -2.3% 106ms 108ms -1.5% ComplexPythonFunctionCalls: 139ms 137ms +1.3% 140ms 139ms +0.1% ConcatStrings: 142ms 151ms -6.0% 156ms 154ms +1.1% ConcatUnicode: 87ms 92ms -5.4% 89ms 94ms -5.7% CreateInstances: 142ms 144ms -1.4% 144ms 145ms -1.1% CreateNewInstances: 107ms 109ms -2.3% 108ms 111ms -2.1% CreateStringsWithConcat: 114ms 137ms -17.1% 117ms 139ms -16.0% CreateUnicodeWithConcat: 92ms 101ms -9.2% 95ms 102ms -7.2% DictCreation: 77ms 81ms -4.4% 80ms 85ms -5.9% DictWithFloatKeys: 91ms 107ms -14.5% 93ms 109ms -14.6% DictWithIntegerKeys: 95ms 94ms +1.4% 108ms 96ms +12.3% DictWithStringKeys: 83ms 88ms -5.8% 84ms 88ms -4.7% ForLoops: 72ms 72ms -0.1% 79ms 74ms +5.8% IfThenElse: 83ms 80ms +3.9% 85ms 80ms +5.3% ListSlicing: 117ms 118ms -0.7% 118ms 121ms -1.8% NestedForLoops: 116ms 119ms -2.4% 121ms 121ms +0.0% NormalClassAttribute: 106ms 115ms -7.7% 108ms 117ms -7.7% NormalInstanceAttribute: 96ms 98ms -2.3% 97ms 100ms -3.1% PythonFunctionCalls: 92ms 95ms -3.7% 94ms 99ms -5.2% PythonMethodCalls: 147ms 147ms +0.1% 152ms 149ms +2.1% Recursion: 135ms 136ms -0.3% 140ms 144ms -2.9% SecondImport: 101ms 99ms +2.1% 103ms 101ms +2.2% SecondPackageImport: 107ms 103ms +3.5% 108ms 104ms +3.3% SecondSubmoduleImport: 134ms 134ms +0.3% 136ms 136ms -0.0% SimpleComplexArithmetic: 105ms 111ms -5.0% 110ms 112ms -1.4% SimpleDictManipulation: 95ms 106ms -10.6% 96ms 109ms -12.0% SimpleFloatArithmetic: 90ms 99ms -9.3% 93ms 102ms -8.2% SimpleIntFloatArithmetic: 78ms 76ms +2.3% 79ms 77ms +2.0% SimpleIntegerArithmetic: 78ms 77ms +1.8% 79ms 77ms +2.0% SimpleListManipulation: 80ms 78ms +2.4% 80ms 79ms +1.9% SimpleLongArithmetic: 110ms 113ms -2.0% 111ms 113ms -2.1% SmallLists: 128ms 117ms +9.5% 130ms 124ms +4.9% SmallTuples: 115ms 114ms +1.7% 117ms 114ms +2.2% SpecialClassAttribute: 101ms 112ms -10.3% 104ms 114ms -8.9% SpecialInstanceAttribute: 173ms 177ms -1.9% 176ms 179ms -1.6% StringMappings: 165ms 167ms -1.2% 168ms 169ms -0.5% StringPredicates: 126ms 134ms -5.7% 127ms 134ms -5.6% StringSlicing: 125ms 123ms +1.9% 131ms 130ms +0.7% TryExcept: 79ms 80ms -0.6% 80ms 80ms -0.8% TryFinally: 110ms 107ms +3.0% 111ms 112ms -1.1% TryRaiseExcept: 99ms 101ms -1.6% 100ms 102ms -1.7% TupleSlicing: 127ms 127ms +0.6% 137ms 137ms +0.0% UnicodeMappings: 144ms 144ms -0.3% 145ms 145ms -0.4% UnicodePredicates: 116ms 114ms +1.3% 117ms 115ms +1.1% UnicodeProperties: 106ms 102ms +3.6% 107ms 104ms +3.1% UnicodeSlicing: 95ms 111ms -14.0% 99ms 112ms -11.8% WithFinally: 157ms 152ms +3.3% 159ms 154ms +3.3% WithRaiseExcept: 123ms 125ms -1.1% 125ms 126ms -1.2% ------------------------------------------------------------------------------- Totals: 6043ms 6182ms -2.2% 6185ms 6301ms -1.9% (this=pybench.out, other=../build_orig/pybench.out) 2to3 times: Before: $ time ./python.exe ~/src/2to3/2to3 -f all ~/src/2to3/ >/dev/null real0m56.685s user0m55.620s sys0m0.380s After: $ time ./python.exe ~/src/2to3/2to3 -f all ~/src/2to3/ >/dev/null real0m55.067s user0m53.843s sys0m0.376s == 3% faster Gory details: The meat of the patch is: < at >< at > -884,11 +891,12 < at >< at > fast_next_opcode: f->f_lasti = INSTR_OFFSET(); /* line-by-line tracing support */ -if (tstate->c_tracefunc != NULL && !tstate->tracing) { +if (_Py_TracingPossible && + tstate->c_tracefunc != NULL && !tstate->tracing) { This converts the generated assembly (produced with `gcc -S -dA ...`, then manually annotated a bit) from: # basic block 17 # ../Python/ceval.c:885 LM541: movl8(%ebp), %ecx LVL319: subl-316(%ebp), %edx movl%edx, 60(%ecx) # ../Python/ceval.c:889 LM542: # %esi = tstate movl-336(%ebp), %esi LVL320: # %eax = tstate->c_tracefunc movl28(%esi), %eax LVL321: # if tstate->c_tracefunc == 0 testl%eax, %eax # goto past-if () jeL567 # more if conditions here to: # basic block 17 # ../Python/ceval.c:889 LM542: movl8(%ebp), %ecx LVL319: subl-316(%ebp), %edx movl%edx, 60(%ecx) # ../Python/ceval.c:893 LM543: # %eax = _Py_TracingPossible movl__Py_TracingPossible-"L00000000033$pb"(%ebx), %eax LVL320: # if _Py_TracingPossible != 0 testl%eax, %eax # goto rest-of-if (nearby) jneL2321 # opcode = NEXTOP(); continues here The branch should be predicted accurately either way, so there are 2 things that may be contributing to the performance change. First, adding the global caching variable halves the amount of memory that has to be read to check the prediction. The memory that is read is still read one instruction before it's used, but adding a local variable to read the memory earlier doesn't affect the performance. Without the global variable, the compiler puts the tracing code immediately after the if; with the global, it moves it away and puts the non-tracing code immediately after the first test in the if. This may affect branch prediction and may affect the icache. I tried using gcc's __builtin_expect() to ensure that the tracing code is always out-of-line. This moved it much farther away and cost about 1% in performance (i.e. 1% instead of 2% faster than "before"). I don't know why the __builtin_expect() version would be slower. If anyone feels inspired to test this out on another processor or compiler version, let me know how it goes. Jeffrey _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Jeffrey Yasskin 2008-12-01T01:54:02 http://comments.gmane.org/gmane.comp.python.devel/98604 This is my first message in this list, therefore I would like to say hello to everyone. I also hope, that I am not breaking any rules or guidelines for sending proposals. I would like to ask, if it is possible to provide type name of the object invoking the exception, when Attribute error is catched. It is done for functions, like: AttributeError: 'function' object has no attribute 'getValue' but for some objects there is only: AttributeError: connectToBases This is cool, when you know exactly what type of object cast the exception. But if there might be many of them, you must do one of two things: add print statement just before the line with the exception and check the type or iterate over all classes that might appear them. Showing the class name would solve this problem and could save a lot of time. Filip Gruszczyński 2008-11-30T18:41:55 http://comments.gmane.org/gmane.comp.python.devel/98599 I just ran "svn up" in my 2.6 sandbox and got this warning about the configure script: svn: File 'configure' has inconsistent newlines svn: Inconsistent line ending style I don't see any newlines other than just LF. Is some property on the file hosed? I should point out that it is 'M'odified in my sandbox, but I just ran autoconf (v2.63) to recreate it from a modified configure.in. Skip _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org skip< at >pobox.com 2008-11-30T04:02:49 http://comments.gmane.org/gmane.comp.python.devel/98598 I just ran "svn up" in my 2.6 sandbox and got this warning about the configure script: svn: File 'configure' has inconsistent newlines svn: Inconsistent line ending style I don't see any newlines other than just LF. Is some property on the file hosed? Skip _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org skip< at >pobox.com 2008-11-30T04:00:24 http://comments.gmane.org/gmane.comp.python.devel/98582 _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Python tracker 2008-11-28T17:06:44 http://comments.gmane.org/gmane.comp.python.devel/98573 A friend pointed out that running python under valgrind (simply "valgrind python") produces a lot of "invalid read" errors. Reading up on Misc/README.valgrind only seems to describe why "uninitialized reads" should occur, not invalid ones. For example: $ valgrind python [... lots of output ...] ==31428== Invalid read of size 4 ==31428== at 0x808EBDF: PyObject_Free (in /usr/bin/python2.5) ==31428== by 0x810DD0A: (within /usr/bin/python2.5) ==31428== by 0x810DD34: PyNode_Free (in /usr/bin/python2.5) ==31428== by 0x80EDAD9: PyRun_InteractiveOneFlags (in /usr/bin/python2.5) ==31428== by 0x80EDDB7: PyRun_InteractiveLoopFlags (in /usr/bin/python2.5) ==31428== by 0x80EE515: PyRun_AnyFileExFlags (in /usr/bin/python2.5) ==31428== by 0x80595E6: Py_Main (in /usr/bin/python2.5) ==31428== by 0x8058961: main (in /usr/bin/python2.5) ==31428== Address 0x43bf010 is 3,112 bytes inside a block of size 6,016 free'd ==31428== at 0x4024B4A: free (vg_replace_malloc.c:323) ==31428== by 0x8059C07: (within /usr/bin/python2.5) ==31428== by 0x80EDAA5: PyRun_InteractiveOneFlags (in /usr/bin/python2.5) ... valgrind claims that Python reads 4 bytes inside a block on which free() has already been called. Is valgrind wrong, or is Python really doing that? Googling revealed previous reports of this, normally answered by a reference to README.valgrind. But README.valgrind justifies reading from ununitialized memory, which doesn't help me understand how reading from the middle of a block of freed memory (more precisely, memory on which the libc free() has already been called) would be okay. I suppose valgrind could be confused by PyFree's pool address validation that intentionally reads the memory just before the allocated block, and incorrectly attributes it to a previously allocated (and hence freed) block, but I can't prove that. Has anyone investigated this kind of valgrind report? _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Hrvoje Niksic 2008-11-28T12:38:14 http://comments.gmane.org/gmane.comp.python.devel/98561 Dear All, Apologies that web site was not working earlier. I believe that I've now fixed it. The patch can still be found at http://www.cats-muvva.net/software/Python-taint-diff-3.0rc3.tar.bz2. CatsMuvva _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Nicole King 2008-11-27T22:57:10 http://comments.gmane.org/gmane.comp.python.devel/98554 Dear All, I found I needed support for taint mode in python and have done some work to realise this. It's by no means complete at this time, but I'm floating this idea on this group to see how much interest there is. The implementation is pretty simple: - an extra field in PyObject to maintain the taint status - a couple of extra functions __gettaint__() that returns the taint status and __settaint__(value) that sets the taint value, returning the previous status - an additional command-line flag -a and environment variable PYTHONIGNORETAINT that suppress taint checking - a few macros defined in Objects/object.h to support taint management - a new built-in exception, PyExc_TaintError, for reporting operations on tainted objects You can pick up the diff at http://www.cats-muvva.net/software/Python-taint-diff-3.0rc3.tar.bz2 This diff is ONLY for 3.0RC3. Given that I have only a limited understanding of the internals of Python (but it's very clearly written), I'm sure there are some behaviours I've overlooked. I have done very little testing at this point, but, at least, everything compiles! Regards Nicole King (aka CatsMuvva) _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Nicole King 2008-11-27T19:52:43 http://comments.gmane.org/gmane.comp.python.devel/98549 Python programmers need to dynamically load submodules instead of top-level modules -- given a string with module hierarchy, e.g. 'foo.bar.baz', access to the tail module 'baz' is required instead of 'foo'. Currently, the common hack for that is to use This, however, is indeed an undocumented hack and, what's worse, causes 'baz' to be imported twice, as 'baz' and 'baz.' (with tail dot). The problem is reported in [1] and the idiom pops up in about 2000 (sic!) lines in Google Code Search [2]. There at least two workarounds: * the getattr approach documented in [3] * use __import__(modname, {}, {}, [modname.rsplit(".", 1)[-1]]) As both of them are clumsy and inefficient, I created a simple patch for __import__ [4] that adds yet another argument, 'submodule' (maybe 'tail_module' would have been more appropriate) that caters for that particular use case: <module 'foo' from 'foo/__init__.py'> <module 'foo.bar.baz' from 'foo/bar/baz.py'> <module 'foo.bar.baz' from 'foo/bar/baz.py'> --- While I was doing that, I noticed that the import_module_level() function that does the gruntwork behind __import__ does not entirely match the documentation [3]. Namely, [3] states "the statement from spam.ham import eggs results in __import__('spam.ham', globals(), locals(), ['eggs'], -1)." This is incorrect: <module 'foo.bar' from 'foo/bar/__init__.py'> i.e. 'bar' is imported, not 'baz' (or 'ham' and not 'eggs'). As a matter of fact, anything can be in 'fromlist' (the reason for the API abuse seen in [2]): ... ['this_is_a_bug'], -1) <module 'foo.bar.baz' from 'foo/bar/baz/__init__.py'> So, effectively, 'fromlist' is already functioning as a boolean that indicates whether the tail or toplevel module is imported. Proposal: * either fix __import__ to behave as documented: # from foo.bar import baz >>> __import__('foo.bar', fromlist=['baz']) <module 'foo.bar.baz' from 'foo/bar/baz/__init__.py'> # from foo.bar import baz, baq >>> __import__('foo.bar', fromlist=['baz', 'baq']) (<module 'foo.bar.baz' from 'foo/bar/baz/__init__.py'>, <module 'foo.bar.baq' from 'foo/bar/baq/__init__.py'>) and add the 'submodule' argument to support the common __import__ use case [4], * or if that is not feasible, retain the current boolean behaviour and make that explicit by renaming 'fromlist' to 'submodule' (and require the latter to be a boolean, not a list). Too bad I couldn't come up with this before, 3.0 would have been a perfect opportunity to get things right (one way or the other). --- References: [1] http://bugs.python.org/issue2090 [2] http://google.com/codesearch?hl=en&lr=&q=__import__.*%5C%5B%27%27%5C%5D [3] http://docs.python.org/library/functions.html#__import__ [4] http://bugs.python.org/issue4438 _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Mart Somermaa 2008-11-27T14:40:39 http://comments.gmane.org/gmane.comp.python.devel/98547 A while ago I contacted Jukka Laurila from the Nokia developer board about IEEE 754 support on S60 phones. The information from Jukka may be useful for future reference. Christian -------- Original Message -------- Subject: Re: Python 2.6/3.0, IEEE 754 floating point semantics and S60 We've been battling with some float problems recently as well in the course of our 2.5 core porting. The principal problem has been imprecise floating point formatting/parsing routines which lead to the unfortunate case of not being able to roundtrip a Python float to string and back bit-exactly. I've been trying to get to the bottom of that problem and so far it seems like the root cause is a poorly written C library. Though on the other hand, no one from Symbian is yet to confirm to me that the software-emulated floating point routines correspond to anything like IEEE 754. Most S60 devices lack floating point hardware. The best guess is that the floating point support is IEEE 754'ish, with imprecise formatting and parsing routines in the standard library. -Jukka On 2.2.2008 3.54, "ext Crys < at > Developer Discussion Boards" _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Christian Heimes 2008-11-27T14:58:43 http://comments.gmane.org/gmane.comp.python.devel/98546 _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Kristján Valur Jónsson 2008-11-27T14:50:07 http://comments.gmane.org/gmane.comp.python.devel/98523 I was experimenting with profiled builds and building more extensions as builtins instead of loadable modules, noticing that Module/Setup.dist is not kept updated. Currently only the section above the *shared* line is really used for the build, but everything else doesn't seem to be updated (extensions are missing or out of date). Is there another way to build extensions as builtins, or should this file kept be up to date? Matthias _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Matthias Klose 2008-11-26T22:55:07 http://comments.gmane.org/gmane.comp.python.devel/98520 I built Python 2.5.2 on RedHat3 and wrapped it up with some other stuff that was deployed on RedHat4. When trying to fire up Idle on RedHat4, there is an error that states a usable init.tcl cannot be found. Python is built on RedHat3 against Tcl/Tk 8.3, so even installing Tcl/Tk on RedHat4 would not work, as it is 8.4. I've noticed when Python is installed on Windows, all the necessary stuff is also installed in the Python25 directory. Is there a similar way to do this on linux? Thanks, -Chris _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org chris 2008-11-26T22:42:30 http://comments.gmane.org/gmane.comp.python.devel/98491 Hi, I encountered a weird problem using distutils. Generally, distutils try to use the same compiler options used for building Python interpreter, but it looks like some of them are omitted sometimes. - CPPFLAGS are not retrieved from the config and only ones in env are used. - OPT is retrieved from the config, but it's only used when env has CFLAGS. See: http://svn.python.org/view/python/trunk/Lib/distutils/sysconfig.py """ def customize_compiler(compiler): ... if compiler.compiler_type == "unix": (cc, cxx, opt, cflags, ccshared, ldshared, so_ext) = \ get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS', 'CCSHARED', 'LDSHARED', 'SO') if 'CC' in os.environ: cc = os.environ['CC'] if 'CXX' in os.environ: cxx = os.environ['CXX'] if 'LDSHARED' in os.environ: ldshared = os.environ['LDSHARED'] if 'CPP' in os.environ: cpp = os.environ['CPP'] else: cpp = cc + " -E" # not always if 'LDFLAGS' in os.environ: ldshared = ldshared + ' ' + os.environ['LDFLAGS'] if 'CFLAGS' in os.environ: cflags = opt + ' ' + os.environ['CFLAGS'] ldshared = ldshared + ' ' + os.environ['CFLAGS'] if 'CPPFLAGS' in os.environ: cpp = cpp + ' ' + os.environ['CPPFLAGS'] cflags = cflags + ' ' + os.environ['CPPFLAGS'] ldshared = ldshared + ' ' + os.environ['CPPFLAGS'] cc_cmd = cc + ' ' + cflags compiler.set_executables( preprocessor=cpp, compiler=cc_cmd, compiler_so=cc_cmd + ' ' + ccshared, compiler_cxx=cxx, linker_so=ldshared, linker_exe=cc) compiler.shared_lib_extension = so_ext """ Are these logics are intentional or just a bug? If this is intentional behavior, why is that being this way? Thanks, _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Akira Kitada 2008-11-26T17:28:47 http://comments.gmane.org/gmane.comp.python.devel/98490 there's an alpha-release interpreter based on cpython2.6 that has support for 2.6 extensions, & can automatically generate & import files using 2to3 (& pure python3.0 scripts as well). it consists of 3 files & is intended to coexist w/ an existing python2.6 installation http://pypi.python.org/pypi/py3to2 enclosed is its patch summary against Python-2.6 (most of it deals w/ backporting 3.0 opcodes). here's a test run of it (version 2008.11.23) importing most of Python2.6's standard library regenerated by 2to3: 2TO3 COMPATIBILITY TEST $ python setup.py dev --py2to3test ... tested 200 2to3 generated scripts from 2.6.py3to2 (r26:66714, Nov 25 2008, 22:10:03) [GCC 3.4.6 20060404 (Red Hat 3.4.6-10)] standard library 0 skipped: 23 couldn't import required modules: BaseHTTPServer CGIHTTPServer cgi cookielib copy dummy_threading HTMLParser httplib macurl2path mimetypes os pdb pydoc re robotparser sgmllib SimpleHTTPServer SimpleXMLRPCServer _strptime tempfile threading urllib2 urllib 6 were non-utf8 compliant scripts: base64 getopt heapq shlex smtpd tarfile 14 failed import due to other reasons: anydbm dbhash doctest fractions _LWPCookieJar _MozillaCookieJar pickle sets socket tabnanny UserList UserString whichdb xmlrpclib 158 passed import: _abcoll abc aifc ast asynchat asyncore atexit audiodev Bastion bdb binhex bisect calendar cgitb chunk cmd codecs codeop code collections colorsys commands compileall ConfigParser contextlib Cookie copy_reg cProfile csv decimal difflib dircache dis DocXMLRPCServer dumbdbm dummy_thread filecmp fileinput fnmatch formatter fpformat ftplib functools __future__ genericpath getpass gettext glob gzip hashlib hmac htmlentitydefs htmllib ihooks imaplib imghdr imputil inspect io keyword linecache locale macpath mailbox mailcap markupbase md5 mhlib mimetools MimeWriter mimify modulefinder multifile mutex netrc new nntplib ntpath nturl2path numbers opcode optparse os2emxpath __phello__.foo pickletools pipes pkgutil platform plistlib popen2 poplib posixfile posixpath pprint profile pstats pty pyclbr py_compile pydoc_topics Queue quopri random repr rexec rfc822 rlcompleter runpy sched sha shelve shutil site smtplib sndhdr SocketServer sre_compile sre_constants sre_parse sre ssl stat statvfs StringIO stringold stringprep string struct subprocess sunaudio sunau symbol symtable telnetlib textwrap this _threading_local timeit toaiff tokenize token traceback trace tty types unittest urlparse UserDict user uuid uu warnings wave weakref webbrowser xdrlib xmllib zipfile PYTHON2.6 COMPATIBILITY TEST $ python setup.py dev --maketest ... 324 tests OK. 36 tests skipped: test_aepack test_al test_applesingle test_bsddb185 test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses test_dl test_gdbm test_gl test_imageop test_imgfile test_kqueue test_linuxaudiodev test_macos test_macostools test_normalization test_ossaudiodev test_pep277 test_py3kwarn test_scriptpackages test_socketserver test_startfile test_sunaudiodev test_timeout test_urllib2net test_urllibnet test_winreg test_winsound test_zipfile64 1 skip unexpected on linux2: test_gdbm completed 2.6 (r26:66714, Nov 17 2008, 13:35:14) [GCC 3.4.6 20060404 (Red Hat 3.4.6-10)] regression tests _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org kai 2008-11-26T08:35:59 http://comments.gmane.org/gmane.comp.python.devel/98483 I looked through the list now and weeded out irrelevant CVEs (by putting them into the ignore list in the script). Also, now the output has descriptions of the CVEs as well, so it's more readable. Improved output: http://dpaste.com/hold/93386/ Improved script (with a proper IGNORED_LIST): http://dpaste.com/hold/93388/ The results are much better: 5 OK's, 8 WARNings, 7 ERRORs. Most of the errors are from 2007 or before, the only error from 2008 is an obscure Tools/faqwiz/move-faqwiz.sh-related one. _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Mart Somermaa 2008-11-24T18:43:17 http://comments.gmane.org/gmane.comp.python.devel/98477 I created a script that parses the http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=python Python-related CVE list and classifies the CVEs as follows: * "ok" -- CVE has references to bugs.python.org * "warnings" -- CVE has references to Python SVN revisions or an issue in bugs.python.org refers to it (i.e. the probelm is probably fixed, but the CVE should really be updated to link to the issue that is probably listed in bugs.python.org) * "errors" -- CVE does have no references to Python issues or SVN nor does any issue in bugs.python.org have references to the CVE id The script is at http://dpaste.com/hold/92930/ The results are at http://dpaste.com/hold/92929/ There were 35 errors, 8 warnings, 5 CVEs were OK. In an ideal world, the references would be symmetric, i.e. every Python-related CVE would have references to one or more issues in bugs.python.org and these issues would also refer back to the CVE id. ### As for the rmtree problem that Gisle Aas raised, this seems to apply as of Python 2.6: --- # emulate removing /etc $ sudo cp -a /etc /root/etc/ $ sudo python2.6 >>> for i in xrange(0, 50000): ... with open("/root/etc/" + str(i), "w") as f: ... f.write("0") ... $ ls /root/etc > orig_list.txt $ mkdir /tmp/attack $ cp -a /root/etc/* /tmp/attack $ sudo python2.6 >>> from shutil import rmtree >>> rmtree('/tmp/attack') >>> # press ctrl-z to suspend execution ^Z [1]+ Stopped sudo python2.6 $ mv /tmp/attack /tmp/dummy; ln -s /root/etc /tmp/attack $ fg sudo python2.6 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.6/shutil.py", line 225, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/local/lib/python2.6/shutil.py", line 223, in rmtree os.rmdir(path) OSError: [Errno 20] Not a directory: '/tmp/attack' $ ls /root/etc > new_list.txt $ diff -q orig_list.txt new_list.txt Files orig_list.txt and new_list.txt differ --- If the attack weren't possible, the lists would not differ. _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Mart Somermaa 2008-11-24T12:05:11 http://comments.gmane.org/gmane.comp.python.devel/98476 subprocess.Popen is much SLOWER than os.popen() on Mac and Solaris. The goal is to read the output of a command (through a pipe) as fast as possible. The problem is the pipe buffering (the reader file in the Python process). Today, subprocess.Popen() uses bufsize=0 by default. It should be bufsize=(-1): use the system default buffer size. ==> http://bugs.python.org/issue4194 Note: On Linux the performances between subprocess (unbuffered) and popen() (buffered) are the same, which is quite strange. Victor _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Victor Stinner 2008-11-24T01:37:59 http://comments.gmane.org/gmane.comp.python.devel/98474 Hi, I'd like to write a small daemon in Python, which should never be swapped out (on Linux, this daemon will be Linux specific, so no need in a platform-independent solution). In C I'd do: #include <sys/mman.h> mlockall(MCL_FUTURE); //do stuff here munlockall(); Is there anything similar in Python? TIA & kind regards Evgeni Golov _______________________________________________ Python-Dev mailing list Python-Dev< at >python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org Evgeni Golov 2008-11-24T00:44:43 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.python.devel