http://permalink.gmane.org/gmane.comp.python.cython.user hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.python.cython.user/9238 <pre> Den 23. mai 2013 kl. 19:14 skrev Zak &lt;cython&lt; at &gt;m.allo.ws&gt;: When we timed the performance of scipy.spatial.cKDTree, it was sometimes 1500x times faster than scipy.spatial.kdtree (its Python prototype). But that is an extreme case. A similar situation is when we write 3D graphics in "old-style" OpenGL and call glVertex4f millions of times in tight loops. Putting those loops into Cython will help enormously. (But graphics wizards will tell you to use vertex arrays and VBOs instead, in which case Cython doesn't help a lot.) BTW: The clue to using Cython efficiently is knowing when to NOT use it! Until you get a feel for it, start by writing Python, and let the profiler tell you the locations of the bottlenecks. Don't touch Cython until you have a wrorking prototype. If the Python code is "fast enough", don't bother to optimize because it is already fast enough (faster than "fast enough" is still just fast enough). If the Python code is "too slow", use the profiler to locate the hotspots. They might not be w</pre> Sturla Molden 2013-05-25T17:51:49 http://permalink.gmane.org/gmane.comp.python.cython.user/9237 <pre> Den 23. mai 2013 kl. 19:14 skrev Zak &lt;cython&lt; at &gt;m.allo.ws&gt;: Algorithmic or numerical code that cannot be solved using performance libraries (e.g. Intel MKL) or expressed as vectorized high-level statements (e.g. with numpy, but not only that). This is an example of code where Cython helps enormously: https://github.com/scipy/scipy/blob/master/scipy/spatial/ckdtree.pyx https://github.com/scipy/scipy/blob/master/scipy/spatial/kdtree.py Ok granted: The Cython code would have used memoryviews if we write it today. But it was written for a version SciPy that needed to maintain Python 2.4 compatibility. It depends on what you do. In my experience, as a rule of thumb: - Normal Python code to Cython: 2 to 4x improvement - NumPy code to Cython: 1 to 20x - Text processing: 1 to 20x - MKL functions in NumPy, etc: Cython slower - Plain algorithmic Python code to Cython: 100 to 2000x 3d graphics: - OpenGL, vertex arrays/buffers, and NumPy: Similar - OpenGL, GSL: Similar - OpenGL, display lists: Similar - Open</pre> Sturla Molden 2013-05-25T17:30:49 http://permalink.gmane.org/gmane.comp.python.cython.user/9236 <pre>Works, and is very quick to build the extensions. Brill! </pre> dbv 2013-05-25T15:21:01 http://permalink.gmane.org/gmane.comp.python.cython.user/9235 <pre>You should have a look at: http://docs.cython.org/src/tutorial/profiling_tutorial.html I usually get numbers like yours (i.e. ~20x speedup) when I naively just add type information to my old python code. Usually that's enough to make something else the bottle-neck, if not you might want to look into how many intermediate python objects you are creating and work on C arrays instead. (either by writing Cython or writing a separate C routine and interfacing) Hope it helps! /Björn On Thursday, 23 May 2013 19:14:55 UTC+2, Zak wrote: </pre> Björn Dahlgren 2013-05-25T11:56:01 http://permalink.gmane.org/gmane.comp.python.cython.user/9234 <pre>Am 18.05.2013 19:02, schrieb Nikita Nemkin: Compiler smartification patches are welcome. Stefan </pre> Stefan Behnel 2013-05-25T08:18:54 http://permalink.gmane.org/gmane.comp.python.cython.user/9233 <pre>Am 24.05.2013 08:25, schrieb Wichert Akkerman: As Robert noted, it's lacking documentation. Another major feature here, besides support for passing options, is that you can do extensions = [ Extension( sources=["module.pyx" if USE_CYTHON else "module.c"], **more_opts) )] if USE_CYTHON: extensions = cythonize(extensions) setup(...) That makes it very easy to ship pre-compiled (and tested!) C sources in source releases. Stefan </pre> Stefan Behnel 2013-05-25T08:07:36 http://permalink.gmane.org/gmane.comp.python.cython.user/9232 <pre> Looks good; thanks! </pre> Robert Bradshaw 2013-05-25T05:07:33 http://permalink.gmane.org/gmane.comp.python.cython.user/9231 <pre> On May 24, 2013, at 21:24, Robert Bradshaw &lt;robertwb&lt; at &gt;gmail.com&gt; wrote: Ok, so trying to trace that: cythonize takes a **option parameter, which it uses to create two CompilationOptions instances (one for C and one for C++, but completely identical in all other respects). CompilationOptions is documented (in its docstring) to take include_path, but no linking options other compiler flags at all. I've updated my pull request to reflect this. I hope it is accurate now. Wichert. </pre> Wichert Akkerman 2013-05-24T22:09:27 http://permalink.gmane.org/gmane.comp.python.cython.user/9230 <pre>On Sat, 25 May 2013 01:24:11 +0600, Robert Bradshaw &lt;robertwb&lt; at &gt;gmail.com&gt; wrote: This code: from distutils.core import setup from Cython.Build import cythonize ext_modules = cythonize( 'test.pyx', include_dirs=['TEST_INCLUDE'], library_dirs=['TEST_LIB']) print ext_modules[0].include_dirs, ext_modules[0].library_dirs produces two empty lists. cythonize does have "include_path", but it is not related to Extension's "include_dirs". Best regards, Nikita Nemkin </pre> Nikita Nemkin 2013-05-24T19:34:50 http://permalink.gmane.org/gmane.comp.python.cython.user/9229 <pre> Yep. It should work for the (very) common case as well, but apparently there's something missing in between? The original proposal is up at http://wiki.cython.org/enhancements/distutils_preprocessing , but feedback is still welcome. - Robert </pre> Robert Bradshaw 2013-05-24T19:25:34 http://permalink.gmane.org/gmane.comp.python.cython.user/9228 <pre>Currently I can declare numpy 1d and 2d arrays both in args and local vars using syntax: ndarray[double, ndim=1] or ndarray[double, ndim=2]. When I write function in .py file and trying to use .pxd for type annotations I can declare 1d and 2d numpy arrays in args. For example: cdef ndarray[double, ndim=1] func(ndarray[double, ndim=2] X, ndarray[double, ndim=1] y) But I don't know howto define in function local var with a such 1d/2d numpy array using &lt; at &gt;locals decorator. Is it doable? </pre> Zaur Shibzukhov 2013-05-24T19:25:45 http://permalink.gmane.org/gmane.comp.python.cython.user/9227 <pre> Correct. In the code. - Robert </pre> Robert Bradshaw 2013-05-24T19:24:11 http://permalink.gmane.org/gmane.comp.python.cython.user/9226 <pre>The $ arch -i386 flag is easy to forget but batch automated builds makes it easier. Need 32-bit Cython for portfolio of 32-bit products but coming to the conclusion that for OSX, 64-bit is the only way to go. </pre> dbv 2013-05-24T19:16:31 http://permalink.gmane.org/gmane.comp.python.cython.user/9225 <pre>On Fri, May 24, 2013 at 11:42 AM, Chris Barker - NOAA Federal &lt;chris.barker&lt; at &gt;noaa.gov&gt; wrote: We do test 32-bit on linux, but don't have a OS X build bot (though it's a common development platform). https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/ Yep. - Robert </pre> Robert Bradshaw 2013-05-24T18:59:23 http://permalink.gmane.org/gmane.comp.python.cython.user/9224 <pre>On Fri, May 24, 2013 at 11:03 AM, Robert Bradshaw True -- but that introduces a whole other level of hassle and confusing performance implications.... Thanks for that factoid -- nice to know. -Chris </pre> Chris Barker - NOAA Federal 2013-05-24T18:48:53 http://permalink.gmane.org/gmane.comp.python.cython.user/9223 <pre> yup -- that is what's supposed to work, but I find it gets confused some times... by the way, without the flag you'd get a 32+64 bit universal Cython, which would work fine with a 32 bit python...why do you need a 32 bit only Cython? well, if no one's testing 32 bit, I'd expect some hick-ups! you will need to be careful to pass on that arch flag whenever running pyton for building, etc... -Chris </pre> Chris Barker - NOAA Federal 2013-05-24T18:42:12 http://permalink.gmane.org/gmane.comp.python.cython.user/9222 <pre>Hi Chris Got it to work before receiving your response. Force Python to run in 32-bit mode (for Intel only) with: $ arch -i386 python Used this to build 32-bit Cython from the tar ball (ie. $ arch -i386 python setup.py install). Managed to build 32-bit extensions but gevent (1.0 beta), which is built using Cython, works in 64-bit Python but not in 32-bit. Not sure why yet. </pre> dbv 2013-05-24T18:04:20 http://permalink.gmane.org/gmane.comp.python.cython.user/9221 <pre>On Mon, May 20, 2013 at 10:51 AM, Chris Barker - NOAA Federal &lt;chris.barker&lt; at &gt;noaa.gov&gt; wrote: If you have enough bools that char vs int makes a difference, you might want to jump for the 32x savings of storing them as bits rather than bytes. But the real reason bint is an int is because that's the the result type of relational and equality operators in C. The value of bint primarily lies in its coercion to/from Python. </pre> Robert Bradshaw 2013-05-24T18:03:36 http://permalink.gmane.org/gmane.comp.python.cython.user/9220 <pre> use a 32 bit Python. Apple's python (I think) is universal, so with the right magic incantations, you can get it to run in 32 bit mode. However, I've found that trickly, particularly for building extensions. My solution is a total hack: 1) Install one of the binaries from Python.org: 32 bit ppc_Intel 32+64 bit whichone depends on if you want to support older systems, and if you are OK with using an older complier. 2) Hack it to run the one you want: - for the 32 bit one, if you're running an intel machine, it will run 32bit intel -- you're done - for the Intel one, you can use lipo to remove the 64 bit binary from the main python executable (I told you this was a hack!) 3) hack the config so that it will only build the platform you want when bulding extensions: edit: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/Makefile remove the "-arch x86_64" or "-arch ppc" from the two places it appears in there. then you'll have a 32 bit python that builds 32 bit only extensions</pre> Chris Barker - NOAA Federal 2013-05-24T17:01:42 http://permalink.gmane.org/gmane.comp.python.cython.user/9219 <pre>How would you build a 32-bit version of Cython on a 64-bit OSX machine? </pre> dbv 2013-05-24T13:29:31 http://permalink.gmane.org/gmane.comp.python.cython.user/9218 <pre> On May 24, 2013, at 13:20 , "Nikita Nemkin" &lt;nikita&lt; at &gt;nemkin.ru&gt; wrote: I'm just going with what Robert said: extensions = cythonize(Extension('*.pyx', [arbitrary Extension arguments])) which seems to work at least for include path. Perhaps those happen to match the Cython options in this case? If so, where can I find the list of valid Cython options? Wichert. </pre> Wichert Akkerman 2013-05-24T12:35:31 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.python.cython.user