http://permalink.gmane.org/gmane.comp.python.numeric.general hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.python.numeric.general/54613 <pre>Hi, On Wed, Jun 19, 2013 at 1:43 AM, Frédéric Bastien &lt;nouiz&lt; at &gt;nouiz.org&gt; wrote: Would someone be able to guide some of the numpy C experts into a room to do some thinking / writing on this at the scipy conference? I completely agree that these kind of optimizations and code sharing seem likely to be very important for the future. I'm not at the conference, but if there's anything I can do to help, please someone let me know. Cheers, Matthew </pre> Matthew Brett 2013-06-19T11:45:42 http://permalink.gmane.org/gmane.comp.python.numeric.general/54612 <pre>Hi, On Mon, Jun 17, 2013 at 5:03 PM, Julian Taylor &lt; jtaylor.debian&lt; at &gt;googlemail.com&gt; wrote: I don't recall a discussion when numexpr was done as this is before I read this list. numexpr do optimization that can't be done by NumPy: fusing element-wise operation in one call. So I don't see how it could be done to reuse it in NumPy. You call your optimization trivial, but I don't. In the git log of NumPy, the first commit is in 2001. It is the first time someone do this in 12 years! Also, this give 1.5-8x speed up (from memory from your PR description). This is not negligible. But how much time did you spend on them? Also, some of them are processor dependent, how many people in this list already have done this? I suppose not many. Yes, your optimization don't cover all cases that minivect do. I see 2 level of optimization. 1) The inner loop/contiguous cases, 2) the strided, broadcasted level. We don't need all optimization being done for them to be useful. Any of them are useful. So what I think is that </pre> Frédéric Bastien 2013-06-19T00:43:53 http://permalink.gmane.org/gmane.comp.python.numeric.general/54611 <pre>Hi all, I assume that this question is addressed somewhere, but I have spent an inordinate amount of time looking around for the answer including digging into the source code a bit. I have tried to put the basic problem in the first paragraph. The rest of the email shows a basic example of the problem. This question is also available here: http://stackoverflow.com/questions/17156125/f2py-exposing-parameters-from-used-modules I am attempting to compile a module that contains a "use" statement that points to another, more general, module. I would prefer to keep the used module separate so that it can be used in several "packages" as a set of general settings. When I compile the two modules using f2py everything works as advertised from the fortran side, but from the python side the use statement appears to be ignored. If I allow f2py to generate a signature file, the file contains a use statement as is appropriate, but if I complete the compilation and import from the resulting library the</pre> Jeremy Solbrig 2013-06-18T19:19:34 http://permalink.gmane.org/gmane.comp.python.numeric.general/54610 <pre>Thank you  Zachary,                    I will try this and see how this goes. with best regards, Sudheer   *************************************************************** Sudheer Joseph Indian National Centre for Ocean Information Services Ministry of Earth Sciences, Govt. of India POST BOX NO: 21, IDA Jeedeemetla P.O. Via Pragathi Nagar,Kukatpally, Hyderabad; Pin:5000 55 Tel:+91-40-23886047(O),Fax:+91-40-23895011(O), Tel:+91-40-23044600(R),Tel:+91-40-9440832534(Mobile) E-mail:sjo.India&lt; at &gt;gmail.com;sudheer.joseph&lt; at &gt;yahoo.com Web- http://oppamthadathil.tripod.com *************************************************************** NumPy-Discussion mailing list NumPy-Discussion&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion </pre> Sudheer Joseph 2013-06-18T14:10:13 http://permalink.gmane.org/gmane.comp.python.numeric.general/54609 <pre> scipy.ndimage.map_coordinates may be exactly what you want. </pre> Zachary Pincus 2013-06-18T12:40:10 http://permalink.gmane.org/gmane.comp.python.numeric.general/54608 <pre> A good thing to try (and unfortunately I don't have time right now to try and work out why my results are different than yours). Remember though that once you get it working you'll want to re-enable optimization (-O2 or -O3 instead of -O0) before you use the profile results for anything serious. -n _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion </pre> Nathaniel Smith 2013-06-18T12:26:12 http://permalink.gmane.org/gmane.comp.python.numeric.general/54607 <pre> You can try using bento, with something like: CFLAGS="-O0 -fno-omit-frame-pointer -g" bentomaker build -i -v -j4 There is a bit of overhead to set it up, though. David </pre> David Cournapeau 2013-06-18T11:40:27 http://permalink.gmane.org/gmane.comp.python.numeric.general/54606 <pre>Dear all, after some code clean-up / testing and a few additions, I've now sent a pull request to numpy:master (#3451). https://github.com/numpy/numpy/pull/3451 I also made a blog post to explain the new typemaps I would like included: http://egorzindy.blogspot.co.uk/2013/06/new-numpyi-typemaps-for-working-with.html Any comments appreciated. Kind regards, Egor On 9 June 2013 09:20, Egor Zindy &lt;ezindy&lt; at &gt;gmail.com&gt; wrote: </pre> Egor Zindy 2013-06-18T11:36:20 http://permalink.gmane.org/gmane.comp.python.numeric.general/54605 <pre> No, this will not be happening. Instead, the focus of effort went into creating the enhanced buffer interface. This allows libraries to communicate ndarray-like information using that interface instead of sharing a particular concrete data structure. http://www.python.org/dev/peps/pep-3118/ -- Robert Kern </pre> Robert Kern 2013-06-18T09:41:22 http://permalink.gmane.org/gmane.comp.python.numeric.general/54604 <pre>I see that the plan to merge old Numeric into the python standard library, PEP 208, is listed as withdrawn, although no reasons are given as far as I could see. Considering how mature Numpy has become, and how common it is as a dependency for python packages, I was wondering if there were still plans on the table to ultimately merge at least the ndarray class into the python standard library, and if so where those plans stand. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion </pre> Todd 2013-06-18T09:33:39 http://permalink.gmane.org/gmane.comp.python.numeric.general/54603 <pre> There's also related things like arr + arr.T which has much less than optimal performance in NumPy (unless there was recent changes). This example was one of the motivating examples for minivect. Dag Sverre </pre> Dag Sverre Seljebotn 2013-06-17T21:29:09 http://permalink.gmane.org/gmane.comp.python.numeric.general/54602 <pre>I am building numpy from source, python setup.py build --fcompiler=gnu95 then installation, python setup.py install --user, on ubuntu 13.04 for analysis results pprof --svg /usr/bin/python py.prof On Mon, Jun 17, 2013 at 10:04 PM, Nathaniel Smith &lt;njs&lt; at &gt;pobox.com&gt; wrote: </pre> Arink Verma 2013-06-17T21:06:22 http://permalink.gmane.org/gmane.comp.python.numeric.general/54601 <pre> Hi, what I vectorized is just the really easy cases of unit stride continuous operations, so the min/max reductions which is now in numpy is in essence pretty trivial. minivect goes much further in optimizing general strided access and broadcasting via loop optimizations (it seems to have a lot of overlap with the graphite loop optimizer available in GCC [0]) so my code is probably not of very much use to minivect. The most interesting part in minivect for numpy is probably the optimization of broadcasting loops which seem to be pretty inefficient in numpy [0]. Concerning the rest I'm not sure how much of a bottleneck general strided operations really are in common numpy using code. I guess a similar discussion about adding an expression compiler to numpy has already happened when numexpr was released? If yes what was the outcome of that? [0] http://gcc.gnu.org/wiki/Graphite [1] ones((5000,100)) - ones((100,) spends about 40% of its time copying stuff around in buffers </pre> Julian Taylor 2013-06-17T21:03:00 http://permalink.gmane.org/gmane.comp.python.numeric.general/54600 <pre> I think it means that pprof is failing to walk to the stack to compute callers. That's consistent with PyArray_DESCR being the only "call" that it can find, because PyArray_DESCR isn't a real function, it always gets inlined, so detecting the caller doesn't require walking the stack. Is your numpy compiled with -fomit-frame-pointer or something like that? Any other weird build options used while building it? Is your binary stripped? If you're using a distro version, do you have the debug symbols installed? Did you compile this numpy yourself? (If not, the simplest thing to do would be to just build it yourself using the default settings and see if that helps.) What OS are you on? When I run your profiling code (but being lazier and only running 12000 samples), then do 'google-pprof --pdf /path/to/python /path/to/my/profile.prof', then I get the graph attached below. -n _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion&lt; at &gt;scipy.org http://mail.scipy.org/mailman/li</pre> Nathaniel Smith 2013-06-17T16:34:47 http://permalink.gmane.org/gmane.comp.python.numeric.general/54599 <pre> I am profiling following code timeit.timeit('x+y',number=1000000000,setup='import numpy as np;x = np.asarray(1.0);y = np.asarray(2.0);') Arink Verma www.arinkverma.in _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion </pre> Arink Verma 2013-06-17T16:08:54 http://permalink.gmane.org/gmane.comp.python.numeric.general/54598 <pre> Not sure what you are profiling. The PyArray_DESCR call just returns a pointer to the descr contained in an ndarray instance, so probably has little relevance here. Chuck _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion </pre> Charles R Harris 2013-06-17T16:01:31 http://permalink.gmane.org/gmane.comp.python.numeric.general/54597 <pre>Hi Nathaniel It's a probabilistic sampling profiler, so if it doesn't have enough I ran code '1000000000' times, which record 229115 samples. Callgraph[1] generated is converging to *PyArray_DESCR*, and rest are unconnected. Does it mean anything? [1] https://docs.google.com/file/d/0B3Pqyp8kuQw0MzRoTTNVbmlaNFE/edit?usp=sharing </pre> Arink Verma 2013-06-17T15:29:05 http://permalink.gmane.org/gmane.comp.python.numeric.general/54596 <pre>Hi, I saw that recently Julian Taylor is doing many low level optimization like using SSE instruction. I think it is great. Last year, Mark Florisson released the minivect[1] project that he worked on during is master thesis. minivect is a compiler for element-wise expression that do some of the same low level optimization that Julian is doing in NumPy right now. Mark did minivect in a way that allow it to be reused by other project. It is used now by Cython and Numba I think. I had plan to reuse it in Theano, but I didn't got the time to integrate it up to now. What about reusing it in NumPy? I think that some of Julian optimization aren't in minivect (I didn't check to confirm). But from I heard, minivect don't implement reduction and there is a pull request to optimize this in NumPy. The advantage to concentrate some functionality in one common package is that more project benefit from optimization done to it. (after the work to use it first!) How this could be done in NumPy? NumPy have its own code</pre> Frédéric Bastien 2013-06-17T15:11:43 http://permalink.gmane.org/gmane.comp.python.numeric.general/54595 <pre>Hi, On Mon, Jun 17, 2013 at 12:07 PM, Bala subramanian &lt;bala.biophysics&lt; at &gt;gmail.com&gt; wrote: Maybe you want something like: e = d[np.tril_indices(5)] ? Best, Matthew </pre> Matthew Brett 2013-06-17T11:19:55 http://permalink.gmane.org/gmane.comp.python.numeric.general/54594 <pre> How about saving numpy.concatenate([x[i,:i+1] for i in range(x.shape[0])]) instead? If you remove the concatenate, you'll get the individual arrays that have the data you need as well. Kumar </pre> Kumar Appaiah 2013-06-17T11:14:56 http://permalink.gmane.org/gmane.comp.python.numeric.general/54593 <pre>Friends, I have to save only the lower half of a symmetric matrix to a file. I used numpy.tril to extract the lower half. However when i use 'numpy.savetxt', numpy saves the whole matrix (with upper half values replaced as zeros) rather than only the lower half. Any better idea to achieve this. as an example array([[ 1, 0, 0, 0, 0], [ 6, 7, 0, 0, 0], [11, 12, 13, 0, 0], [16, 17, 18, 19, 0], [21, 22, 23, 24, 25]]) output test also contains the zeros, what i want is only the lower half like below. 1 6 7 11 12 13 - - - - etc Thanks, Bala </pre> Bala subramanian 2013-06-17T11:07:34 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.python.numeric.general