gmane.comp.python.numeric.general

Re: how to avoid re-shaping

Massimo DiPierro — 2012-05-22T15:21:26

On May 22, 2012, at 10:12 AM, Robert Kern wrote:


That's what I was looking for. Thank you.

Re: how to avoid re-shaping

Robert Kern — 2012-05-22T15:12:32

On Tue, May 22, 2012 at 4:09 PM, Massimo DiPierro
<massimo.dipierro< at >gmail.com> wrote:

Note that the .reshape() method does not work in-place. It just
returns a new ndarray object viewing the same data using the different
shape.

That said, just iterate over data.flat, if you must iterate manually.

how to avoid re-shaping

Massimo DiPierro — 2012-05-22T15:09:18

One more questions (since this list is very useful. ;-)

If I have a numpy array of arbitrary shape, is there are a way to sequentially loop over its elements without reshaping it into a 1D array?

I am trying to simplify this:

n=product(data.shape)
oldshape = data.shape
newshape = (n,)
data.reshape(newshape)
for i in xrange(n):
     do_something_with(data[i])
data.reshape(oldshape)

Massimo

Re: question about in-place operations

Massimo DiPierro — 2012-05-22T15:06:31

Thank you this does it.



On May 22, 2012, at 9:59 AM, Robert Kern wrote:

Re: question about in-place operations

Robert Kern — 2012-05-22T14:59:31

On Tue, May 22, 2012 at 3:47 PM, Massimo DiPierro
<massimo.dipierro< at >gmail.com> wrote:

BLAS has the xAXPY functions, which will do this for float and complex.

import numpy as np
from scipy.linalg import fblas

def add_scaled_inplace(a, scale, b):
    if np.issubdtype(a.dtype, complex):
        fblas.zaxpy(b, a, a=scale)
    else:
        fblas.daxpy(b, a, a=scale)

Re: question about in-place operations

Massimo DiPierro — 2012-05-22T14:54:40

For now I will be doing this:

import numpy
import time

a=numpy.zeros(2000000)
b=numpy.zeros(2000000)
c=1.0

# naive solution
t0 = time.time()
for i in xrange(len(a)):
    a[i] += c*b[i]
print time.time()-t0

# possible solution
n=100000
t0 = time.time()
for i in xrange(0,len(a),n):
    a[i:i+n] += c*b[i:i+n]
print time.time()-t0

the second "possible" solution appears 1000x faster then the former in my tests and uses little extra memory. It is only 2x slower than b*=c.

Any reason not to do it?

On May 22, 2012, at 9:32 AM, Dag Sverre Seljebotn wrote:

Re: Internationalization of numpy/scipydocstrings...

Tim Cera — 2012-05-22T14:51:22

I understand that this is non-trivial, for me anyway, because I can't
figure out how to make my way around numpydoc, and documentation editor
code (not quite true, as Pauli accepted a couple of my pull requests, but
I definitely can't make it dance).  This is why I asked for interest and
help on the mailing list.  I think for the people that worked on the
documentation editor, or know Django, or are cleverer than I, the required
changes to the documentation editor might by mid-trivial.  That is my hope
anyway.

Would probably have the high-level docs separate from the docstring
processing anyway since the high-level docs are already in a sphinx source
directory.  So I agree that the high-level docs would be the best place to
start and in-fact that is what I was working with and found the problem
with the sphinx gettext builder mentioned in the original post.

I do want to defend and clarify the docstring processing though.
 Docstrings, in the code, will always be English. The documentation editor
is the ful

Re: question about in-place operations

Massimo DiPierro — 2012-05-22T14:47:35

Thank you. I will look into numexpr. 

Anyway, I do not need arbitrary expressions. If there were something like

numpy.add_scaled(a,scale,b)

with support for scale in int, float, complex, this would be sufficient for me.

Massimo

On May 22, 2012, at 9:32 AM, Dag Sverre Seljebotn wrote:

Re: un-silencing Numpy's deprecation warnings

Jonathan T. Niehof — 2012-05-22T14:39:53

For SpacePy we took a similar but slightly different approach; this is 
in the top-level __init__:
if config['enable_deprecation_warning']:
     warnings.filterwarnings('default', '', DeprecationWarning,
                             '^spacepy', 0, False)
enable_deprecation_warning is True by default, but can be changed in the 
user's config file. This keeps everything as DeprecationWarning but only 
fiddles with the filter for spacepy (and it's set to default, not always.)

Re: question about in-place operations

Dag Sverre Seljebotn — 2012-05-22T14:32:07

I don't think NumPy support this; if you can't modify b[i] in-place, I 
think your only option will be one of numexpr/Theano/Cython.

Dag

question about in-place operations

Massimo DiPierro — 2012-05-22T14:25:25

hello everybody,

first of all thanks to the developed for bumpy which is very useful. I am building a software that uses numpy+pyopencl for lattice qcd computations. One problem that I am facing is that I need to perform most operations on arrays in place and I must avoid creating temporary arrays (because my arrays are many gigabyte large).

One typical operation is this

a[i] += const * b[i]

What is the efficient way to do is when a and b are arbitrary arrays?  const is usually a complex number.
a and b have the same shape but are not necessarily uni-dimensional.

Massimo

Building error with ATLAS

Magician — 2012-05-22T13:52:59

Hi all,


I'm now trying to build NumPy with ATLAS on CentOS 6.2.
I'm going to use them with SciPy.

My CentOS is installed as "Software Development Workstation"
on my Virtual Machine (VMware Fusion 4, Mac OS 10.7.4).
I already installed Python 2.7.3 on /usr/local/python-2.7.3 from sources,
and no other tools are installed yet.

I tried but failed to build ATLAS, so I got ATLAS by using yum command:


Then I tried to build NumPy.
But I got these errors:

Before I built NumPy, I uncommented and modified site.cfg as below:

And "setup.py config" dumped these messages:

When I built NumPy before "yum install", it was successfully finished.
I didn't use site.cfg, but I had to export CFLAGS="-L/usr/local/python-2.7.3/lib".

How can I build and install NumPy with yum-installed ATLAS?


Magician

Re: un-silencing Numpy's deprecation warnings

Robert Kern — 2012-05-22T13:49:46

Sure.

Re: un-silencing Numpy's deprecation warnings

Nathaniel Smith — 2012-05-22T13:45:59

Yes, good points -- though I think there is a also real cost/benefit
trade-off that depends on the details of how often these warnings are
issued, the specific user base, etc.

Compared to stdlib, a *much* higher proportion of numpy-using code
consists of scripts whose only users are their authors, who didn't
think very carefully about error handling, and who will continue to
use these scripts for long periods of time (i.e. over multiple
releases). So I feel like we should have a higher threshold for making
warnings silent by default.

OTOH, the distinction you suggest does make sense. I would summarize it as:

 - If a function or similar will just disappear in a future release,
causing obvious failures in any code that depends on it, then
DeprecationWarning is fine. People's code will unexpectedly break from
time to time, but in safe ways, and anyway downgrading is easy.
- Otherwise FutureWarning is preferred

Does that sound like a reasonable rule of thumb?

Problem with str.format() and np.recarray

Tom Aldcroft — 2012-05-22T13:39:41

I came across this problem which appears to be new in numpy 1.6.2 (vs. 1.6.1):

In [17]: a = np.array([(1, )], dtype=[('a', 'i4')])

In [18]: ra = a.view(np.recarray)

In [19]: '{}'.format(ra[0])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/data/baffin/tom/git/eng_archive/<ipython-input-19-cbdd26e3ea78> in <module>()
----> 1 '{}'.format(ra[0])

RuntimeError: maximum recursion depth exceeded while calling a Python object

In [20]: str(ra[0])
Out[20]: '(1,)'

In [21]: ra[0]
Out[21]: (1,)

There are obvious workarounds but it seems something is not right.
I'm running Python 2.7 on linux x86_64.

Cheers,
Tom

assign a float number to a member of integerarray always return integer

Chao YUE — 2012-05-22T13:33:21

Dear all,

Just in case some one didn't know this. Assign a float number to an integer
array element will always return integer.

In [4]: a=np.arange(2,11,2)

In [5]: a
Out[5]: array([ 2,  4,  6,  8, 10])

In [6]: a[1]=4.5

In [7]: a
Out[7]: array([ 2,  4,  6,  8, 10])

so I would always do this if I expected a transfer from integer to float?
In [18]: b=a.astype(float)

In [19]: b
Out[19]: array([  2.,   4.,   6.,   8.,  10.])

In [20]: b[1]=4.5

In [21]: b
Out[21]: array([  2. ,   4.5,   6. ,   8. ,  10. ])

thanks et cheers,

Chao

Re: un-silencing Numpy's deprecation warnings

Robert Kern — 2012-05-22T10:39:44

On Tue, May 22, 2012 at 11:14 AM, Dag Sverre Seljebotn
<d.s.seljebotn< at >astro.uio.no> wrote:


Sure.

Re: un-silencing Numpy's deprecation warnings

Dag Sverre Seljebotn — 2012-05-22T10:14:56

I guess the diagonal() change would at least be a FutureWarning then? 
(When you write to the result?)

Dag

Re: un-silencing Numpy's deprecation warnings

Robert Kern — 2012-05-22T10:06:50

That's not quite it, I think, since this change was also made in
Python 3.2 and will remain for all future versions of Python.
DeprecationWarning *is* used for things that will definitely be going
away, not just things that are no longer recommended but will continue
to live. Note that the 3.2 moratorium was for changes to the language
proper. The point was to encourage stdlib development, including the
removal of deprecated code. It was not a moratorium on removing
deprecated things. The silencing discussion just came up first in a
discussion on the moratorium.

The main problem they were running into was that the people who saw
these warnings the most were not people directly using the deprecated
features; they were users of packages written by third parties that
used the deprecated features. Those people can't do anything to fix
the problem, and many of them think that something is broken when they
see the warning (I don't know why people do this, but they do). This
problem is exacerbated by the standard

Re: un-silencing Numpy's deprecation warnings

Peter — 2012-05-22T09:50:38

We had the same discussion for Biopython two years ago, and
introduced our own warning class to avoid our deprecations being
silent (and thus almost pointless). It is just a subclass of Warning
(originally we used a subclass of UserWarning).

Re: Separating out the maskna code

Nathaniel Smith — 2012-05-22T08:30:22

We can talk about this one and everyone's welcome to look at the
patch, of course. (In fact it'd be useful if anyone catches any issues
now, so I can roll them into the final rebase.) But I'll rebase it
again after the PyArray_diagonal thing has been sorted to sort out
conflicts, and also fix some docs that I missed, so I don't want to
create an actual PR yet.