gmane.comp.python.numeric.general

Re: Fwd: Named dtype array: Difference betweena[0]['name'] and a['name'][0]?

diehose — 2012-05-24T21:07:50

thanks a lot. I updated the question on stackoverflow and opened a ticket
 
http://projects.scipy.org/numpy/ticket/2139
 
björn
 

Am Montag, 21. Mai 2012, 15:37:36 schrieb Travis Oliphant:

This is the right place to ask, it's just that it can take time to get an 
answer because people who might know the answer may not have the time to 
respond immediately.  


The short answer is that this is not really a "normal" bug, but it could be 
considered a "design" bug (although the issues may not be straightforward to 
resolve).    What that means is that it may not be changed in the short term 
--- and you should just use the first spelling. 



Structured arrays can be a confusing area of NumPy for several of reasons.   
You've constructed an example that touches on several of them.   You have a 
data-type that is a "structure" array with one member ("tuple").  That member 
contains a 2-vector of integers.  


First of all, it is important to remember that with Python, doing a['tuple']
[0] = (1,2) is equivalent to b = a['tuple']; b[0] = (1,2).   In like manner, 
a[0]['tuple'] = (1,2) is equivalent to b = a[0]; b['tuple'] = (1,2). 


To understand the behavior, we need to dissect both code paths and what 
happens.   You built a (3,) array of those elements in 'a'.  When you write b 
= a['tuple'] you should probably be getting a (3,) array of (2,)-integers, but 
as there is currently no formal dtype support for (n,)-integers as a general 
dtype in NumPy, you get back a (3,2) array of integers which is the closest 
thing that NumPy can give you.    Setting the [0] row of this object via 


a['tuple'][0] = (1,2)


works just fine and does what you would expect. 


On the other hand, when you type: 


b = a[0]


you are getting back an array-scalar which is a particularly interesting kind 
of array scalar that can hold records.    This new object is formally of type 
numpy.void and it holds a "scalar representation" of anything that fits under 
the "VOID" basic dtype.  


For some reason: 


b['tuple'] = [1,2] 


is not working.   On my system I'm getting a different error: TypeError: object 
of type 'int' has no len()


I think this should be filed as a bug on the issue tracker which is for the 
time being here:    http://projects.scipy.org/numpy


The problem is ultimately the void->copyswap function being called in 
voidtype_setfields if someone wants to investigate.   I think this behavior 
should work. 


-Travis








On May 21, 2012, at 1:50 PM, bmu wrote:

dear all,

can anybody tell me, why nobody is answering this question? is this the wrong 
place to ask? or does nobody know an answer?

björn

From: bmu <diehose< at >freenet.de>

Subject: Named dtype array: Difference between a[0]['name'] and a['name'][0]?

Date: May 20, 2012 6:45:03 AM CDT

To: numpy-discussion< at >scipy.org




I came acroos a question on stackoverflow (http://stackoverflow.com/q/9470604) 
and I am wondering if this is a bug

import numpy as np
dt = np.dtype([('tuple', (int, 2))])
a = np.zeros(3, dt)
type(a['tuple'][0])  # ndarray
type(a[0]['tuple'])  # ndarray

a['tuple'][0] = (1,2)  # ok
a[0]['tuple'] = (1,2)  # ValueError: shape-mismatch on array construction

Could somebody explain this behaviour (either in this mailing list or on 
stackoverflow)?

bmu


_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion





_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: Checking for views (was: Should arr.diagonal() return a copy or aview?)

2012-05-24T19:20:12

as example:

I checked pandas recently and IIRC, I needed three .base to get a True

<class 'pandas.core.series.Series'>
False
False
True
True

Josef

Re: Checking for views (was: Should arr.diagonal() return a copy or aview?)

Nathaniel Smith — 2012-05-24T17:59:38

If using the current development version of numpy, that answer is
actually wrong... if you do
  a = np.arange(10)
  b = a.view()
  c = b.view()
then in the development version, c.base is a, not b. This is the
source of some contention and confusion right now...:
  https://github.com/numpy/numpy/pull/280#issuecomment-5888154

In any case, if "b.base is a" is True, then you can be pretty certain
that b and a share memory, but if it is False, it doesn't tell you
much at all. AFAICT np.may_share_memory would be strictly more useful.

Re: Checking for views (was: Should arr.diagonal()return a copy or aview?)

Larsen, Brian A — 2012-05-24T17:07:37

This is the stack overflow discussion mentioned.

http://stackoverflow.com/questions/9164269/can-you-tell-if-an-array-is-a-view-of-another

I basically implemented the answer from SO.  I feel like the "is" gives you a good handle on things since to be true they are actually the same location in memory.

Brian






On May 24, 2012, at 8:56 AM, Jonathan T. Niehof wrote:

On 05/23/2012 05:31 PM, T J wrote:

It seems that there are a number of ways to check if an array is a view.
Do we have a preferred way in the API that is guaranteed to stay
available? Or are all of the various methods "here to stay"?

We've settled on checking array.base, which I think was the outcome of a
stackoverflow thread that I can't dig up. (I'll check with the guy who
wrote the code.)

--
Jonathan Niehof
ISR-3 Space Data Systems
Los Alamos National Laboratory
MS-D466
Los Alamos, NM 87545

Phone: 505-667-9595
email: jniehof< at >lanl.gov<mailto:jniehof< at >lanl.gov>

Correspondence /
Technical data or Software Publicly Available
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org<mailto:NumPy-Discussion< at >scipy.org>
http://mail.scipy.org/mailman/listinfo/numpy-discussion





--

Brian A. Larsen
ISR-1 Space Science and Applications
Los Alamos National Laboratory
PO Box 1663, MS-D466
Los Alamos, NM 87545
USA

(For overnight add:
SM-30, Bikini Atoll Road)

Phone: 505-665-7691
Fax:   505-665-7395
email: balarsen< at >lanl.gov<mailto:balarsen< at >lanl.gov>

Correspondence /
Technical data or Software Publicly Available



_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: Checking for views (was: Should arr.diagonal() return a copy or aview?)

Robert Kern — 2012-05-24T16:52:09

numpy.may_share_memory() gets closer, but it can be defeated by
certain striding patterns. At least, it is conservative and reports
false positives but not false negatives. Implementing
numpy.does_share_memory() correctly involves some number theory and
hairy edge cases.

(Hmm, now that I think about it, the edge cases are when the strides
are 0 or negative. 0-stride axes can simply be removed, and I think we
should be able to work back to a first item and flip the sign on the
negative strides. The typical positive-stride solution can be found in
an open source C++ global array code, IIRC. Double-hmmm...)

Re: Checking for views (was: Should arr.diagonal() return a copy or aview?)

Nathaniel Smith — 2012-05-24T15:10:18

The problem is that "is a view" isn't a very meaningful concept...
checking .base will tell you whether writes to an array are likely to
affect some object that existed before that array was created. But it
doesn't tell you whether writes to that array can affect any
*particular* other object (at least without a fair amount of groveling
around the innards of both objects), and it can happen that an object
has base == None yet writes to it will affect another object, and it
can happen that an object has base != None and yet writes to it won't
affect any object that was ever accessible to your code. AFAICT it's
really these other questions that one would like to answer, and
checking .base won't answer them.

Re: Checking for views (was: Should arr.diagonal() return a copy or aview?)

Benjamin Root — 2012-05-24T15:04:55

Just as a quick word to the wise.  I think I can recall a situation where
this could be misleading.  In particular, I think it had to do with
boolean/fancy indexing of an array.  In some cases, what you get is a view
of the copy of the original data.  So, if you simply check to see if it is
a view, and then assume that because it is a view, it must be a view of the
original data, then that assumption can come back and bite you in strange
ways.

Cheers!
Ben Root
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Checking for views (was: Should arr.diagonal()return a copy or aview?)

Jonathan T. Niehof — 2012-05-24T14:56:33


We've settled on checking array.base, which I think was the outcome of a 
stackoverflow thread that I can't dig up. (I'll check with the guy who 
wrote the code.)

Re: Some numpy funcs for PyPy

Peter — 2012-05-24T13:37:39

I don't follow the PyPy IRC so that would explain it. I don't know how
much they use that rather than their mailing list, but both seem a
better place to discuss their handling or external contributions than
on the numpy-discussion and scipy-user lists.

Still, I hope you are able to make some contributions to numpypy,
because so far I've also found PyPy's numpy implementation too
limited for my usage.

Regards,

Peter

Re: Some numpy funcs for PyPy

Dmitrey — 2012-05-24T13:07:25




On your website you wrote:


I assume that is paraphrased with a little hyperbole, but it
isn't so different from numpy (other than using git), or many
other open source projects. 

Of course, many opensource projects do like that, but in the case of
numpypy IMHO the things are especially bad.





Unit tests are important, and
taking patches without them is risky.

Yes, but at first, things required from numpypy newcomers are TOO
complicated - and no guarrantee is provided, that elapsed efforts will
not be just a waste of time; at 2nd, the high-quality standards are
especially cynic when compared with their own code quality, e.g.
numpypy.all(True) doesn't work yet, despite it hangs in bug tracker for a
long time; a[a<0] = b[b<0] works incorrectly etc.
These are reasons that forced me to write some required for my purposes
missing funcs and some bug walkarounds (like for that one with
numpypy.all and any).





 I've been subscribed to the pypy-dev list for a while, 

I had been subsribed IIRC for a couple of months





but I
don't recall seeing you posting there.

I had made some, see my pypy activity here





 Have you tried to submit
any of your work to PyPy yet?

yes: I had spent lots of time for concatenate() (pypy developers said
noone works on it) - and finally they have committed code for this func
from other trunc. Things like this were with some other my proposed code
for PyPy and all those days spent for it.





 Perhaps you should have
sent this message to pypy-dev instead?

I had explained them my point of view in mail list and irc channel, their
answer was like "don't borther horses, why do you in a hurry? All will be
done during several months", but I see it (porting whole numpy)
definitely won't be done during the term. IIRC during ~ 2 months only ~10
new items were added to numpypy; also, lots of numpypy items, when
calling, e.g. searchsorted, just raise NotImplementedError: wainting for
interplevel routine, or don't work with high-dimensional arrays and/or
some other corner cases.

numpypy developers go (rather slowly) their own way, while I just propose
temporary alternative, till proper PyPy-numpy implementation

regards, D.
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: Some numpy funcs for PyPy

Peter — 2012-05-24T12:19:23

As a NumPy user interested in PyPy it is great to know more people
are trying to contribute in this area. I myself have only filed PyPy
bugs about missing NumPy features rendering the initial numpypy
support useless to me.

On your website you wrote:


I assume that is paraphrased with a little hyperbole, but it
isn't so different from numpy (other than using git), or many
other open source projects. Unit tests are important, and
taking patches without them is risky.

I've been subscribed to the pypy-dev list for a while, but I
don't recall seeing you posting there. Have you tried to submit
any of your work to PyPy yet? Perhaps you should have
sent this message to pypy-dev instead?

(I am trying to be constructive, not critical.)

Regards,

Peter

Re: [SciPy-User] Some numpy funcs for PyPy

Flavio Coelho — 2012-05-24T11:43:45

That's very usefull! I hope these features get included upstream in the
next release of numpypy.

thanks,

Flávio

On Thu, May 24, 2012 at 8:32 AM, Dmitrey <tmp50< at >ukr.net> wrote:

Re: question about in-place operations

Francesc Alted — 2012-05-24T08:32:43

Yeah, this what common sense seems to indicate, that RAM IO bound 
problems do not benefit from using multiple cores.  But reality is 
different:

 >>> import numpy as np
 >>> a = np.arange(1e8)
 >>> c = 1.0
 >>> time a*c
CPU times: user 0.22 s, sys: 0.20 s, total: 0.43 s
Wall time: 0.43 s
array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00, ...,
          9.99999970e+07,   9.99999980e+07,   9.99999990e+07])

Using numexpr with 1 thread:

 >>> import numexpr as ne
 >>> ne.set_num_threads(1)
8
 >>> time ne.evaluate("a*c")
CPU times: user 0.20 s, sys: 0.25 s, total: 0.45 s
Wall time: 0.45 s
array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00, ...,
          9.99999970e+07,   9.99999980e+07,   9.99999990e+07])

while using 8 threads (the machine has 8 physical cores):

 >>> ne.set_num_threads(8)
1
 >>> time ne.evaluate("a*c")
CPU times: user 0.39 s, sys: 0.68 s, total: 1.07 s
Wall time: 0.14 s
array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00, ...,
          9.99999970e+07,   9.99999980e+07,   9.99999990e+07])

which is 3x faster than using 1 single thread (look at wall time figures).

It has to be clear that this is purely due to the fact that several 
cores can transmit data to/from memory from/to CPU faster than one 
single core.  I have seen this behavior lots of times; for example, in 
slide 21 of this presentation:

http://pydata.org/pycon2012/numexpr-cython/slides.pdf

one can see how using several cores can speed up not only a polynomial 
computation, but also the simple expression "y = x", which is 
essentially a memory copy.

Another example where this effect can be seen is the Blosc compressor.  
For example, in:

http://blosc.pytables.org/trac/wiki/SyntheticBenchmarks

the first points on each of the plots means that Blosc is in compression 
level 0, that is, it does not compress at all, and it basically copies 
data from origin to destination buffers.  Still, one can see that using 
several threads can accelerate this copy well beyond memcpy speed.

So, definitely, several cores can make your memory I/O bounded 
computations go faster.

Re: Should arr.diagonal() return a copy or aview? (1.7 compatibility issue)

Luis Pedro Coelho — 2012-05-24T06:25:56

On one of my papers, we put up the code online. Years afterwards, I still get 
emails every six months or so because the version of the code which was used 
for the paper now returns the wrong result!

The problem is that it was written for the old histogram and, although I have 
a new version of the code, sometimes people still download the old version.

just my two cent,
Luis

On Wednesday, May 23, 2012 12:06:45 AM Travis Oliphant wrote:
wrote:

Re: Should arr.diagonal() return a copy or aview? (1.7 compatibility issue)

T J — 2012-05-23T23:31:52

On Wed, May 23, 2012 at 4:16 PM, Kathleen M Tacina <
Kathleen.M.Tacina< at >nasa.gov> wrote:


 It seems that there are a number of ways to check if an array is a view.
 Do we have a preferred way in the API that is guaranteed to stay
available?  Or are all of the various methods "here to stay"?
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: Should arr.diagonal() return a copy oraview? (1.7 compatibility issue)

Kathleen M Tacina — 2012-05-23T23:16:06



As a "real user", if I care about whether an array arr2 is a copy or a
view, I usually either check arr2.flags.owndata or append copy() to the
statement that created arr2, e.g., arr2 = arr.diagonal().copy().  

Numpy rules on views vs. copies sometimes require a bit of thought, and
so I'll frequently just check the flags or make a copy instead of
thinking.  (More foolproof :).)  

Kathy
-

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: Should arr.diagonal() return a copy or a view? (1.7 compatibility issue)

Nathaniel Smith — 2012-05-23T22:31:48

I'm not really worried about users who have a problem with the
double-copy. It's a totally legitimate concern, but anyone who has
that concern has already understood the issues well enough to be able
to take care of themselves, and decided that it's worth the effort to
special-case this. They can check whether the returned array has .base
set to tell whether it's an array or a view, use a temporary hack to
check for the secret warning flag in arr.flags.num, check the numpy
version, all sorts of things to get them through the one version where
this matters. The suggestion in the docs to make a copy is not exactly
binding :-).

Re: Should arr.diagonal() return a copy or aview? (1.7 compatibility issue)

Travis Oliphant — 2012-05-23T21:53:00

To be clear, I'm not opposed to the change, and it looks like we should go forward.   

In my mind it's not about developers vs. users as satisfying users is the whole point.   The purpose of NumPy is not to make its developers happy :-).   But, users also want there to *be* developers on NumPy so developer happiness is not irrelevant.  

In this case, though, there are consequences for users because of the double copy if a user wants to make their code future proof.   We are always trading off predicted user-experiences.    I hope that we all don't have the same perspective on every issue or more than likely their aren't enough voices being heard from real users. 

-Travis



On May 23, 2012, at 3:01 PM, Dag Sverre Seljebotn wrote:

Re: Should arr.diagonal() return a copy or a view? (1.7 compatibility issue)

Olivier Delalleau — 2012-05-23T21:04:11

2012/5/23 Nathaniel Smith <njs< at >pobox.com>


It wasn't just stricter rules. Some operations involving in particular
mixed scalar / array computations resulted in different outputs (with no
warning).

-=- Olivier
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: command for retrieving unmasked data from a mask array?

Chao YUE — 2012-05-23T20:34:39

Thanks Olivier. it works.

chao

2012/5/23 Olivier Delalleau <shish< at >keba.be>

Re: Should arr.diagonal() return a copy or a view? (1.7 compatibility issue)

Nathaniel Smith — 2012-05-23T20:13:42

I never understood exactly what changed with the casting rules, but
yeah, maybe. Still, the question of what our deprecation rules
*should* be is somewhat separate from the question of what we've
actually done (or even will do). You have to have ideals before you
can ask whether you're living up to them :-).

Didn't the casting rules become strictly stricter, i.e. some
questionable operations that used to succeed now throw an error? If so
then that's not a *major* violation of my suggested rules, but yeah, I
guess it'd probably be better if they did warn. I imagine it wouldn't
be terribly difficult to implement (add a new
NPY_WARN_UNSAFE_CASTING_INTERNAL value, use it everywhere that used to
be UNSAFE but now will be SAFE?), but someone who understands better
what actually changed (Mark?) would have do it.