gmane.comp.python.numeric.general

why Segmentation fault (core dumped)?

Chao YUE — 2012-05-26T11:51:24

Dear all,

Previously I am able to run a script on our server but now it gave me a
Segmentation fault (core dumped) error.
try I tried the script with same type of netcdf file but with much smaller
file size and it works. So I think the error is related with memory stuff.
I guess it's because our system administrator make some change somewhere
and that cause my problem?
the file size that cause the error to appear is 2.6G (in the script I read
this file with NetCDF4 to numpy array and make some manipulation),
the small one without error is only 48M.

I tried to use "limit" command to list the following, then I change
stacksize to unlimited and the problem still occurs.

ychao< at >obelix2 - ...CRU_NEW - 12 >limit
cputime      unlimited
filesize     unlimited
datasize     unlimited
stacksize    10240 kbytes
coredumpsize 0 kbytes
memoryuse    unlimited
vmemoryuse   unlimited
descriptors  1024
memorylocked 64 kbytes
maxproc      1024

would anybody be able to give me a short explanation  or direct me to some
webpage

indexes in an array where value is greater than1?

Chris Withers — 2012-05-25T15:17:27

Hi All,

I have an array:

arrrgh = numpy.zeros(100000000)

A sparse collection of elements will have values greater than zero:

arrrgh[9999] = 2
arrrgh[3453453] =42

The *wrong* way to do this is:

for i in xrange(len(arrrgh)):
     if arrrgh[i] > 1:
         print i

What's the right way?

Chris

Retrieving and flattening lower triangular components of inner axes of 3d array

Damien Moore — 2012-05-25T14:19:34

Hi List,

I have an array, x, with dimensions (K,M,M). I would like to compute a
new array, y ~ (K,M*(M+1)/2), such that

y[k] = numpy.tril(x[k]).flatten() for each k = 0,..,K-1

Is there a way to do this without looping? I was trying with
tril_indices but couldn't quite get the logic right.

Thanks,
dm

.max(0) on reshaped array returns inconsistentresults.

Thouis (Ray) Jones — 2012-05-25T11:46:31

I'm seeing some strange behavior from .max() on a reshaped array in
the current master, and wanted to raise it here to make sure it's not
something uniquely broken in my setup.

This code fails for me, though changing the context (adding a counter
to the loop, or running under "python -i") sometimes prevents it from
failing.  This code doesn't fail under 1.6.2.

---------------
import numpy as np

b = np.array([0, 1, 2, 3, 4, 5], np.int64)
a = b.reshape(3, 2)

while True:
    np.testing.assert_array_equal(np.atleast_1d(np.array(a.max(0), np.float)),
                                  np.atleast_1d(np.array(a.max(0), np.float)))
---------------

I spent several hours with valgrind trying to track down what was
causing this, but had no luck.  Perhaps someone with more knowledge of
the numpy ufunc internals can track it down faster than me.

I went ahead and filed a bug for it:
http://projects.scipy.org/numpy/ticket/2144

libnpysort.a and PyDataMem_NEW/FREE

Thouis (Ray) Jones — 2012-05-23T15:37:54

The "private" libnpysort.a (see https://github.com/numpy/numpy/pull/89
for its history) uses PyDataMem_NEW/FREE.  I'm trying to convert these
to actual functions to allow tracing numpy memory allocations (see
https://github.com/numpy/numpy/pull/284).

However, these new functions have to be in the API to allow
third-party extensions to use them.  This prevents libnpysort.a from
finding them, as adding NUMPY_API to the declaration makes them static
during the build process.

There are several ways this could be dealt with:
- convert all the NEW/FREE calls in the sort library to malloc/free
(or the Python memory equivalents)
- create a separate library for the memory functions (like
libnpysort.a), but linked earlier.
- something else?

I'm not familiar enough with the internal of numpy and its API to know
which is the best approach.  Since the sorting functions could
allocate significant amounts of data, I think it would be best for
allocations within them to be tracable.

Ray Jones

command for retrieving unmasked data from a maskarray?

Chao YUE — 2012-05-23T14:49:25

Dear all,

is there a command for  retrieving unmasked data from a mask array?
excepting using dt3[~dt3.mask].flatten()?

thanks,

Chao

Numpy inverse giving incorrect result

shbr — 2012-05-23T09:22:10

I am running the following script as a basic input-output model but the
inverse function is bringing back an incorrect result (I have checked this
by inputting the data manually).I am very new to python but I assume the
error occurs in how the data is gathered from the csv but I don’t know how
to fix this. Any advice would be much appreciated. 

import numpy 
from numpy import * 
from numpy.linalg import * 
from numpy import genfromtxt 

conMatrix = genfromtxt("iopy.csv", delimiter=',') 
print "Consumption Matrix (C)" 
print conMatrix 
print "" 

#print "Consumption Matrix (C)" #manually entered data 
#conMatrix = numpy.matrix('0.4102 0.0301 0.0257; 0.0624 0.3783 0.1050;
0.1236 0.1588 0.1919') 
#print conMatrix 

identityMatrix = numpy.identity(3) 
print "Identity Matrix (I)" 
print identityMatrix 
print "" 

print "I-C" 
eyesee = numpy.subtract(identityMatrix, conMatrix) 
numpy.matrix(eyesee) 
print eyesee 
print "" 

print "(I-C)-1" 
inv = numpy.matrix(eyesee).I 
print inv 
print "" 

print "(I-C)-1*d"

question about in-place operations

Massimo DiPierro — 2012-05-22T14:25:25

hello everybody,

first of all thanks to the developed for bumpy which is very useful. I am building a software that uses numpy+pyopencl for lattice qcd computations. One problem that I am facing is that I need to perform most operations on arrays in place and I must avoid creating temporary arrays (because my arrays are many gigabyte large).

One typical operation is this

a[i] += const * b[i]

What is the efficient way to do is when a and b are arbitrary arrays?  const is usually a complex number.
a and b have the same shape but are not necessarily uni-dimensional.

Massimo

Building error with ATLAS

Magician — 2012-05-22T13:52:59

Hi all,


I'm now trying to build NumPy with ATLAS on CentOS 6.2.
I'm going to use them with SciPy.

My CentOS is installed as "Software Development Workstation"
on my Virtual Machine (VMware Fusion 4, Mac OS 10.7.4).
I already installed Python 2.7.3 on /usr/local/python-2.7.3 from sources,
and no other tools are installed yet.

I tried but failed to build ATLAS, so I got ATLAS by using yum command:


Then I tried to build NumPy.
But I got these errors:

Before I built NumPy, I uncommented and modified site.cfg as below:

And "setup.py config" dumped these messages:

When I built NumPy before "yum install", it was successfully finished.
I didn't use site.cfg, but I had to export CFLAGS="-L/usr/local/python-2.7.3/lib".

How can I build and install NumPy with yum-installed ATLAS?


Magician

Problem with str.format() and np.recarray

Tom Aldcroft — 2012-05-22T13:39:41

I came across this problem which appears to be new in numpy 1.6.2 (vs. 1.6.1):

In [17]: a = np.array([(1, )], dtype=[('a', 'i4')])

In [18]: ra = a.view(np.recarray)

In [19]: '{}'.format(ra[0])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/data/baffin/tom/git/eng_archive/<ipython-input-19-cbdd26e3ea78> in <module>()
----> 1 '{}'.format(ra[0])

RuntimeError: maximum recursion depth exceeded while calling a Python object

In [20]: str(ra[0])
Out[20]: '(1,)'

In [21]: ra[0]
Out[21]: (1,)

There are obvious workarounds but it seems something is not right.
I'm running Python 2.7 on linux x86_64.

Cheers,
Tom

assign a float number to a member of integerarray always return integer

Chao YUE — 2012-05-22T13:33:21

Dear all,

Just in case some one didn't know this. Assign a float number to an integer
array element will always return integer.

In [4]: a=np.arange(2,11,2)

In [5]: a
Out[5]: array([ 2,  4,  6,  8, 10])

In [6]: a[1]=4.5

In [7]: a
Out[7]: array([ 2,  4,  6,  8, 10])

so I would always do this if I expected a transfer from integer to float?
In [18]: b=a.astype(float)

In [19]: b
Out[19]: array([  2.,   4.,   6.,   8.,  10.])

In [20]: b[1]=4.5

In [21]: b
Out[21]: array([  2. ,   4.5,   6. ,   8. ,  10. ])

thanks et cheers,

Chao

un-silencing Numpy's deprecation warnings

Nathaniel Smith — 2012-05-22T08:27:31

So starting in Python 2.7 and 3.2, the Python developers have made
DeprecationWarnings invisible by default:
  http://docs.python.org/whatsnew/2.7.html#the-future-for-python-2-x
  http://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html
  http://bugs.python.org/issue7319
The only way to see them is to explicitly request them by running
Python with -Wd.

The logic seems to be that between the end-of-development for 2.7 and
the moratorium on 3.2 changes, there were a *lot* of added
deprecations that were annoying people, and deprecations in the Python
stdlib mean "this code is probably sub-optimal but it will still
continue to work indefinitely". So they consider that deprecation
warnings are like a lint tool for conscientious developers who
remember to test their code with -Wd, but not something to bother
users with.

In Numpy, the majority of our users are actually (relatively
unsophisticated) developers, and we don't plan to support deprecated
features indefinitely. Our deprecations seem to bet

Trivial pull request: tox support

Nathaniel Smith — 2012-05-21T22:48:34

I got tired of juggling virtualenvs, and probably everyone else would
soon get tired of pointing out Python 2.4 incompatibilities I'd
forgotten to test, so:
  https://github.com/numpy/numpy/pull/285

- N

Fwd: Named dtype array: Difference betweena[0]['name'] and a['name'][0]?

bmu — 2012-05-21T18:50:25

dear all,

can anybody tell me, why nobody is answering this question? is this the wrong 
place to ask? or does nobody know an answer?

björn_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

subclassing ndarray subtleties??

Tom Aldcroft — 2012-05-21T17:47:56

Over on the scipy-user mailing list there was a question about
subclassing ndarray and I was interested to see two responses that
seemed to imply that subclassing should be avoided.


"Subclassing ndarray is a very tricky business -- I did it once and
regretted having done it for years, because there's so much you can't
do etc.. You're almost certainly better off with embedding an array as
an attribute, and then forward properties etc. to it."

"Yes, it's almost always the wrong thing..."

So my question is whether there are subtleties or issues that are not
covered in the standard NumPy documents on subclassing ndarray.  What
are the "things you can't do etc"?  I'm working on a project that
relies heavily on an ndarray subclass which just adds a few attributes
and slightly tweaks __getitem__.  It seems fine and I really like that
the class is an ndarray with all the built-in methods already there.
Am I missing anything?

override __getslice__ in addition to __getitem__ to be safe.

Thanks,
Tom

Named dtype array: Difference betweena[0]['name'] and a['name'][0]?

bmu — 2012-05-20T11:45:03

I came acroos a question on stackoverflow (http://stackoverflow.com/q/9470604) 
and I am wondering if this is a bug

import numpy as np
dt = np.dtype([('tuple', (int, 2))])
a = np.zeros(3, dt)
type(a['tuple'][0])  # ndarray
type(a[0]['tuple'])  # ndarray

a['tuple'][0] = (1,2)  # ok
a[0]['tuple'] = (1,2)  # ValueError: shape-mismatch on array construction

Could somebody explain this behaviour (either in this mailing list or on 
stackoverflow)?

bmu_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

ANN: NumPy 1.6.2 released

Ralf Gommers — 2012-05-20T09:34:46

Hi,

I'm pleased to announce the availability of NumPy 1.6.2.  This is a
maintenance release. Due to the delay of the NumPy 1.7.0, this release
contains far more fixes than a regular NumPy bugfix release.  It also
includes a number of documentation and build improvements.

Sources and binary installers can be found at
<https://sourceforge.net/projects/numpy/files/NumPy/1.6.2rc1/>
http://sourceforge.net/projects/numpy/files/NumPy/1.6.2/, release notes are
copied below.

Thanks to everyone who contributed to this release.

Enjoy,
The NumPy developers



=========================
NumPy 1.6.2 Release Notes
=========================

This is a bugfix release in the 1.6.x series.  Due to the delay of the NumPy
1.7.0 release, this release contains far more fixes than a regular NumPy
bugfix
release.  It also includes a number of documentation and build improvements.


``numpy.core`` issues fixed
---------------------------

#2063  make unique() return consistent index
#1138  allow creating arrays from empty buffers

why not zerodivision error?

Chao YUE — 2012-05-20T07:21:39

Dear all,

could anybody give one sentence about this? why in the loop I didn't get
zerodivision error by when I explicitly do this, I get a zerodivision
error? thanks.

In [7]: for i in np.arange(-10,10):
        print 1./i
   ...:
-0.1
-0.111111111111
-0.125
-0.142857142857
-0.166666666667
-0.2
-0.25
-0.333333333333
-0.5
-1.0
inf
1.0
0.5
0.333333333333
0.25
0.2
0.166666666667
0.142857142857
0.125
0.111111111111

In [8]: 1/0.
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
/mnt/f/data/DROUGTH/<ipython-input-8-7e0bf5b37da6> in <module>()
----> 1 1/0.

ZeroDivisionError: float division by zero

In [9]: 1./0.
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
/mnt/f/data/DROUGTH/<ipython-input-9-3543596c47ff> in <module>()
----> 1 1./0.

ZeroDivisionError: float division by zero

In [10]: 1./0
---------------

Separating out the maskna code

Nathaniel Smith — 2012-05-19T20:54:38

Hi all,

Since Mark's original missingdata branch made so many changes, I
figured it would be a useful exercise to figure out what code in
master is actually related to masked arrays, and which isn't. The
easiest way seemed to be to delete the new fields, then keep removing
any code that depended on them until everything built again, which
gives this:
  https://github.com/njsmith/numpy/commits/separate-maskna

Possible uses:
  - Use the diff as a checklist for going through to change the API
  - Use the diff as a checklist for adding an experimental-API-only flag
  - Merge into master, then use as a reference to cherrypick the
pieces that we want to save (there is some questionable stuff in here

python import question

Chao YUE — 2012-05-18T21:49:56

Dear all,

This is only a small python import question. I think I'm right but just
want some confirmation.

Previously I have installed numpy 1.5.1. and then I used pip install
--upgrade numpy
to install numpy 1.6.1

But when I try to import numpy as np within ipython shell, I still get the
version 1.5.1

then I checked my sys.path:

In [21]: sys.path
Out[21]:
['',
 '/usr/local/bin',
 '/usr/local/lib/python2.7/dist-packages/pupynere-1.0.15-py2.7.egg',
 '/usr/lib/pymodules/python2.7',
 '/usr/local/lib/python2.7/dist-packages/scikits.statsmodels-0.3.1-py2.7.egg',
 '/usr/local/lib/python2.7/dist-packages/Shapely-1.2.13-py2.7-linux-i686.egg',
 '/usr/local/lib/python2.7/dist-packages/pandas-0.7.3-py2.7-linux-i686.egg',
 '/home/chaoyue/python/python_lib',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-pac

Masked Array for NumPy 1.7

Travis Oliphant — 2012-05-18T21:47:12

Hey all, 

After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. 

This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master.   So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments.  

The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged.   While the details need to be worked out here,  a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from).     This might also be a good way for Dag to