gmane.comp.python.numeric.general

why Segmentation fault (core dumped)?

Chao YUE — 2012-05-26T11:51:24

Dear all,

Previously I am able to run a script on our server but now it gave me a
Segmentation fault (core dumped) error.
try I tried the script with same type of netcdf file but with much smaller
file size and it works. So I think the error is related with memory stuff.
I guess it's because our system administrator make some change somewhere
and that cause my problem?
the file size that cause the error to appear is 2.6G (in the script I read
this file with NetCDF4 to numpy array and make some manipulation),
the small one without error is only 48M.

I tried to use "limit" command to list the following, then I change
stacksize to unlimited and the problem still occurs.

ychao< at >obelix2 - ...CRU_NEW - 12 >limit
cputime      unlimited
filesize     unlimited
datasize     unlimited
stacksize    10240 kbytes
coredumpsize 0 kbytes
memoryuse    unlimited
vmemoryuse   unlimited
descriptors  1024
memorylocked 64 kbytes
maxproc      1024

would anybody be able to give me a short explanation  or direct me to some
webpage which can help to understand the problem?

thanks et cheers,

Chao

indexes in an array where value is greater than1?

Chris Withers — 2012-05-25T15:17:27

Hi All,

I have an array:

arrrgh = numpy.zeros(100000000)

A sparse collection of elements will have values greater than zero:

arrrgh[9999] = 2
arrrgh[3453453] =42

The *wrong* way to do this is:

for i in xrange(len(arrrgh)):
     if arrrgh[i] > 1:
         print i

What's the right way?

Chris

Retrieving and flattening lower triangular components of inner axes of 3d array

Damien Moore — 2012-05-25T14:19:34

Hi List,

I have an array, x, with dimensions (K,M,M). I would like to compute a
new array, y ~ (K,M*(M+1)/2), such that

y[k] = numpy.tril(x[k]).flatten() for each k = 0,..,K-1

Is there a way to do this without looping? I was trying with
tril_indices but couldn't quite get the logic right.

Thanks,
dm

.max(0) on reshaped array returns inconsistentresults.

Thouis (Ray) Jones — 2012-05-25T11:46:31

I'm seeing some strange behavior from .max() on a reshaped array in
the current master, and wanted to raise it here to make sure it's not
something uniquely broken in my setup.

This code fails for me, though changing the context (adding a counter
to the loop, or running under "python -i") sometimes prevents it from
failing.  This code doesn't fail under 1.6.2.

---------------
import numpy as np

b = np.array([0, 1, 2, 3, 4, 5], np.int64)
a = b.reshape(3, 2)

while True:
    np.testing.assert_array_equal(np.atleast_1d(np.array(a.max(0), np.float)),
                                  np.atleast_1d(np.array(a.max(0), np.float)))
---------------

I spent several hours with valgrind trying to track down what was
causing this, but had no luck.  Perhaps someone with more knowledge of
the numpy ufunc internals can track it down faster than me.

I went ahead and filed a bug for it:
http://projects.scipy.org/numpy/ticket/2144

libnpysort.a and PyDataMem_NEW/FREE

Thouis (Ray) Jones — 2012-05-23T15:37:54

The "private" libnpysort.a (see https://github.com/numpy/numpy/pull/89
for its history) uses PyDataMem_NEW/FREE.  I'm trying to convert these
to actual functions to allow tracing numpy memory allocations (see
https://github.com/numpy/numpy/pull/284).

However, these new functions have to be in the API to allow
third-party extensions to use them.  This prevents libnpysort.a from
finding them, as adding NUMPY_API to the declaration makes them static
during the build process.

There are several ways this could be dealt with:
- convert all the NEW/FREE calls in the sort library to malloc/free
(or the Python memory equivalents)
- create a separate library for the memory functions (like
libnpysort.a), but linked earlier.
- something else?

I'm not familiar enough with the internal of numpy and its API to know
which is the best approach.  Since the sorting functions could
allocate significant amounts of data, I think it would be best for
allocations within them to be tracable.

Ray Jones

command for retrieving unmasked data from a maskarray?

Chao YUE — 2012-05-23T14:49:25

Dear all,

is there a command for  retrieving unmasked data from a mask array?
excepting using dt3[~dt3.mask].flatten()?

thanks,

Chao

Numpy inverse giving incorrect result

shbr — 2012-05-23T09:22:10

I am running the following script as a basic input-output model but the
inverse function is bringing back an incorrect result (I have checked this
by inputting the data manually).I am very new to python but I assume the
error occurs in how the data is gathered from the csv but I don’t know how
to fix this. Any advice would be much appreciated. 

import numpy 
from numpy import * 
from numpy.linalg import * 
from numpy import genfromtxt 

conMatrix = genfromtxt("iopy.csv", delimiter=',') 
print "Consumption Matrix (C)" 
print conMatrix 
print "" 

#print "Consumption Matrix (C)" #manually entered data 
#conMatrix = numpy.matrix('0.4102 0.0301 0.0257; 0.0624 0.3783 0.1050;
0.1236 0.1588 0.1919') 
#print conMatrix 

identityMatrix = numpy.identity(3) 
print "Identity Matrix (I)" 
print identityMatrix 
print "" 

print "I-C" 
eyesee = numpy.subtract(identityMatrix, conMatrix) 
numpy.matrix(eyesee) 
print eyesee 
print "" 

print "(I-C)-1" 
inv = numpy.matrix(eyesee).I 
print inv 
print "" 

print "(I-C)-1*d" 
d = numpy.matrix('39.24 60.02 130.65') 
d.shape = (3,1) 
print inv*d 


raw_input() 


Thank you 

Shaun

question about in-place operations

Massimo DiPierro — 2012-05-22T14:25:25

hello everybody,

first of all thanks to the developed for bumpy which is very useful. I am building a software that uses numpy+pyopencl for lattice qcd computations. One problem that I am facing is that I need to perform most operations on arrays in place and I must avoid creating temporary arrays (because my arrays are many gigabyte large).

One typical operation is this

a[i] += const * b[i]

What is the efficient way to do is when a and b are arbitrary arrays?  const is usually a complex number.
a and b have the same shape but are not necessarily uni-dimensional.

Massimo

Building error with ATLAS

Magician — 2012-05-22T13:52:59

Hi all,


I'm now trying to build NumPy with ATLAS on CentOS 6.2.
I'm going to use them with SciPy.

My CentOS is installed as "Software Development Workstation"
on my Virtual Machine (VMware Fusion 4, Mac OS 10.7.4).
I already installed Python 2.7.3 on /usr/local/python-2.7.3 from sources,
and no other tools are installed yet.

I tried but failed to build ATLAS, so I got ATLAS by using yum command:


Then I tried to build NumPy.
But I got these errors:

Before I built NumPy, I uncommented and modified site.cfg as below:

And "setup.py config" dumped these messages:

When I built NumPy before "yum install", it was successfully finished.
I didn't use site.cfg, but I had to export CFLAGS="-L/usr/local/python-2.7.3/lib".

How can I build and install NumPy with yum-installed ATLAS?


Magician

Problem with str.format() and np.recarray

Tom Aldcroft — 2012-05-22T13:39:41

I came across this problem which appears to be new in numpy 1.6.2 (vs. 1.6.1):

In [17]: a = np.array([(1, )], dtype=[('a', 'i4')])

In [18]: ra = a.view(np.recarray)

In [19]: '{}'.format(ra[0])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/data/baffin/tom/git/eng_archive/<ipython-input-19-cbdd26e3ea78> in <module>()
----> 1 '{}'.format(ra[0])

RuntimeError: maximum recursion depth exceeded while calling a Python object

In [20]: str(ra[0])
Out[20]: '(1,)'

In [21]: ra[0]
Out[21]: (1,)

There are obvious workarounds but it seems something is not right.
I'm running Python 2.7 on linux x86_64.

Cheers,
Tom

assign a float number to a member of integerarray always return integer

Chao YUE — 2012-05-22T13:33:21

Dear all,

Just in case some one didn't know this. Assign a float number to an integer
array element will always return integer.

In [4]: a=np.arange(2,11,2)

In [5]: a
Out[5]: array([ 2,  4,  6,  8, 10])

In [6]: a[1]=4.5

In [7]: a
Out[7]: array([ 2,  4,  6,  8, 10])

so I would always do this if I expected a transfer from integer to float?
In [18]: b=a.astype(float)

In [19]: b
Out[19]: array([  2.,   4.,   6.,   8.,  10.])

In [20]: b[1]=4.5

In [21]: b
Out[21]: array([  2. ,   4.5,   6. ,   8. ,  10. ])

thanks et cheers,

Chao

un-silencing Numpy's deprecation warnings

Nathaniel Smith — 2012-05-22T08:27:31

So starting in Python 2.7 and 3.2, the Python developers have made
DeprecationWarnings invisible by default:
  http://docs.python.org/whatsnew/2.7.html#the-future-for-python-2-x
  http://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html
  http://bugs.python.org/issue7319
The only way to see them is to explicitly request them by running
Python with -Wd.

The logic seems to be that between the end-of-development for 2.7 and
the moratorium on 3.2 changes, there were a *lot* of added
deprecations that were annoying people, and deprecations in the Python
stdlib mean "this code is probably sub-optimal but it will still
continue to work indefinitely". So they consider that deprecation
warnings are like a lint tool for conscientious developers who
remember to test their code with -Wd, but not something to bother
users with.

In Numpy, the majority of our users are actually (relatively
unsophisticated) developers, and we don't plan to support deprecated
features indefinitely. Our deprecations seem to better match what
Python calls a "FutureWarning": "warnings about constructs that will
change semantically in the future."
  http://docs.python.org/library/warnings.html#warning-categories
FutureWarning is displayed by default, and available in all versions of Python.

So maybe we should change all our DeprecationWarnings into
FutureWarnings (or at least the ones that we actually plan to follow
through on). Thoughts?

- N

Trivial pull request: tox support

Nathaniel Smith — 2012-05-21T22:48:34

I got tired of juggling virtualenvs, and probably everyone else would
soon get tired of pointing out Python 2.4 incompatibilities I'd
forgotten to test, so:
  https://github.com/numpy/numpy/pull/285

- N

Fwd: Named dtype array: Difference betweena[0]['name'] and a['name'][0]?

bmu — 2012-05-21T18:50:25

dear all,

can anybody tell me, why nobody is answering this question? is this the wrong 
place to ask? or does nobody know an answer?

björn_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

subclassing ndarray subtleties??

Tom Aldcroft — 2012-05-21T17:47:56

Over on the scipy-user mailing list there was a question about
subclassing ndarray and I was interested to see two responses that
seemed to imply that subclassing should be avoided.


"Subclassing ndarray is a very tricky business -- I did it once and
regretted having done it for years, because there's so much you can't
do etc.. You're almost certainly better off with embedding an array as
an attribute, and then forward properties etc. to it."

"Yes, it's almost always the wrong thing..."

So my question is whether there are subtleties or issues that are not
covered in the standard NumPy documents on subclassing ndarray.  What
are the "things you can't do etc"?  I'm working on a project that
relies heavily on an ndarray subclass which just adds a few attributes
and slightly tweaks __getitem__.  It seems fine and I really like that
the class is an ndarray with all the built-in methods already there.
Am I missing anything?

override __getslice__ in addition to __getitem__ to be safe.

Thanks,
Tom

Named dtype array: Difference betweena[0]['name'] and a['name'][0]?

bmu — 2012-05-20T11:45:03

I came acroos a question on stackoverflow (http://stackoverflow.com/q/9470604) 
and I am wondering if this is a bug

import numpy as np
dt = np.dtype([('tuple', (int, 2))])
a = np.zeros(3, dt)
type(a['tuple'][0])  # ndarray
type(a[0]['tuple'])  # ndarray

a['tuple'][0] = (1,2)  # ok
a[0]['tuple'] = (1,2)  # ValueError: shape-mismatch on array construction

Could somebody explain this behaviour (either in this mailing list or on 
stackoverflow)?

bmu_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

ANN: NumPy 1.6.2 released

Ralf Gommers — 2012-05-20T09:34:46

Hi,

I'm pleased to announce the availability of NumPy 1.6.2.  This is a
maintenance release. Due to the delay of the NumPy 1.7.0, this release
contains far more fixes than a regular NumPy bugfix release.  It also
includes a number of documentation and build improvements.

Sources and binary installers can be found at
<https://sourceforge.net/projects/numpy/files/NumPy/1.6.2rc1/>
http://sourceforge.net/projects/numpy/files/NumPy/1.6.2/, release notes are
copied below.

Thanks to everyone who contributed to this release.

Enjoy,
The NumPy developers



=========================
NumPy 1.6.2 Release Notes
=========================

This is a bugfix release in the 1.6.x series.  Due to the delay of the NumPy
1.7.0 release, this release contains far more fixes than a regular NumPy
bugfix
release.  It also includes a number of documentation and build improvements.


``numpy.core`` issues fixed
---------------------------

#2063  make unique() return consistent index
#1138  allow creating arrays from empty buffers or empty slices
#1446  correct note about correspondence vstack and concatenate
#1149  make argmin() work for datetime
#1672  fix allclose() to work for scalar inf
#1747  make np.median() work for 0-D arrays
#1776  make complex division by zero to yield inf properly
#1675  add scalar support for the format() function
#1905  explicitly check for NaNs in allclose()
#1952  allow floating ddof in std() and var()
#1948  fix regression for indexing chararrays with empty list
#2017  fix type hashing
#2046  deleting array attributes causes segfault
#2033  a**2.0 has incorrect type
#2045  make attribute/iterator_element deletions not segfault
#2021  fix segfault in searchsorted()
#2073  fix float16 __array_interface__ bug


``numpy.lib`` issues fixed
--------------------------

#2048  break reference cycle in NpzFile
#1573  savetxt() now handles complex arrays
#1387  allow bincount() to accept empty arrays
#1899  fixed histogramdd() bug with empty inputs
#1793  fix failing npyio test under py3k
#1936  fix extra nesting for subarray dtypes
#1848  make tril/triu return the same dtype as the original array
#1918  use Py_TYPE to access ob_type, so it works also on Py3


``numpy.f2py`` changes
----------------------

ENH:   Introduce new options extra_f77_compiler_args and
extra_f90_compiler_args
BLD:   Improve reporting of fcompiler value
BUG:   Fix f2py test_kind.py test


``numpy.poly`` changes
----------------------

ENH:   Add some tests for polynomial printing
ENH:   Add companion matrix functions
DOC:   Rearrange the polynomial documents
BUG:   Fix up links to classes
DOC:   Add version added to some of the polynomial package modules
DOC:   Document xxxfit functions in the polynomial package modules
BUG:   The polynomial convenience classes let different types interact
DOC:   Document the use of the polynomial convenience classes
DOC:   Improve numpy reference documentation of polynomial classes
ENH:   Improve the computation of polynomials from roots
STY:   Code cleanup in polynomial [*]fromroots functions
DOC:   Remove references to cast and NA, which were added in 1.7


``numpy.distutils`` issues fixed
-------------------------------

#1261  change compile flag on AIX from -O5 to -O3
#1377  update HP compiler flags
#1383  provide better support for C++ code on HPUX
#1857  fix build for py3k + pip
BLD:   raise a clearer warning in case of building without cleaning up first
BLD:   follow build_ext coding convention in build_clib
BLD:   fix up detection of Intel CPU on OS X in system_info.py
BLD:   add support for the new X11 directory structure on Ubuntu & co.
BLD:   add ufsparse to the libraries search path.
BLD:   add 'pgfortran' as a valid compiler in the Portland Group
BLD:   update version match regexp for IBM AIX Fortran compilers.


``numpy.random`` issues fixed
-----------------------------

BUG:  Use npy_intp instead of long in mtrand
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion< at >scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

why not zerodivision error?

Chao YUE — 2012-05-20T07:21:39

Dear all,

could anybody give one sentence about this? why in the loop I didn't get
zerodivision error by when I explicitly do this, I get a zerodivision
error? thanks.

In [7]: for i in np.arange(-10,10):
        print 1./i
   ...:
-0.1
-0.111111111111
-0.125
-0.142857142857
-0.166666666667
-0.2
-0.25
-0.333333333333
-0.5
-1.0
inf
1.0
0.5
0.333333333333
0.25
0.2
0.166666666667
0.142857142857
0.125
0.111111111111

In [8]: 1/0.
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
/mnt/f/data/DROUGTH/<ipython-input-8-7e0bf5b37da6> in <module>()
----> 1 1/0.

ZeroDivisionError: float division by zero

In [9]: 1./0.
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
/mnt/f/data/DROUGTH/<ipython-input-9-3543596c47ff> in <module>()
----> 1 1./0.

ZeroDivisionError: float division by zero

In [10]: 1./0
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
/mnt/f/data/DROUGTH/<ipython-input-10-523760448f92> in <module>()
----> 1 1./0

ZeroDivisionError: float division by zero


Chao

Separating out the maskna code

Nathaniel Smith — 2012-05-19T20:54:38

Hi all,

Since Mark's original missingdata branch made so many changes, I
figured it would be a useful exercise to figure out what code in
master is actually related to masked arrays, and which isn't. The
easiest way seemed to be to delete the new fields, then keep removing
any code that depended on them until everything built again, which
gives this:
  https://github.com/njsmith/numpy/commits/separate-maskna

Possible uses:
  - Use the diff as a checklist for going through to change the API
  - Use the diff as a checklist for adding an experimental-API-only flag
  - Merge into master, then use as a reference to cherrypick the
pieces that we want to save (there is some questionable stuff in here

python import question

Chao YUE — 2012-05-18T21:49:56

Dear all,

This is only a small python import question. I think I'm right but just
want some confirmation.

Previously I have installed numpy 1.5.1. and then I used pip install
--upgrade numpy
to install numpy 1.6.1

But when I try to import numpy as np within ipython shell, I still get the
version 1.5.1

then I checked my sys.path:

In [21]: sys.path
Out[21]:
['',
 '/usr/local/bin',
 '/usr/local/lib/python2.7/dist-packages/pupynere-1.0.15-py2.7.egg',
 '/usr/lib/pymodules/python2.7',
 '/usr/local/lib/python2.7/dist-packages/scikits.statsmodels-0.3.1-py2.7.egg',
 '/usr/local/lib/python2.7/dist-packages/Shapely-1.2.13-py2.7-linux-i686.egg',
 '/usr/local/lib/python2.7/dist-packages/pandas-0.7.3-py2.7-linux-i686.egg',
 '/home/chaoyue/python/python_lib',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PIL',
 '/usr/lib/pymodules/python2.7/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/gst-0.10',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/pymodules/python2.7/ubuntuone-client',
 '/usr/lib/pymodules/python2.7/ubuntuone-control-panel',
 '/usr/lib/pymodules/python2.7/ubuntuone-storage-protocol',
 '/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode',
 '/usr/local/lib/python2.7/dist-packages/IPython/extensions']

Actually I found I have numpy 1.5.1 in /usr/lib/pymodules/python2.7

and numpy 1.6.1 in /usr/local/lib/python2.7/dist-packages/numpy/

but because the first path is before the second one in sys.path, so ipython
imports only the first one and ignore the second one.
Then I delete the directory of /usr/lib/pymodules/python2.7/numpy and redo
the import, I get the version 1.6.1

This means that import will try to find the first occurrence of the module
and will ignore the ones with same name in later occurrences?

cheers,

Chao

Masked Array for NumPy 1.7

Travis Oliphant — 2012-05-18T21:47:12

Hey all, 

After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. 

This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master.   So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments.  

The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged.   While the details need to be worked out here,  a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from).     This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. 

One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure.     I think this is the easiest thing to do and requires the least amount of code-change.      It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from.     That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. 

This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date.   The timing will largely depend on what time is available from people interested in resolving the situation.   Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us).    If there are others who can step in and help, it will help accelerate the process.  

Best regards,

-Travis