gmane.comp.python.scientific.devel

Re: Circular reference in interp1d causing memory errors

2013-05-18T11:14:54

more general question (I think patch is fine)

interp._call is an instance method assigned in the __init__
This is a pattern we use quite a few times (stats.distribution does it
a lot, statsmodels in some models).

I know we have problems with pickling in these cases, but didn't know
about the garbage collection.

I didn't find much explicit information after googling for an hour.
http://stackoverflow.com/questions/9329232/internal-reference-prevents-garbage-collection
looks similar but doesn't use create instance methods. The solution
with calling through the class might be interesting in some cases.
I found a few more threads on various mailing lists but never a clear
answer or link.

Does anyone know a more explicit explanation?
Or, is it really just the cyclical reference, even though it's a
method (which always hold self)

searching for "python pickle instance method" gives a lot more direct
information on that problem.

Thanks,

Josef

Re: Sparse Boolean Specification

Pauli Virtanen — 2013-05-18T09:31:44

18.05.2013 05:23, Blake Griffith kirjoitti:
[clip]

Yes. int8 is also the data type of numpy.bool_ and the sparsetools data
type should correspond to this.

Re: Circular reference in interp1d causing memory errors

Robert Kern — 2013-05-18T09:23:11


Sounds about right. The patch needs a comment explaining why, though,
because the current code is a common pattern.

--
Robert Kern

Re: Circular reference in interp1d causing memory errors

Pauli Virtanen — 2013-05-18T09:20:29

18.05.2013 10:11, Matthew Brett kirjoitti:
[clip]

They will get freed, but the Python gc only counts the number of
allocations, not their size.

If you can change the code so that it doesn't rely on the cyclic gc,
that's OK.

Circular reference in interp1d causing memory errors

Matthew Brett — 2013-05-18T07:11:09

Hi,

In a class I'm teaching, Dan Bliss (Cc'ed) noticed that he was getting
memory errors while running multiple interpolations on large matrices.

After a little futzing, it came down to something like this:

<try_interpolators.py>

import numpy as np

import scipy.interpolate as spi

N_ITERS = 50
N = 1000

for i in range(N_ITERS):
    x = np.arange(N)
    y = np.empty((N, N))
    interp = spi.interp1d(x, y)

</try_interpolators.py>

Running this through ``time`` for a snapshot of memory usage gives:

/usr/bin/time --format='(%DK data %MK max)' python try_interpolators.py
(0K data 410088K max)

If I add a ``gc.collect()`` as the last line of the loop above I get:

[mb312< at >angela ~/tmp/dan_code]$ /usr/bin/time --format='(%DK data %MK
max)' python try_interpolators.py
(0K data 42396K max)

- a ten-fold memory difference.

Investigating scipy.interpolate.interp1d, it holds a reference to
self, for the function it will call for the interpolation.  Thus in
the default case:

interp = spi.interp1d(x, y)

interp._c

Re: New scipy.org site

Matti Pastell — 2013-05-17T07:05:13

Hi,
I've added a conversion to IPython notebook script format 
(http://ipython.org/ipython-doc/stable/interactive/htmlnotebook.html) in 
ipython folder in my 
repository(https://github.com/mpastell/SciPy-CookBook).

The script files can be imported to IPython notebook 
http://ipython.org/ipython-doc/stable/interactive/htmlnotebook.html#exporting-a-notebook-and-importing-existing-scripts.

Let me know what you think, it would be nice to know what format will 
used for the Cookbook in the future before doing any more work.

Best,
Matti


On 16.5.2013 11:13, Matti Pastell wrote:

Sparse Boolean Specification

Blake Griffith — 2013-05-17T04:45:46

Hello SciPy,
I've been writing up how I think adding support for boolean operations and
the bool dtype should work. You can read the document on my GitHub, here
https://github.com/cowlicks/scipy-sparse-boolean-spec/blob/master/spec.rst

I'd love some feedback.

Thanks!
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: Unbundling arpack

Pauli Virtanen — 2013-05-16T08:49:36

[clip]

ARPACK has nowadays a semi-alive fork with patches pooled from Scilab,
Debian, Octave, and also from us:
http://forge.scilab.org/index.php/p/arpack-ng/

The bundled Qhull is not modified. 

The bundled SuperLU has small modifications, and cannot be unbundled
due to its error reporting mechanism.

Unbundling anything under scipy.special is out of the question ---
with the exception of AMOS, the original libraries are too buggy.

I'm not sure what the patch situation is with fftpack, quadpack, 
fitpack, and the others.

Re: Unbundling arpack

Pauli Virtanen — 2013-05-16T08:37:28

We are using almost unmodified arpack-ng, so if the system version is
also that, things should be fine.

I wrote in README.scipy under scipy/sparse/linalg/eigen/arpack/ARPACK:

"""
This directory contains a bundled version of ARPACK-NG 3.1.2,
http://forge.scilab.org/index.php/p/arpack-ng/

NOTE FOR VENDORS: it is in general safe to use a system version of ARPACK
instead. Note, however, that earlier versions of ARPACK and ARPACK-NG had
certain bugs, so using those over the bundled version is not recommended.

The bundled version has the following patches applied:

(i) Replacing calls to certain Fortran functions with wrapper
functions, to avoid various ABI mismatches on OSX. These changes are
made with the following command:

perl -pi -e '
s< at >\bsdot\b< at >wsdot< at >g;
s< at >\bcdotc\b< at >wcdotc< at >g;
s< at >\bzdotc\b< at >wzdotc< at >g;
s< at >\bcdotu\b< at >wcdotu< at >g;
s< at >\bzdotu\b< at >wzdotu< at >g;
s< at >\bcladiv\b< at >wcladiv< at >g;
s< at >\bzladiv\b< at >wzladiv< at >g;
' \
SRC/*.f UTIL/*.f

(ii) Using a UTIL/second_cputime.f which calls the Fortran intrinsic
CPU_TIME function instead

Re: New scipy.org site

Matti Pastell — 2013-05-16T08:13:52

Hello all,
I did some work in converting the Cookbook to Sphinx: I scraped the 
Cookbook from the archive Robert sent to the list earlier and used a 
couple of scripts to make the conversion, it's not perfect but not that 
bad either.

You can see the results at: http://mpastell.github.io/

I put the various stages of conversion to GitHub with a brief explanation:
https://github.com/mpastell/SciPy-CookBook

Conversion scripts currently require a forked version of Pweave 
http://mpastell.com/pweave (I'm the author) that only I have, but of 
course I can share it.

I'd like to get some feedback before I do any more work and I think a 
lot of the pages will require a bit of manual editing in the end.

Also what do you think about using Pweave for the Cookbook? It can be 
used to capture figures and code and can be intgrated to sphinx build 
process.  An example (http://mpastell.com/pweave/examples.html).

Matti

Re: Unbundling arpack

David Cournapeau — 2013-05-16T07:18:48

Unfortunately, most the bundled libraries are not proper libraries as
we are used to in open source, but are really just a bunch of files
with no real notion of upstream (no versioning, etc...). In practice,
unbundling them would be quite tedious for that reason alone. They
also tend to be quite buggy, and because there is no real 'upstream'
here in most cases, we just patch our own version. I don't think we
have the bandwidth to actually take care of those separately and
'become' the upstream.

That's the case for arpack, but also fftpack, quadpack, specfunc, etc...

David

Re: Unbundling arpack

Jacob Vanderplas — 2013-05-16T04:46:36

Hi Orion,
One potential issue is that there were some small patches required for
ARPACK that we've made within the version bundled with scipy.  Since there
is no official ARPACK repository anywhere, it's unclear whether these bugs
have been fixed in the other versions available out there.  If you dig into
the scipy source, you'll see that because of these bugs we actually prepend
"scipy" to the ARPACK install to make sure that scipy *won't* use the
system version if it's available.
   Jake


On Wed, May 15, 2013 at 8:24 PM, Orion Poplawski <orion< at >cora.nwra.com>wrote:

_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Unbundling arpack

Orion Poplawski — 2013-05-16T03:24:12

I'm starting to take a look at the bundled libraries in scipy from the 
Fedora package, starting with arpack and qhull.  Are there any 
mechanisms in place at the moment for building scipy with the system 
versions of these libraries?  I don't see any.

I'm attaching my first attempt at a patch to allow this by specifying 
options in the site.cfg file.  But it appears that get_info only works 
with certain pre-defined strings.  Is that right?  This seems pretty 
cumbersome.

Can anyone help me with this?  Let me know if I'm on the right track or 
not with this.

Thanks!

PS - If anyone can think of other libraries bundled with scipy, please 
let me know.

Re: issues trac migration review

Thouis (Ray) Jones — 2013-05-15T13:40:22

Now on github:
https://github.com/ljosa/issues-with-attachments

Re: New scipy.org site

Pauli Virtanen — 2013-05-15T12:40:48

[clip]

That would be useful. If someone wants to give a hand, dumping
the Cookbook pages at

    http://scipy.org/Cookbook

to IPython notebooks is something that probably won't go to waste
in any case. This format could be better than e.g. RST.

Re: issues trac migration review

Thouis (Ray) Jones — 2013-05-13T14:11:16

The CellProfiler project uses exactly this approach for handling
attachments to issues:
http://cellprofiler.org/issues/

The code behind the page is here:
http://www.broadinstitute.org/~ljosa/issues-with-attachments/

Ray Jones

Re: doc string of brentq

nicky van foreest — 2013-05-12T19:53:42

Thanks.


On 12 May 2013 00:31, Charles R Harris <charlesr.harris< at >gmail.com> wrote:

_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: doc string of brentq

Charles R Harris — 2013-05-11T22:31:33

See PR #2462 <https://github.com/scipy/scipy/pull/2462>

Chuck
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

doc string of brentq

nicky van foreest — 2013-05-11T20:20:27

Hi,

Perhaps I am just checking an old documentation site... Anyway, on this
page:

http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.brentq.html#scipy.optimize.brentq

the argument rtol is mentioned as an valid argument to brenqt, but it is
not explained on the page.  Is this a bug?

Nicky
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

issue comments out of order

2013-05-11T12:54:31

Just an observation (too late to do anything about it)

Some comments to trac tickets are in a different sequence in github issues
https://github.com/scipy/scipy/issues/2108
http://projects.scipy.org/scipy/ticket/1583

But maybe that's a special case, because the trac ticket comments
might have been simultaneously edited during a time when trac was very
slow.

Josef

Re: scipy github slow?

Charles R Harris — 2013-05-10T22:11:57

On Fri, May 10, 2013 at 3:43 PM, Warren Weckesser <
warren.weckesser< at >gmail.com> wrote:

Too much coffee can make things seem slow ;)

Chuck
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev