gmane.comp.python.scientific.devel

Re: Unbundling arpack

Ralf Gommers — 2013-05-19T22:30:53


I think you'd have to add an ARPACK section in
numpy/distutils/system_info.py, similar to what is done for UMFPACK. Then
use a site.cfg to specify where to find arpack-ng, and in the arpack
setup.py file use system_info.get_info to get at that site.cfg info.

Were you planning on maintaining a patch for that though? I'm not sure we'd
want to add this in numpy/scipy itself - it would be yet another build
complexity that's hard to test.

Ralf
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: BLAS level 3

Evgeni Burovski — 2013-05-19T19:01:42

Done.
Would be happy to hear any feedback.
Evgeni

On Sun, May 19, 2013 at 7:39 PM, Pauli Virtanen <pav< at >iki.fi> wrote:

_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: BLAS level 3

Pauli Virtanen — 2013-05-19T18:39:33

Hi,

19.05.2013 21:33, Evgeni Burovski kirjoitti:
[clip]

Looks sensible --- could you make it a pull request?

Best,
Pauli

BLAS level 3

Evgeni Burovski — 2013-05-19T18:33:42

Dear All,

I've recently found myself needing some BLAS level 3 functionality, esp
syrk/herk (also requested here: https://github.com/scipy/scipy/issues/2135).

Which turned to be rather easy to wrap, but being no f2py expert, I'd like
to ask for a review of the patch:

https://github.com/bburlo/scipy/compare/master...blas3

Best,

Evgeni
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: Sparse Boolean Specification

Pauli Virtanen — 2013-05-19T17:25:06

Hi,

17.05.2013 07:45, Blake Griffith kirjoitti:

I think the following things need some thought:

- How to avoid code duplication between the different operations?

- Dense vs. sparse matrix as a return value. Another undecided issue,
  but it might make sense to make the return type undefined in the
  sense that the most efficient format in used in all cases.

- Figuring out how to make SWIG automatically pick the right routines
  if input is bool-type, or whether this needs special-casing logic
  on the Python side.

- Figuring out what sort of routines would be needed to be added to
  sparsetools for broadcasting to work.

Also:

- Indexing with sparse boolean matrices? This probably boils down to
  using .nonzero() to get the nonzero indices.

- There was the suggestion to use a mostly-true matrix as an output
  for the "mostly True" operations. I'm undecided on this idea.


Pauli

Re: Circular reference in interp1d causing memory errors

Matthew Brett — 2013-05-19T07:47:41

Hi,

On Sat, May 18, 2013 at 5:00 PM, Matthew Brett <matthew.brett< at >gmail.com> wrote:

Added context managers in scipy.misc, added interp1d patch

https://github.com/scipy/scipy/pull/2502

Cheers,

Matthew

Re: Unbundling arpack

Orion Poplawski — 2013-05-19T02:52:45

Fedora is using arpack-ng, so we should be okay.  But we need a 
mechanism to link to it properly.  Can someone please comment on this?

Yeah, I'm working with the Fedora Qhull maintainer to figure out how to 
deal with this.  We will need a way to link scipy to it as well.


Thanks for the overview.

Re: Circular reference in interp1d causing memory errors

Matthew Brett — 2013-05-19T00:17:06

Hi,

On Sat, May 18, 2013 at 5:07 PM, Aron Ahmadia <aron< at >ahmadia.net> wrote:

:)  - I mean a context manager in the sense of a thing that does the
with statement:

http://docs.python.org/2/library/stdtypes.html#typecontextmanager

Is that what you meant?


Not a bad name - thanks.


Well - on entry to the with-block it:

Turns off garbage collection
Creates the object using the input arguments to 'check_refs_for'
Collects a weak reference to the new object
Passes the new object into the context of the with-block - the 'as
obj' part of the with statement

After the with-block is done it

deletes its own reference to the object
asserts that the weakref is now None
turns garbage collection back on (or rather, restores it to the previous state).

I've attached the code too ...

Cheers,

Matthew
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: Circular reference in interp1d causing memory errors

Aron Ahmadia — 2013-05-19T00:07:32


Newbie question but what is context manager in this, ermnn, context?

That's a cool test, maybe rename it to:

assert_no_refs_to(C):

And I haven't seen the code, but I assume it...

- asserts no C refs
- disables garbage collector

- code

- asserts no C refs
- re-enables garbage collector

Cheers,
Aron
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: Circular reference in interp1d causing memory errors

Matthew Brett — 2013-05-19T00:00:06

Hi,

On Sat, May 18, 2013 at 1:22 PM, Matthew Brett <matthew.brett< at >gmail.com> wrote:

Following on from Josef's and Robert's posts, I wrote a
`check_refs_for` context manger that does this:

    >>> class C(object): pass
    >>> with check_refs_for(C) as c:
    ...     # do something
    ...     del c

    >>> class C(object):
    ...     def __init__(self):
    ...         self._circular = self # Make circular reference
    >>> with check_refs_for(C) as c:
    ...     # do something
    ...     del c
    Traceback (most recent call last):
        ...
    ReferenceError: Remaining reference(s) to object

I was thinking of proposing it for scipy.misc in order to test my
interpolate.py patch, but I wonder if it belongs in numpy.testing.
What do y'all think?

Cheers,

Matthew

Re: Circular reference in interp1d causing memory errors

Matthew Brett — 2013-05-18T20:22:09

Hi,

On Sat, May 18, 2013 at 2:23 AM, Robert Kern <robert.kern< at >gmail.com> wrote:

Yes - maybe it's easier to make that clear by using unbound references
rather than doing it the crude way as in the previous patch.

Cheers,

Matthew
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: Circular reference in interp1d causing memory errors

Matthew Brett — 2013-05-18T20:13:08

Hi,

On Sat, May 18, 2013 at 5:17 AM, Robert Kern <robert.kern< at >gmail.com> wrote:

Thanks - I didn't realize this - that each time I do this:

obj.method

- I get a new method object.   Because I had to explain this to
myself, and more or less repeating Robert's exposition:

In [1]: class C(object):
   ...:     def method(self):
   ...:         pass
   ...:

In [2]: c = C()

In [3]: m1 = c.method

In [4]: m2 = c.method

In [5]: m1
Out[5]: <bound method C.method of <__main__.C object at 0x1958550>>

In [6]: m2
Out[6]: <bound method C.method of <__main__.C object at 0x1958550>>

In [7]: m1 is m2
Out[7]: False

In [8]: id(m1)
Out[8]: 26609456

In [9]: id(m2)
Out[9]: 26609696

http://docs.python.org/2/reference/datamodel.html

Thanks for the discussion,

Matthew

Re: Circular reference in interp1d causing memory errors

2013-05-18T18:39:10

Thank you, I will try this out

I got started with len(gc.get_objects()) which shows that a simple
loop with 20 OLS instantiations leaves 320 uncollected objects for
delayed garbage collection if there is a reference cycle. (and zero
with the current OLS class)

https://github.com/statsmodels/statsmodels/issues/839

the main parts of my script:
---
import gc
import numpy as np
import statsmodels.api as sm

def func(nrep):
    for i in range(nrep):
        res = sm.OLS(np.random.randn(100),
np.vander(np.linspace(-1,1,100), 4)).fit();
        #res.model.res = res

def func_cycle(nrep):
    for i in range(nrep):
        res = sm.OLS(np.random.randn(100),
np.vander(np.linspace(-1,1,100), 4)).fit();
        res.model.res = res

nrep = 20
g6 = len(gc.get_objects())

func(nrep)
g6a = len(gc.get_objects())

func_cycle(nrep)
g7 = len(gc.get_objects())

print g6a, g6a - g6, "OLS func"
print g7, g7 - g6, "OLS func cycle"

---
106722 0 OLS func
107042 320 OLS func cycle
--

Matthew, Thanks for raising the point, this t

Re: Circular reference in interp1d causing memory errors

Robert Kern — 2013-05-18T17:28:46


Sure, you can use weakrefs to determine if something has been actually
collected, as long as the object is weakrefable. Most of the objects
you are concerned with testing will be, just not objects like tuples.

http://docs.python.org/2/library/weakref#weakref.ref


import gc
import weakref


# You will probably want to disable cyclical garbage collection.
gc.disable()
try:
    # Create the object you want to test.
    obj = ...
    # Create a weak reference to the object.
    ref = weakref.ref(obj)
    # Do stuff to the object that you want to test.
    do_stuff(obj)
    # Do whatever is necessary to dispose of the object in normal use,
usually nothing.
    dispose_of_object(obj)
    # But you must at least delete it's name from the current namespace.
    del obj
    # Now see if the weak reference still can access it.
    assert ref() is None
finally:
    # Restore garbage collection.
    gc.enable()


--
Robert Kern

Re: Circular reference in interp1d causing memory errors

2013-05-18T13:56:34


Follow-up question:

Is there a way to check whether a class gets garbage collected by reference
counting, and gets immediately collected when the reference count is zero?
Something that can be used in unittests.

I never went over the details of gc.

Josef


_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: Circular reference in interp1d causing memory errors

2013-05-18T13:27:39


That's the problem with applications with large numpy arrays, especially in
loops, cyclic garbage collection doesn't kick in soon enough.

We didn't realize this in statsmodels until a user pointed out that we have
an (unnecessary) cyclical reference.

scipy.stats.distributions are safe, there shouldn't be memory problems,
since they reuse the instance and don't hold any data.
But we need to check this in statsmodels, where we still use instance
method assignment in the __init__ of one or three models.

Thanks,

Josef


_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: Circular reference in interp1d causing memory errors

2013-05-18T13:14:23


Thank, took me a while but I think I understand now

True
False

I didn't understand or looked at this part of the python documentation
before (which I guess is the relevant part):
"Note that the transformation from function object to (unbound or bound)
method object happens each time the attribute is retrieved from the class
or instance. In some cases, a fruitful optimization is to assign the
attribute to a local variable and call that local variable. "

Josef


_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: Circular reference in interp1d causing memory errors

Pauli Virtanen — 2013-05-18T12:24:19

18.05.2013 14:14, josef.pktd< at >gmail.com kirjoitti:
[clip]

Python gc is capable of breaking most reference cycles automatically.
The point however is that it does not run on every allocation. If you
allocate big objects that are kept around by a reference cycle at a too
rapid rate, you run into a memoryerror before the gc gets around to
breaking the cycles.

Re: Circular reference in interp1d causing memory errors

Robert Kern — 2013-05-18T12:17:33


Yes, this is the same issue, just storing the bound methods in a dict
rather than directly as an attribute on the instance.


The bound method is not created with the .im_self reference until it
is requested. Until then, the *unbound* method lives on the class
without a reference to any particular instance.

[~]
|1> from scipy.interpolate import interp1d

[~]
|2> f = interp1d(np.linspace(0,1), np.linspace(0,1))

[~]
|3> f._call_linear is f._call_linear
False


Requesting the method creates a new bound method that refers to the instance.

[~]
|4> f._call_linear.im_self
<scipy.interpolate.interpolate.interp1d at 0x10b4f8710>

[~]
|5> f
<scipy.interpolate.interpolate.interp1d at 0x10b4f8710>


Storing that newly created bound method as a new attribute on itself
creates a cycle.

--
Robert Kern

Re: Circular reference in interp1d causing memory errors

2013-05-18T11:14:54

more general question (I think patch is fine)

interp._call is an instance method assigned in the __init__
This is a pattern we use quite a few times (stats.distribution does it
a lot, statsmodels in some models).

I know we have problems with pickling in these cases, but didn't know
about the garbage collection.

I didn't find much explicit information after googling for an hour.
http://stackoverflow.com/questions/9329232/internal-reference-prevents-garbage-collection
looks similar but doesn't use create instance methods. The solution
with calling through the class might be interesting in some cases.
I found a few more threads on various mailing lists but never a clear
answer or link.

Does anyone know a more explicit explanation?
Or, is it really just the cyclical reference, even though it's a
method (which always hold self)

searching for "python pickle instance method" gives a lot more direct
information on that problem.

Thanks,

Josef

Re: Sparse Boolean Specification

Pauli Virtanen — 2013-05-18T09:31:44

18.05.2013 05:23, Blake Griffith kirjoitti:
[clip]

Yes. int8 is also the data type of numpy.bool_ and the sparsetools data
type should correspond to this.