gmane.comp.python.scientific.devel

Re: bool for sparse matrices

Pauli Virtanen — 2013-06-19T17:44:24

19.06.2013 20:18, Blake Griffith kirjoitti:

Yea, it may be possible to hack around it, but I think this is best 
tackled later together with the second half of your GSoc proposal, as 
the solution will be the same for the comparison ops.

Pauli

Re: bool for sparse matrices

Blake Griffith — 2013-06-19T17:18:48

Okay, I will do this for now. I was exploring using numpy.set_numeric_ops,
as discussed in this SO question:

http://stackoverflow.com/questions/14619449/how-can-i-override-comparisons-between-numpys-ndarray-and-my-type


On Wed, Jun 19, 2013 at 11:22 AM, Pauli Virtanen <pav< at >iki.fi> wrote:

_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: bool for sparse matrices

Pauli Virtanen — 2013-06-19T16:22:20

19.06.2013 18:31, Blake Griffith kirjoitti:

That's my understanding of the issue too.

I think for the present, we can just ignore the dense matrix comparison 
tests that do not work, and just raise knownfailures in the tests. This 
is not a regression in functionality, as the ndarray comparisons with 
ndarray as left-hand-side op did not previously produce useful results.

Re: bool for sparse matrices

Blake Griffith — 2013-06-19T15:31:50

It looks like the ndarray inequality methods are basically ufuncs. So I'm
running into the problem of overriding ufuncs.


On Tue, Jun 18, 2013 at 6:39 PM, Blake Griffith
<blake.a.griffith< at >gmail.com>wrote:

_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

bool for sparse matrices

Blake Griffith — 2013-06-18T23:39:44

Recently I've been implementing boolean comparisons for sparse matrices,
I've run into a problem supporting dense matrices.

If A is a dense ndarray, and B is a sparse matrix. And I do:

    A bool_op B

The ndarray calls B.__bool__() for some reason, and I have not figured out
how to set __bool__ to work appropriately for all bool ops. In my last PR I
set __bool__ to raise a ValueError, like ndarrays do. And this is ok for A
== B and A != B. In these cases, the sparse matrix B handles the operation,
like it should. With __bool__ set to True or False, the ndarray tries to
handle the operation and fails.

But with A < B, A > B, the ValueError in bool is raised. So I'm not sure
what to do.

Any suggestions? I'm currently looking for the rich comparison
implementation for ndarrays.
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: [stats] discrete vs continuous distributions,arguments

2013-06-15T00:27:16

On Fri, Jun 14, 2013 at 8:11 PM, Evgeni Burovski
<evgeny.burovskiy< at >gmail.com> wrote:

I assume you have a recent scipy. this was recently fixed.
with scipy 0.9 it's possible to get weird results with too many parameters

1.4867195147342979e-06

does this raise an error ?
0.1562934518505317


the explanation

each continuous distribution requires shape parameter and optional loc and scale
loc and scale are keyword parameters, but can be given as positional parameters

numargs shows the number of shape parameters

0
1

so we need at least numargs distribution parameters and at most numargs + 2

For the discrete distribution there is no scale, so only loc is an
extra keyword, which means discrete distributions take either numargs
arguments with loc=0 as default, or numargs + 1 parameters when loc is
also given.

the normal distribution has numargs=0, so it can either take 0, 1 or 2
arguments, interpreted as loc and scale with defaults 0, 1.
3 arguments is too many and raises an exception.

Poisson should raise an exception if there are 3 or more parameters
given, and needs at least one for the shape parameter (loc=0 is
optional).

What I don't remember is whether the discrete distribution have been
fixed so they raise an exception if there are too many arguments.

Josef

[stats] discrete vs continuous distributions, arguments

Evgeni Burovski — 2013-06-15T00:11:19

Dear experts,

Could somebody comment on the way how continuous and discrete distributions
handle their positional arguments --- especially where there are too many
of them. For example:

0.060697388909241624
1.7221234070193835e-35
1.7221234070193835e-35
0.12098536225957168
0.12098536225957168
x=39+101?
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/home/br/.local/lib/python2.7/site-packages/scipy/stats/distributions.py",
line 1212, in pdf
    args, loc, scale = self._fix_loc_scale(args, loc, scale)
  File
"/home/br/.local/lib/python2.7/site-packages/scipy/stats/distributions.py",
line 545, in _fix_loc_scale
    raise TypeError("Too many input arguments.")
TypeError: Too many input arguments.

I understand what happens in the code (the `loc` parameter is free for
`poisson` and is fixed at `mu` for norm), but I'm at loss whether this
disparity is by design --- is it a feature or a bug or just my
misunderstanding?

Zhenya
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: stats, distributions, design choices

Nathaniel Smith — 2013-06-14T14:55:30

Surely the ideal solution in this case would be to unconditionally
respect the value of np.geterr()["invalid"]? That way the
distributions would handle invalid input in the same as ufuncs like
np.log.

(Of course this would be easier if numpy also exposed some simple API
for raising such errors.)

-n

Re: stats, distributions, design choices

2013-06-14T14:16:10

On Fri, Jun 14, 2013 at 10:07 AM, Evgeni Burovski
<evgeny.burovskiy< at >gmail.com> wrote:

Checking at freezing time is just 2 or 3 lines in the freeze method.
That's not a problem.

However, we would have to add the check to all methods, pdf, cdf, ...,
if we want to raise a warning with non-frozen usage. We don't want to
fiddly with the function specific _xxx methods, since they should be
simple and should not include code that is generic, and there are a
huge number of the _xxx methods.

Josef

Re: stats, distributions, design choices

Evgeni Burovski — 2013-06-14T14:07:19

Changing all the _argchecks sounds fishy indeed, but since all the pdf-type
methods call _argcheck anyway, the additional cost would not be large.
Moreover, extracting `loc` and `scale` and _argcheck-ing can be factored
out from individual methods to a special method of rv_distrubution. This
would require a bit of fiddling with  the arguments of private methods
(_pdf and its relatives), but the upshot is that for frozen all this would
only be done once, at the freezing time.
What would you say?

Zhenya


On Fri, Jun 14, 2013 at 1:44 AM, <josef.pktd< at >gmail.com> wrote:

_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: New test errors in sparse

Blake Griffith — 2013-06-14T13:45:45



Ah, it was that simple. Thank you Pauli.
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: disabling Accelerate on OS X

Pauli Virtanen — 2013-06-14T08:00:26

12.06.2013 00:29, Ralf Gommers kirjoitti:
[clip]

It goes like this:

     https://github.com/pv/numpy/tree/fortran-abicheck

Interested parties on OSX should check whether they manage to build this 
version of Numpy with Accelerate when this is enabled, and whether 
Scipy's ARPACK tests still fail when building against this version of Numpy.

Re: New test errors in sparse

Pauli Virtanen — 2013-06-14T07:50:19

then SciPy master I run into a bunch of problems with scipy.test(). It looks 
ilke:

You need to do a full rebuild of Scipy, `rm -rf build`.

Re: New test errors in sparse

Blake Griffith — 2013-06-14T04:28:31

I'm having a hard time getting things working now. When I build NumPy 1.7.1
then SciPy master I run into a bunch of problems with scipy.test(). It
looks ilke:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 8 but this version of
numpy is 7

...

FAILED (KNOWNFAIL=2, SKIP=16, errors=22)

Most of the erros seem to be import errors or related to import errors. I'm
not sure what this import error means since I rebuilt scipy after building
numpy 1.7.1.





On Thu, Jun 13, 2013 at 4:50 PM, Pauli Virtanen <pav< at >iki.fi> wrote:

_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: stats, distributions, design choices

2013-06-14T00:44:08

On Thu, Jun 13, 2013 at 7:46 PM, Evgeni Burovski
<evgeny.burovskiy< at >gmail.com> wrote:

raising an exception is too drastic as change in behavior/pattern in
my opinion, but I'm thinking whether we could issue a special warning.
A warning could be set by users to raise an exception.

Checking the scale would be easy, that's in one central location.
However, checking the shape parameter relies on the individual
`_argcheck` and on the calls to them in each method. To change those,
we would have to either change all _argchecks which I don't think is a
good idea, or to change all methods.
In the methods (at least most of them) it would not incur any
additional cost to check if there are any invalid parameters, or it
would not cost much, depending on where the check is done.
(based on what I remember, I didn't check the current code.)


I wouldn't be opposed to that, but Ralf is the expert on warnings.

Josef

Re: stats, distributions, design choices

Evgeni Burovski — 2013-06-13T23:46:52

That I understand --- I just like signalling NaNs much more than quite
ones, so was trying to sneak one in.

Anyway, if it doesn't match the usage, it's over; thanks for the
clarification!


Zhenya
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: stats, distributions, design choices

2013-06-13T23:11:00

On Thu, Jun 13, 2013 at 5:42 PM, Evgeni Burovski
<evgeny.burovskiy< at >gmail.com> wrote:

I don't know of many cases, I usually try to avoid wrong parameters

however zero (which could be floating point negative -1e-30) might
show up as special case in some calculations

array([        nan,  0.18393972])

The only case I remember right now was that I wanted nan propagation
for the t distribution for t-tests at one point.

In general, it's just the pattern that we follow, why do we have nans
and masked arrays?

If we have lots of samples, then there might be some cases that don't
satisfy the parameter restriction, but we still want all the other
results. For t-test the case is when the variance is zero, i.e. only
constant values in an array.

nan
nan
nan

other possible use cases
optimization, when the optimizer tries invalid values (but I think
this is now handled explicitly)
I find it easier to debug if I can see from the result which values
are valid and which are nan, especially when it's a larger array.

Josef


No pull request, this mailing list is the right place for design
discussions like this.
The pull request would not be merged without a previous discussion on
the mailing list.

Josef

Re: New test errors in sparse

Pauli Virtanen — 2013-06-13T21:50:42

Hi,

14.06.2013 00:12, Blake Griffith kirjoitti:
[clip]
[clip]

I suggest that you use Numpy 1.7.1 for development in the meanwhile, so 
that you can concentrate on the "meaningful" failures.

There's also a failure on Python 3 from a different source in your pull 
request, I just commented on that.

Pauli

Re: stats, distributions, design choices

Evgeni Burovski — 2013-06-13T21:42:36


When, say, trying to evaluate a pdf outside of a distribution support, yes,
I understand. But what about the case where there's no chance of getting a
meaningful answer: say, trying to use a normal distribution with sigma=-1,
like in an example I mentioned.

Vectorization... hmmm, what would be a use case for something like this:

array([ 0.18393972,         nan])

while rvs() throws anyway:
File
"/home/br/virtualenvs/scipy-dev/local/lib/python2.7/site-packages/scipy/stats/distributions.py",
line 7472, in _rvs
    Pk = k*log(mu)-gamln(k+1) - mu
  File "mtrand.pyx", line 3681, in mtrand.RandomState.poisson
(numpy/random/mtrand/mtrand.c:17130)
ValueError: lam < 0

The fix is trivial, and I can turn this into a pull request, if that's a
more appropriate medium for this discussion.

Zhenya
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: New test errors in sparse

Blake Griffith — 2013-06-13T21:12:41


This should help, every traceback for the type errors ends in the same
place:

...

======================================================================
ERROR: test_expm (test_base.TestLIL)
----------------------------------------------------------------------
Traceback (most recent call last):
  File
"/home/blake/.virtualenvs/scipy/local/lib/python2.7/site-packages/scipy/sparse/tests/test_base.py",
line 168, in test_expm
    Mexp = scipy.linalg.expm(M)
  File
"/home/blake/.virtualenvs/scipy/local/lib/python2.7/site-packages/scipy/linalg/matfuncs.py",
line 57, in expm
    return scipy.sparse.linalg.expm(A)
  File
"/home/blake/.virtualenvs/scipy/local/lib/python2.7/site-packages/scipy/sparse/linalg/matfuncs.py",
line 331, in expm
    s = s + _ell(2**-s * A, 13)
  File
"/home/blake/.virtualenvs/scipy/local/lib/python2.7/site-packages/scipy/sparse/linalg/matfuncs.py",
line 493, in _ell
    alpha = est / (_exact_1_norm(A) * abs_c_recip)
TypeError: unsupported operand type(s) for *: 'numpy.float64' and 'long'

----------------------------------------------------------------------
Ran 6558 tests in 81.194s

FAILED (KNOWNFAIL=23, SKIP=169, errors=34)
Out[2]: <nose.result.TextTestResult run=6558 errors=34 failures=0>
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev< at >scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-dev

Re: stats, distributions, design choices

2013-06-13T21:02:31

On Thu, Jun 13, 2013 at 4:46 PM, Evgeni Burovski
<evgeny.burovskiy< at >gmail.com> wrote:

The same reason we also add nans instead of raising an exception in
other places.

When we calculate vectorized results, we still return the valid
results, and nan at the invalid results.
If there is only a scalar answer, then we raise an exception if inputs
are invalid.



I don't think there is a reason not to check the arguments on freezing
time. I guess it's just a call to _argcheck and checking that the
scale is strictly positive.

They will be checked again in the calls to individual methods, which,
I think, is unavoidable because they use the same methods as
non-frozen distributions.

Josef

gmane.comp.python.scientific.devel

Re: __bool__ for sparse matrices

Re: __bool__ for sparse matrices

Re: __bool__ for sparse matrices

Re: __bool__ for sparse matrices

__bool__ for sparse matrices

Re: [stats] discrete vs continuous distributions,arguments

[stats] discrete vs continuous distributions, arguments

Re: stats, distributions, design choices

Re: stats, distributions, design choices

Re: stats, distributions, design choices

Re: New test errors in sparse

Re: disabling Accelerate on OS X

Re: New test errors in sparse

Re: New test errors in sparse

Re: stats, distributions, design choices

Re: stats, distributions, design choices

Re: stats, distributions, design choices

Re: New test errors in sparse

Re: stats, distributions, design choices

Re: New test errors in sparse

Re: stats, distributions, design choices

Re: bool for sparse matrices

Re: bool for sparse matrices

Re: bool for sparse matrices

Re: bool for sparse matrices

bool for sparse matrices