http://permalink.gmane.org/gmane.comp.python.scientific.user hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.python.scientific.user/34134 <pre>Hi, I want to compute the shortest path distances in a sparse directed graph with 2M nodes but scipy.csgraph.shortest_path immediately throws a MemoryError -- regardless of the chosen method. This is what I do: And this is what I get: It seems that there are two distinct issues: 1. floyd_warshall() calls validate_graph with csr_output = False (_shortest_path.pyx:218), causing the graph to be converted to dense. I believe this a bug. 2. dijkstra creates a dense distance matrix (_shortest_path.pyx:409). I understand that one cannot make any assumption about the connectivity of the graph, and thus of the sparsity of the distance matrix itself; and of course I can get around this calling dijkstra multiple times with a manageable chunk of indices, and discarding the values that are equal to inf, but it would be nonetheless nice if the code tried to do something similar, at least for the cases when one knows that most of the distances will be inf. Best, </pre> Giovanni Luca Ciampaglia 2013-05-21T18:49:34 http://permalink.gmane.org/gmane.comp.python.scientific.user/34133 <pre> Jerry, This as actually really valuable feedback for us on the Spyder team. I've posted your question to the Spyder list here if you'd like to follow it: https://groups.google.com/d/msg/spyderlib/jjRHQSi6qFE/qDrRUtjggeoJ Jed _______________________________________________ SciPy-User mailing list SciPy-User&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user </pre> Jed Ludlow 2013-05-21T13:54:00 http://permalink.gmane.org/gmane.comp.python.scientific.user/34132 <pre>I am getting closer. Here are some observations: * If I don't imresize() the original image, then visually I can't tell the difference in the output image from Matlab or Scipy. * Empirically the results using different kernels are the same. That is the difference between two images is essentially zero. That is doesn't seem to matter if I use the imported the "fspecial()" kernel from matlab; generate my own kernel using a fgaussion(), or use scipy.ndimage.gaussian_filter(). So look like the problem is with ndimage.resize(). Michael. -- On 16/05/13 1:25 PM, Travis Oliphant wrote: _______________________________________________ SciPy-User mailing list SciPy-User&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user </pre> Brickle Macho 2013-05-17T00:23:54 http://permalink.gmane.org/gmane.comp.python.scientific.user/34131 <pre>Hi Travis, Thanks for the response, it has given a better understanding of the functions. Once working, I will post the new version. The code comes from [1], but [2] questions the use/need of log-amplitude and experimentally show that just using the phase produces the same result. This new approach reduces the computation by about 1/3. I have been able to confirm that using just the phase component produces the same result within either environment, Matlab or Scipy. Obviously I still have the problem that the output from Matlab differs from Scipy. I also discovered that if I don't resize the image (it is currently being reduced to approx 64x64) the output from Matlab is almost the same as Python. This suggests to me that, since keeping the kernels the same and using a smaller/larger images implies the differences in output is due to the kernel being used (I think). So for now I will focus my effort on implementing the the fspecial('gaussian', [10,10], 2.5) using the one provided b</pre> Brickle Macho 2013-05-16T06:12:09 http://permalink.gmane.org/gmane.comp.python.scientific.user/34130 <pre>I accidentally hit send before I finished the original email (Gmail interface quirk strikes again...) I'll including the entire original email.... Hi Michael, Thanks for sharing your code and example. The imfilter command is equivalent to scipy.signal.correlate and scipy.ndimage.correlate (the one in scipy.ndimage is faster I believe). These implement simple correlation-based filtering given a finite kernel. To get the same output you would need to generate the same kind of kernel in Python as the Matlab fspecial command is producing. one approach to get you started and to help separate the problem into 2 stages (reproducing the imfilter and reproducing the fspecial filter) is to export the results of the Matlab fspecial command and use that kernel in Python code (save it as a .mat file and read that .mat file with Python). SciPy does not have the equivalent to the fspecial command but you can generate all kinds of 1-d special filters with scipy.signal.get_window. You can also "generate" the fil</pre> Travis Oliphant 2013-05-16T05:25:56 http://permalink.gmane.org/gmane.comp.python.scientific.user/34129 <pre>Hi Michael, Thanks for sharing your code and example. The imfilter command is equivalent to scipy.signal.correlate and scipy.ndimage.correlate (the one in scipy.ndimage is faster I believe). These implement simple correlation-based filtering given a finite kernel. To get the same output you would need to generate the same kind of kernel in Python as the Matlab fspecial command is producing. one approach to get you started and to help separate the problem into 2 stages (reproducing the imfilter and reproducing the fspecial filter) is to export the results of the Matlab fspecial command and use that kernel in Python code (save it as a .mat file and read that .mat file with Python). SciPy does not have the equivalent to the fspecial command but you can generate all kinds of 1-d special filters with scipy.signal.get_window. You can also "generate" the filters you need directly from code. Here is some untested code for generating something close to what fspecial('gaussian", [10,10], 2.5) would be prod</pre> Travis Oliphant 2013-05-16T05:11:26 http://permalink.gmane.org/gmane.comp.python.scientific.user/34128 <pre>I am porting some Matlab code to python. When I run the ported version the output differs. It appears the difference may be due to how I have translated Matlab's imfilter() and/or fspecial(). Below are my translation, are there any Scipy/Matlab gurus that can let me know if there are correct or if I have made some errors, or could suggest better translations. Matlab: imfilter(saliencyMap, fspecial('gaussian', [10, 10], 2.5)) Scipy: ndimage.gaussian_filter(saliencyMap, sigma=2.5) Also the following. At the rsik of highlighting my lack of understanding of the temrinology, Is uniform_filter the same as 'average'? Matlab: imfilter(myLogAmplitude, fspecial('average', 3), 'replicate'); Scipy: scipy.ndimage.uniform_filter(mylogAmplitude, size=3, mode="nearest") Thanks, Michael. -- If curious here are the 8 lines of matlab code I am trying to port: %% Read image from file inImg = im2double(rgb2gray(imread('yourImage.jpg'))); inImg = imresize(inImg, 64/size(inImg, 2)); %% Spectral Residual myFF</pre> Brickle Macho 2013-05-16T02:19:35 http://permalink.gmane.org/gmane.comp.python.scientific.user/34127 <pre>Dear all, Matching Pursuit Toolkit (MPTK) is a fast and efficient library for the sparse decomposition of multichannel audio signals. Version 0.7 is now officially released: https://gforge.inria.fr/frs/?group_id=36 New in this version is a Python wrapper, so you can decompose/reconstruct signals directly from within Python. Best Dan </pre> Dan Stowell 2013-05-15T08:35:59 http://permalink.gmane.org/gmane.comp.python.scientific.user/34126 <pre>SciPy gurus, I am facing a strange problem using weave on 64 bit machine. Specifically with weave's inline function. It has something to do with weave's catalog. Similar issues I found in the past (very old) http://mail.scipy.org/pipermail/scipy-dev/2006-June/005908.html http://mail.scipy.org/pipermail/scipy-dev/2005-June/003042.html Common things I have in my observation are: - Already working setup in 32 bit doesn't work in same manner in 64 bit env - Weave recompiles inline code which is already compiled and doesn't require recompilation. This doesn't happen every time but at random. Whenever weave recompiles I see a notification "repairing catalog by removing key" in the output which ends up in the error message "ImportError: DLL load failed: Invalid access to memory location". If I don't see repair catalog message, then it doesn't fail. It compiles correctly and generates the output. - Sometimes gcc gets into an infinite loop printing error message "L</pre> Jadhav, Alok 2013-05-15T06:06:04 http://permalink.gmane.org/gmane.comp.python.scientific.user/34125 <pre> I think I left some tests in the test suite that are not run. The problems is that these are coarse statistical tests that have several false failures. Also there are problems if the moments don't even exist. https://github.com/scipy/scipy/issues/1329#issuecomment-17022751 is the list when I looked at this the last time, IIRC We should also be able to get the moments by numerical integration, but I haven't tried that yet. I think some of the entropies have possibly the wrong sign. Essentially, it requires going through the list and checking all the suspicious skew, kurtosis and entropy, to see whether they are a false alarm or a bug. related issues: non-existing moments are not correctly specified searching the issues for skew and kurtosis finds these two https://github.com/scipy/scipy/issues/2401 https://github.com/scipy/scipy/issues/1866 This is a generic warning. Some cases are known and have tickets, like truncnorm if you work only far out in the right tail. There are some function where the pdf i</pre> josef.pktd< at >gmail.com 2013-05-14T20:13:40 http://permalink.gmane.org/gmane.comp.python.scientific.user/34124 <pre>I am wondering whether there are specific examples one could run to check what exactly is wrong with skew and kurtosis on scipy as mentioned in the "remaining Issuess" [1] section of the documention [ http://docs.scipy.org/doc/scipy/reference/tutorial/stats.html#remaining-issues ]. It is also mentioned there that there is a range of values on which the scipy distributions gives wrong results. Is there any other document explaining further this (and the previous) issue? Thanks in advance, Sergio [1] Remaining Issues * skew and kurtosis, 3rd and 4th moments and entropy are not thoroughly tested and some coarse testing indicates that there are still some incorrect results left. * * the distributions have been tested over some range of parameters, however in some corner ranges, a few incorrect results may remain. _______________________________________________ SciPy-User mailing list SciPy-User&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user </pre> Sergio Rojas 2013-05-14T19:34:10 http://permalink.gmane.org/gmane.comp.python.scientific.user/34123 <pre> On Fri, May 10, 2013 at 7:41 PM, Jerry &lt;lanceboyle&lt; at &gt;qwest.net&gt; wrote: No need to apologize. This list doesn't really support Spyder. You might have better luck and Sypder's Google Code page. -p _______________________________________________ SciPy-User mailing list SciPy-User&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user </pre> Paul Hobson 2013-05-13T18:24:34 http://permalink.gmane.org/gmane.comp.python.scientific.user/34122 <pre>Geez, sorry.... :-\ Jerry On May 9, 2013, at 1:59 AM, Jerry wrote: </pre> Jerry 2013-05-11T02:41:32 http://permalink.gmane.org/gmane.comp.python.scientific.user/34121 <pre>Oops, my bad. Meant to sent this to MacPorts list. Jerry On May 9, 2013, at 2:29 AM, Robert Kern wrote: </pre> Jerry 2013-05-11T02:37:20 http://permalink.gmane.org/gmane.comp.python.scientific.user/34120 <pre>Hi, Yes, as Joon wrote, you can use the two sets as parameters for your function. Cheers, 2013/5/10 Arnaldo Russo &lt;arnaldorusso&lt; at &gt;gmail.com&gt; </pre> Matthieu Brucher 2013-05-10T15:17:52 http://permalink.gmane.org/gmane.comp.python.scientific.user/34119 <pre>Thank you so much for explanations. Now I understood the minimum solution. My question is, if I put those minimized values as parameters of my pp_min function, I'll get values of adjusted curve? Arnaldo. --- *Arnaldo D'Amaral Pereira Granja Russo* Lab. de Estudos dos Oceanos e Clima Instituto de Oceanografia - FURG 2013/5/10 Johann Cohen-Tanugi &lt;johann.cohentanugi&lt; at &gt;gmail.com&gt; _______________________________________________ SciPy-User mailing list SciPy-User&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user </pre> Arnaldo Russo 2013-05-10T15:10:13 http://permalink.gmane.org/gmane.comp.python.scientific.user/34116 <pre>On 2013-05-08 15:15:20 +0000, Pierre Raybaut said: The download link for the Mac App is still in RC phase (2.2.0rc) is that on purpose? I am wondering because you announced the release on May 8th, while this RC has been uploaded on April 6th? Cheers, Michael _______________________________________________ SciPy-User mailing list SciPy-User&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user </pre> K.-Michael 2013-05-10T10:00:15 http://permalink.gmane.org/gmane.comp.python.scientific.user/34115 <pre>Hi, I'll have to code multilinear interpolation in n dimensions, n~7. My data space is quite large, ~10**9 points. The values are given on a rectangular (but not square) grid. The values are numbers in a range of approx. [0.0, 100.0]. The challenge is to do this efficiently, and it would be great if the whole thing would be able to run fast on a machine with only 8G (or better 4G) RAM. A common task will be to interpolate 10**6 points, which souldn't take too long. Any ideas on how to do this efficiently are welcome: * which dtype to use? * is using pytables/blosc an option? How can this be integrated in the interpolation? * you name it ... ;) Cheers, Andreas. </pre> Andreas Hilboll 2013-05-10T09:58:45 http://permalink.gmane.org/gmane.comp.python.scientific.user/34114 <pre>Dear Arnaldo, I think what Pauli is trying to say is that the algorithm does not guarantee that the *global* minimum will be found, it proceeds with conjugate gradients or the like to find a local minimum starting from the initial parameter, values, so you might very well converge to a local minimum, but miss the global one. Maybe your function is ill-behaved, it seems to be "scale invariant" : X=x/gamma Y=y/gamma turns your function into g(X,Y) that depends on gamma only through an overall factor, if I read your formula correctly. cheers, Johann On 05/09/2013 11:43 PM, Arnaldo Russo wrote: </pre> Johann Cohen-Tanugi 2013-05-10T08:18:23 http://permalink.gmane.org/gmane.comp.python.scientific.user/34113 <pre>Hi Arnaldo, On Thu, May 9, 2013 at 4:43 PM, Arnaldo Russo &lt;arnaldorusso&lt; at &gt;gmail.com&gt;wrote: The solver tries to find the parameter (alpha, beta, gama) values which minimizes the sum squared value of the return of your function, `pp_min()`. When I try the Python and R solution: sum(pp_min(array([ 9.82259647e-02, -1.58338152e-02, 5.03125198e+01]), x, y), axis=0) Out[23]: 153.42871102569131 &gt;&gt;&gt; pp_R = array([0.09157204, 0.02129695, 148.89173924]) sum(pp_min(pp_R, x, y), axis=0) Out[21]: 155.07002552970221 So it seems like Python solver found better solution than the R one. You might want to fix alpha, beta parameters and draw plots with different values of gama to see what is going on. Best, Joon _______________________________________________ SciPy-User mailing list SciPy-User&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user </pre> Joon Ro 2013-05-10T04:11:25 http://permalink.gmane.org/gmane.comp.python.scientific.user/34112 <pre>Hi Pauli, I didn't understand. I thought that the output was the solution of my best values of "alpha", "beta" and "gama". But a much lower value of gama is a best choice? The R solver picked a double value while comparing with python results. I'm asking these things because I want to plot a fit curve with these parameters and I don't know how. Thank you --- *Arnaldo D'Amaral Pereira Granja Russo* Lab. de Estudos dos Oceanos e Clima Instituto de Oceanografia - FURG 2013/5/9 Pauli Virtanen &lt;pav&lt; at &gt;iki.fi&gt; _______________________________________________ SciPy-User mailing list SciPy-User&lt; at &gt;scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user </pre> Arnaldo Russo 2013-05-09T21:43:25 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.python.scientific.user