http://permalink.gmane.org/gmane.comp.python.numeric.general hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.python.numeric.general/26570 I installed version 0.3.0 back in August on WindowsXP, and as far as I remember there were no problems at all with the install, and all tests pass. I thought the interface was really easy to use. But after trying it out I realized that my matlab is too old to understand the generated hdf5 files in an easy-to-use way, and I had to go back to csv-files. Josef josef.pktd< at >gmail.com 2008-12-02T02:53:01 http://permalink.gmane.org/gmane.comp.python.numeric.general/26569 On Dec 1, 2008, at 6:09 PM, Eric Firing wrote: Eric, Should be fixed in SVN (r6130). There were indeed problems with nested dtypes. Tricky beasts they are. Thanks for reporting! Pierre GM 2008-12-02T02:42:53 http://permalink.gmane.org/gmane.comp.python.numeric.general/26568 ===================================== Announcing HDF5 for Python (h5py) 1.0 ===================================== What is h5py? ------------- HDF5 for Python (h5py) is a general-purpose Python interface to the Hierarchical Data Format library, version 5. HDF5 is a versatile, mature scientific software library designed for the fast, flexible storage of enormous amounts of data. store data, organized by name in a tree-like fashion. You can create datasets (arrays on disk) hundreds of gigabytes in size, and perform random-access I/O on desired sections. Datasets are organized in a filesystem-like hierarchy using containers called "groups", and accesed using the tradional POSIX /path/to/resource syntax. This is the fourth major release of h5py, and represents the end of the "unstable" (0.X.X) design phase. Why should I use it? -------------------- H5py provides a simple, robust read/write interface to HDF5 data from Python. Existing Python and NumPy concepts are used for the interface; for example, da Andrew Collette 2008-12-02T01:53:46 http://permalink.gmane.org/gmane.comp.python.numeric.general/26567 _______________________________________________ Numpy-discussion mailing list Numpy-discussion< at >scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion frank wang 2008-12-02T01:38:11 http://permalink.gmane.org/gmane.comp.python.numeric.general/26566 On Dec 1, 2008, at 6:21 PM, Christopher Barker wrote: Good question. Having a dtype != None does skip a secondary loop. Once again, I;m sure there's plenty of room for optimization (eg, different loops whether the dtype is defined or not, whether missing values have to be taken into account or not, etc...). I just want to make sure that we're not missing any functionality and/or corner cases and that the usage is intuitive enough before spending some time optimizing... Pierre GM 2008-12-01T23:28:38 http://permalink.gmane.org/gmane.comp.python.numeric.general/26565 Does all that get bypassed if the dtype(s) is specified? Is it still slow in that case? -Chris Christopher Barker 2008-12-01T23:21:07 http://permalink.gmane.org/gmane.comp.python.numeric.general/26564 ascii I/O is slow, so that's a reason in itself to want it not to be slower! I agree -- for the simple cases, fromfile() could work very well -- perhaps it could even be used to speed up some special cases of loadtxt. But is anyone working on fromfile()? By the way, I think overloading fromfile() for text files is a bit misleading for users -- I propose we have a fromtextfile() or something instead. -Chris Christopher Barker 2008-12-01T23:19:24 http://permalink.gmane.org/gmane.comp.python.numeric.general/26563 Pierre, ma.masked_all does not seem to work with fancy dtypes and more then one dimension: In [1]:import numpy as np In [2]:dt = np.dtype({'names': ['a', 'b'], 'formats': ['f', 'f']}) In [3]:x = np.ma.masked_all((2,), dtype=dt) In [4]:x Out[4]: masked_array(data = [(--, --) (--, --)], mask = [(True, True) (True, True)], fill_value=(1.0000000200408773e+20, 1.0000000200408773e+20)) In [5]:x = np.ma.masked_all((2,2), dtype=dt) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/efiring/<ipython console> in <module>() /usr/local/lib/python2.5/site-packages/numpy/ma/extras.pyc in masked_all(shape, dtype) 78 """ 79 a = masked_array(np.empty(shape, dtype), ---> 80 mask=np.ones(shape, bool)) 81 return a 82 /usr/local/lib/python2.5/site-packages/numpy/ma/core.pyc in __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, k Eric Firing 2008-12-01T23:09:59 http://permalink.gmane.org/gmane.comp.python.numeric.general/26562 I agree, genloadtxt is a bit blotted, and it's not a surprise it's slower than the initial one. I think that in order to be fair, comparisons must be performed with matplotlib.mlab.csv2rec, that implements as well the autodetection of the dtype. I'm quite in favor of keeping a lite version around. On Dec 1, 2008, at 4:47 PM, Stéfan van der Walt wrote: Well, one of the issues is that we need to keep the function compatible w/ urllib.urlretrieve (Ryan, am I right?), which means not being able to go back to the beginning of a file (no call to .seek). Another issue comes from the possibility to define the dtype automatically: you need to keep track of the converters, then have to do a second loop on the data. Those converters are likely the bottleneck, as you need to check whether each value can be interpreted as missing or not and respond appropriately. I thought about creating a base class, with a specific subclass taking care of the missing values. I found out it would have dupli Pierre GM 2008-12-01T22:55:43 http://permalink.gmane.org/gmane.comp.python.numeric.general/26561 2008/12/1 Ryan May <rmay31< at >gmail.com>: More "I" than "O"! But I think numpy.fromfile, once fixed up, could fill this niche nicely. I haven't investigated the code in too much detail, but wouldn't it be possible to implement the current set of functionality in a base-class, which is then specialised to add the rest? That way, one could always instantiate TextReader yourself for some added speed. That's neat! Cheers Stéfan Stéfan van der Walt 2008-12-01T21:47:00 http://permalink.gmane.org/gmane.comp.python.numeric.general/26560 I've wondered about this being an issue. On one hand, you hate to make existing code noticeably slower. On the other hand, if speed is important to you, why are you using ascii I/O? I personally am not entirely against having two versions of loadtxt-like functions. However, the idea seems a little odd, seeing as how loadtxt was already supposed to be the "swiss army knife" of text reading. I'm seeing a similar slowdown with Pierre's version of the code. The version of loadtxt that I cobbled together with the StringConverter class (and no missing value support) shows about a 50% slowdown, so clearly there's a performance penalty for trying to make a generic function that can be all things to all people. On the other hand, this approach reduces code duplication. I'm not really opinionated on what the right approach is here. My only opinion is that this functionality *really* needs to be in numpy in some fashion. For my own use case, with the old version, I could read a text file and by h Ryan May 2008-12-01T21:23:18 http://permalink.gmane.org/gmane.comp.python.numeric.general/26559 Hi Pierre 2008/12/1 Pierre GM <pgmdevlist< at >gmail.com>: I see the code length increased from 200 lines to 800. This made me wonder about the execution time: initial benchmarks suggest a 3x slow-down. Could this be a problem for loading large text files? If so, should we consider keeping both versions around, or by default bypassing all the extra hooks? Regards Stéfan Stéfan van der Walt 2008-12-01T20:47:06 http://permalink.gmane.org/gmane.comp.python.numeric.general/26558 The docstring is correct in 1.2.1 and in the documentation; I suppose you have an older version of Numpy. Pauli Virtanen 2008-12-01T19:56:51 http://permalink.gmane.org/gmane.comp.python.numeric.general/26557 Says it takes a default dtype arg, but doesn't act like it's an optional arg: fromiter (iterator or generator, dtype=None) Construct an array from an iterator or a generator. Only handles 1-dimensional cases. By default the data-type is determined from the objects returned from the iterator. ---> 20 z = fromiter (y) TypeError: function takes at least 2 arguments (1 given) Neal Becker 2008-12-01T19:43:11 http://permalink.gmane.org/gmane.comp.python.numeric.general/26556 On Dec 1, 2008, at 2:26 PM, John Hunter wrote Quite agreed. Personally, I'd ditch the default dtype=float in favor of dtype=None, but compatibility is an issue. However, if we all agree on genloadtxt, we can use tailored-made version in different modules, like you suggest. There's an extra issue for which we have an solution I'm not completely satisfied with: names=True. It might be simpler for basic user not to set names=True, and have the first header recognized as names or not if needed (by processing the first line after the others, and using it as header if it's found to be a list of names, or inserting it back at the beginning otherwise)... Pierre GM 2008-12-01T19:42:27 http://permalink.gmane.org/gmane.comp.python.numeric.general/26555 OK, that worked great. I do think some a default impl in np.rec which returned a recarray would be nice. It might also be nice to have a method like np.rec.fromcsv which defaults to a delimiter=',', names=True and dtype=None. Since csv is one of the most common data interchange format in the world, it would be nice to have some obvious function that works with it with little or no customization required. Fernando and I have taught a scientific computing course on a number of occasions, and on the last round we taught to undergrads. Most of these students have little or no programming, for many the concept of an array is something they struggle with, dtypes are a difficult concept, but we found that they responded very well to our csv2rec example, because with no syntactic cruft they were able to load a file and do some stats on the columns, and I would like to see that ease of use preserved. JDH John Hunter 2008-12-01T19:26:40 http://permalink.gmane.org/gmane.comp.python.numeric.general/26554 (Sorry about that, I pressed "Reply" instead of "Reply all". Not my day for emails...) Pierre GM 2008-12-01T19:14:19 http://permalink.gmane.org/gmane.comp.python.numeric.general/26553 It looks like I am doing something wrong -- trying to parse a CSV file with dates formatted like '2008-10-14', with:: import datetime, sys import dateutil.parser StringConverter.upgrade_mapper(dateutil.parser.parse, default=datetime.date(1900,1,1)) r = loadtxt(sys.argv[1], delimiter=',', names=True) print r.dtype I get the following:: Traceback (most recent call last): File "genload_proposal.py", line 734, in ? r = loadtxt(sys.argv[1], delimiter=',', names=True) File "genload_proposal.py", line 711, in loadtxt (output, _) = genloadtxt(fname, **kwargs) File "genload_proposal.py", line 646, in genloadtxt rows[i] = tuple([conv(val) for (conv, val) in zip(converters, vals)]) File "genload_proposal.py", line 385, in __call__ raise ValueError("Cannot convert string '%s'" % value) ValueError: Cannot convert string '2008-10-14' In debug mode, I see the following where the error occurs ipdb> vals ('2008-10-14', '116.26', '116.40', '103.14', '104.08', '70749800', '104 John Hunter 2008-12-01T18:54:27 http://permalink.gmane.org/gmane.comp.python.numeric.general/26552 Well, looks like the attachment is too big, so here's the implementation. The tests will come in another message. _______________________________________________ Numpy-discussion mailing list Numpy-discussion< at >scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion Pierre GM 2008-12-01T18:21:08 http://permalink.gmane.org/gmane.comp.python.numeric.general/26551 2008/12/1 Pierre GM <pgmdevlist< at >gmail.com>: Struggling to comply! Cheers Stéfan Stéfan van der Walt 2008-12-01T18:22:15 http://permalink.gmane.org/gmane.comp.python.numeric.general/26550 And now for the tests: _______________________________________________ Numpy-discussion mailing list Numpy-discussion< at >scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion Pierre GM 2008-12-01T18:21:32 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.python.numeric.general