gmane.comp.lang.smalltalk.squeak.beginners

Re: Re: Binary file I/O performance problems

David Finlayson — 2008-09-06T15:29:35

I have implemented a number of signal processing programs in both C99 and Python (with psyco jit). I have an 8-core Mac Pro workstation which I can use as for parallel processing by launching multiple instances of the code using Make scripts. An interesting thing happened when I compared the performance of the C code to the Python code: The C code became I/O bound at 4 cores saturating either the disks or the memory bus (I am not sure exactly where the bottleneck is). While the Python version never became I/O bound at 8 cores, it did however close to within a factor of 10 of the performance of the C code. This suggested to me that If I had enough processors to saturate the I/O there was no speed advantage of writing the code in C. The next generation of workstations we buy will probably have dozens of cores but hard drives and memory will only be marginally faster (if history is any indication). So, if I/O is the rate limiting factor, not cpu speed, why not look for the most productive programing environme

Re: Re: Binary file I/O performance problems

Herbert König — 2008-09-06T09:45:39

Hello David, YO> I'm sure that there are other implications, but it sounds like you YO> do need some primitives to make it efficient. I would make a YO> primitive that is equivalent of read_xyza_ping() that fills a Squeak YO> object, or if you are dealing with array of XYZA_Ping structure, YO> making an array of homogeneous arrays so that all linenames are stored YO> in a ByteArray, all pingnums are stored in a WordArray, etc. In this YO> way, you may still be able to utilize the vector primitives. this approach seems to give a chance of solving the sped problem. In your original post you talked about 10 significant figures, so be aware that float array only is 32 bit floats with only about 8 significant figures. The second caveat is if many of your floats are in the range of 1e-38 (the closet to zero number of 32 Bit Float) Float array gets very slow (speed degradation by a factor of 8). I'm talking about FloatArray>>* and *= here. Sorry if I sound negative I just think its bad to ignore problems

Re: Integrating Seaside and Flash

Andy Burnett — 2008-09-06T07:35:47

_______________________________________________ Beginners mailing list Beginners< at >lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/beginners

Re: Arrays within arrays

Yoshiki Ohshima — 2008-09-05T23:36:13

At Fri, 5 Sep 2008 02:59:08 -0700, Charles Gray wrote: I still wonder what version of image you are using. In the image I'm using, I can evalute: #((0 0 0 0) (0 0 0 0) (0 0 0 0) (0 0 0 0) (0 0 0 0) (0 0 0 0) (2 1 0 0) (8 4 0 1) (14 6 0 2) (30 12 1 4)) = #(#(0 0 0 0) #(0 0 0 0) #(0 0 0 0) #(0 0 0 0) #(0 0 0 0) #(0 0 0 0) #(2 1 0 0) #( 8 4 0 1) #(14 6 0 2) #(30 12 1 4)) and get true. Thank you!

Re: Pre 1900 Date problems

Rob Rothwell — 2008-09-05T23:28:36

_______________________________________________ Beginners mailing list Beginners< at >lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/beginners

Re: Re: Binary file I/O performance problems

Yoshiki Ohshima — 2008-09-05T23:27:57

At Fri, 5 Sep 2008 14:49:29 -0700, David Finlayson wrote: I'm sure that there are other implications, but it sounds like you do need some primitives to make it efficient. I would make a primitive that is equivalent of read_xyza_ping() that fills a Squeak object, or if you are dealing with array of XYZA_Ping structure, making an array of homogeneous arrays so that all linenames are stored in a ByteArray, all pingnums are stored in a WordArray, etc. In this way, you may still be able to utilize the vector primitives.

Re: Pre 1900 Date problems

Yoshiki Ohshima — 2008-09-05T22:48:54

At Fri, 5 Sep 2008 17:54:19 -0400, Rob Rothwell wrote: It is only #readFrom: is trying to be too helpful, but the system itself can surely represent these earlier dates. Date class>>year:month:day: does the right thing.

Re: Re: Binary file I/O performance problems

David Finlayson — 2008-09-05T22:12:33

Coastal and marine geology, USGS. But this isn't an official project. Just a pipe dream of mine right now. I am not even sure I am competent enough to pull it off by myself. However, I figure the best way to get support for this is to build a semi-working prototype and then show it off and see what happens. I do wish Cog were further along though. Without Croquet, VW isn't really an option. I don't know if other languages support the 3D collaboration that Croquet promises. Meanwhile, I need to learn more Smalltalk. David

Pre 1900 Date problems

Rob Rothwell — 2008-09-05T21:54:19

_______________________________________________ Beginners mailing list Beginners< at >lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/beginners

Re: Re: Binary file I/O performance problems

David Finlayson — 2008-09-05T21:49:29

Unfortunately, the data is not a simple block of floats. For example, in C here is how I read a "ping" header block from one of our vendors formats: /* read_xyza_ping: read ping block, returns 1 if successful, EOF if * end of file */ int read_xyza_ping(FILE *fin, XYZA_Ping *pp) { int8_t byte[4]; fread(&pp->linename, sizeof(int8_t), MAX_LINENAME_LEN, fin); fread(&pp->pingnum, sizeof(uint32_t), 1, fin); fread(&byte, sizeof(int8_t), 4, fin); fread(&pp->time, sizeof(double), 1, fin); fread(&pp->notxers, sizeof(int32_t), 1, fin); fread(&byte, sizeof(int8_t), 4, fin); read_posn(fin, &pp->posn); fread(&pp->roll, sizeof(double), 1, fin); fread(&pp->pitch, sizeof(double), 1, fin); fread(&pp->heading, sizeof(double), 1, fin); fread(&pp->height, sizeof(double), 1, fin); fread(&pp->tide, sizeof(double), 1, fin); fread(&pp->sos, sizeof(double), 1, fin); if (ferror(fin) != 0) { perror("sxpfile: error: (read_xyza_ping)"); abort(); } /

Re: Re: Binary file I/O performance problems

Yoshiki Ohshima — 2008-09-05T21:21:23

At Fri, 05 Sep 2008 23:00:07 +0200, nicolas cellier wrote: I assume the answer is USGS, because of his email address! Yes, it sounds like something cool is going on.

Re: Binary file I/O performance problems

nicolas cellier — 2008-09-05T21:19:20

nicolas cellier a écrit : forgot to provide some timing (Athlon 32bits 1GHz) for write/read access: | a b c | { [a := FloatArray withAll: (1 to: 100000)] timeToRun. [b := ExternalFloatArray withAll: (1 to: 100000)] timeToRun. [c := ExternalDoubleArray withAll: (1 to: 100000)] timeToRun. [a do: [:e | ]] timeToRun. [b do: [:e | ]] timeToRun. [c do: [:e | ]] timeToRun. }. #(142 312 335 80 181 182)

Re: Binary file I/O performance problems

nicolas cellier — 2008-09-05T21:00:07

Yoshiki Ohshima a écrit : Hi David, your applications is exciting my curiosity. Which company/organization are you working for, if not indiscreet? I think you will solve most performances problems following good advices from Yoshiki. You might also want to investigate FFI as a way for handling C-layout-like ByteArray memory from within Smalltalk as an alternative. I made an example of use in Smallapack-Collections (search Smallapack in squeaksource, http://www.squeaksource.com/Smallapack/) . ExternalArray is an abstract class for handling memory filled as a C-Arrays of any type from within Smalltalk (only float double and complex are programmed in subclasses, but you can extend), and in fact FFI can handle any structure (though you'll might have to resolve alignment problems by yourself). There's a trade-off between fast reading (no conversion) and slower access (conversion at each access), however with ByteArray>>#doubleAt: and #floatAt: primitives (from FFI), and fast hacks to eventually rev

Re: Binary file I/O performance problems

Yoshiki Ohshima — 2008-09-05T19:03:45

At Fri, 5 Sep 2008 11:33:37 -0700, David Finlayson wrote: Ah, cool. In the OLPC Etoys image, there is a more efficient version of Sonogram called WsSonogram, and it is about 2 times faster than the original, and if you just add a primitive that takes a float array and calculate the sqrt of all entries and store them into the array, that will be 4-5 times faster or such. The code is of course perfectly portable across the platform (i.e., not tied to OLPC) so probably it might be an interest of you. It could be quite practical with a few extra primitives. One could of course imagine to utilize GPU. That would be fairly viable. That could be certainly an option, too.

Re: Binary file I/O performance problems

Yoshiki Ohshima — 2008-09-05T18:44:09

At Fri, 5 Sep 2008 10:59:03 -0700, David Finlayson wrote: If the integers or floats are in the layout of C's int[] or float[], there is a better chance to make it much faster. Look at the method Bitmap>>asByteArray and Bitmap>>copyFromByteArray:. You can convert a big array of non-pointer words from/to a byte array. data := (1 to: 1000000) as: FloatArray. words := Bitmap new: data size. words replaceFrom: 1 to: data size with: data. bytes := words asByteArray. "and you write out the bytes into a binary file." "to get them back:" words copyFromByteArray: bytes. data replaceFrom: 1 to: words size with: words. Obviously, you can recycle some of the intermediate buffer allocation and that would speed it up. FloatArray has some vector arithmetic primitives, and the Kedama system in OLPC Etoys image have more elaborated vector arithmetic primitives on integers and floats including operations with masked vectors.

Re: Binary file I/O performance problems

David Finlayson — 2008-09-05T18:33:37

For the most part, these numbers represent instrument measurements (swath bathymetry from sonar systems). Precision ranges from 5 to 10 significant figures depending on the specific instrument being recorded. So it wouldn't really be practical to form a look-up table in most cases. What attracted me to Squeak was that I was on the boat a few months ago and got a functional navigation system built (sort-of like a Garmin console on a pleasure boat) in about 2 days (used morphic and the UDPSocket stuff)! That was awesome. Then I modified the sonogram class to display sonar backscatter data (like a black-and-white image of the sea floor) in about 2 hours. Very cools stuff. The only problem was that the sonar data is time consuming to parse in Squeak and so the sonogram scrolled about 1 row per second (our system is collecting data at 8 pings per second) So it would take me 8 hours to display 1 hour of sonar data. The distant dream is to paint the sonar data into a Croquet world in real time where scientists f

Re: Binary file I/O performance problems

Matthias Berth — 2008-09-05T18:07:09

David, How many possible float values do you have? Maybe a lookup strategy for the conversion is feasible... Cheers Matthias On Fri, Sep 5, 2008 at 7:59 PM, David Finlayson usgs.gov> wrote:

Re: Binary file I/O performance problems

David Finlayson — 2008-09-05T17:59:03

I re-wrote the test application to load the test file entirely into memory before parsing the data. The total time to parse the file decreased by about 50%. Now that I/O is removed from the picture, the new bottle neck is turning bytes into integers (and then integers into Floats). I know that Smalltalk isn't the common language for number crunching, but if I can get acceptable performance out of it, then down the road I would like to tap into the Croquet environment. That is why I am trying to learn a way that will work. David

RE: Integrating Seaside and Flash

Ramon Leon — 2008-09-05T16:47:40

Both options are viable. In your blog post you didn't exactly say what you were wanting to parse the request for, I was trying to tell you that under normal circumstances you don't need to do that, post your form and let Seaside bind the values up to your model for you. Flash talking to the server and saving data is not a normal circumstance. I'd use Scriptaculous to hook some event you're interested in so when that event occurred a callback was done to pass data back to the server using some JavaScript to read the data from the client side element if it's a simple value... html div onMouseOut: (html request callback: [ :value | value inspect ] value: (SUStream new nextPutAll: 'fetchFlashData()')) If it's more complex with a bunch of fields, I'd post a form back on some event as you state in #2... html div onMouseOut: (html updater id: #someStatusIndicator ; triggerForm: #someId ; on: #renderStatusOn: of: self) Or just use a request instead of updater if yo

Re: Integrating Seaside and Flash

Cédrick Béler — 2008-09-05T12:46:12

I have no experience with Flash in general. What I would try is to use scriptaculous request fonction but I don't know at all if that's possible. Something like in AS: ExternalInterface.call("request", "Params"); My 2 cents Cédrick ps: you might ask the seaside list for this one though _______________________________________________ Beginners mailing list Beginners< at >lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/beginners

Integrating Seaside and Flash

Andy Burnett — 2008-09-05T12:00:06

_______________________________________________ Beginners mailing list Beginners< at >lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/beginners