gmane.comp.lang.erlang.general

Re: [eeps] Joe Armstrong's suggestion forJSON<->Erlang BIFs

Jim Larson — 2008-07-25T08:15:27

Richard, Thanks for the great work on the JSON EEPS proposal! Robert Chu and I wrote the first public Erlang library for JSON (released thanks to A2Z Development, "an Amazon.com Company"), later the basis for the parser in Yaws, and inspiration for dozens of other independently-developed libraries whose authors must have thought "I can do better than _this_!" With that dubious introduction, let me offer some feedback. In message <87B91711-F378-4A2B-BE43-75D388A354E6< at >cs.otago.ac.nz> you write: This is assuming some sort of framing of the Io_List to limit it to a single JSON term. Since JSON is mostly self-framing (everything but numbers declare their own ends), and since at least one historical JSON-based transport took advantage of this (JSON-RPC 1.0 over raw TCP), is there any interest in a flavor of the parser that allows continuations of unparsed input? For example, our parsing interface was: decode(Continuation, CharList) => {done, Term, LeftoverChars} or {more, Continuation} I assume th

Re: fast JSON parser in C

Richard A. O'Keefe — 2008-07-25T04:42:32

On 25 Jul 2008, at 4:20 pm, Chris Anderson wrote: OK, the biggest question I see there is what to do about "objects". It seemed intuitively obvious to me (is there a common abbreviation for this (:-)?) that it would be really nice if a JSON "object" had an Erlang representation that an Erlang programmer would normally use for that kind of stuff. So I expect {"foo":1, "bar":2} to become [{foo,1},{bar,2}]

Re: fast JSON parser in C

Chris Anderson — 2008-07-25T04:20:10

Richard, Thanks for taking up the charge on this JSON stuff. I agree that interface, rather than speed, is what we should be thinking about now. We can always speed it up later. I posted to the couchdb-dev mailing list with a summary of some of the format opinions I'd found in various discussions. This might be helpful for the EEP. http://mail-archives.apache.org/mod_mbox/incubator-couchdb-dev/200807.mbox/%3ce282921e0807200924k256685f6r4b5651219f16d7b5< at >mail.gmail.com%3e Chris On Thu, Jul 24, 2008 at 11:55 PM, Richard A. O'Keefe cs.otago.ac.nz> wrote:

Re: fast JSON parser in C

Richard A. O'Keefe — 2008-07-25T03:55:27

I've printed that to study over the weekend. Thing is, it doesn't have to be *better*. It could even *be* the mochijson2 code. Part of what Joe was getting at, I think, is that it would be nice if JSON support were a standard part of Erlang/OTP in *some* fashion. That way everyone who wants to use JSON builds (for export) and sees (from import) the same kind of Erlang data. From a quick glance at the that code, I believe that a BIF *could* be usefully faster. Whether it is worth putting the effort into it is another matter. There are also some interface issues; I would value your comments on my EEP. -- If stupidity were a crime, who'd 'scape hanging?

Re: comment on my erlang Spamfilter

Richard A. O'Keefe — 2008-07-25T03:33:03

On 25 Jul 2008, at 2:18 am, James Hague wrote: I guess another comment is appropriate. "What's a word?" (This takes most of a 2-hour lecture in our information retrieval paper!) This is not going to identify "The" with "the". It is going to take "time-to-live" as one word. It is not going to realise that "these, and those" contains the word "these". (It's going to think that the first word is "these,".) The obvious thing is to run some separate program that splits a document into words, and writes them out one per line, but of course string:tokens("foo\nbar\n", " ") is going to think the whole string is one token. Oh, and let's consider a popular device used by spammers. Let us suppose that Erlang is a "naughty word". They might write it as "3r1ang" and rely on the human eye to read what they meant. The simplest definition of a "word", that works much of the time, is a sequence of letters and apostrophes such that each apostrophe has a letter on each side. (This will be confused by "3r1ang"; I d

Re: can mnesia be made to become something likeBigTable or SimpleDB ?

Rick R — 2008-07-25T03:33:49

Re: can mnesia be made to become something likeBigTable or SimpleDB ?

Eric Ho — 2008-07-25T02:17:43

Re: comment on my erlang Spamfilter

Richard A. O'Keefe — 2008-07-25T03:12:47

Lev Walkin has already commented on this. I'm just going to cover a few points he omitted. On 24 Jul 2008, at 10:58 pm, hask ellian wrote: (1) Where is the comment saying what this does? (2) It is a bad idea to use 'andalso' or 'orelse' in guards; the current compiler generates MUCH worse code for them than the classic ',' and ';'. (3) You wouldn't use head, tail, and not null in Haskell, so why do it in Erlang? (4) Indenting the auxiliary function is something I used to do back when I first learned Prolog. Lawrence Byrd commented wryly, "You really miss Algol, don't you?" I learned not to do that, because Prolog predicates don't nest, and NEITHER DO ERLANG FUNCTIONS. (Funs, yes. Named functions, no.) To use indentation when there is no actual nesting is misleading. (5) Jamming two function definitions together with no blank lines between makes them hard to read. (6) Oh yeah, you would count DOWN in Haskell, not up. Erlang is no different. Let's f

Re: fast JSON parser in C

Richard A. O'Keefe — 2008-07-25T02:16:16

I've written this up as an EEP and posted it to the eeps mailing list. Joe, when it turns up at the EEPs site, could you have a really good look at it and see if it fits what you meant? If not, maybe you could write a better EEP. Basically, erlang:term_to_json(Term[, Options]) is seen as a device for generating JSON to be consumed by programs in other languages (such as Javascript and Perl), and erlang:json_to_term(Io_List[, Options]) is seen as a device for accepting JSON produced by such programs. In writing my EEP, I cared about JSON->term->JSON round trip consistency, but not about term->JSON->term. -- If stupidity were a crime, who'd 'scape hanging?

Re: how: info on how to traverse abstract formattrees?

Robert Virding — 2008-07-25T00:58:01

Re: fast JSON parser in C

Chris Anderson — 2008-07-25T00:50:10

The version of mochijson2 that is in CouchDB's trunk (not in use) is about twice as fast as cjson - which is used by CouchDB currently. After spending a day writing my own leex/yecc parser, it turns out to be about 3 times slower than cjson, and about 6 times slower than mochijson2. I could probably optimize the grammar definition to try to make it faster. I was hoping that the magic of parser-generators would give me a big jump on the hand-crafted code. Since it didn't, I don't plan to pursue it any further. If anyone wants to see the leex/yaac code, I can put it online.

Re: fast JSON parser in C

Bob Ippolito — 2008-07-24T22:56:24

I'd be curious to know if leex/yecc can do any better than mochijson2 (which is written by hand), especially considering that it uses binaries instead of strings. http://code.google.com/p/mochiweb/source/browse/trunk/src/mochijson2.erl -bob

Re: fast JSON parser in C

Dominic Williams — 2008-07-24T22:31:13

Hi Joe, Joe Armstrong wrote: I'm a great fan of your principle of removing something for everything you add to the language. What would you want to remove in exchange for JSON bifs? JSON doesn't seem ubiquitous to me, but maybe I haven't been getting out enough lately. Cheers, Dominic Williams http://dominicwilliams.net ----

Re: fast JSON parser in C

Chris Anderson — 2008-07-24T21:53:05

I researched some into the C way of doing things, and wasn't sure how much overhead the ei communication would consume. Currently I'm working on a leex/yecc parser for JSON. The output format is quite flexible, so for now I'm just building it to pass the CouchDB cjson test_suite. Once it is working, it should be trivial to alter the format to fit an agreed-upon convention. I'll probably finish in the next day or two, and then I'll have an idea of whether using leex/yecc to generate Erlang provides a big speed boost. If it doesn't, at least I had fun! Joe's BIF idea does seem like the long-term solution.

Re: how: info on how to traverse abstract formattrees?

Tim Fletcher — 2008-07-24T21:46:29

Sure, but for the case where you just want to rewrite a particular function call a full parse transform seems a bit too much. Do you mean that your modified version is somewhere in the Erlang distribution, or am I misunderstanding you?

Re: fast JSON parser in C

Bob Ippolito — 2008-07-24T20:43:22

There are pretty decent pure Erlang JSON libraries available. They're pretty fast, relatively speaking, and they certainly don't crash the Erlang interpreter ;) I would worry about C when you actually need to. 2008/7/24 Rick R gmail.com>:

Re: Ideas for a new Erlang

Ulf Wiger — 2008-07-24T20:28:49

2008/7/24 Sven-Olof Nystr|m it.uu.se>: Not much, but the vital use of selective receive is in the start_tone() and similar functions. The key part is that the function blocks and waits for a specific response to its particular signal, and doesn't return to the surrounding control loop until it knows whether or not the operation succeeded. Since selective receive makes this possible, the top-level control loop is free to assume that start_tone() is atomic. This particular assumption is illegal in OHaskell, for example. No, that's the interpretation of synchronous that I meant. Many programming frameworks do not allow operations like start_tone() to block, which means that such operations must be carried out as explicit request-reply pairs, which become visible in the global state-event matrix. The last example in the presentation implements a filtering event loop, which might be similar to a solution using channels. Sure, but that doesn't solve the problem. Take the case of admission control.

Re: How to extend mensia lock to supportconditionally lock?

Ulf Wiger — 2008-07-24T20:09:41

As far as I know there are no docs on the locker implementation in mnesia, but I do recall some tricky bugs in this area. For one thing, the locker also handles the deadlock prevention logic, so it must keep track of dependencies to some extent. You can read the code and try to form your own opinion, but before you start trying to extend this part of mnesia, possibly introducing deadlock bugs in your application, you should carefully measure the performance of the existing locking solutions: record locks, sticky locks and table locks. You should also write a wrapper and see if it gives you the kind of performance improvement you're after. If it doesn't, you may be able to come up with some really nifty extension of mnesia_locker through deep meditation over the code. The next challenge, after ensuring that it's stable, would then be to try to get it accepted as an official extension to mnesia - otherwise, you'll be stuck maintaining your own mnesia patches, and that's no fun (been there, done that). Table

Re: fast JSON parser in C

Rick R — 2008-07-24T18:25:03

Re: Ideas for a new Erlang

Sven-Olof Nystr|m — 2008-07-24T18:24:28

> 2008/7/22 Sven-Olof Nystr|m it.uu.se>: > > Ulf Wiger writes: > > > > > > > A good place to start might be the POTS control program, which was > > > presented in the Erlang book, and also used (in a slightly different form) > > > in the Introductory Erlang course. > > > > > > I used it in my "Structured Network Programming" presentation to > > > show the consequences of different programming styles in multiway > > > signaling: > > > > > > http://www.erlang.se/euc/05/1500Wiger.ppt > > > > > > I'm not convinced that it will be sufficient to reveal significant differences > > > between the standard erlang style and channels, but it's a good place to > > > start, and the code can be expanded in a number of interesting ways. > > > > > > BR, > > > Ulf W > > > > > > (There's a working simulator to go with the program, but the OTP team > > > would have to give permission to release it in public. I'm sure you can > > > get your hands on it anyway.) > > > > >

Re: fast JSON parser in C

Jonathan Gray — 2008-07-24T17:19:00

Joe, It would certainly ease our integration. JSON is certainly becoming ubiquitous and tighter integration with Erlang would be very useful. One note, in my current implementation and in general, there's no real direct mapping for JSON objects/dictionaries. While Erlang does have dicts, we were never able to "create" them using erl_interface/ei, though I'm sure it's possible by dissecting the dict representation. I use a {obj, [{},{}]} type representation now to handle JSON objects in Erlang. Any thoughts? +1 on new JSON BIFs :) Willing to help in any way I can. Jonathan Gray -----Original Message----- From: erlang-questions-bounces< at >erlang.org [mailto:erlang-questions-bounces< at >erlang.org] On Behalf Of Joe Armstrong Sent: Thursday, July 24, 2008 3:49 AM To: Martin Carlson Cc: erlang-questions< at >erlang.org Subject: Re: [erlang-questions] fast JSON parser in C Since JSON seem to be ubiquitous is seems to me that there would be a strong case for a couple of new BIFs, term_to_json and json_to_term. thes