gmane.comp.parsers.spirit.devel

Re: [spirit2] namespaces

Carl Barron — 2008-12-02T03:00:38

On Dec 1, 2008, at 9:10 PM, Joel de Guzman wrote: great idea ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/

Re: [spirit2] namespaces

Matthias Vallentin — 2008-12-02T02:49:00

I found myself many times in this situation. This looks clean and helpful, from a usability perspective. My vote goes for it. Matthias

Re: [spirit 2] create and fusionize struct

Joel de Guzman — 2008-12-02T02:17:17

Cool! I think that's sufficient. But, hey, don't forget the docs, ok. You can simply rip the adapt_struct docs, if you want. Regards,

Re: [spirit 2] create and fusionize struct

Joel de Guzman — 2008-12-02T02:13:07

Hey, you should keep your copyright ;-) Regards,

[spirit2] namespaces

Joel de Guzman — 2008-12-02T02:10:59

Hi, namespaces... I find it a bit cumbersome to use lots of components in spirit2 placed in different namespaces. For example, in what namespace is "rule" in? How about "unused"? "lit"? etc. Many times, it's not clear where one component lives. I do like the way we have ascii, iso8859_1 and soon unicode. But others are rather, ehm, placed adhoc-y. For example, while arg_names::_1 is ok, arg_names::_val and arg_names::_a are not! Those are not arguments! I want to clean these up, so be prepared for namespace name changes. Also, I'm not quite happy with the situation where the user needs to refer to the docs to know which namespace a particular component lives in. So... I am really contemplating on hoisting all public components used by a particular module, except the character-set related components, in the namaspaces qi, karma, or lex. For example: namespace boost { namespace spirit { namespace qi { using boost::spirit::int_; }}} That way, when using Qi, for example, you can use the boost::spirit::qi namespace exclusively and still get the right components. No need to memorize which components live in which namespace. I'm inclined to do this unless there are objections. Your thoughts? Regards,

Re: [spirit 2] create and fusionize struct

Carl Barron — 2008-12-02T01:53:29

On Dec 1, 2008, at 7:02 PM, Joel de Guzman wrote: Ok tthe extra includes go away ok DEFINE_STRUCT -> BOOST_FUSION_STRUCT and your copywrite /license added. Probably since it was once cut from a testing main file:) its not needed Docs same as BOOST_FUSION_ADAPT_STRUCT except it also creates the struct. attached is modifed crteate_strict.h. maiin.cpp - simple tests results.txt my results of my runs. tested with fusion:::for_each,fusion::at_c any others fusion operations needed to test?? Last login: Mon Dec 1 19:22:57 on console localhost:~ carlbarron$ cd xcode_worrk/test_create -bash: cd: xcode_worrk/test_create: No such file or directory localhost:~ carlbarron$ cd xcode_work/test_create localhost:test_create carlbarron$ cd build/release localhost:release carlbarron$ ls test_createtest_create.dSYM localhost:release carlbarron$ ./test_create test of ex1 2 example 1 end of test of ex1 test of ex2 2 example 1 4 example 2 end of test of ex2 localhost:release carlbarron$ ./test_create test of ex1 2 example 1 end of test of ex1 test of ex2 2 example 1 4 example 2 end of test of ex2 start of test 3 10 example 1 using at_c end of test 3 start of test4 10 example 1 using at_c 20 example 2 using at_c end of test 4 localhost:release carlbarron$ ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________ Spirit-devel mailing list Spirit-devel< at >lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/spirit-devel

Re: [spirit 2] create and fusionize struct

Joel de Guzman — 2008-12-02T00:02:27

Cool! Some comments: * Don't include . That is expensive. Include only what's needed. Actually, simply including adapt_struct.hpp is enough to give you what you need from PP. * If you intend that to be part of fusion, then the name should be BOOST_FUSION_STRUCT * Why do you need to include ? If you want them added to fusion, I'll need: 1) Some tests 2) Docs I'll add a page for acknowledging contributions. Thanks! Regards,

[spirit 2] create and fusionize struct

Carl Barron — 2008-12-01T19:22:57

Atteached is a macro I use to create 'simple' structs and make fusion sequences of them ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________ Spirit-devel mailing list Spirit-devel< at >lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/spirit-devel

howto feed inputs to examples? (was Re: Spirit2x inSpirit SVN

Larry Evans — 2008-12-01T17:53:19

In: https://spirit.svn.sourceforge.net/svnroot/spirit/trunk/Spirit2x/libs/spirit/example/qi/ there's subdirectories with, apparently, inputs files to some examples: mini_c mini_xml* For example, mini_xml1.cpp contains: -{--cut here-- int main(int argc, char **argv) { char const* filename; if (argc > 1) { filename = argv[1]; } else { std::cerr << "Error: No input file provided." << std::endl; return 1; } std::ifstream in(filename, std::ios_base::in); if (!in) { std::cerr << "Error: Could not open input file: " << filename << std::endl; return 1; } -{--cut here-- However, the Jamfile doesn't show how to feed those input files to the programs. Is there some way, other than manually, to do that? ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/

Re: Rethinking the "what"

Joel de Guzman — 2008-11-23T01:49:01

That's what I did with Spirit2x. It works ok, until I had to write the expect parser where we have to supply a stream. Recall that the expectation_failure exception requires the parser to know "what" it is. There is no way in our interface to make that user configurable, unless we change the expect API into something else, which I do not want to do. No, that's a different parse tree. This is a grammar parse tree. In fact, the what function is emitting a flat version of it. In terms of complexity, it should be the same. Not sure I agree. It was never sufficient. Beyond some toy examples, it will be difficult to coax the error result into something more useful (*** but see below) Sure, that was what I planned in 2x. Alas, it's not doable unless we make expect a directive, like: p >> expect()[p2] ala classic Spirit. I don't like that. IMO, the spirit2 expect syntax is as ideal as it can be. Let's not mess with perfection ;-) I'd like to tweak something else to cater to the expect syntax, not the other way around. Yes, that's a lesser issue. That's not my main concern. I'm not adding more complexity to parsing. "what" is not called at parse time. Besides, the info object I suggested is not a template. It is instantiated only once. It's as cheap (at compile time) as any plain OOP class. (***) Having said all that, a minimalist approach would be to generate something like: struct info { std::string tag; boost::any data; std::list children; }; tag: parser tag, e.g. "int_" data: typically a std::string or wide-string children: non-empty for composites like sequence or alternative It is also my observation that when you have an expect expression like: a > b you do not want b to be a complex expression. In order to generate simple error meessages like: "Error. Expecting 'begin'" // a literal or "Error. Expecting 'int'" // int_ or "Error. Expecting conditional-expression" // a rule you'll want b above to be a simple terminal or a rule. So, if tag is for a moderately complex expression like "alternative", the user can have a std::map of functions that walks the info.children transforming that into something like, e.g.: // alternates "Error. Expecting conditional-expression or integer or 'begin'" it's probably a good idea to suggest proper usage of the expect expression to avoid overly complex error messages. Regards,

Re: Rethinking the "what"

Hartmut Kaiser — 2008-11-22T15:20:18

A couple of (possibly unrelated) thought: I'ld say, make it as simple as possible and let the user supply the unknowns (like the type of the stream to use in your example above). Letting what() to create a parse tree duplicates the parse tree the user might want to create in the first place. And it adds complexity even if the user doesn't need it. The scheme we have had in Spirit2 was not ideal, but sufficient. What's perhaps missing there is the possibility for the user to customize the behavior. Creating expensive object instances during error handling is generally not an issue, at least that's the way I'm approaching error handling... Don't add more complexity to the parsing, compilation times are already scary without this. My 2€ cents. Regards Hartmut ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Spirit-devel mailing list Spirit-devel< at >lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/spirit-devel

Re: Rethinking the "what"

Joel de Guzman — 2008-11-22T12:40:03

Exactly. Well, almost... Regarding pre-skipping, e will pre-skip only if it is a terminal. Perhaps it's a good idea to always do a pre-skip before throwing the exception. That way, the location of the error is dead accurate. Regards,

Re: Rethinking the "what"

Carl Barron — 2008-11-22T09:42:46

------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________ Spirit-devel mailing list Spirit-devel< at >lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/spirit-devel

Re: Rethinking the "what"

Joel de Guzman — 2008-11-22T04:50:59

Bad example. That wasn't gibberish, it seems. But I think you get my point :P .. Just imagine a bit more complex expression. Regards,

Rethinking the "what"

Joel de Guzman — 2008-11-22T04:45:23

Hi, I have mixed feelings with the way we have the "what" member functions. In spirit2, it simply returns a std::string. I've had problems when I had to return a wide string. No good. In spirit2x, I initially redesigned it to take in a templated stream:. Here's the int parser parser's what function, for example: template void what(OStream& out, Context const& /*context*/) const { out << "integer"; } Here's for alternative: template void what(OStream& out, Context const& context) const { out << "alternatives["; fusion::for_each(elements, spirit::detail::what_function(out, context)); out << "]"; } Iterates over the alternatives and collects the "what"s of each. So far so good. But then I had to implement the expect parser which throws an exception. Somewhere in its internals, it has to call the "what" function of the parser that fails and make an exception out of it. In spirit2 (with the what function returning string) the code was: Exception x = {first, last, component.what(context) }; throw x; Now, with spirit2x, I had to create a string stream. Yaiks! 1) We all know that this is an expensive operation 2) What string type? Base it on the iterator type? What if the iterator type points to a stream of tokens? Also, I'm not quite happy that the what function returns an unstructured blob. Say, we have this parser: start = '[' > (a | b | c); The default error message extracted from this when > fails is something like: "Error. Expecting alternatives[a, b, c] here" Gibberish, to say the least. We'd want to transform that to something more meaningful. Alas, the unformatted stream (or string) makes it difficult. So what do we want? I'm thinking we want to return an AST of the parser! Something that can be travesed. So, I came up with something python/lua/-ish dynamic type info: struct info { struct nil {}; struct nary; typedef boost::variant< nil , int , double , std::basic_string , std::basic_string , boost::recursive_wrapper > type; info(std::string const& tag) : tag(tag), value(nil()) {} template info(std::string const& tag, T const& value) : tag(tag), value(value) {} std::string tag; type value; }; struct info::nary { template nary(Iterator first, Iterator last) : children(first, last) {} std::vector children; }; with this structure, we can easily transform the error message into something like: "Error. Expecting a or b or c" we can even use karma to walk the tree and therefore provide more powerful output transformations. Your thoughts? Regards,

Re: Anatomy of.. installment: extended terminals

OvermindDL1 — 2008-11-20T20:45:36

Definitely nicely done. Cannot wait till the next installment. On Sun, Nov 16, 2008 at 10:11 PM, Joel de Guzman boost-consulting.com> wrote: ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/

Re: Anatomy of.. installment: extended terminals

Joel de Guzman — 2008-11-17T05:11:25

Hi! here's an (important) update... So far, I've been writng code with the attribute metafunction writen as (example): This will have to be changed to: template struct attribute; The iterator type is needed in certain (rare) occasions. An example is the raw[p] directive which creates an iterator range as its attribute. There might be more in karma. Regards,

Re: spirit2x speedup

Hartmut Kaiser — 2008-11-16T19:29:13

Thanks Eric! Regards Hartmut ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/

Re: spirit2x speedup

Eric Niebler — 2008-11-16T19:03:34

Spirit's test results were looking ok on trunk so I've merged this change to the release branch and deleted the old Proto v2 code from both trunk and release. I'll be keeping an eye on spirit's regression results on release just to make sure.

Anatomy of.. installment: extended terminals

Joel de Guzman — 2008-11-16T18:35:17

Hi, We had an example of a terminal, the int parser. next, well investigate extended terminals, those that accept parameters. First, we'll have another example of a simple terminal, the bare eps parser. Like before, we start of by enabling our place holder. But wait, we haven't really seen how we define our place holders. It's actually quite simple. For the int_ (and friends) we have this code: BOOST_SPIRIT_DEFINE_TERMINALS( /*...*/ ( short_ ) ( long_ ) ( int_ ) ( ulong_long ) ( long_long ) /*...*/ ) Placed in support/common_terminals.hpp. Each of these is a call to the macro, BOOST_SPIRIT_TERMINAL. Example: BOOST_SPIRIT_TERMINAL(int_) which expands to: namespace tag { struct int_{}; } typedef boost::proto::terminal::type int__type; int__type const int_ = {{}}; Our eps parser also has its eps placeholder. Unlike the int_ placeholder, however, eps can stand in its own, or can accept a parameter. eps basicallly has two forms: eps // as is eps(c) // with a condition the condition can be an immediate boolean, or a function or function object (typically phoenix expressions) that return a bool. Some samples: eps eps(true) eps(f) The parameterized forms are called semantic predicates. The parser fails or succeeds depending on the bool condition or the result of the function f, in the last example. The function f is evaluated not at grammar defintion time, but at parse time. In other words, it is lazy. All function-like parsers, that take in functions as its arguments, like the eps(f), char_(f), lit(f), etc., are lazily evaluated at parse time. For terminals with function-like forms, we use a related macro: BOOST_SPIRIT_TERMINAL_EX(name) Hence, for eps: BOOST_SPIRIT_TERMINAL_EX(eps) Similarly, you'll see in support/common_terminals.hpp a macro call to BOOST_SPIRIT_DEFINE_TERMINALS_EX which lists all the extended terminals used by Spirit: // Our extended terminals BOOST_SPIRIT_DEFINE_TERMINALS_EX( /*...*/ ( repeat ) ( eps ) /*...*/ ) Ok, let's proceed. As before, we enable the eps terminal: template <> struct use_terminal // enables eps : mpl::true_ {}; Note again that this is in namespace boost::spirit. Now, the actual parser (in boost::spirit::qi namespace): struct eps_parser : primitive_parser { template struct attribute { typedef unused_type type; }; template bool parse(Iterator& first, Iterator const& last , Context& context, Skipper const& skipper , Attribute& attr) const { qi::skip(first, last, skipper); return true; } template void what(OStream& out, Context const& context) const { out << "eps"; } }; No need to explain, I presume? It's even simpler than the int parser. eps always returns true, but eats nothing. Well, ok, it does a pre-skip, as all terminals do. Then, our generator (in namespace boost::spirit::qi): template struct make_primitive { typedef eps_parser result_type; result_type operator()(unused_type, unused_type) const { return result_type(); } }; Ok, there you go. But we're not done yet. We have so far dealt with the unparameterized form of eps. For the parameterized form, we present the semantic_predicate: struct semantic_predicate : primitive_parser { template struct attribute { typedef unused_type type; }; semantic_predicate(bool predicate) : predicate(predicate) {} template bool parse(Iterator& first, Iterator const& last , Context& context, Skipper const& skipper , Attribute& attr) const { qi::skip(first, last, skipper); return predicate; } template void what(OStream& out, Context const& context) const { out << "semantic-predicate"; } bool predicate; }; Again, I guess there's no need to explain how this parser functions. It's a very trivial parser. What's interesting is how this is hooked up. First, we enable the parameterized form of eps: template struct use_terminal > > : is_convertible {}; This fella enables the eps form: eps(bool-condition) Notice that we are picking up the: terminal_ex > (terminal_ex is the one used for "extended" terminals when we use the macro BOOST_SPIRIT_TERMINAL_EX) and we enable the use_terminal only if A0 is convertible to a bool. The fusion::vector holds the arguments passed into our eps terminal. Here, we care only for single arguments. eps only accepts a single argument. Likewise, we provide its generator: template struct make_primitive< terminal_ex > , Modifiers> { typedef semantic_predicate result_type; template result_type operator()(Terminal const& term, unused_type) const { return result_type(fusion::at_c<0>(term.args)); } }; We create a semantic_predicate passing in the boolean predicate we obtained from the argument to eps (contained in the fusion vector). We're almost done. But we have to take care of one last detail. So far, we've handled the /immediate/ form of eps. For eps, what's more important is the lazy form: eps(f) For that to happen, we simply need to enable the lazy form: template <> // enables eps(f) struct use_lazy_terminal< qi::domain, tag::eps, 1 /*arity*/ > : mpl::true_ {}; Then all the lazy magic happens. use_lazy_terminal is only concerned about 1) the domain 2) the terminal tag and 3) the arity. The code above reads as: use lazy terminal in qi::domain with tag eps and an arity of 1. At parser construction time, this is all that's needed. At parse time, the function f, passed to the eps terminal, will be converted to a semantic_predicate. Again, take note, this happens at parse time. The actual creation of the semantic_predicate is delayed. Only at parse time can we truly know the boolean predicate. For example, this may be a rule local variable, a rule inherited attribute, and so on. ----------------------- Attached is the full eps.hpp file. Peruse the code. Try to see how it compares to the one in "classic' spirit. You'll be amazed :-) Regards,

Re: [Spirit2X] How to add function-like parsers and directives to Spirit2X (+ remarks)

Joel de Guzman — 2008-11-12T23:06:52

Potentially, yes, it can have an impact. Here's the pseudo code for that: iterator save = first; if (subject.parse(first, last)) { if (action()) return true; } first = save; // restore return false; The iterator is held for the whole duration of the parse. If the parse involves multi-megabytes of input, and if the iterator is a multi_pass, then all that information must be cached in memory. The bad thing about it is that regardless if the flag is used or not, the client will still pay. Regards,