gmane.comp.parsers.spirit.devel

[spirit2] namespaces

Joel de Guzman — 2008-12-02T02:10:59

Hi, namespaces... I find it a bit cumbersome to use lots of components in spirit2 placed in different namespaces. For example, in what namespace is "rule" in? How about "unused"? "lit"? etc. Many times, it's not clear where one component lives. I do like the way we have ascii, iso8859_1 and soon unicode. But others are rather, ehm, placed adhoc-y. For example, while arg_names::_1 is ok, arg_names::_val and arg_names::_a are not! Those are not arguments! I want to clean these up, so be prepared for namespace name changes. Also, I'm not quite happy with the situation where the user needs to refer to the docs to know which namespace a particular component lives in. So... I am really contemplating on hoisting all public components used by a particular module, except the character-set related components, in the namaspaces qi, karma, or lex. For example: namespace boost { namespace spirit { namespace qi { using boost::spirit::int_; }}} That way, when using Qi, for example, you can use the boost::spirit::qi namespace exclusively and still get the right components. No need to memorize which components live in which namespace. I'm inclined to do this unless there are objections. Your thoughts? Regards,

[spirit 2] create and fusionize struct

Carl Barron — 2008-12-01T19:22:57

Atteached is a macro I use to create 'simple' structs and make fusion sequences of them ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________ Spirit-devel mailing list Spirit-devel< at >lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/spirit-devel

Rethinking the "what"

Joel de Guzman — 2008-11-22T04:45:23

Hi, I have mixed feelings with the way we have the "what" member functions. In spirit2, it simply returns a std::string. I've had problems when I had to return a wide string. No good. In spirit2x, I initially redesigned it to take in a templated stream:. Here's the int parser parser's what function, for example: template void what(OStream& out, Context const& /*context*/) const { out << "integer"; } Here's for alternative: template void what(OStream& out, Context const& context) const { out << "alternatives["; fusion::for_each(elements, spirit::detail::what_function(out, context)); out << "]"; } Iterates over the alternatives and collects the "what"s of each. So far so good. But then I had to implement the expect parser which throws an exception. Somewhere in its internals, it has to call the "what" function of the parser that fails and make an exception out of it. In spirit2 (with the what function returning string) the code was: Exception x = {first, last, component.what(context) }; throw x; Now, with spirit2x, I had to create a string stream. Yaiks! 1) We all know that this is an expensive operation 2) What string type? Base it on the iterator type? What if the iterator type points to a stream of tokens? Also, I'm not quite happy that the what function returns an unstructured blob. Say, we have this parser: start = '[' > (a | b | c); The default error message extracted from this when > fails is something like: "Error. Expecting alternatives[a, b, c] here" Gibberish, to say the least. We'd want to transform that to something more meaningful. Alas, the unformatted stream (or string) makes it difficult. So what do we want? I'm thinking we want to return an AST of the parser! Something that can be travesed. So, I came up with something python/lua/-ish dynamic type info: struct info { struct nil {}; struct nary; typedef boost::variant< nil , int , double , std::basic_string , std::basic_string , boost::recursive_wrapper > type; info(std::string const& tag) : tag(tag), value(nil()) {} template info(std::string const& tag, T const& value) : tag(tag), value(value) {} std::string tag; type value; }; struct info::nary { template nary(Iterator first, Iterator last) : children(first, last) {} std::vector children; }; with this structure, we can easily transform the error message into something like: "Error. Expecting a or b or c" we can even use karma to walk the tree and therefore provide more powerful output transformations. Your thoughts? Regards,

Anatomy of.. installment: extended terminals

Joel de Guzman — 2008-11-16T18:35:17

Hi, We had an example of a terminal, the int parser. next, well investigate extended terminals, those that accept parameters. First, we'll have another example of a simple terminal, the bare eps parser. Like before, we start of by enabling our place holder. But wait, we haven't really seen how we define our place holders. It's actually quite simple. For the int_ (and friends) we have this code: BOOST_SPIRIT_DEFINE_TERMINALS( /*...*/ ( short_ ) ( long_ ) ( int_ ) ( ulong_long ) ( long_long ) /*...*/ ) Placed in support/common_terminals.hpp. Each of these is a call to the macro, BOOST_SPIRIT_TERMINAL. Example: BOOST_SPIRIT_TERMINAL(int_) which expands to: namespace tag { struct int_{}; } typedef boost::proto::terminal::type int__type; int__type const int_ = {{}}; Our eps parser also has its eps placeholder. Unlike the int_ placeholder, however, eps can stand in its own, or can accept a parameter. eps basicallly has two forms: eps // as is eps(c) // with a condition the condition can be an immediate boolean, or a function or function object (typically phoenix expressions) that return a bool. Some samples: eps eps(true) eps(f) The parameterized forms are called semantic predicates. The parser fails or succeeds depending on the bool condition or the result of the function f, in the last example. The function f is evaluated not at grammar defintion time, but at parse time. In other words, it is lazy. All function-like parsers, that take in functions as its arguments, like the eps(f), char_(f), lit(f), etc., are lazily evaluated at parse time. For terminals with function-like forms, we use a related macro: BOOST_SPIRIT_TERMINAL_EX(name) Hence, for eps: BOOST_SPIRIT_TERMINAL_EX(eps) Similarly, you'll see in support/common_terminals.hpp a macro call to BOOST_SPIRIT_DEFINE_TERMINALS_EX which lists all the extended terminals used by Spirit: // Our extended terminals BOOST_SPIRIT_DEFINE_TERMINALS_EX( /*...*/ ( repeat ) ( eps ) /*...*/ ) Ok, let's proceed. As before, we enable the eps terminal: template <> struct use_terminal // enables eps : mpl::true_ {}; Note again that this is in namespace boost::spirit. Now, the actual parser (in boost::spirit::qi namespace): struct eps_parser : primitive_parser { template struct attribute { typedef unused_type type; }; template bool parse(Iterator& first, Iterator const& last , Context& context, Skipper const& skipper , Attribute& attr) const { qi::skip(first, last, skipper); return true; } template void what(OStream& out, Context const& context) const { out << "eps"; } }; No need to explain, I presume? It's even simpler than the int parser. eps always returns true, but eats nothing. Well, ok, it does a pre-skip, as all terminals do. Then, our generator (in namespace boost::spirit::qi): template struct make_primitive { typedef eps_parser result_type; result_type operator()(unused_type, unused_type) const { return result_type(); } }; Ok, there you go. But we're not done yet. We have so far dealt with the unparameterized form of eps. For the parameterized form, we present the semantic_predicate: struct semantic_predicate : primitive_parser { template struct attribute { typedef unused_type type; }; semantic_predicate(bool predicate) : predicate(predicate) {} template bool parse(Iterator& first, Iterator const& last , Context& context, Skipper const& skipper , Attribute& attr) const { qi::skip(first, last, skipper); return predicate; } template void what(OStream& out, Context const& context) const { out << "semantic-predicate"; } bool predicate; }; Again, I guess there's no need to explain how this parser functions. It's a very trivial parser. What's interesting is how this is hooked up. First, we enable the parameterized form of eps: template struct use_terminal > > : is_convertible {}; This fella enables the eps form: eps(bool-condition) Notice that we are picking up the: terminal_ex > (terminal_ex is the one used for "extended" terminals when we use the macro BOOST_SPIRIT_TERMINAL_EX) and we enable the use_terminal only if A0 is convertible to a bool. The fusion::vector holds the arguments passed into our eps terminal. Here, we care only for single arguments. eps only accepts a single argument. Likewise, we provide its generator: template struct make_primitive< terminal_ex > , Modifiers> { typedef semantic_predicate result_type; template result_type operator()(Terminal const& term, unused_type) const { return result_type(fusion::at_c<0>(term.args)); } }; We create a semantic_predicate passing in the boolean predicate we obtained from the argument to eps (contained in the fusion vector). We're almost done. But we have to take care of one last detail. So far, we've handled the /immediate/ form of eps. For eps, what's more important is the lazy form: eps(f) For that to happen, we simply need to enable the lazy form: template <> // enables eps(f) struct use_lazy_terminal< qi::domain, tag::eps, 1 /*arity*/ > : mpl::true_ {}; Then all the lazy magic happens. use_lazy_terminal is only concerned about 1) the domain 2) the terminal tag and 3) the arity. The code above reads as: use lazy terminal in qi::domain with tag eps and an arity of 1. At parser construction time, this is all that's needed. At parse time, the function f, passed to the eps terminal, will be converted to a semantic_predicate. Again, take note, this happens at parse time. The actual creation of the semantic_predicate is delayed. Only at parse time can we truly know the boolean predicate. For example, this may be a rule local variable, a rule inherited attribute, and so on. ----------------------- Attached is the full eps.hpp file. Peruse the code. Try to see how it compares to the one in "classic' spirit. You'll be amazed :-) Regards,

[Spirit2X] How to add function-like parsers anddirectives to Spirit2X (+ remarks)

Francois Barel — 2008-11-11T15:23:44

Hi, I started playing around with Spirit2X, and have a few minor remarks as well as one "how should I" question (to which the answer is probably "just wait for it to be implemented" :p). First off, 3 remarks: 1. In support/placeholder.hpp I suggest applying the attached patch (adding the boost namespace) so that the BOOST_SPIRIT_PLACEHOLDER and BOOST_SPIRIT_DEFINE_PLACEHOLDERS macros can be used in contexts which do not have "using namespace boost;". 2. Small copy/paste error in plus's what() function, patch attached. 3. Setting "pass" to false in an action (e.g. my_parser[pass = false]) fails parsing as expected, yet the input iterator is still advanced (it isn't reset to the value it had before my_parser was called). Just wondering, since this is contrary to usual Spirit behavior -- is that intentional? (Looks like it was the same in Spirit2.) And the question: in Spirit2 I had several custom directives with arguments, using the following syntax: my_directive(first_arg, second_arg, ...)[ ...parser... ] (think classic Spirit if_p / limit_d / ...), and also custom "function-like" parsers: my_function(first_arg, second_arg, ...) Depending on the cases, first_arg, second_arg, ... are either "immediates" (number, ...) or Phoenix actors or Spirit parsers (well, Proto exprs). Figuring out how to hook those into the Spirit2 meta-grammar was a lot of fun at the time :o) Now I am wondering how that should be registered into Spirit2X's grammar. Basically I see two options, and I'm wondering which one is best: 1. - Make my_xxx an instance of a custom type (let's say my_xxx_terminal_spec) - which has an operator() which receives first_arg, second_arg, ... and returns an instance of another custom type which stores the values of first_arg, second_arg, ... (let's say my_xxx_extended_terminal) - And register that second type as a directive (with use_directive for my_directive_extended_terminal) or as a terminal (with use_terminal for my_function_extended_terminal) IOW, do it much like it is done for char_ now (with terminal_spec having an operator() which returns an extended_terminal, and that type getting registered "normally" via use_terminal / use_directive). That seems a little "harder" than the rest of extending Spirit2X which is made very easy with use_terminal, use_directive, ... (having to write an operator() by hand -- it's like for the operator() overloads of parameterized rules and grammars, it's a little surprising that the function-call syntax is not handled via Proto like the rest). 2. Next to use_operator, use_terminal and use_directive, I see there is a use_function, which is not implemented / used yet AFAICT. Maybe it is meant to provide the way to declare this kind of constructs in the future... Can you confirm -- and should I just wait for it to be implemented? :) TIA, François ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________ Spirit-devel mailing list Spirit-devel< at >lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/spirit-devel

Boostcon 2009 call for sessions

dan marsden — 2008-11-09T20:00:41

3rd annual Boost Conference 2009 Aspen CO, USA, May 3-9, 2009, www.boostcon.com ---------------------------------------------------------------------------- Call for participation ---------------------- Promising to be the main face-to-face event for all things Boost (www.boost.org), BoostCon'09 opens the door to your C++ future. From using the Boost libraries to writing and maintaining them, from evangelizing to deploying Boost within your organization, from infrastructure and process to vision and mission, and from TR1 to TR2 and the new C++0x Standard, BoostCon brings together the sessions, the colleagues, and the inspiration to support your work with C++ and Boost in particular for the next year. To reflect the breadth of the Boost community, the conference includes sessions aimed at two constituencies: Boost end-users and hard-core Boost library and tool developers. The program fosters interaction and engagement within and across those two groups, with an emphasis on hands-on, participatory sessions. Session topics -------------- * Topics of interest include, but are not restricted to, the following: * General tutorial sessions introducing one or more Boost libraries * In-depth sessions on using specific libraries * Case studies on using Boost * Experts panels * Advanced sessions on implementation techniques used within Boost libraries * C++0x and how it will change life for users and library writers * TR1 and TR2 * Development workshops to extend or enhance existing Boost libraries * Workshops on design process * Infrastructure workshops such as Build tools, Website, Testing * Concepts and Generic Programming * Hardware and infrastructure presentations focused on how libraries can make better use of the technology * Other topics likely to be of great interest to Boost users and Developers. Interactive and collaborative sessions are encouraged, as this is the nature of both the on-line Boost community and the style of learning and participation that has proven most successful at such events. Sessions can be tutorial based, with an emphasis on interaction and participant involvement, or workshop based, whether hands-on programming or paper-based, discussion-driven collaborative work. Session formats --------------- Presentations ............. Presentations focus on a practitioner's ideas and experience with anything relevant to Boost and Boost users. Panels ...... Panels feature three or four people presenting their ideas and experiences relating to Boost's relevant, controversial, emerging, or unresolved issues. Panels may be conducted in several ways, such as comparative, analytic, or historic. Tutorials ......... Tutorials are sessions at which instructors teach conference participants specific Boost-relevant skills. Workshops ......... Workshops provide an active arena for advancements in Boost-relevant topics. Workshops provide the opportunity for experienced practitioners to develop new ideas about a topic of common interest and experience. Author's Corner Presentations ............................. These were introduced at BoostCon 2008, and were a great success. They are short (30 minute) sessions, focusing on tips on usage and design. In addition, we're looking to uncover the hidden design gems boost libraries. Tool Vendors Presentations .......................... We actively encourage tool vendors and ISP's to submit proposals for a special Tool Vendors Session Track aimed at products related to Boost and C++ (compilers, libraries, tools, etc.). Other formats may also be of interest. Don't hold back a proposal just because it doesn't fit into a pigeonhole. Submitting a proposal --------------------- Standard Sessions are 90 minutes. You may submit a proposal for fractions or multiples of 90-minutes. Fractional proposals will be grouped into 90 minute sessions covering related topics. Longer sessions, such as tutorials and classes, will be assigned 90 minute, three hour (i.e. half day), or six hour (i.e. full day) time slots. Please include: * The working title. * Type of session: presentation, panel, tutorial, workshop, authors corner, vendor track, other * A paragraph or two describing the topic covered, suitable for the conference web site * Proposed length: 10-20 minute short talks, 45 minutes, 90 minutes, half-day, full day * Alternate lengths, if you are willing to make adjustments: 10-20 minute lightning-talks, 45 minutes, 90 minutes, half-day, full day * Audience: users, developers, both * Level: basic, intermediate, advanced * A biography, suitable for the conference web site * Your contact information (will not be made public) Submission details ------------------ All submissions have to be done through the EasyChair conference management system: http://www.easychair.org/conferences/?conf=boostcon09. If you have not already registered at EasyChair, you will need to do so in order to submit your proposal. All submissions will go through a peer review process. Authors are invited to submit PDF versions of full papers of up to 10 pages in ACM conference proceedings format (see http://www.acm.org/sigs/publications/proceedings-templates). The full papers are not required unless you want them published in the proceedings. All accepted papers will be made available in the Association for Computing Machinery (ACM) Digital Library (approval pending). Best papers, after further reviews, will be considered to be book chapters or journal articles in a renowned journal. The session materials go on the BoostCon CD handed out to attendees. For general information on the BoostCon 2009 paper submission or the scope of technical papers solicited, please refer to the conference website at www.boostcon.com. For any other questions about the submission process or paper format, please contact the Program Committee at boostcon09< at >easychair.com. If you have any technical problems with EasyChair, please contact EasyChair for help. Note: Presenters must agree to grant a non-exclusive perpetual license to publish submitted materials, either electronically or in print, in any media related to BoostCon. Important dates --------------- Proposals for submissions due: January 10th 2009. Proposals acceptances sent (tentative program available): February 10th, 2009 Fully scheduled program available: March 1st, 2009 Session materials due: April 1st 2009. Hartmut Kaiser, email: hartmut.kaiser< at >gmail.com (Program Committee Chair) David Abrahams, email: dave< at >boostpro.com (Conference Chair) On behalf of the conference organizers ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/

spirit2x speedup

Joel de Guzman — 2008-11-05T12:35:07

Hi Y'all, Ok, the informal count is in. Heh, not the U.S. race, but for Spirit2x vs. Spirit2 :-) I'm getting a 24% speedup in the rule.cpp test when comparing the old and the new. The syntax and usage is nearly identical. I haven't optimized yet. Well, ok, I did a bit: converting the proto transforms into its primitive forms. Next step: profiling the compiler. Dan, any news in the Fusion side of things? Regards,

spirit2 permutations and attributes

Joel de Guzman — 2008-11-04T16:00:14

Hi Y'all, Ok, so we know that spirit2 collapses unused attributes. Some terminals have unused attributes. For example literals have unused attributes so as to be able to collapse attributes of expressions like: char_ >> ',' >> int_ into: tuple The collapsing rules work fine for sequences and alternatives. Most of the time, we do not care about the literals. Yet, I know at least one use case for alternatives where you want to have an attribute for the literals. Example: lit("apple") | "banana" | "cherry" but, of course, you can use a symbol table for that. But what if we had: int_ | "banana" | "cherry" I think this is clearly a case where we definitely want to know if it's an int, a "banana" or "cherry" we parsed. I think we need an attributed literal. Example: attr("banana") (can anyone suggest a better name?) I'm guessing the raw directive allows you to do the same: raw["banana"] but the intent is not the same. Anyway, so... permutations... We all know (or do you?) that spirit2 reuses the seldom (if ever) used ^ operator (I've never used it, has anyone?) for permutations. It's useful for parsing items in any order. For example: char_('a') ^ 'b' ^ 'c' allows us to parse: "a", "ab", "abc", "cba", "bca" ... etc. yeah, permutations. (aside: we also probably need strict permutations where elements need to occur at least once). Now, the same problem with unused attribute collapsing arises. There's no way to extract any usable info from the parse when some of the attributes (all attributes in this case) are suppressed. What happens? For example, the attribute of the above expression is unused_type. It starts out as: fusion::vector< optional , optional , optional > then some Spirit smarts collapses this to: fusion::vector<> then, to: unused_type based on Spirit's attribute collapsing rules. This information is useless. We can never know if we parsed "abc" or "cba" or "a" or "ab". Again, the "raw" or "attr" (as suggested above) solves this problem with the price of extra verbosity. Your thoughts? Is there a better way? Regards,

modifying symbols

Carl Barron — 2008-10-30T04:18:19

I have symbols working for adder::operator () (const std::basic_string &) remover::operator () (const std::basic_string &) working. Problem is that symbols::operator += (const std::basic_string &str) { return add(str); } does nothing have not added operator , (const std::basic_string &) to either adder or remover at this time don't see much gain with strings but as soon as I frind a solution to this forwarding problem I will make similiar additons to operator , ( const std::basic_string &), Any Ideas why symbolstable; std::basic_stringstr; table.add(str) worksl but table += str; should not work ???? ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/

Anatomy of a Spirit2x composite parser

Joel de Guzman — 2008-10-26T12:10:38

Hi, As part of my "Inside Spirit2x" installments, here's some info on how to write a Spirit composite. I'll put all installements so far in the Spirit site soon. Required reading: "Anatomy of a Spirit2x primitive parser" /////////////////////////////////////////////////////////////////////////////// Composites: /////////////////////////////////////////////////////////////////////////////// Let me present the kleene star (also in namespace spirit::qi): template struct kleene : unary_parser > { typedef Subject subject_type; template struct attribute { // Build a std::vector from the subject's attribute. Note // that build_std_vector may return unused_type if the // subject's attribute is an unused_type. typedef typename traits::build_std_vector< typename traits::attribute_of::type >::type type; }; kleene(subject_type const& subject) : subject(subject) {} template bool parse(Iterator& first, Iterator const& last , Context& context, Skipper const& skipper , Attribute& attr) const { typename traits::attribute_of::type val; while(subject.parse(first, last, context, skipper, val)) { traits::push_back(attr, val); } return true; } template void what(OStream& out, Context const& context) const { out << "kleene["; subject.what(out, context); out << ']'; } subject_type subject; }; Looks similar in form to its primitive cousin, the int_parser. It has the same basic ingredients: * The nested attribute metafunction * The parse member function * The what member function kleene is a composite parser. It is a parser that composes another parser, its "subject". It is a unary_parser and subclasses from it. Like primitive_parser, unary_parser derives from parser As mentioned, these base classes, like in classic Spirit, provide some necessary facilities for e.g. parser detection, introspection, transformation and visitation. unary_parser, has these expression requirements on Derived: Requirement: p.subject -> subject parser p: a unary parser Requirement: P::subject_type -> subject parser type P: a unary parser type /parse/ is the main parser entry point. Since this is not a primitive parser, we do not need to call qi::skip(first, last, skipper); The subject, if it is a primitive, will do the pre-skip. If if it is another composite parser, it will eventually call a primitive parser somewhere down the line which will do the pre-skip. This makes it a lot more efficient than Spirit1. Spirit1 puts the skipping business into the so-called "scanner" which blindly attempts a pre-skip everytime we increment the iterator. What is the attribute of the kleene? In general, it is a std::vector where T is the attribute of the subject. There is a special case though. If T is an unused_type, then the attribute of kleene is also unused_type. traits::build_std_vector takes care of that minor detail. So, let's parse. First, we need to provide a local attribute of for the subject: typename traits::attribute_of::type val; traits::attribute_of simply calls the subject's struct attribute nested metafunction. /val/ starts out default initialized. This val is the one well pass to the subject's parse function. The kleene repeats indefinitely while the subject parser is successful. On each successful parse, we push_back the parsed attribute to the kleen's attribute, which is expected to be, at the very least, compatible with a std::vector. In other words, although we say that we want our attribute to be a std::vector, we try to be more lenient than that. The caller of kleene's parse may pass a different attribute type. For as long as it is also a conforming STL container with push_back, we are ok. Here is the kleene loop: while(subject.parse(first, last, context, skipper, val)) { traits::push_back(attr, val); } return true; Take note that we didn't call attr.push_back(val). Instead, we called a Spirit provided function: traits::push_back(attr, val); This is a recurring pattern. The reason why we do it this way is because attr *can* be unused_type. traits::push_back takes care of that detail. The overload for unused_type is a no-op. Now, you can imagine why Spirit2 is fast! The parsers are so simple and the generated code is as efficient as a hand rolled loop. All these parser compositions and recursive parse invocations are extensively inlined by a modern C++ compiler. In the end, you get a tight loop when you use the kleene. No more. No excess baggage. If the attribute is unused, then there is no code generated for that. That's how Spirit2 is designed. The /what/ function should be obvious. It provides some information about "what" the parser is. For the kleene, we simply wrap the output of the subject in a "kleene[" ... "]". Ok, now, like the int_parser, we have to hook our parser to the spirit engine. Here's how we do it: First, we enable the prefix star operator. In proto, it's called the "dereference": template <> struct use_operator // enables *p : mpl::true_ {}; This is done in namespace boost::spirit like its friend, the use_terminal specialization for our int_parser. Obviously, we use /use_operator/ to enable the dereference for the qi::domain. Then, we need to write our generator (in namespace qi): template struct make_composite { typedef typename fusion::result_of::value_at_c::type element_type; typedef kleene result_type; result_type operator()(Elements const& elements, unused_type) const { return result_type(fusion::at_c<0>(elements)); } }; This essentially says: for all expressions of the form: *p build a kleene parser. Elements is a fusion sequence. For the kleene, which is a unary operator, expect only one element in the sequence. That element is the subject of the kleene. We still don't care about the Modifiers. We'll see how the modifiers is all about when we get to deep directives. Ok, that's it. Next time, directives! :P Regards,

Collapsing rule not applied to optional variant

Allan Odgaard — 2008-10-22T12:47:09

If I have a rule like this: rule %= -(':' >> (char_ | double_)); I would expect the attribute type to be: optional< variant > But instead it is: optional< fusion::vector< variant > > A full example is attached below which shows that this problem is not present in variations of the above. This is running spirit 2 as included with boost 1.36.0 ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________ Spirit-devel mailing list Spirit-devel< at >lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/spirit-devel

[Spirit2][Spirit2x] Rules / skippers and impact oncompilation times

Francois Barel — 2008-10-19T15:49:30

Not sure if this is the place for a little question about Spirit2 / Spirit2x, but I'll try... I thought I should at least raise the question before it's re-written :) IIUC Spirit2's implementation, for a rule object whose type has a given skipper type: - whenever an expr is assigned to that rule, - a virtual_component gets instantiated, - and two parse() worker methods always get instantiated: one for parsing with an instance of that skipper type (or of another skipper type convertible to the former), but also one for parsing with no skipper (in case that rule ever gets used in lexeme[...] context). The result is very nice from a user's pov, since the user doesn't have to worry whether a rule is used "directly" or in lexeme[...] context, both just work. However, if I look at the grammars I've written, this means almost 1/2 of those instantiations are never used: most rules (especially the high-level, more complex ones) only get used either directly or in lexeme[...] context, but not in both (only very few rules get used in both contexts) so this feels like a bit of a waste. I assume it can increase: - compilation times: the "longest" parts may be the Proto transforms and so on which are done only once before those instantiations? I don't really know... but still, it feels disturbing that instantiations for the parse() methods, and thus a good amount of parsing code generation and optimization, happen twice for every parser used from a rule with skipper, - executable size: with the virtual parse_main() mechanism, I don't think the compiler/linker is able to detect the instantiations that will never get used and simply discard them? Note that this is all just a hunch, I haven't actually measured anything... Basically I'm wondering if the resource impact of these double instantiations is known and knowingly considered negligible, or if it hasn't really been measured? (in which case if it turns out to be not so negligible, it might be interesting to find a way to say "I can guarantee this rule will only ever be called _with_ its skipper" for a given rule in Spirit2x, to prevent instantiation of its no-skipper parse() method) Thanks, François ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/

Anatomy of a Spirit2x primitive parser

Joel de Guzman — 2008-10-18T23:40:56

Hi, Let me present an exemplar for a Spirit2x primitive, the int_parser in the boost::spirit::qi namespace: template < typename T , unsigned Radix = 10 , unsigned MinDigits = 1 , int MaxDigits = -1> struct int_parser : primitive_parser > { // check template parameter 'Radix' for validity BOOST_MPL_ASSERT_MSG( Radix == 2 || Radix == 8 || Radix == 10 || Radix == 16, not_supported_radix, ()); template struct attribute { typedef T type; }; template bool parse(Iterator& first, Iterator const& last , Context& /*context*/, Skipper const& skipper , Attribute& attr) const { qi::skip(first, last, skipper); return extract_int ::call(first, last, attr); } template void what(OStream& out, Context const& /*ctx*/) const { out << "integer"; } }; It inherits from primitive_parser using the CRTP technique. primitive_parser derives from parser. These base classes, like in classic Spirit, provides some necessary facilities for e.g. parser detection, introspection, transformation and visitation. parser, the most base class, has these expression requirements on Derived: Requirement: p.parse(f, l, ctx, skip, attr) -> bool p: a parser f, l: first/last iterator pair ctx: enclosing rule context (can be unused_type) skip: skipper (can be unused_type) attr: attribute (can be unused_type) Requirement: p.what(out, ctx) -> void p: a parser ctx: enclosing rule context (can be unused_type) Requirement: P::template attribute::type P: a parser type Ctx: A context type (can be unused_type) int_parser fulfills these requirements. /parse/ is the main parser entry point. For primitive parsers, our first thing to do is call: qi::skip(first, last, skipper); to do a pre-skip. Note that /skipper/ can be an unused_type. It's a type used everyweher in Spirit2 to signify "don't-care". There is an overload for /skip/ for unused_type that is simply a no-op. That way, we do not have to write multiple parse functions for phrase and character level parsing. After pre-skipping, the parser proceeds to do its thing. The actual parsing code is placed in extract_int::call(first, last, attr); Here are the basic rules for parsing: * The parser returns `true` if successful, `false` otherwise. * If successful, `first` is incremented N number of times, where N is the number of characters parsed. N can be zero --an empty (epsilon) match. * If successful, the parsed attribute is assigned to /attr/ * If unsuccessful, `first` is reset to its position before entering the parser function. /attr/ is untouched. This simple no-frills protocol is one of the reasons why Spirit2 is fast. If you know the internals of Spirit1 (classic) and perhaps even wrote some parsers with it, this simple Spirit2 mechanism is a joy to work with. There are no scanners and all that crap. The /what/ function should be obvious. It provides some information about "what" the parser is. It is used as a debugging aid, for example. The /attribute/ metafunction returns the expected attribute type of the parser. In some cases, this is context dependent. In our case, it is simply the T template parameter, usually an int. So, that's basically it. Now, we have to associate the int_parser to some placeholders: short_, int_, long_ and long_long First, we enable these placeholders in namespace boost::spirit: template <> struct use_terminal // enables short_ : mpl::true_ {}; template <> struct use_terminal // enables int_ : mpl::true_ {}; template <> struct use_terminal // enables long_ : mpl::true_ {}; template <> struct use_terminal // enables long_long : mpl::true_ {}; Notice that int_parser is placed in the namespace boost::spirit::qi while these /enablers/ are in namespace boost::spirit. The reason is that these placeholders are shared by other spirit /domains/. Qi, the parser is one domain. Karma, the generator is another domain. Other parser technologies may be developed and placed in yet another domain. Yet, all these can potentially share the same placeholders for interoperability. The interpretation of these placeholders is domain-specific. Now that we enabled the placeholders, we have to write generators for them. The make_xxx stuff (in boost::spirit::qi namespace): template struct make_int { typedef int_parser result_type; result_type operator()(unused_type, unused_type) const { return result_type(); } }; This one above is our main generator. It's a simple function object with 2 (unused) arguments. These arguments are 1) The actual terminal value obtained by proto. In this case, either a short_, int_, long_ or long_long. We don;t care about this. 2) Modifiers. We also don't care about this. This allows directives such as no_case[p] to pass information to inner parser nodes. We'll see how that works later. Now: template struct make_primitive : make_int {}; template struct make_primitive : make_int {}; template struct make_primitive : make_int {}; template struct make_primitive : make_int {}; These, specializes qi:make_primitive for specific tags. They all inherit from make_int which does the actual work. Ok, that's it for this installment... You can see the code in Spirit's SVN at: https://spirit.svn.sourceforge.net/svnroot/spirit/trunk/Spirit2x I encourage you to peek into the code, peruse it, play with it. I also encougarage feedback and comments. The installments I am providing here shall form the basis of the "inside" docs. I'd love to have it reviewed incrementally as we go. Regards,

Spirit2x in Spirit SVN

Joel de Guzman — 2008-10-17T14:20:30

Hi, To those interested, FYI, I have the Spirit2x code in the Spirit SourceForge SVN at: https://spirit.svn.sourceforge.net/svnroot/spirit/trunk/Spirit2x This is not the Boost SVN. Since this is a total rewrite, I do not want to touch Spirit2 in the Boost SVN. What do we have there? Well, more or less the same code that I posted a few days ago but structured into modules and directories like Spirit2. For those who want to keep up with the development, are interseted in contributing somehow in the future, or simply want to see how a library like Spirit is being developed, this is it. I'll try to provide tech information as it's being developed. This log might be useful in the future to document the design and evolution of the library. To those who really, really want to keep abreast, the basic requirement is knowledge of key Boost libraries, most importantly: 1) MPL 2) Fusion 3) Proto 4) Phoenix Oh, did you know that Phoenix is now a full fledged Boost library? Cheers!!!,

Spirit2x development

Joel de Guzman — 2008-10-16T02:00:14

Hi, (posting to both Spirit general and Spirit devel. I'd like to invite those interested in Spirit-general to join in the development, the discussions, etc. I intend to have another round of dev-talk-and-coding. All these will happen in the spirit-devel list.) For some time now, we had Spirit2 in beta. The library is just fine. The generated code is trim and fit and fast! The interface is nice and intuitive, and IMO, is quite elegant. Like it's predecessor, classic Spirit, Spirit2 is again breaking new grounds on what you can do with template metaprogramming and expresion templates. It is quite stable, despite its beta status, perhaps in large, due to its undrelying infrastructure: MPL, Fusion and Proto (and a couple more boost libraries here and there). We've focused on optimizing runtime. And indeed, as shown by Overmind's recent tests, and by some of my (and Hartmut's) original tests, Spirit2 is fast. I'm planning to have more tests done, surely. We've also focused on usability and perfected the interface you see now. There are however, some deficiencies that we want to address... 1) Compile time is a problem. No effort was invested in that area (yet). 2) Documentation is sparse. 3) The extension interface is poor. You need to be a mad scientist of some sort in order to write the simplest parser (e.g. char_). While working on 2 (docs), I started inverstigating on 1 and 3. There are two ways, I can see, to address the compile time issue. 1) Improve the Proto-Spirit linkup. I've seen Eric's work on Phoenix3. I'd like to use some of the techniques he used there (e.g. use proto::switch_). 2) Improve Fusion. There has been some minimal effort to make fusion go faster at compile time. More work needs to be done. Working on 1, I think, should also address the extension interface. So, I came up with an experimental prototype with an improved proto linkup. See attached. The cpp file is divided into 1) Generic spirit code (for qi and karma, etc.) 2) The meta-compiler (proto). You don't need to understand this part to write your own parser. 3) Qi hooks and code (doesn't do any parsing yet, but shows how one can setup a parser struct (ala spirit1) and hook it up with the extension mechanism without having to fiddle with proto. Basically, you have 1) a struct that defines the parser and 2) a make_xxx specialization for hooking the parser into the engine. 3) a use_xxx specialization to enable terminals (primitives) and operators. Here's how to write the any_char parser, for example: template <> struct use_terminal // enables char_ : mpl::true_ {}; namespace qi { struct any_char_parser { // parse code goes in here (just like in spirit classic) }; template <> struct make_primitive { typedef any_char_parser result_type; any_char_parser operator()(tag::char_) const { return any_char_parser(); } }; } 4) Driver code that shows the generated (meta-compiled) struct compositions. I am posting this code in the hope that we can persuade more developers to actively participate in the development, much like the Spirit1 times when we had a community of developers actively involved. I'm guessing that Spirit2 has become too complex, internally, in structure and code, compared to the original Spirit 1.0's seven header file library that spurred a lot of active development. Now, I intend to make Spirit2 as simple as possible, easily understandable and easily extensible. This is a good chance to see something fresh and simple evolve into a powerful library. Don't miss the action! ;-) See you in the Spirit-devel list. https://lists.sourceforge.net/lists/listinfo/spirit-devel Regards,

Using position_iterator with mixed newlines

Daniel James — 2008-10-08T20:00:47

Hello, Because of the recent problems with quickbook and windows newlines, I've been having a look at how position_iterator in classic spirit deals with windows newlines and it has some problems. position_iterator treats as a single newline '\r', '\n', '\r\n' and '\n\r' - the last of which isn't generally treated as a single newline, and isn't by classic spirit elsewhere. Also, in the case of '\r\n' and '\n\r' it skips over the second character, which means that it transforms '\r\n' to '\r' and '\n\r' to '\n'. This leads to interesting situation with '\r\n\n', this gets transformed to '\r\n' which spirit treats as a single newline, but position_iterator will have counted two lines. Here's a demonstration that spirit with and without position_iterator acts differently: #include #include #include using namespace boost::spirit::classic; template bool position_parse(char const* str, Grammar grammar) { char const* end = str; while(*end) ++end; position_iterator pos_begin(str, end); position_iterator pos_end; return parse(pos_begin, pos_end, grammar).full; } int main() { assert(parse("\n\n", (eol_p >> eol_p)).full); assert(position_parse("\n\n", (eol_p >> eol_p))); assert(parse("\r\n\n", (eol_p >> eol_p)).full); assert(position_parse("\r\n\n", (eol_p))); assert(parse("\n\r\r", (eol_p >> eol_p >> eol_p)).full); assert(position_parse("\n\r\r", (eol_p >> eol_p))); } So, I think position_iterator should be changed so that it doesn't treat '\n\r' as a single newline, and doesn't transform the sequence at all. But this is a breaking change, and I guess you might not be keen on them for classic spirit. So a slight change to the behaviour, that I think will pass the current unit tests, might be to change it so that it transforms '\r\n' to '\n' (i.e. it skips that '\r' not the '\n'), although that will be more expensive than the current iterator as the iterator would have to store some extra state. I'll create a patch for anything that's considered acceptable. Daniel ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/

Minor BOOST_SPIRIT_RULE fixes

Vygintas Daugmaudis — 2008-09-15T14:49:14

Hello, I tried to recompile my project with g++ -pedantic-errors option, and then I encountered some problems with BOOST_SPIRIT_RULE macro (the compiler complained about extra semicolons). The attached patch fixes the problem. Cheers, Vygintas ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/_______________________________________________ Spirit-devel mailing list Spirit-devel< at >lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/spirit-devel

[proto v4]compile time equation soln prototype fails to compile with "when" map for "variables"

Larry Evans — 2008-05-14T18:23:22

In the vault at: http://www.boost-consulting.com/vault/index.php?action=downloadfile&filename=pass_thru_tag_xform.cpp&directory=Strings%20-%20Text%20Processing&PHPSESSID=ab51206c9d980155d142f5bcef8e00ee there's a partial prototype for compile time solving a system of equations. The partial prototype is partial because it just tries to simplify the rhs of the equation. The domain is calculating the empty attributes of a set of grammar productions (i.e. the equations). The typedef ... xfrm_empty_varbl_map0 is supposed to implement a substitution operation which substitutes a literal value for a variable. In this particular case, the variable is empty::tag_varbl and the value to be substituted is: empty::tag_literal As indicated in the comments following: //NOTE:varbl_empty_map_bug: it fails to reduce the rhs to a terminal expression of type tag_literal at the valu_empty_not_leaves statement. However, this only happens when the xfrm_empty_varbl_map0 is included as the 1st term in xfrm_reduce_empty_n. Does anybody have any ideas of why that happens? TIA. -regards Larry ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Proto v4

Eric Niebler — 2008-04-08T00:43:33

No library ever made it through a review unscathed and proto is no exception. There's a new version of proto in branches/proto/v4, and Spirit-2 will need some changes to work with it. This is the version that will be merged into trunk, eventually. First the simple things: * Proto lives at boost/proto/, not boost/xpressive/proto * boost/proto/proto.hpp now includes all of proto with the exception of the typeof registrations. That includes the contexts, the transforms and the debugging utilities. If you just want the core of proto without the other stuff, there is boost/proto/core.hpp * s/posit/unary_plus/ * s/arg/child, s/arg_c/child_c/, s/_argN/_childN * s/bind/lazy/ * s/_visitor/_data/ * The proto::transform namespace is no more. Some bigger changes: The protocol for defining a primitive transform has changed. Previously, primitive transforms were just ternary function objects like this: struct MyTransform : proto::callable { template struct result; template struct result { typedef ... type; }; template typename result::type operator()(Expr const &expr, State const &state, Data &data) const { return ...; } }; You would now write this as: struct MyTransform : proto::transform { template struct impl : proto::transform_impl { typedef ... result_type; result_type operator()( typename impl::expr_param expr , typename impl::state_param state , typename impl::data_param data ) const { return ...; } }; }; With proto v4, the Expr, State, and Data parameters may be reference types. With v3 they could not. In a related change, when using result_of to calculate the return type of a transform, you need to think about the constness and lvalue-ness of the arguments you pass. So instead of: result_of::type you might have to say result_of::type Finally, the behavior of result_of::child_c (formerly known as result_of::arg_c) has changed. It is now sensitive to const and reference. Old ------------------------------------------ 1) result_of::arg_c::type 2) result_of::arg_c::reference 3) result_of::arg_c::const_reference New ------------------------------------------ 1) result_of::child_c::type 2) result_of::child_c::type 3) result_of::child_c::type These are some internal changes that probably should affect anybody: * ref_ is no more. Children are held by plain reference, not by reference wrapper. * s/args0/term, s/argsN/listN/ That's it, I think. The result is a *much* more expressive, powerful and optimal system of transforms.

skipper rules silently ignored (spirit2)

Cédric Venet — 2008-02-24T17:16:18

Hi, Using the latest svn version of spirit2, I have had a small problem with the skipper rule. Since I wanted to use a rule more complexe than space (and that I didn't have its type): charset::space | ('%' >> *~spirit::char_('\n') >> '\n') I defined my grammar like this: template struct my_grammar : qi::grammar_def > {...}; but doing: bool r = phrase_parse(iter, storage.end(), make_parser(def), charset::space | ('%' >> *~spirit::char_('\n') >> '\n')); ignore the skipping rule in qi/nonterminal/rule.hpp:170 template bool parse( Iterator_& first, Iterator_ const& last , Context& context, Skipper const& skipper) const { // in this function, the skipper is ok // but this call loose it (select the unused_type for the skipper overload of the function) return ptr->parse(first, last, context, skipper); } Finally, I did: qi::rule skipper; skipper = charset::space | ('%' >> *~spirit::char_('\n') >> '\n'); bool r = phrase_parse(iter, storage.end(), make_parser(def), skipper ); which work. It would be great to either make work the first or generate a compile time error, lost quite some time thinking the problem was in my grammar. Regards,

proto breaking changes (and patch for spirit2)

Eric Niebler — 2008-01-12T06:48:31

I've committed to boost svn head a new version of proto with improved transforms. This is a breaking change. There isn't documentation yet for the new transforms, so I made a patch that gets Spirit-2 working again. (At least, it works with msvc, but there appear to be other problems with spirit-2 and gcc which are unrelated.) Patch attached. Cheers,