gmane.comp.parsers.antlr.devel

What is the status of options parser=precedence?

Maciej Pilichowski — 2012-08-10T17:16:53

Hello all,

  I was pretty happy when I've read about precedence option in ANTLR 
3.2:

http://www.antlr.org/wiki/display/ANTLR3/Operator+precedence+parser

however despite now we have 3.4 as stable one, this option is not 
present:

http://www.antlr.org/wiki/display/ANTLR3/Grammar+options

  So what is the status of this option?



  Sidenote: I am asking, because Yacc-like grammar is dead simple when 
dealing with precedence, the most common solution in ANTLR is 
introducing several levels of rules -- not straightforward at all -- 
and writing AST from those rules requires a lot of typing. Clarity of 
the rules suffers, and so productivity of language developer.

Thank you in advance,

ANTLR3.4: maven tests failure onorg.antlr.test.TestSemanticPredicates

Francis ANDRE — 2012-08-09T15:25:37

Hi

Building antlr 3.4 from the latest version of the Git repository 
git://github.com/antlr/antlr3.git, one gets 3 failures on the test 
org.antlr.test.TestSemanticPredicates.

Francis

Results :

Failed tests:
testGatedPredDoesNotForceAllToBeGated2(org.antlr.test.TestSemanticPredicates): 
expected:<.s0-B->.s1
   testORGatedPred(org.antlr.test.TestSemanticPredicates): 
expected:<.s0-B->.s1
testSynPredWithSemPredReferencingLabeledRule(org.antlr.test.TestSyntacticPredicateEvaluation): 
expected:<[alt1

Tests run: 977, Failures: 3, Errors: 0, Skipped: 1

[INFO] 
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] ANTLR Master build control POM .................... SUCCESS [2.271s]
[INFO] Antlr Runtime 3.4.1-SNAPSHOT ...................... SUCCESS [4.107s]
[INFO] ANTLR Tool 3.4.1-SNAPSHOT ......................... FAILURE 
[3:06.185s]
[INFO] Maven plugin for ANTLR V3 ......................... SKIPPED
[INFO] ANTLR gUnit ..................................

ANTLR 3.4: missing parameter in the method generated for a syntactic predicate for a rule with a parameter

Francis ANDRE — 2012-07-30T06:41:40

Hi

The generation of the method for the syntactic predicate of a rule with 
a parameter is missing the parameter in its signature, which leads to a 
compile error.

Rgds

Francis

Here the CMinus grammar modified with the variable rule having a 
parameter and conditionned by a syntactic predicate:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
program
scope {
   List globals;
   List functions;
}
< at >init {
   $program::globals = new ArrayList();
   $program::functions = new ArrayList();
}
     : _declaration[2]_+
         -> 
program(globals={$program::globals},functions={$program::functions})
     ;

declaration[int i]
     : _(variable[i])=>variable[i]_ {$program::globals.add($variable.st);}
     |   f=function {$program::functions.add($f.st);}
     ;


_variable[int i]_
     :   type declarator ';'
         -> {$function.size()>0 && $function::name==null}?
            globalVariable(ty

antlr 3.4: command line option -nda leads to IllegalArgumentException: No such directory: org/antlr/tool/templates/dot/dfa

Francis ANDRE — 2012-07-28T13:57:45

Hi

Running antlr 3.4 with a -nfa command line argument produces this exception:

Buildfile: C:\.....\cmp\antlr3\build.xml
antlr:
[antlr:ant-antlr3] Failed to change file modification time
[antlr:ant-antlr3] ANTLR Parser Generator  Version 3.4
[antlr:ant-antlr3] C:\...\cmp\antlr3\ProgramBaseParser.g
[antlr:ant-antlr3] Exception in thread "main" 
java.lang.ExceptionInInitializerError
[antlr:ant-antlr3]     at org.antlr.Tool.generateNFAs(Tool.java:694)
[antlr:ant-antlr3]     at org.antlr.Tool.generateRecognizer(Tool.java:651)
[antlr:ant-antlr3]     at org.antlr.Tool.process(Tool.java:468)
[antlr:ant-antlr3]     at org.antlr.Tool.main(Tool.java:93)
[antlr:ant-antlr3] Caused by: java.lang.IllegalArgumentException: No 
such directory: org/antlr/tool/templates/dot/dfa
[antlr:ant-antlr3]     at 
org.stringtemplate.v4.STGroupDir.<init>(STGroupDir.java:73)
[antlr:ant-antlr3]     at 
org.stringtemplate.v4.STGroupDir.<init>(STGroupDir.java:48)
[antlr:ant-antlr3]     at 
org.antlr.tool.DOTGenerator.<clinit>(DOTGenerator

Rewriting a rule into a nested tree

Nic Wolfe — 2012-06-26T21:34:33

Hey all,

I'm fairly new to ANTLR and grammars/parsing in general so apologies if
this question doesn't make a whole lot of sense. I'll try to explain what
I'm trying to achieve as best I can.

I'm trying to make a rule that will rewrite into a nested tree (similar to
a binary tree).

For example:

a + b + c + d;

Would parse to a tree like:

  a
  +
  b
      +
      c
           +
           d

Or typed another way: ( ( (a + b) + c) + d). Basically each root node would
have three children (LHS '+' RHS) where LHS could be more nested nodes.

My first attempt was something like:

rule: lhs '+' ID;
lhs: ID | rule;

(with some tree rewrites) which gives me an error about it being
left-recursive. I'm not sure how to solve this without some type of
recursion.

How can I achieve this? Any input is appreciated.

Thanks,

Nic

Re: Back after a long absence :)

Terence Parr — 2012-06-21T23:16:03

On Jun 20, 2012, at 8:47 AM, Matt Benson wrote:


yep. it will be there. also grab parrt/antlr4 not antlr/antlr4 as it is the most up-to-date.


Yeah, we may have to get to this as we go to other targets.  Note that the graph simulation to get parsing and lexing to work for adaptive LL(*) are extremely complicated and need to be very efficiently implemented.


:)

Ter

Back after a long absence :)

Matt Benson — 2012-06-20T15:47:18

Hi Ter, all!

  I never did get fully functional with ANTLR 3, and returning after a
long hiatus I find ANTLR 4 on the horizon.  :)  How time flies.  But
getting down to business, I've been looking at ANTLR 4 for a personal
project I'm wanting to do, have cloned it on github, and am playing
around with it.  Some observations:

* Seemingly the codegen in v4's current code is driven by a "language"
option in the grammar.  Notwithstanding the fact that I haven't yet
figured out precisely where to set this if I want to toy with a new
codegen target, if one of v4's goals is to decouple a given grammar
from its targets, wouldn't it be more to the point to add a Tool
option to specify the desired output language?

* Taking further some of the extensibility requests I came across in
the ML archives, what about the idea of abstracting the notion of a
Grammar processor, e.g.:

public interface GrammarProcessor {
  void process(Grammar g);
}

and

public class CodeGenPipeline implements GrammarProcessor {
  public void

Re: What alternatives to this left recursiverequirement???

Arun N Kumar — 2012-06-18T04:31:49

Hello

Thanks for the reply.

My requirement wouldnt allow for an unwrap at the grammar. Yes all leaves
wud be 'simple' but complex dont mean simple. Meaning

Complex : expr ( join expr)+

Helps in

Complex1 and complex2
Or
Simple1 or complex1
Etc

Thanks n Regards
Arun
On Jun 14, 2012 6:15 PM, "John B. Brodie" <jbb-HInyCGIudOg< at >public.gmane.org> wrote:

Re: Including whitespace in AST

Jim Idle — 2012-06-15T05:07:51

Each tree node will tell you the starting and ending token (the tokens that
get what the node represents). Hidden tokens (rather than skipped tokens)
are still within the span, but the parser does not see them. So, if you
want the whitespace, you will need to access the tokens directly using code
at the point in the tree walk where you need to see it.



Jim



*From:* antlr-dev-bounces-ErkRXerACLvYtjvyW6yDsg< at >public.gmane.org [mailto:antlr-dev-bounces-ErkRXerACLvYtjvyW6yDsg< at >public.gmane.org] *On
Behalf Of *Mike Lischke
*Sent:* Thursday, June 14, 2012 7:47 PM
*To:* ANTLR Mailing Mailing List
*Subject:* [antlr-dev] Including whitespace in AST



Hi,



I couldn't find an answer to this in the archive, so here we go. What's
needed to have the parser include the hidden channel when constructing the
AST? The white spaces there should remain hidden wrt. parsing, but I would
like to get the tokens in my resulting AST. Is this possible or do I have
to make white spaces not hidden and handle them manually everywhere?

Re: What alternatives to this left recursiverequirement???

John B. Brodie — 2012-06-14T12:45:43

Greetings!

On 06/14/2012 04:44 AM, Arun N Kumar wrote:

first note that your `complex` rule can be simplified to:

complex : expr ( join expr )+ ;

we can easily see that the base case for the recursion is the `simple` rule.
and so no matter how one parses (an ambiguous) sequence of `join`s one 
always ends up with:

complex : simple ( join simple )+ ;

of course, if the `join` rule contains `ID` and/or `operator` rule 
references ambiguity will probably still be present.

and, if you feel like it, we can now remove the `complex` rule entirely  
(assuming no other usage) via:

expr : simple | simple ( join simple )+ ;

written better as:

expr : simple ( join simple )* ;
simple : ID operator ID ;

Hope this helps...
    -jbb

Including whitespace in AST

Mike Lischke — 2012-06-14T11:47:11

Hi,

I couldn't find an answer to this in the archive, so here we go. What's needed to have the parser include the hidden channel when constructing the AST? The white spaces there should remain hidden wrt. parsing, but I would like to get the tokens in my resulting AST. Is this possible or do I have to make white spaces not hidden and handle them manually everywhere?

Thanks,

Mike

What alternatives to this left recursive requirement???

Arun N Kumar — 2012-06-14T08:44:02

Hi All,

I have a recursive rule, would like to know what can be done to make it
behave !

expr : ( simple | complex ) ;

simple : ( ID operator ID ) ;
complex : ( expr join expr (join expr)* ) ;

Cheers!
Arun

Recursive rule with warning

Mike Lischke — 2012-05-25T15:40:34

Hi list,

I have a rule that produces a warning, which I would like to get rid of. However, it seems I can't get further without some help, so I would much appreciate any hint to solve this. The actual grammar is of course more complex but I condensed the problem to this:

a: b (d1 (e)? | d2 a e)*;
b: 'b';
d1: 'd1';
d2: 'd2';
e: 'e';

The problem is the recursion of rule a which may end in rule e, but via two alternatives (obviously). I was thinking about extracting the (e)? part  somehow like this:

a: b ((d1 | d2 a e) (e)?)*

but this doesn't feel right. Any better idea?

Mike

Re: Forcing a lexer rule

Mike Lischke — 2012-05-23T07:42:26

Hi Sam,


Thanks for the idea. I added a new lexer rule like this:

ANSI_QUOTED_ID_OPEN:
DOUBLE_QUOTE 
(
ESCAPE_SEQUENCE
| ~(DOUBLE_QUOTE | '\\')
)*
;

and my identifier parser rule is now:

identifier:
IDENTIFIER
| BACK_TICK_QUOTED_ID
| ANSI_QUOTED_ID
| ANSI_QUOTED_ID_OPEN
;

but unfortunately this works only for very few cases.

select 1 from "gives no tree at all (just an error node) + error (extraneous input at "from")
select 1 from "aworks almost as I want it (except I don't get an error)
select 1 from "a""AST + error saying everything starting with "from" is extraneous input
select 1 from "a" "is the only case which really does what I need (AST + open id + error)

In case this matters: I have backtracking active for the entire grammar (I'll work later to get rid of this but for now I need it).

Mike

Re: Forcing a lexer rule

Sam Harwell — 2012-05-22T12:56:25

Hi Mike,

 

The best way to handle this is add a lexer rule ANSI_QUOTED_ID_OPEN (or
similar) which matches the quoted string when it's missing the final
character.

 

--

Sam Harwell

Owner, Lead Developer

Description: Description: C:\Users\sam\Documents\Work\TVL\tvl_logo_small.png

 <http://tunnelvisionlabs.com/> http://tunnelvisionlabs.com

 

From: Mike Lischke [mailto:mike-gIurkOZOMMMEZ6m2XrtfILNAH6kLmebB< at >public.gmane.org] 
Sent: Tuesday, May 22, 2012 7:43 AM
To: ANTLR Mailing Mailing List
Subject: [antlr-dev] Forcing a lexer rule

 

Hi all,

 

for code completion I need a way to force a lexer to match somehow even if
it is not complete. Sounds weird, I know. What I have in mind is this:
Imagine you have defined special identifiers which can be quoted (double
quote char). When the user has started typing something like this:

 

select "abc

 

and now invokes code completion I want to show all possible identifiers. But
with a lexer rule like this:

 

ANSI_QUOTED_ID:

            DOUBLE_QUOTE

Forcing a lexer rule

Mike Lischke — 2012-05-22T12:42:36

Hi all,

for code completion I need a way to force a lexer to match somehow even if it is not complete. Sounds weird, I know. What I have in mind is this: Imagine you have defined special identifiers which can be quoted (double quote char). When the user has started typing something like this:

select "abc

and now invokes code completion I want to show all possible identifiers. But with a lexer rule like this:

ANSI_QUOTED_ID:
DOUBLE_QUOTE 
( options {greedy=false;} :
ESCAPE_SEQUENCE
| .
)* DOUBLE_QUOTE
;

I only get an error node plus a not so helpful error message:

1 errors found
-end of input-(1)  : error 3 : , at offset 0, at <EOF> : cannot match to any predicted input...

The AST looks like:

(1, 0, nil)    nil
(1, 0, SELECT_SYM55 [418])    select
(0, 0, <invalid> [0])    Tree Error Node
(1, 0, (null) [-1])    <EOF>

However, what I would like is to get ANSI_QUOTED_ID to match the entire rest of the input and return this as the ID (plus the error, so I know it was incorrect). This way I kno

help

김영수 — 2012-05-21T08:04:15

How could a new contributor get started? (cleanerresend)

George S. Cowan — 2012-05-18T22:20:57

I would like to contribute to ANTLR 3.4 but the Maven build still fails (I
was using antlr-antlr3-release-3.4-150-g8312471.zip); see message 
  http://www.antlr.org/pipermail/antlr-dev/2012-February/002719.html

I'm not really interested in becoming a Maven expert. Is there some other
way that I can help?

George

How could a new contributor get started?

George S. Cowan — 2012-05-18T21:43:40

I would like to contribute to ANTLR 3.4 but the Maven build still fails (I
was using antlr-antlr3-release-3.4-150-g8312471.zip). In addition, the tests
failed for the previous committed code. I'm not really interested in
becoming a Maven expert. Is there some other way that I can help?
 
George
 
 
Terry Parr said:
 
wiped .m2 antlr files and tried again:
 
Downloading:
<http://antlr.org/antlr-snapshot/org/antlr/ST4/4.0.4/ST4-4.0.4.jar>
http://antlr.org/antlr-snapshot/org/antlr/ST4/4.0.4/ST4-4.0.4.jar
[INFO] Unable to find resource 'org.antlr:ST4:jar:4.0.4' in repository
antlr-snapshot ( <http://antlr.org/antlr-snapshot>
http://antlr.org/antlr-snapshot)
Downloading:
<http://repo1.maven.org/maven2/org/antlr/ST4/4.0.4/ST4-4.0.4.jar>
http://repo1.maven.org/maven2/org/antlr/ST4/4.0.4/ST4-4.0.4.jar
 
but still 
 
[INFO] org/stringtemplate/v4/STGroup
org.stringtemplate.v4.STGroup
[INFO]
------------------------------------------------------------------------
 
but for gunit.  maybe it doesn't have the appropriate

Re: Error while building ANTLR C runtime from source onZRHEL/ZSUSE

Jim Idle — 2012-05-08T18:28:45

Google: http://blog.zortrium.net/archives/date/2010/01


Though the description of manually configuring the runtime here will
probably work for you, I think that this probably not the 'correct'
solution. I suspect that what is really wrong is that either you have the
wrong version of configure related tools, have not looked at the
./configure --help (where you can turn off the CPU 32/64 but flag for
instance (and the author did not spot that), or that there is a version of
automake that would work for OS390 based Linux.

Generally it seems like the headers are not installed, perhaps not
correctly, or don't exist in standard places, hence configure cannot find
them. Post the output of configure if you cannot work it out after
checking that you have the headers installed correctly.



Jim

Error while building ANTLR C runtime from source onZRHEL/ZSUSE

Vinit S Metange — 2012-05-08T11:36:24

Hello,

I am getting following error while building the ANTLR C runtime from source
on ZRHEL and ZSUSE machine:

[nodcosta< at >lnxec439 libantlr3c-3.4]$ gmake 2>&1 |tee
$HOME/vinit/ANTLR/antlr34_ZRHEL_gmake_output_32bit.txt
gmake  all-am
gmake[1]: Entering directory `/home/nodcosta/vinit/ANTLR/libantlr3c-3.4'
/bin/sh ./libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I.
-Iinclude    -m31  -O2  -Wall -D_SUSE -D_INT64_T -D_GNU_SOURCE -fPIC -MT
antlr3baserecognizer.lo -MD -MP -MF .deps/antlr3baserecognizer.Tpo -c -o
antlr3baserecognizer.lo `test -f 'src/antlr3baserecognizer.c' || echo
'./'`src/antlr3baserecognizer.c
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -Iinclude -m31 -O2 -Wall -D_SUSE
-D_INT64_T -D_GNU_SOURCE -fPIC -MT antlr3baserecognizer.lo -MD -MP
-MF .deps/antlr3baserecognizer.Tpo -c src/antlr3baserecognizer.c  -fPIC
-DPIC -o .libs/antlr3baserecognizer.o
In file included from /usr/include/stdio.h:34,
                 from include/antlr3defs.h:248,
                 from include/antlr3baserecognizer