http://blog.gmane.org/gmane.comp.parsers.sparse hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.parsers.sparse/2861 <pre> That is a great tip. Thanks. Chris -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Christopher Li 2012-05-23T18:30:31 http://permalink.gmane.org/gmane.comp.parsers.sparse/2860 <pre>Hi, Replying to my own post (for future reference): --- On Tue, Mar 20, 2012 at 08:55:48PM +0530, Shakthi Kannan wrote: | But, is it really required to generate eth.o? Is it possible to just | parse through the sources, produce the .xml output without having to | generate the object files? \-- I had to use: MAKEFLAGS+="--dry-run" make C=2 CHECK="c2xml &gt; foo.xml" net/ethernet/eth.o &gt; output.txt eval `tail -2 output.txt | head -1` SK </pre> Shakthi Kannan 2012-05-23T09:59:03 http://permalink.gmane.org/gmane.comp.parsers.sparse/2859 <pre> substitute_arguments() is because I am not allowed extra token TOKEN_M_EMPTY. it is unrelated to the upper cases... I'm not shure who is wrong here or weather maybe my example is not general enought, maybe you are right... Kind-of:-) http://gcc.gnu.org/ml/gcc/2012-03/msg00208.html It disapears in the gcc list noise however... I dump everything, however I use a perl script to reconstruct... -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Konrad Eisele 2012-05-15T13:03:49 http://permalink.gmane.org/gmane.comp.parsers.sparse/2858 <pre>Hi, I am resending the patch with the reference to Linus' email added to Documentation/sparse.txt. Thanks for all your feedback and replies. Signed-off-by: Shakthi Kannan &lt;shakthimaan&lt; at &gt;gmail.com&gt; --- Documentation/sparse.txt | 45 +++++++++++++++++++++++++++++++++++++++++++++ sparse.1 | 14 +++++++++++++- 2 files changed, 58 insertions(+), 1 deletions(-) create mode 100644 Documentation/sparse.txt diff --git a/Documentation/sparse.txt b/Documentation/sparse.txt new file mode 100644 index 0000000..1ee3b39 --- /dev/null +++ b/Documentation/sparse.txt &lt; at &gt;&lt; at &gt; -0,0 +1,45 &lt; at &gt;&lt; at &gt; +Sparse +~~~~~~ + +__nocast vs __bitwise: + +__nocast warns about explicit or implicit casting to different types. + +HOWEVER, it doesn't consider two 32-bit integers to be different +types, so a __nocast 'int' type may be returned as a regular 'int' +type and then the __nocast is lost. + +So "__nocast" on integer types is usually not that powerful. It just +gets lost too easily. It's more useful for things like pointers. It +also doesn't warn about the mixing: you can add integers to __nocast +integer types, and it's not really considered anything wrong. + +__bitwise ends up being a "stronger integer separation". That one +doesn't allow you to mix with non-bitwise integers, so now it's much +harder to lose the type by mistake. + +So the basic rule is: + + - "__nocast" on its own tends to be more useful for *big* integers +that still need to act like integers, but you want to make it much +less likely that they get truncated by mistake. So a 64-bit integer +that you don't want to mistakenly/silently be returned as "int", for +example. But they mix well with random integer types, so you can add +to them etc without using anything special. However, that mixing also +means that the __nocast really gets lost fairly easily. + + - "__bitwise" is for *unique types* that cannot be mixed with other +types, and that you'd never want to just use as a random integer (the +integer 0 is special, though, and gets silently accepted iirc - it's +kind of like "NULL" for pointers). So "gfp_t" or the "safe endianness" +types would be __bitwise: you can only operate on them by doing +specific operations that know about *that* particular type. + +Generally, you want __bitwise if you are looking for type safety. +"__nocast" really is pretty weak. + +Reference: + +* Linus' e-mail about __nocast vs __bitwise: + + http://article.gmane.org/gmane.linux.kernel.mm/75784 diff --git a/sparse.1 b/sparse.1 index bde6b6d..ae85b54 100644 --- a/sparse.1 +++ b/sparse.1 &lt; at &gt;&lt; at &gt; -53,7 +53,19 &lt; at &gt;&lt; at &gt; arithmetic operations other than bitwise operations, and on any conversion of one restricted type into another, except via a cast that includes \fB__attribute__((force))\fR. -Sparse does not issue these warnings by default. +__bitwise ends up being a "stronger integer separation". That one +doesn't allow you to mix with non-bitwise integers, so now it's much +harder to lose the type by mistake. + +__bitwise is for *unique types* that cannot be mixed with other +types, and that you'd never want to just use as a random integer (the +integer 0 is special, though, and gets silently accepted iirc - it's +kind of like "NULL" for pointers). So "gfp_t" or the "safe endianness" +types would be __bitwise: you can only operate on them by doing +specific operations that know about *that* particular type. + +Generally, you want bitwise if you are looking for type safety. Sparse +does not issue these warnings by default. . .TP .B \-Wcast\-to\-as </pre> Shakthi Kannan 2012-05-15T10:06:36 http://permalink.gmane.org/gmane.comp.parsers.sparse/2857 <pre> OK, I mistaken that as apply request. Never mind some of the coding style comment then. At this point I think 1) is actually better for your requirement. I start out as hoping the expand_macro() can abstract away the internal implementation detail of macro expand and just give you the text before and after the macro expand. But with all this extra manipulations of the macro tokens, especially the substitute_argument() is clear indicate tightening into the implementation details. I would take 1) over substitute_arguments() if 1) don't need call back like substitute_arguments(). That is surprising to me. I previously have different assumptions. I assume you want to back trace all the way back to the source macro. I would just say main() depend on xdef, which depend one S1 and S2. S1 also depend on E1 and S2 depend on E2. If you bypass xdef and directly extract S1(sx). Why can't you do one step further bypass S1() as well, and say main depend on "struct sx { E1 int d1;}? This is even more complicated than I original though. How about this case: #define S1(a) struct a { E1 int d1; }; E3 struct #define S2(a) a { E2 int d2; }; The rest is the same. Now xdef will expand to the same text. What should main() depend on? S1, E1 and E3? Notice that without S2, macro expand by S1 can't compile at all. Yes. Did that patch submit to gcc? I can see trace empty expansion one level. You track empty macro pass that one multiple level macro expand as well? Chris -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Christopher Li 2012-05-15T09:44:49 http://permalink.gmane.org/gmane.comp.parsers.sparse/2856 <pre>The last patch with mdep.c and test-mdep.c was also nothing to apply, only a work in progress to go further... One thing you can see is how I propagate the token sources using: 137: n-&gt;from = list-&gt;pos; ... 143: list-&gt;pos.line = id; 144: list-&gt;pos.stream = pps; here you get an argument why it is better to have a (1) -&gt;macro_begin(a) -&gt;macro_end(b) instead of only one (2) expand_macro(a,b) If you want to use the preprocessorhooks to output human readable macro expansion history line similar to LLVM you probably want a pointer to the "pre-expanded" token location, that is in (a). In (1) you can buildup the token-source-paths going from post-expand buffer (b) through the pre-expanded buffers (a) in (2) you only have the paths going through the expanded buffers, can use (a) to reason about where (b) came from, but not in a simple way... Kindof something like this: #define E1 #define E2 #define S1(a) struct a { E1 int d1; }; #define S2(a) struct a { E2 int d2; }; #define xdef S1(sx) S2(sy) xdef main() { struct sx v; } The xdef expands in one-line to "struct sx { int d1; }; struct sy { int d1; };" main() only uses "struct sx". Therefore the dependency analysis should not have "E2 and S2" as dependencies. I think you need the location of empty expansions... Do you mean http://cfw.sourceforge.net/htmltag/init_32.c.pinfo.html ? This is a gcc-based patch even there you need to trace empty expansion positions. See above example. I'm not shure if it works... -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Konrad Eisele 2012-05-15T07:52:23 http://permalink.gmane.org/gmane.comp.parsers.sparse/2855 <pre> I am very sorry that my speed of absorbing patches is much slower than your speed of producing it :( About propagating the empty expansions, may I ask a silly question? Is that the goal is you are able to track that, after the recursive expansion, you are able to tell in the result stream, there is an macro expand to nothing in the this location? Obviously, if the empty expansion is happen in the top level macro expand, you can always keep track of where is the original location of the expand using macro token-&gt;pos. The tricky part is the, if the empty expansion happen inside a multi-level macro expand. Then it is hard to keep track of where that empty macro should have been landed if it is not empty. Is that the problem you are trying to solve? About the two example usage you give. The first one is using the html to show how macro expand recursively. I like that demo. It only need to remember at which level the macro expand to empty. It don't need to remember where the empty macro need to land in the result stream. For the second usage example, the shrinking program. I think it only need to remember empty macro expand at the top level. Which is easy because you have the macro token-&gt;pos pointing to where this macro used in the source stream. For the empty macro expand inside another macro, you only need to remember it is a dependent of the the top level macro. Because when you trim the source code, you can't split a top level macro. I think a lot of the complexity is introduced try to remember where that empty macro will land, if it is not being empty. However, exactly because the macro is being empty, it will not show up in the result token list. So where it should land is actually not very useful information, the parser never see it any way. We can relax the requirement a little bit, only need to remember where the empty macro will land in the top level case. That will greatly simplify the solution. Is my understanding correct? Thanks Chris -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Christopher Li 2012-05-15T06:30:20 http://permalink.gmane.org/gmane.comp.parsers.sparse/2854 <pre> Here is some feed back for the patch you send out: The test file should be in validation/ directory. Please give a.h a name represent the test as well. + if (!strncmp(arg, "buildin", 7)) { + fnobuildin = 1; + } Is there a stand gcc option to handle nobuildin? I haven't found one. If gcc has one, we can duplicate the gcc behavior. Otherwise, I don't see it is necessary to add one. The dependence program needs to able to handle the symbol from built in stream any way. If it is for easier to debug, your program can trivially skip out the builtin. stream. + * BSD-License + * Redistribution and use in source and binary forms are permitted Can you make it cover by the sparse license file as well? I don't mind you grant extra license to your code. But please at least make it cover by the license of the project itself otherwise I won't apply it. It is pure overhead to figure out what license is compatiable to spase and maintain different file at different license. Have all file under the project cover by the same license will keep it simple. +void mdep_init(void) { + preprocess_hook = &amp;pp; + pps = init_stream("&lt;pp&gt;", -1, 0); +} Please use the same coding standing as the other part of the project. It is actually the same as linux kernel. The "{" for the function should be on a new line. Indentation is tab and 8 char wide. There is a lot of those little one on this file. I am not going to repeat myself here. +struct hash { + struct hash_v *f; +} h[HASH_LEN]; +struct pp { + enum pp_typ t; + union { + unsigned int argi; + }; + struct pp_e *f; + struct symbol *sym; + struct token *tok; + struct token *s, *d; +}; + I hate those one letter member name. You have to guess what it is all the time. pp_dope_list() is full of those one letter variable name as well, make it very hard to read. The variable name need to be a little bit more meaningful. + show_tokenstream(token, 0); Please don't use 0 for NULL pointer. -void show_tokenstream(struct token *token) +void show_tokenstream(struct token *token, struct token *end) - while (!eof_token(token)) { + while (token != end &amp;&amp; !eof_token(token)) { I don't see you use the show_tokenstream with two arguments. You write your own show_tokenstream for mdep any way. You can leave it one unchanged. &lt; at &gt;&lt; at &gt; -194,7 +196,7 &lt; at &gt;&lt; at &gt; void show_tokenstream(struct token *token) - printf("%s%.*s", show_token(token), prec, separator); + fprintf(stderr,"%s%.*s", show_token(token), prec, separator); Can't do that, the "-E" will use show_tokenstream() as well. You just change it to stderr for "-E". To be continue... Chris -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Christopher Li 2012-05-14T10:53:35 http://permalink.gmane.org/gmane.comp.parsers.sparse/2853 <pre> I have thought about how to implement empty expansion tracing without introducing a new token type. I came up with a solution, however I need one callback, I called it substitute_arg(), see patch attached. What do you think, is it apply-able? I think I can use the address of the pointer to token (strict token **, which is normally &amp;tok-&gt;next) as a hashing to propagate the empty expansions... Im not 100% shure it works but I need the extra hook to be able to propagate the empty expansion from the arguments into the substitution body... diff --git a/pre-process.c b/pre-process.c index fb3430a..73a58be 100644 --- a/pre-process.c +++ b/pre-process.c &lt; at &gt;&lt; at &gt; -573,6 +573,9 &lt; at &gt;&lt; at &gt; static struct token **substitute(struct token **list, struct token *body, struct case TOKEN_MACRO_ARGUMENT: arg = args[body-&gt;argnum].expanded; count = &amp;args[body-&gt;argnum].n_normal; +if (preprocess_hook) { +preprocess_hook-&gt;substitute_arg (&amp;added, &amp;args[body-&gt;argnum].expanded); +} if (eof_token(arg)) { state = Normal; continue; &lt; at &gt;&lt; at &gt; -650,7 +653,7 &lt; at &gt;&lt; at &gt; static int expand(struct token **list, struct symbol *sym) if (preprocess_hook &amp;&amp; preprocess_hook-&gt;expand_arg) { int i; for (i = 0; i &lt; nargs; i++) { -preprocess_hook-&gt;expand_arg(token, sym, i, args[i].orig, args[i].expanded); +preprocess_hook-&gt;expand_arg(token, sym, i, args[i].orig, &amp;args[i].expanded); free_preprocessor_line(args[i].orig); } } diff --git a/token.h b/token.h index 985d1f5..c45d6be 100644 --- a/token.h +++ b/token.h &lt; at &gt;&lt; at &gt; -175,7 +175,8 &lt; at &gt;&lt; at &gt; struct preprocess_hook { void (*expand_macro)(struct token *macro, struct symbol *sym, struct token *parent, struct token **replace, struct token **replace_tail); void (*expand_arg)(struct token *macro, struct symbol *sym, int arg, - struct token *orig, struct token *expanded); + struct token *orig, struct token **expanded); +void (*substitute_arg)(struct token **dest, struct token **argp); }; #define MAX_STRING 4095 </pre> Konrad Eisele 2012-05-13T08:52:21 http://permalink.gmane.org/gmane.comp.parsers.sparse/2852 <pre> To explain mdep.c: There are in fact only 3 lines that are of interest: ... 137: n-&gt;from = list-&gt;pos; ... ... 143: list-&gt;pos.line = id; 144: list-&gt;pos.stream = pps; ... Line 137 saves the last token.pos , (143+144) insert a new id into token.pos. This will generate the path for each token through the expansions. mdep_trace() traverses the path... -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Konrad Eisele 2012-05-12T17:57:16 http://permalink.gmane.org/gmane.comp.parsers.sparse/2851 <pre> Appended is a test-patch that adds test-mdep testcase. The file mdep.c is used to record that macro expansion, each token will have a reference to its source. test-mdep.c does pre-process (as test-macro.c) then prints out the token trace through macros for each token: &lt; at &gt;{ } is used to mark the active path. An example file is added: a.h $test-mdep a.h ... 0004: 8 body in D1 :4 &lt; at &gt;{8} 10 9 5 &lt;untaint: D1&gt; arg0 in D1 :&lt; at &gt;{8} 10 9 body in D0 :1 &lt; at &gt;{D1}(8 10 9) 2 D2(11) 3 &lt;untaint: D0&gt; a.h:6:6 ... Token nr 4 of the preprocess stream is "8". The generation path of "8" is marked &lt; at &gt;{8}... Not 100%, still, I think already readable. (Actually the printout order should be reversed (starting from file scope and drilling down the macro expansions...) I still dont handle empty expansions. I'll see weather I can come up with something here... No, only a linked list that model the nexting levels. Then a preprocessor hook that can register lookup_macro() macro lookups inside # preprocessor lines. An example makes it clear: #if defined(a) &amp;&amp; defined(b) #if defined(c) #endif #if defined(e) #endif #endif Result in: [a b]+&lt;-[c] +&lt;-[e] This can be easily done with a push-pop brackets and a callback in lookup_macro(). Also: #if defined(a) #elif defined(c) #endif [a]+&lt;-[c] #if defined(a) #else #endif &lt;-[empty]&lt;-[a] ... Another point I also need is to have an option so that inside do_handle_define() the symbol structures are never reused but alloc_symbol() is always used for undef and define, this is because I need to be able to also track the undef and define history for a macro at a certain position. I think this should be easy to add because you just need to define define-undef on top of each other... I thought about taking "struct symbol_list *syms = sparse(file)" as the root. Then mark all elements that are used by them as dependent. I dont have enough insight to say how I can determine things like which "static inline" are used or how to traverse the "typedef" dependency. The goal is to have a "shrink" application that can strip away all c-lines (pre-pre-process level) that are not used by a specific command invocation of the compiler. Also a tool that can quickly show for a specific identifier everything that is connected to it, again on pre-preprocessor source level. kind-of something like: ... func1() { struct string_list *filelist = NULL; int i; } .. I point to "string_list" and then all lines that are related to struct string_list, (#ifdef nestings, macros, all member typedefs) etc are shown and all the rest stripped away, again on human readable c source level. From aff7f53ce89d24512c0ba2f66b981718538ae1c8 Mon Sep 17 00:00:00 2001 From: Konrad Eisele &lt;eiselekd&lt; at &gt;gmail.de&gt; Date: Sat, 12 May 2012 18:43:16 +0200 Subject: [PATCH] mdep.c test --- Makefile | 5 +- a.h | 6 ++ a2.h | 2 + lib.c | 19 +++-- mdep.c | 248 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ pre-process.c | 3 +- test-macro.c | 10 +- test-mdep.c | 62 ++++++++++++++ token.h | 8 ++- tokenize.c | 8 +- 10 files changed, 351 insertions(+), 20 deletions(-) create mode 100644 a.h create mode 100644 a2.h create mode 100644 mdep.c create mode 100644 test-mdep.c diff --git a/Makefile b/Makefile index 4abcbdd..b688054 100644 --- a/Makefile +++ b/Makefile &lt; at &gt;&lt; at &gt; -44,7 +44,7 &lt; at &gt;&lt; at &gt; PKGCONFIGDIR=$(LIBDIR)/pkgconfig PROGRAMS=test-lexing test-parsing obfuscate compile graph sparse \ test-linearize example test-unssa test-dissect ctags \ - test-macro + test-mdep test-macro INST_PROGRAMS=sparse cgcc INST_MAN1=sparse.1 cgcc.1 &lt; at &gt;&lt; at &gt; -96,7 +96,8 &lt; at &gt;&lt; at &gt; LIB_H= token.h parse.h lib.h symbol.h scope.h expression.h target.h \ LIB_OBJS= target.o parse.o tokenize.o pre-process.o symbol.o lib.o scope.o \ expression.o show-parse.o evaluate.o expand.o inline.o linearize.o \ sort.o allocate.o compat-$(OS).o ptrlist.o \ - flow.o cse.o simplify.o memops.o liveness.o storage.o unssa.o dissect.o + flow.o cse.o simplify.o memops.o liveness.o storage.o unssa.o dissect.o \ + mdep.o LIB_FILE= libsparse.a SLIB_FILE= libsparse.so diff --git a/a.h b/a.h new file mode 100644 index 0000000..63da9e8 --- /dev/null +++ b/a.h &lt; at &gt;&lt; at &gt; -0,0 +1,6 &lt; at &gt;&lt; at &gt; +//#include &lt;stdio.h&gt; +#define D0(d0a0,d0a1) 1 D1(d0a0) 2 D2(d0a1) 3 +#define D1(d1a0) 4 d1a0 5 +#define D2(d2a0) +#define D3(d3a0) 8 d3a0 9 +1 2 D0(D3(10),11) 3 4 diff --git a/a2.h b/a2.h new file mode 100644 index 0000000..098fbc2 --- /dev/null +++ b/a2.h &lt; at &gt;&lt; at &gt; -0,0 +1,2 &lt; at &gt;&lt; at &gt; +#define A(a,b) b 3 +A(1,2) diff --git a/lib.c b/lib.c index 1876fc9..51879d9 100644 --- a/lib.c +++ b/lib.c &lt; at &gt;&lt; at &gt; -577,6 +577,7 &lt; at &gt;&lt; at &gt; static char **handle_switch_ftabstop(char *arg, char **next) return next; } +int fnobuildin = 0; static char **handle_switch_f(char *arg, char **next) { arg++; &lt; at &gt;&lt; at &gt; -588,6 +589,9 &lt; at &gt;&lt; at &gt; static char **handle_switch_f(char *arg, char **next) if (!strncmp(arg, "no-", 3)) { arg += 3; +if (!strncmp(arg, "buildin", 7)) { +fnobuildin = 1; +} } /* handle switch here.. */ return next; &lt; at &gt;&lt; at &gt; -875,7 +879,7 &lt; at &gt;&lt; at &gt; static struct symbol_list *sparse_tokenstream(struct token *token) token = preprocess(token); if (preprocess_only) { -show_tokenstream(token); +show_tokenstream(token, 0); putchar('\n'); return NULL; } &lt; at &gt;&lt; at &gt; -963,12 +967,13 &lt; at &gt;&lt; at &gt; struct symbol_list *sparse_initialize(int argc, char **argv, struct string_list // Initialize type system init_ctype(); -create_builtin_stream(); -add_pre_buffer("#define __CHECKER__ 1\n"); -if (!preprocess_only) -declare_builtin_functions(); - -list = sparse_initial(); +if (!fnobuildin) { +create_builtin_stream(); +add_pre_buffer("#define __CHECKER__ 1\n"); +if (!preprocess_only) +declare_builtin_functions(); +list = sparse_initial(); +} /* * Protect the initial token allocations, since diff --git a/mdep.c b/mdep.c new file mode 100644 index 0000000..6af8467 --- /dev/null +++ b/mdep.c &lt; at &gt;&lt; at &gt; -0,0 +1,248 &lt; at &gt;&lt; at &gt; +/* + * Copyright (C) 2012 Konrad Eisele &lt;eiselekd&lt; at &gt;gmail.com&gt; + * BSD-License + * Redistribution and use in source and binary forms are permitted + * provided that the above copyright notice and this paragraph are + * duplicated in all such forms and that any documentation, + * advertising materials, and other materials related to such + * distribution and use acknowledge that the software was developed + * by the &lt;organization&gt;. The name of the + * University may not be used to endorse or promote products derived + * from this software without specific prior written permission. + * THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR + * IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + */ + +#include &lt;stdlib.h&gt; +#include &lt;stdio.h&gt; +#include &lt;string.h&gt; +#include &lt;assert.h&gt; +#include "token.h" +#include "allocate.h" +#include "compat.h" +#include "parse.h" +#include "symbol.h" +#include "token.h" +#include "lib.h" + +static void expand_macro(struct token *macro, struct symbol *sym, + struct token *parent, struct token **replace, + struct token **replace_tail, struct token *last); +static void expand_arg(struct token *macro, struct symbol *sym, int arg, + struct token *orig, struct token *expanded); +struct pp; +struct pp_e; + +struct preprocess_hook pp = { + .expand_macro = expand_macro, + .expand_arg = expand_arg +}; + +unsigned int pps = 0; + +void mdep_init(void) { + preprocess_hook = &amp;pp; + pps = init_stream("&lt;pp&gt;", -1, 0); +} + +enum tags { + ATTR_TOK = 1, + ATTR_TOKP = 2, +}; + +struct hash_v { + struct hash_v *n; + long key; + enum tags tag; + void *v; +}; + +__DECLARE_ALLOCATOR(struct hash_v, hash_v); +__ALLOCATOR(struct hash_v, "hash value", hash_v); + +#define HASH_LEN (1024*4) +struct hash { + struct hash_v *f; +} h[HASH_LEN]; + +static int hash_func(long key, enum tags tag) { + unsigned int k = ((unsigned int)key) &gt;&gt; 4; + return ((k) ^ (k &gt;&gt; 16) ^ (k &gt;&gt; 24) ^ tag) &amp; (HASH_LEN-1); +} + +void **lookup_attr(long key, enum tags tag, int create) { + int i = hash_func(key, tag); + struct hash *hp = &amp;h[i]; + struct hash_v *p; + struct hash_v **c = &amp;hp-&gt;f; + while((p = *c)) { + if ((p -&gt;tag == tag) + &amp;&amp; (p -&gt;key == key)) { + return &amp;p-&gt;v; + } + c = &amp;p-&gt;n; + } + if (create) { + p = __alloc_hash_v(0); + p-&gt;key = key; + p-&gt;tag = tag; + p-&gt;v = 0; + *c = p; + return &amp;p-&gt;v; + } + return 0; +} + +enum pp_typ { + MARG = 1, + MBODY, +}; + +struct pp { + enum pp_typ t; + union { + unsigned int argi; + }; + struct pp_e *f; + struct symbol *sym; + struct token *tok; + struct token *s, *d; +}; + +struct pp_e { + struct pp_e *n; + struct pp *p; + struct position from; + int idx; +}; + +__DECLARE_ALLOCATOR(struct pp, pp_v); +__ALLOCATOR(struct pp, "pp trace", pp_v); +__DECLARE_ALLOCATOR(struct pp_e, pp_e_v); +__ALLOCATOR(struct pp_e, "pp trace element", pp_e_v); +int n_tokid = 1; + +void pp_dope_list(struct pp *p, struct token **d, int dope, struct token *list, struct token *end, int prt) +{ + struct pp_e **e = &amp;p-&gt;f; + struct token *n; + int idx = 0; + while ((!eof_token(list)) &amp;&amp; list != end ) { + if (dope) { + void **v; + int id = n_tokid++; + struct pp_e *n = __alloc_pp_e_v(0); + n-&gt;from = list-&gt;pos; + n-&gt;idx = idx; + n-&gt;n = 0; + n-&gt;p = p; + *e = n; + v = lookup_attr(id, ATTR_TOK, 1); + *v = n; + list-&gt;pos.line = id; + list-&gt;pos.stream = pps; + e = &amp;n-&gt;n; + } + n = __alloc_token(0); + *n = *list; + /*printf(" %s\n", show_token(list));*/ + n-&gt;next = &amp;eof_token_entry; + *d = n; + d = &amp;n-&gt;next; + list = list-&gt;next; + idx++; + + + } +} + +struct pp *new_pp(struct token *m, int t) { +struct pp *n = __alloc_pp_v(0); +n-&gt;t = t; +n-&gt;f = 0; +return n; +} + +static void expand_macro(struct token *macro, struct symbol *sym, struct token *parent, + struct token **replace, struct token **replace_tail, struct token *last) +{ + struct pp *p = new_pp(macro, MBODY); + p-&gt;sym = sym; + p-&gt;tok = macro; + pp_dope_list(p, &amp;p-&gt;s, 0, sym-&gt;expansion, 0, 0); + pp_dope_list(p, &amp;p-&gt;d, 1, *replace, last, 0); +} + +static void expand_arg(struct token *macro, struct symbol *sym, int arg, + struct token *orig, struct token *expanded) +{ + struct pp *p = new_pp(macro, MARG); + p-&gt;argi = arg; + p-&gt;sym = sym; + p-&gt;tok = macro; + pp_dope_list(p, &amp;p-&gt;s, 0, orig, 0, 0); + pp_dope_list(p, &amp;p-&gt;d, 1, expanded, 0, 0); +} + +void mdep_show_tokenstream(struct token *token, struct token *end, int idx) +{ + int i = 0; + while (token != end &amp;&amp; !eof_token(token)) { + int prec = 1; + struct token *next = token-&gt;next; + const char *separator = ""; + if (next-&gt;pos.whitespace) + separator = " "; + if (next-&gt;pos.newline) { + separator = "\n\t\t\t\t\t"; + prec = next-&gt;pos.pos; + if (prec &gt; 4) + prec = 4; + } + if (i == idx) + fprintf(stderr,"&lt; at &gt;{%s}%.*s", show_token(token), prec, separator); + else + fprintf(stderr,"%s%.*s", show_token(token), prec, separator); + token = next; + i++; + } +} + +void mdep_trace (struct token *tok, char *pre) +{ + void **v; int id; struct position pos = tok-&gt;pos; + struct pp_e *e; struct pp *p; + pre = pre ? pre : ""; + if(!tok || eof_token (tok)) + return; + while(1) { + if (pos.stream != pps) { + char *name = stream_name(pos.stream); + fprintf(stderr, "%s%s:%d:%d\n", pre, + name, pos.line, pos.pos); + break; + } + id = pos.line; + if (!(v = lookup_attr(id, ATTR_TOK, 0))) { + break; + } + e = (struct pp_e *)*v; + p = e-&gt;p; + fprintf(stderr, "%s",pre); + if (p-&gt;t == MARG) { + fprintf(stderr,"arg%d in %s :", p-&gt;argi, show_token(p-&gt;tok)); + } else { + fprintf(stderr,"body in %s :", show_token(p-&gt;tok)); + } + mdep_show_tokenstream(p-&gt;d, 0,e-&gt;idx); fprintf (stderr,"\n"); + pos = e-&gt;from; + } +} + +/* +Local Variables: +c-basic-offset:4 +indent-tabs-mode:nil +End: +*/ diff --git a/pre-process.c b/pre-process.c index fb3430a..4eee864 100644 --- a/pre-process.c +++ b/pre-process.c &lt; at &gt;&lt; at &gt; -385,6 +385,7 &lt; at &gt;&lt; at &gt; static void expand_arguments(int count, struct arg *args) struct token *arg = args[i].arg; if (!arg) arg = &amp;eof_token_entry; +args[i].expanded = &amp;eof_token_entry; if (args[i].n_str) args[i].str = stringify(arg); if (args[i].n_normal) { &lt; at &gt;&lt; at &gt; -661,7 +662,7 &lt; at &gt;&lt; at &gt; static int expand(struct token **list, struct symbol *sym) last = token-&gt;next; tail = substitute(list, sym-&gt;expansion, args); if (preprocess_hook &amp;&amp; preprocess_hook-&gt;expand_macro) -preprocess_hook-&gt;expand_macro(token, sym, parent, list, tail); +preprocess_hook-&gt;expand_macro(token, sym, parent, list, tail, last); *tail = last; return 0; diff --git a/test-macro.c b/test-macro.c index b30ee50..3115bef 100644 --- a/test-macro.c +++ b/test-macro.c &lt; at &gt;&lt; at &gt; -22,9 +22,9 &lt; at &gt;&lt; at &gt; static void expand_arg(struct token *macro, struct symbol *sym, int i, struct token *orig, struct token *expanded) { printf("arg%d in %s :", i, show_token(macro)); -show_tokenstream(orig); +show_tokenstream(orig, 0); printf(" -&gt; "); -show_tokenstream(expanded); +show_tokenstream(expanded, 0); printf("\n"); } &lt; at &gt;&lt; at &gt; -35,7 +35,7 &lt; at &gt;&lt; at &gt; static void expand_macro(struct token *macro, struct symbol *sym, struct token * printf("macro %s inside", show_token(macro)); printf(" %s\n", show_token(parent)); printf("expand result: "); -show_tokenstream(*replace); +show_tokenstream(*replace, 0); printf("\n"); } &lt; at &gt;&lt; at &gt; -55,11 +55,11 &lt; at &gt;&lt; at &gt; void test_macro(char *filename) die("No such file: %s", filename); token = tokenize(filename, fd, NULL, includepath); -show_tokenstream(token); +show_tokenstream(token, 0); printf("\n"); token = preprocess(token); printf("After preprocessing\n"); -show_tokenstream(token); +show_tokenstream(token, 0); } int main(int argc, char **argv) diff --git a/test-mdep.c b/test-mdep.c new file mode 100644 index 0000000..eca0357 --- /dev/null +++ b/test-mdep.c &lt; at &gt;&lt; at &gt; -0,0 +1,62 &lt; at &gt;&lt; at &gt; +/* + * Parse and linearize the tree for testing. + * + * Copyright (C) 2012 Christophre Li + * + */ +#include &lt;stdarg.h&gt; +#include &lt;stdlib.h&gt; +#include &lt;stdio.h&gt; +#include &lt;string.h&gt; +#include &lt;ctype.h&gt; +#include &lt;unistd.h&gt; +#include &lt;fcntl.h&gt; + +#include "lib.h" +#include "allocate.h" +#include "token.h" +#include "parse.h" +#include "symbol.h" +#include "expression.h" + +void test_mdep(char *filename) +{ + struct token *token; + int fd; int idx = 0; + fd = open(filename, O_RDONLY); + if (fd &lt; 0) + die("No such file: %s", filename); + + token = tokenize(filename, fd, NULL, includepath); + token = preprocess(token); + printf("Dump token stream:\n"); + + while (!eof_token(token)) { + struct token *next = token-&gt;next; + printf("%04d: %s\n", idx, show_token(token)); + mdep_trace (token, " "); + token = next; idx++; + } + +} + +int main(int argc, char **argv) +{ + struct string_list *filelist = NULL; + char *file; + + mdep_init(); + + sparse_initialize(argc, argv, &amp;filelist); + FOR_EACH_PTR_NOTAG(filelist, file) { + test_mdep(file); + } END_FOR_EACH_PTR_NOTAG(file); + return 0; +} + +/* +Local Variables: +c-basic-offset:4 +indent-tabs-mode:nil +End: +*/ diff --git a/token.h b/token.h index 985d1f5..8ddccd5 100644 --- a/token.h +++ b/token.h &lt; at &gt;&lt; at &gt; -173,7 +173,7 &lt; at &gt;&lt; at &gt; struct token { struct preprocess_hook { void (*expand_macro)(struct token *macro, struct symbol *sym, struct token *parent, - struct token **replace, struct token **replace_tail); + struct token **replace, struct token **replace_tail, struct token *last); void (*expand_arg)(struct token *macro, struct symbol *sym, int arg, struct token *orig, struct token *expanded); }; &lt; at &gt;&lt; at &gt; -206,7 +206,7 &lt; at &gt;&lt; at &gt; extern const char *show_special(int); extern const char *show_ident(const struct ident *); extern const char *show_string(const struct string *string); extern const char *show_token(const struct token *); -extern void show_tokenstream(struct token *token); +extern void show_tokenstream(struct token *token, struct token *end); extern struct token * tokenize(const char *, int, struct token *, const char **next_path); extern struct token * tokenize_buffer(void *, unsigned long, struct token **); &lt; at &gt;&lt; at &gt; -223,4 +223,8 &lt; at &gt;&lt; at &gt; static inline int match_ident(struct token *token, struct ident *id) return token-&gt;pos.type == TOKEN_IDENT &amp;&amp; token-&gt;ident == id; } +/* mdep.c */ +extern void mdep_init (void); +extern void mdep_trace (struct token *tok, char *pre); + #endif diff --git a/tokenize.c b/tokenize.c index b626f3f..6d0978f 100644 --- a/tokenize.c +++ b/tokenize.c &lt; at &gt;&lt; at &gt; -127,6 +127,8 &lt; at &gt;&lt; at &gt; const char *show_token(const struct token *token) if (!token) return "&lt;no token&gt;"; +if (token == &amp;eof_token_entry) +return "&lt;eof&gt;"; switch (token_type(token)) { case TOKEN_ERROR: return "syntax error"; &lt; at &gt;&lt; at &gt; -180,9 +182,9 &lt; at &gt;&lt; at &gt; const char *show_token(const struct token *token) } } -void show_tokenstream(struct token *token) +void show_tokenstream(struct token *token, struct token *end) { -while (!eof_token(token)) { +while (token != end &amp;&amp; !eof_token(token)) { int prec = 1; struct token *next = token-&gt;next; const char *separator = ""; &lt; at &gt;&lt; at &gt; -194,7 +196,7 &lt; at &gt;&lt; at &gt; void show_tokenstream(struct token *token) if (prec &gt; 4) prec = 4; } -printf("%s%.*s", show_token(token), prec, separator); +fprintf(stderr,"%s%.*s", show_token(token), prec, separator); token = next; } } </pre> Konrad Eisele 2012-05-12T17:46:48 http://permalink.gmane.org/gmane.comp.parsers.sparse/2850 <pre> The expand_macro call back use the parent argument which get from expanding_macro list. The caller should be able to create tree from the leaf node using the parent pointer. Feel free to change to use the expanding_macro instead if that make building the tree easier. The body expansion can't be recursive on same macro otherwise it can result in unlimited expansion. The C stander specify the macro expand this way. No problem at all. I figure you just want to the patch to get included. That is great. I have a test-macro program in that branch which is very close to print out all the tokens. Let me think about this. Just thinking out lound, The #include and #ifdef can consider as a special kind of predefine macro as well. For symbol, it relative easy because symbol has pos range and aux pointer. Do you need to attach the dependency for the statment and expression as well? Chris -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Christopher Li 2012-05-12T11:02:17 http://permalink.gmane.org/gmane.comp.parsers.sparse/2849 <pre> This seems ok. expanding_macro has to be global not static to be used... (?) I think the fact that argument expansion is recursive and body expansion is non-recursive is one of the things that make the preprocessor kindof hard to grasp. I cannot say this before I've tried it. I'd like to straighten things out a bit: My last emails where a bit too harsh and I'd like to apologize. Sorry for that. The next step then is: I'll write a patch to add a test-prog that uses this api to trace the token generation and generate a tree for it. For a start I'll printout for all tokens of a preprocessor run all macros-expansions that generated them. Now, I've learned not to run too fast towards the goal, (which is still "dependency tee from c parser entities downto token"), maybe you can think about how to achieve the next steps in an API : - An #include #ifdef #else #endif pushdown-stack to record the nestings for each token - How to connect all this to the AST. </pre> Konrad Eisele 2012-05-11T21:48:27 http://permalink.gmane.org/gmane.comp.parsers.sparse/2848 <pre> You have a valid point that B(B(x)) will break the sym-&gt;parent list. I remove the sym-&gt;parent and just use a token_list to maintain the expanding macro. The macro is append to the list when enter and remove from the list when untaint token is reached. The change is in the same review branch. That should solve this problem? That is right. The macro expansion has 3 stages. The first stage is expand the arguments list while the caller macro does not consider "tainted" during the argument expansion. The macro can recursively appear in the argument list. That is some thing I haven't consider previously. The second stage is just replace the expanded arguments into body. The third stage is rescan the replacement string for macro expand. In this third stage, the macro itself is consider tainted and can't be expand again during the rescan. Can you take a look at this modify version will fit your need or not? I am curious the part weather that will remove the need to add empty token to the list for expansion. In order words, can we design the API clever enough that you don't need to jump through hoops to handle the empty expansion token. Thanks Chris -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Christopher Li 2012-05-11T19:40:57 http://permalink.gmane.org/gmane.comp.parsers.sparse/2847 <pre> I've found lkml.org somewhat unreliable. I'd suggest liking to http://mid.gmane.org/&lt;the-message-ID&gt; ; in this case, http://mid.gmane.org/CA+55aFzbhYvw7Am9EYgatpjTknBFm9eq+3jBWQHkSCUpnb3HRQ&lt; at &gt;mail.gmail.com - Josh Triplett -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Josh Triplett 2012-05-10T18:24:40 http://permalink.gmane.org/gmane.comp.parsers.sparse/2846 <pre> I have to revise my previous assumption though: #define B(y) A(x) B(1) i thought that in the body expansion of B recursion is involved, so that it would yield: expand_macro(A); expand_macro(B); but that was wrong, so its expand_macro(B); expand_macro(A); you might be right here. expand() is flat when substituting the body part of the macro. It might work. </pre> Konrad Eisele 2012-05-10T12:28:23 http://permalink.gmane.org/gmane.comp.parsers.sparse/2845 <pre> As far as I can tell, the type can be recovered reliably, one way or the other, and missing pointer cast was one of a few (if not only one) source of type loss in the instruction stream (e.g., with those pointers returned by function call as demonstrated). Thanks. </pre> Jan Pokorný 2012-05-10T12:20:49 http://permalink.gmane.org/gmane.comp.parsers.sparse/2844 <pre> Do a B(B(x)) and your sym-&gt;parent linked-list will fail. </pre> Konrad Eisele 2012-05-10T12:14:42 http://permalink.gmane.org/gmane.comp.parsers.sparse/2843 <pre> CCing Linus, Not sure about the rules of quoting other people's email. Does it need the full copyright notice? I would just add these two lines to the end of the sparse.txt Reference: * Linus' email about __nocast vs __bitwise: https://lkml.org/lkml/2012/3/22/409 Chris -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Christopher Li 2012-05-10T11:42:22 http://permalink.gmane.org/gmane.comp.parsers.sparse/2842 <pre> Well, the __cannot__ part is base on your reply you seems don't wish to continue this discussion. A change like this is bound to need some careful discussion and planing. Yes, I am guilt of only accepting patches meet some subjective stander of mine. But so is to any self respect project maintainers. I would rather spend some time to do it right than commit some thing I would regret later on. I heard you that this discussion is taking long. That is why I offer to write up the core sparse part of the change myself and let you provide feed back to shape it the way we both can happy. That is the agreement we have earlier right? So I did exactly what I said I am going to do, now you are calling me my way vs your way? My evaluation function is straightly technical merit: - I prefer patch minimize performance impact on other clients don't use this feature. - I prefer simpler interface over complicate one. To me, believe it or not, It is never about my way vs your way. If you submit a perfect patch, I would more than happy to apply it. Apply a patch is much easier than writing one myself. Chris -- To unsubscribe from this list: send the line "unsubscribe linux-sparse" in the body of a message to majordomo&lt; at &gt;vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html </pre> Christopher Li 2012-05-10T11:25:05 http://permalink.gmane.org/gmane.comp.parsers.sparse/2841 <pre>Hi Chris: --- On Thu, May 10, 2012 at 4:20 PM, Christopher Li &lt;sparse&lt; at &gt;chrisli.org&gt; wrote: | Looks good. Thanks for doing that. | | Can you add some reference (in sparse.txt) to the original Linus email | where some | of this document is base on? Just to give credit where it is due. A link to the | Linus original email would be perfect. \-- Is the following reference sufficient if added to the sparse.txt file? === sparse.txt === Sparse ~~~~~ Copyright (C) 2012 Linus Torvalds Source: https://lkml.org/lkml/2012/3/22/409 __nocast vs __bitwise: ... ... ... === END === I will re-submit the patch with the approved changes. Thanks for your quick response! SK </pre> Shakthi Kannan 2012-05-10T11:06:28 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.parsers.sparse