http://permalink.gmane.org/gmane.comp.syslog-ng hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.comp.syslog-ng/15119 <pre>Thanks Robert. I figured as much. I am going to look at some intermediary that can transform binary to base64 and then stream to syslog. On Tue, May 21, 2013 at 11:48 AM, Fekete Róbert &lt;frobert&lt; at &gt;balabit.hu&gt; wrote: ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq </pre> Xuri Nagarin 2013-05-22T04:14:10 http://permalink.gmane.org/gmane.comp.syslog-ng/15118 <pre> On Saturday, May 18, 2013 02:35 CEST, Xuri Nagarin &lt;secsubs&lt; at &gt;gmail.com&gt; wrote: Hi, I currently do not know about any way to handle binary data within a messages. Create a log statement that uses the flags(catchall) option. Regards, Robert ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq </pre> Fekete Róbert 2013-05-21T18:48:38 http://permalink.gmane.org/gmane.comp.syslog-ng/15117 <pre>https://bugzilla.balabit.com/show_bug.cgi?id=235 Balazs Scheidler &lt;bazsi&lt; at &gt;balabit.hu&gt; changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bazsi&lt; at &gt;balabit.hu --- Comment #1 from Balazs Scheidler &lt;bazsi&lt; at &gt;balabit.hu&gt; 2013-05-21 15:32:23 --- I have just pushed a "fix" for this in the 3.5 branch. The fix is quoted as it contains a largish refactoring in the area, so that unit testing becomes possible. The change itself is not very simple, as the change that broke this functionality allows the use of free-form text as block arguments, which means that quoted strings are injected as a quoted string, e.g. together with the quotation marks. The solution was to detect if the block argument is a single string literal, and if that's the case use the string's value if it is substituted within another string literal. e.g. block destination fooblock foobar() { filename("/var/lo</pre> bugzilla< at >bugzilla.balabit.com 2013-05-21T13:32:24 http://permalink.gmane.org/gmane.comp.syslog-ng/15116 <pre>Hi, My log source sends data over syslog in CEF (Arcsight format). I have a simple parser written in Syslog-NG that takes advantage of the CEF format (that is all messages are in eight fields separated by a pipe char). -----------xxxxxxxxxxxxxxxxx------------------- parser p_cef { csv-parser(columns("cef.ff","cef.vendor","cef.product","cef.c4","cef.c5","cef.c6","cef.c7","cef.c8") delimiters("|") flags(drop-invalid) ); }; destination d_file { file("/var/log/net/${cef.vendor}/${cef.product}/logfile"); }; log { source(s_tcp); parser(p_cef); destination(d_file); }; -----------xxxxxxxxxxxxxxxxx------------------- Some events coming in contain binary data that get translated into control characters and create thousands of directories with garbage in the names. How do I handle binary data in the message? Should I use the sanitize function in the destination/file definition or is there a better way to do it? Also, how do I create a catch-all destination for all the messages that do not match my</pre> Xuri Nagarin 2013-05-18T00:35:43 http://permalink.gmane.org/gmane.comp.syslog-ng/15115 <pre>Turned out to be an application inspection rule on the firewall between the log source and syslog-ng. Turning it off did the trick. On Wed, May 8, 2013 at 11:13 AM, Xuri Nagarin &lt;secsubs&lt; at &gt;gmail.com&gt; wrote: ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq </pre> Xuri Nagarin 2013-05-18T00:24:22 http://permalink.gmane.org/gmane.comp.syslog-ng/15114 <pre> On 13/05/2013, at 3:58 PM, Martin Holste &lt;mcholste&lt; at &gt;gmail.com&gt; wrote: yes. that was the issue. I had multiple log{} statements and I reordered them not realising that one was missing parser(p_db) Now the question is: If I have two log{} clauses both with parser(p_db) does the parsing get done twice? Russell ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq </pre> Russell Fulton 2013-05-17T08:03:45 http://permalink.gmane.org/gmane.comp.syslog-ng/15113 <pre>Hi, Merged, thanks Gergely. On Thu, 2013-05-16 at 14:35 +0200, Gergely Nagy wrote: ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq </pre> Balazs Scheidler 2013-05-16T17:58:22 http://permalink.gmane.org/gmane.comp.syslog-ng/15112 <pre>Hello, Mongodb support is available in syslog-ng for more than a year now. We are aware, that there are many people using it, but now we would like to ask for some feedback about the use cases and environments. Here are just a few sample questions, but any feedback regarding mongodb support is very welcome: - average and peak message rates sent to the mongodb destination - what is your related syslog-ng.conf - how did you tune your mongodb server (if you did) - how and how often do you query your logs in mongodb - etc. Thanks for your help! Bye, </pre> Peter Czanik 2013-05-16T13:08:12 http://permalink.gmane.org/gmane.comp.syslog-ng/15111 <pre>In certain environments (vserver and OpenVZ come to mind), /proc/kmsg is not readable, not even for root. On these systems, emit a warning, and skip it, instead of aborting the startup. Reported-by: Tamas Pal &lt;folti&lt; at &gt;balabit.hu&gt; Signed-off-by: Gergely Nagy &lt;algernon&lt; at &gt;balabit.hu&gt; --- modules/system-source/system-source.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/modules/system-source/system-source.c b/modules/system-source/system-source.c index 3c72859..955ade7 100644 --- a/modules/system-source/system-source.c +++ b/modules/system-source/system-source.c &lt; at &gt;&lt; at &gt; -167,8 +167,17 &lt; at &gt;&lt; at &gt; system_sysblock_add_linux_kmsg(GString *sysblock) close (fd); } - system_sysblock_add_file(sysblock, kmsg, -1, - "kernel", "kernel", format); + if (access(kmsg, R_OK) == -1) + { + msg_warning("system(): The kernel message buffer is not readable, " + "please check permissions if this is unintentional.", + evt_tag_s</pre> Gergely Nagy 2013-05-16T12:35:43 http://permalink.gmane.org/gmane.comp.syslog-ng/15110 <pre>Dear syslog-ng users, This is the 24th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. FEATURED NEWS syslog-ng configurator app for Android -------------------------------------- There is now a new configurator app available for Android, developed as a hobby project by one of the syslog-ng team members. The focus of the application is to create a syslog-ng.conf wich provides optimum performance based on a number of questions. For more details and download locations read the authors blog at http://pzolee.blogs.balabit.com/2013/04/little-syslog-ng-configurator-application-for-android/ syslog-ng and Splunk -------------------- We often receive questions, how to use syslog-ng and Splunk together in a logging infrastructure. We collected the most popular usage scenarios into a white paper, together with example configurations to make testing and integration even more easy. The WP is available at http://www.balabit.com/support/documentation/pdf/sy</pre> Peter Czanik 2013-05-16T11:12:52 http://permalink.gmane.org/gmane.comp.syslog-ng/15109 <pre>Yeah, two parser references cause it to be parsed twice. On May 15, 2013 5:08 PM, "Martin Holste" &lt;mcholste&lt; at &gt;gmail.com&gt; wrote: ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq </pre> Balazs Scheidler 2013-05-15T19:07:38 http://permalink.gmane.org/gmane.comp.syslog-ng/15108 <pre>That's a great question--I have no idea if two parser entries mandate double parsing. If you want to make sure that only your custom log {} statement will be used, you can use flags(final) in your log {} stanza to ensure that no messages will continue on to the other log statements. On Tue, May 14, 2013 at 11:22 PM, Russell Fulton &lt;r.fulton&lt; at &gt;auckland.ac.nz&gt;wrote: ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq </pre> Martin Holste 2013-05-15T15:08:29 http://permalink.gmane.org/gmane.comp.syslog-ng/15107 <pre> I finally figured out what the issue was here. It had to be something totally, idiotically simple and it was. Martin was on the right track with the order of filters relative to parser(p_db); What had happened was that I had originally the filter in a second log {} clause after one that contained the parser() entry so everything worked. Martin introduce the elsa_syslog.conf include and I moved all my local mods into there so now the filter was in a log{} clause that did not have a parser() entry and was now before the one that had it. I won't tell how many hours careful elimination it took to track this down. For elsa users if you put new log{} clauses in the include file you must have a parse() entry in them if you want to do anything with the classifier results. Question: Will having two parser() entries result in the log message being parsed twice? My guess is that it will. R ______________________________________________________________________________ Member info: https://lists.balabit.</pre> Russell Fulton 2013-05-15T04:22:21 http://permalink.gmane.org/gmane.comp.syslog-ng/15106 <pre> I've seen something similar recently, but have not had the time to research the problem further yet. It's on the immediate roadmap, however, and I hope to be able to come up with something sensible within a week. </pre> Gergely Nagy 2013-05-13T09:49:45 http://permalink.gmane.org/gmane.comp.syslog-ng/15105 <pre>The issue is probably where the filter resides. I use that filter (in fact, it's in an optional ELSA config right now) and it works, but you have to remember that ${.classifier.class} isn't set until after the patterndb parser is run, so the filter() statement has to be after parser(p_db); On Fri, May 10, 2013 at 11:51 PM, Evan Rempel &lt;erempel&lt; at &gt;uvic.ca&gt; wrote: ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq </pre> Martin Holste 2013-05-13T03:58:07 http://permalink.gmane.org/gmane.comp.syslog-ng/15104 <pre>Ok, after getting some rest and experimenting, I determined the solution. Creating a custom logformat had potential for a solution, but really wasn't the ideal way. After digging in the code more and stumbling on the logproto.c file (specifically the _frame_ functions), I was able to create my own frame handler for my module. Essentially, Created a custom framed_client_post that encapsulated messages to the destination protocol. This seems to keep true to the logproto intent. Assigned the custom proto when issuing a log_writer_reopen and life is good. Next challenge is how to prevent dropped messages in high volume scenarios... On Wed, May 8, 2013 at 5:38 PM, Dylan Kulesza &lt;dylan.kulesza&lt; at &gt;gmail.com&gt;wrote: ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq </pre> Dylan Kulesza 2013-05-12T21:42:18 http://permalink.gmane.org/gmane.comp.syslog-ng/15103 <pre>This seems to be a lib net compilation issue. The __be16 type is not defined for some reason. It might be caused by mismatching kernel, glibc versions. Try to locate the be16 type in your header files and add its inclusion before the libnet include file to afinet.c We do compile syslog-ng on rhel ourselves so this problem might be specific to your system. If you can find the root cause it'd be nice to know. Alternatively you can disable spoof source support if you don't need it. On 6/05/2013, at 6:31 AM, Balazs Scheidler &lt;bazsi77&lt; at &gt;gmail.com&gt; wrote: Sorry! missed the vital last line with the error in the cutnpaste! DOH!!! /usr/include/linux/if_ether.h:122: error: expected specifier-qualifier-list before '__be16' `/home/rful011/syslog-ng-3.4.1/modules/afsocket' -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/local/include/eventlog -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H -I../../lib/ivykis/src/</pre> Balazs Scheidler 2013-05-11T10:21:10 http://permalink.gmane.org/gmane.comp.syslog-ng/15102 <pre>Wait a second. Version 3.2.x ... really? That's quite old. There was a bug with the .classifier.X tags some time in the past, and it might have been in those old versions. Certainly version 3.3 would be recommended, and all of y work is done with 3.4.x My advice my be specific to version 3.4 :-( Evan Rempel 250.271.7691 University Systems, University of Victoria Evan Rempel &lt;erempel&lt; at &gt;uvic.ca&gt; wrote: This definitely works. I'm using it right now. If it isn't working, then your pattern in the patterndb is not matching. We literally run millions of messages per hour through this exact filter ... I copied and pasted it from our pattern database. Evan Rempel 250.271.7691 University Systems, University of Victoria Russell Fulton &lt;r.fulton&lt; at &gt;auckland.ac.nz&gt; wrote: On 11/05/2013, at 2:26 PM, Evan Rempel &lt;erempel&lt; at &gt;uvic.ca&gt; wrote: This always appears to return true. I.e. this filter includes everything. Negating it includes nothing. I have tried to install 3.2.5 as this is</pre> Evan Rempel 2013-05-11T04:51:48 http://permalink.gmane.org/gmane.comp.syslog-ng/15101 <pre>This definitely works. I'm using it right now. If it isn't working, then your pattern in the patterndb is not matching. We literally run millions of messages per hour through this exact filter ... I copied and pasted it from our pattern database. Evan Rempel 250.271.7691 University Systems, University of Victoria Russell Fulton &lt;r.fulton&lt; at &gt;auckland.ac.nz&gt; wrote: On 11/05/2013, at 2:26 PM, Evan Rempel &lt;erempel&lt; at &gt;uvic.ca&gt; wrote: This always appears to return true. I.e. this filter includes everything. Negating it includes nothing. I have tried to install 3.2.5 as this is the last version that ELSA is confirmed to work with but that does not start: Starting syslog-ng /usr/local/syslog-ng/sbin/syslog-ng: error while loading shared libraries: libsyslog-ng.so.0: cannot open shared object file: No such file or directory So far as I can tell all the lib files are present and correct and in the same place as the previous version? I have syslog-ng installed in /usr/local/syslog-ng-&lt;version&gt; and a symlink </pre> Evan Rempel 2013-05-11T04:36:49 http://permalink.gmane.org/gmane.comp.syslog-ng/15100 <pre> I don't know what the original message looks like that you are trying to substitute in, but I think the issue may be with the \s+ you could try \s* in case there is not any whitespace at the beginning. Also since pcre is greedy by default the \| is not necessary. The [^|]+ (which should probably be [^|]* in case the | follows immediately) will by definition be followed by a | or the end of line. Evan Rempel 250.271.7691 University Systems, University of Victoria Russell Fulton &lt;r.fulton&lt; at &gt;auckland.ac.nz&gt; wrote: I got this going with: rewrite r_snarex { subst("\s+This event is generated when.+", "", value("MSGONLY") type("pcre"));}; Does this mean that my syslog_ng does not support pcre? Russell On 11/05/2013, at 2:29 PM, Russell Fulton &lt;r.fulton&lt; at &gt;auckland.ac.nz&gt; wrote: ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: </pre> Evan Rempel 2013-05-11T04:33:35 http://permalink.gmane.org/gmane.comp.syslog-ng/15099 <pre> On 11/05/2013, at 2:26 PM, Evan Rempel &lt;erempel&lt; at &gt;uvic.ca&gt; wrote: This always appears to return true. I.e. this filter includes everything. Negating it includes nothing. I have tried to install 3.2.5 as this is the last version that ELSA is confirmed to work with but that does not start: Starting syslog-ng /usr/local/syslog-ng/sbin/syslog-ng: error while loading shared libraries: libsyslog-ng.so.0: cannot open shared object file: No such file or directory So far as I can tell all the lib files are present and correct and in the same place as the previous version? I have syslog-ng installed in /usr/local/syslog-ng-&lt;version&gt; and a symlink /usr/local/syslog-ng pointing to the version to use. Russell ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq </pre> Russell Fulton 2013-05-11T04:25:55 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.comp.syslog-ng