gmane.network.argus

Re: Full docs about ra output?

Matt Brown — 2012-05-25T22:31:24

Here is a pastebin of some of the records I am talking about:
http://pastebin.com/c12KvNjk

I'm guessing, from your reply, that <? and ?> mean "I tried, some things
tell me that it goes in this direction, but I'm not 52-100% sure?"  Or is
that 52 a 99?

This is skype traffic for sure.

Thanks for your assistance, Mark.


-Matt


On Thu, May 24, 2012 at 10:33 PM, Matt Brown <matthewbrown< at >gmail.com> wrote:

Re: Full docs about ra output?

Matt Brown — 2012-05-25T02:33:07

Thanks Mark.

I'll grab some stuff out of the DB I populated and reply to this thread.

I'm really focusing on creating a variety of macro views and trying to
figure out how to consider 'dir' in those derived views.

I'm currently focused on the easy stuff as pivot points, source and
destination: bytes, packet count, address and port. But I am interested in
leverage other parts of the DSR as well, if they are useful. (am I using
DSR right?)

I'll spend more time reviewing other threads as well as the NSMwiki, but
any further examples of how people create macro views of the data, versus
considering solely 'dir,' would be appreciated.


Thanks,

Matt


On Thu, May 24, 2012 at 9:39 PM, Mark Poepping <poepping< at >cmu.edu> wrote:

Re: Full docs about ra output?

Mark Poepping — 2012-05-25T01:39:09

Taking a stab (trying to relieve Carter of some of the burden)...

For directionality specifically, if it's a well-defined protocol and argus saw most (if not all) of the packets from the beginning, it will know the direction, but there are many examples of ordinary and hybrid protocols where you won't necessarily know the direction in all cases: peer-to-peer, ICMP, UDP can all make it hard to understand direction - or direction might not have meaning.  Packet loss (esp. packet sampling) often causes this output, and multi-path routing will 'look like' packet loss too, depending on where you're watching and how your paths are advertised or have evolved over time.

On a simple, lightly loaded network (my house), long-running argus probes generally get the directionality right.
At my work, it's not so simple so it helps to interact with questions that we have for the data and considerations of probe location and efficiency given the use cases.

Hope that helps some, it takes a little getting used to.  If you h

Full docs about ra output?

Matt Brown — 2012-05-25T01:00:08

Hello,

I see the man page for ra, but it seems lacking for some DSR value output.
For instance, there are somethings that aren't implicit, but appear like
they should/were intended to be.

Specifically, I see this with 'dir's possible values.

The cases if confusion are <? And ?>.  How can argus "not" know the
direction of the transaction "sort of?"

Thanks,

Matt

Re: Cisco non-flow data.

Carter Bullard — 2012-05-16T13:57:22

Hey Torbjorn,
Yes netflowV9 and full sflow support is a todo item for argus-3.0.8, which should start in June.
I restructured all the "foreign" flow support into the argus-import.c file in 3.0.6, and put a huge
push into flow-tools (100%) and sflow support (60%).  I actually have much of the netflowV9
stuff done, but the sample netflowV9 data that I was using had some errors in it, so I didn't
have it ready for the 3.0.6 release.

So, do you want to be a test site?

I'm hoping to have all of this done in a developers release by the end of June.

Carter


On May 16, 2012, at 8:01 AM, Torbjorn Wictorin wrote:

Cisco non-flow data.

Torbjorn Wictorin — 2012-05-16T12:01:46

Hello,

how are the plans regarding argus handling of cisco non-flow data?
In common/argus_import.c there is a routine ArgusParseCiscoRecordV9Data 
that I assume is supposed to do that, but it is a dummy in 3.0.6.
Any plans for the (reasonably near) future?

Torbjörn Wictorin,
Uppsala university

new client patches for cygwin compile problems

Carter Bullard — 2012-05-09T17:05:44

Gentle people,
We are now in a period where I'm fixing gotcha's after the official release of argus-3.0.6.
We now have two sets of fixes, a set of memory leaks and, now, a set of fixes for cygwin.
This looks to be the only issues right now.

The next round of released code will be argus-clients-3.0.6.1, and because
these are show stopper problems, I'll officially release this next version soon.

Until I hear from the list, my strategy for testing is to put up a single patch file 
that represents the 3.0.6 to 3.0.6.1 upgrade, patch-argus-clients-3-0-6-1.diff, and
to provide an accompanying argus-clients-3.0.6.1.tar.gz file.   These are only
on the developers side of the site.  When all is well and good, I'll move these two
files to the stable side of the server.

As a result, I've uploaded patch-argus-clients-3-0-6-1-diff to the developers site,
and a new argus-clients-3.0.6.1.tar.gz.  The patch file should be applied to the
original argus-clients-3.0.6 distribution.  But, you can apply it to a previously
pat

Re: problems using ARGUS_OUTPUT_FILE and 'filter'

Russell Fulton — 2012-05-09T04:48:50

This gets weirder -- my argus instance on other sensors are also ignoring the filter on the ARGUS_OUTPUT_FILE line too.  I just never noticed that I have a heap more flows than I should in all my argus files :(

Russell


On 9/05/2012, at 12:47 PM, Russell Fulton wrote:

problems using ARGUS_OUTPUT_FILE and 'filter'

Russell Fulton — 2012-05-09T00:47:05

When I use this filter on the command line it works as expected but when applied in the -F argus.conf I get all sorts of stuff that I does not match the filter?

argus-3.0.6.

Makes no difference if I put the filter on the FILTER = or the ARGUS_OUTPUT_FILE.

ARGUS_OUTPUT_FILE=/home/sensors/data/wireless/argus/user-data "tcp and src net ( 172.23.0.0/16 or 172.24.0.0/18 ) and not dst net ( 172.23.0.0/16 or 172.24.0.0/18 )  and dst port 80"

see attached file...

I've used this before often so I am very puzzled - on another sensor I have an identical config but with a ( 130.216.0.0/16 )  rather than the 172 addresses.

Clearly I'm doing something stupid, but what?

Any ideas?

Russell

PS:

sample of captured flows with -F argus-user-data.conf (attached)

   12:12:31.589938  e         tcp       207.46.61.90.80        ?>    130.216.181.194.57572         1         60   RST
   12:12:31.766180  e         tcp     74.125.237.102.443       ?>     130.216.215.70.1629          1         60   RST
   12:12:31.810594  e

updated patch for memory leaks on the server

Carter Bullard — 2012-05-08T22:11:41

Gentle people,
I have uploaded a new patch-all-memory-leak-clients-3-0-6.diff to the developers site,
which adds a memory leak fix for radium, when it is configured to generate geo labels
for flows its processing.

This fixes all known memory leak issues in argus-clients-3.0.6.
I've updated a new argus-clients-3.0.6.1.tar.gz with these new memory leak fixes.
If you could give these a try on your systems, that would be great.  

I will release these final tarfiles and patches tomorrow or Thur.

   http://qosient.com/argus/dev/argus-clients-3.0.6.1.tar.gz
   http://qosient.com/argus/dev/patch-all-memory-leak-clients-3-0-6.diff

Apply the patch by being in the parent directory that contains argus-clients-3.0.6,
and then:

   % patch -p0 < patch-memory-leak-clients-3-0-6.diff

For those that have a patch that is sensitive to RCS, SCCS, Perforce, whatever, try:

   % patch -g0 -p0 < patch-memory-leak-clients-3-0-6.diff

The you will just need to make, as there are no configure.ac changes.

   % cd argus-clients-3

patch file for rabins and rastream memory leak fix

Carter Bullard — 2012-05-07T16:45:31

Gentle people,
I have uploaded  argus-clients-3.0.6.1 to the developers site, and I
have uploaded a patch, for those that like patches, that fixes the reported
problems with rabins and rastream.  The fixes addresses the "-f racluster.conf"
issues with rabins.1, as well as the memory leak problems with long
term use of rabins and rastream, which occur primarily with ARGUS_EVENT
records.  

Apply the patch by being in the parent directory that contains argus-clients-3.0.6,
and then:

   % patch -p0 < patch-memory-leak-clients-3-0-6.diff

For those that have a patch that is sensitive to RCS, SCCS, Perforce, whatever, try:

   % patch -g0 -p0 < patch-memory-leak-clients-3-0-6.diff

The you will just need to make, as there are no configure.ac changes.

   % cd argus-clients-3.0.6; make

The patch will upgrade argus-clients-3.0.6 to argus-clients-3.0.6.1.

If you use rastream.1, you will want to upgrade to argus-3.0.6.1.
For those that want to use rabins.1 with the "-f racluster.conf", you will also
need to upgrad

argus-clients-3.0.6 rabins, rastream memory leak

Carter Bullard — 2012-05-02T22:57:16

Gentle people,
We've got a memory leak in the newest versions of rabins.1 and rastream.1.
I'm working on them now, but I have confirmed that long lived rastream processes
do collection the memory.  If you are using rastream.1, please consider testing
the patches when I make them available.

Should have a fix for you later tomorrow, which I will send to the list.

Hope all is most excellent,
Carter

Re: rabins and racluster.conf

Mark E. Mallett — 2012-04-30T19:58:03

That does seem to fix it for this test.  Thanks!

mm

Re: rabins and racluster.conf

Carter Bullard — 2012-04-30T19:50:27

Hey Mark,
May not be a complete solution still, but give this additional patch a try.
This moves the variable declarations for filter matching into the nested
aggregator loop.

==== //depot/argus/clients/clients/rabins.c#71 - /Volumes/Users/carter/argus/clients/clients/rabins.c ====
996d995
<    int retn = 0, fretn = -1, lretn = -1;
1001a1001


I get this kind of result from argus.simple.data.out and your configuration file.

thoth:clients carter$ ../bin/rabins -r ~/argus/data/argus.simple*out -f /tmp/racluster.conf \
      -M time 1d -s stime dur proto dport spkts dpkts sbytes dbytes state

                 StartTime        Dur  Proto  Dport  SrcPkts  DstPkts     SrcBytes     DstBytes State 
2012/02/13.17:48:36.592155  27.203648    tcp http         38       47         5922        44901   FIN
2012/02/13.17:48:36.589949   0.000608    udp *             2        2          148          291   CON
2012/02/13.17:48:36.589413 213.552917    arp               8        4          424          256   CON

Carter 




On

Re: rabins and racluster.conf

Carter Bullard — 2012-04-30T19:24:02

OK, well some progress is better news.   Yes, my data is generating the
same inconsistency, so I'm on it.
Carter



On Apr 30, 2012, at 2:50 PM, Mark E. Mallett wrote:

Re: rabins and racluster.conf

Mark E. Mallett — 2012-04-30T18:50:06

Not a problem.



Looks better but still not completely joyful. I applied your patches (and also
did a diff against your rabins.c to make sure that one was ok) and did a
complete remake and reinstall.

It now does the aggregation for the first few lines in my c3.conf, which
it did not do before, but still does not for the final "catch-all"
filter line.

Taking your insight that racluster should do the same thing as rabins with
one bin, here's what I find with your argus.simple.data.out file

$ racluster -n -f c3.conf -r argus.simple.data.out -s stime proto bytes dport
         StartTime  Proto   TotBytes  Dport 
   17:48:36.592155    tcp      50823 80
   17:48:36.589949    udp        439 0
   17:48:36.589413    arp        680

$ rabins -n -f c3.conf -r argus.simple.data.out -M time 1d -s stime proto bytes dport
         StartTime  Proto   TotBytes  Dport 
   17:48:36.592155    tcp      50823 80
   17:48:36.589949    udp        439 0

If you don't get the same results then maybe I've done something wrong
her

Re: rabins and racluster.conf

Carter Bullard — 2012-04-30T13:38:53

Hey Mark,
Sorry for the delayed response !!!!!   OK, here are a few patches that should fix the problem.
Looked to be a "cut, copy and no paste" style of omission in the rabins.c source.

The patch to ./common/argus_client.c is not related to your problem, but can be included,
especially if you've gotten any filter error messages when running rabins with the cluster.conf
approach.

Please give these a test, and if all is well, I'll put out the first round of argus-client-3.0.6 fixes !!!
I've included a new copy of rabins.c if you aren't comfortable with patch files.

Hope all is most excellent,

Carter

==== //depot/argus/clients/clients/rabins.c#70 - /Volumes/Users/carter/argus/clients/clients/rabins.c ====
453a454,458

==== //depot/argus/clients/common/argus_client.c#268 - /Volumes/Users/carter/argus/clients/common/argus_client.c ====
11620,11621c11620,11621
<                   if (ArgusFilterCompile (&agg->filter, agg->filterstr, ArgusParser->Oflag) < 0)
<                      ArgusLog (LOG_ERR, "ArgusNew

Re: rabins and racluster.conf

Carter Bullard — 2012-04-24T07:13:39

Hey Mark,
So there is sample data available on the web site to test these types of issues.

   http://qosient.com/argus/data

If you grab the argus.simple.data.out and do a few tests, you'll see, like I did,
that your example demonstrates a bug in rabins(), I very much hate to say.
Using your c3.conf as racluster.conf in these examples, here you go:

thoth:data carter$ racount -r argus.simple.data.out
racount   records     total_pkts     src_pkts       dst_pkts       total_bytes        src_bytes          dst_bytes
    sum   10          101            48             53             51942              6494               45448     

thoth:data carter$ racluster -f racluster.conf -r argus.simple.data.out -w - | racount
racount   records     total_pkts     src_pkts       dst_pkts       total_bytes        src_bytes          dst_bytes
    sum   4           101            48             53             51942              6494               45448    

thoth:data carter$ rabins -f racluster.conf -M time 1h -r argus.simp

Re: rabins and racluster.conf

Mark E. Mallett — 2012-04-23T21:03:01

yes, 3.0.6, on various systems.

And fantastic, that's what I hoped. Let me describe my simple tests
and perhaps somebody can tell me what dumb thing I am doing.

It looked like the racluster.conf aggregation rules could be pretty
useful so I thought I'd try it out with some simple reports where the
details aren't too important but the method is. I figured if the method
works I can go onwards from there. But I got stuck early on.

The simple test is to get some per-hour totals broken down into
potentially high-interest categories in different ways.  I chose http
and https and ssh and "everything else" as categories for the test.
Here's my file c3.conf

=======
# I've tried this on or off
RACLUSTER_PRESERVE_FIELDS=no
#
filter="tcp and dst port eq 80" model="proto dport"
filter="tcp and dst port eq 443" model="proto dport"
filter="tcp and dst port eq 22" model="proto dport"
# filter="tcp or udp" model="proto"
filter="" model="proto"
=========

I processed this with rabins doing time binning (see command below

Re: rabins and racluster.conf

Carter Bullard — 2012-04-23T19:40:52

Hey Mark,
Yes, the "-f " is for a racluster.conf file.  It should aggregate within the bin, so its as if you ran racluster on each bin as the data arrives. 

Sorry about the man page, I'll take a look later today.  Any updates will go on the web server.  You're using 3.0.6 code?

Carter

On Apr 23, 2012, at 2:46 PM, "Mark E. Mallett" <mem< at >mv.mv.com> wrote:

rabins and racluster.conf

Mark E. Mallett — 2012-04-23T18:46:52

Hi,

Is rabins supposed to be able to work with a racluster.conf file?
The man page doesn't specifically say that it does, nor does rabins give
"-f" as a usage option when you ask for --help. However:

 - the man page does say that rabins incorporates functions of racluster,
 - experimentation shows that rabins does do *something* with an
   racluster.conf style file when '-f xx.conf' is used. It's just
   that the something that it does with it doesn't really match what
   I'd think it should do.
 - I do see some code in rabins referencing the flow model file, but
   I haven't tried to follow it very far.

If the answer is "yes" then I may have followup questions and examples,
probably produced by operator error. If "no," then never mind. :-)

Yours,
-mm-

BTW, the racluster.conf (racluster(5)) man page has some holes in it;
there are some paragraphs that just end mid-