gmane.network.argus

Re: Full docs about ra output?

Matt Brown — 2012-05-25T02:33:07

Thanks Mark.

I'll grab some stuff out of the DB I populated and reply to this thread.

I'm really focusing on creating a variety of macro views and trying to
figure out how to consider 'dir' in those derived views.

I'm currently focused on the easy stuff as pivot points, source and
destination: bytes, packet count, address and port. But I am interested in
leverage other parts of the DSR as well, if they are useful. (am I using
DSR right?)

I'll spend more time reviewing other threads as well as the NSMwiki, but
any further examples of how people create macro views of the data, versus
considering solely 'dir,' would be appreciated.


Thanks,

Matt


On Thu, May 24, 2012 at 9:39 PM, Mark Poepping <poepping< at >cmu.edu> wrote:

Re: Full docs about ra output?

Mark Poepping — 2012-05-25T01:39:09

Taking a stab (trying to relieve Carter of some of the burden)...

For directionality specifically, if it's a well-defined protocol and argus saw most (if not all) of the packets from the beginning, it will know the direction, but there are many examples of ordinary and hybrid protocols where you won't necessarily know the direction in all cases: peer-to-peer, ICMP, UDP can all make it hard to understand direction - or direction might not have meaning.  Packet loss (esp. packet sampling) often causes this output, and multi-path routing will 'look like' packet loss too, depending on where you're watching and how your paths are advertised or have evolved over time.

On a simple, lightly loaded network (my house), long-running argus probes generally get the directionality right.
At my work, it's not so simple so it helps to interact with questions that we have for the data and considerations of probe location and efficiency given the use cases.

Hope that helps some, it takes a little getting used to.  If you have specific questions or confusions, it does help to snap a packet capture that displays your confusion - that way others may be work with them directly and try to help you (with no explicit promises, of course).
Mark.


From: argus-info-bounces+poepping=cmu.edu< at >lists.andrew.cmu.edu [mailto:argus-info-bounces+poepping=cmu.edu< at >lists.andrew.cmu.edu] On Behalf Of Matt Brown
Sent: Thursday, May 24, 2012 9:00 PM
To: argus-info< at >lists.andrew.cmu.edu
Subject: [ARGUS] Full docs about ra output?


Hello,

I see the man page for ra, but it seems lacking for some DSR value output.  For instance, there are somethings that aren't implicit, but appear like they should/were intended to be.

Specifically, I see this with 'dir's possible values.

The cases if confusion are <? And ?>.  How can argus "not" know the direction of the transaction "sort of?"

Thanks,

Matt

Full docs about ra output?

Matt Brown — 2012-05-25T01:00:08

Hello,

I see the man page for ra, but it seems lacking for some DSR value output.
For instance, there are somethings that aren't implicit, but appear like
they should/were intended to be.

Specifically, I see this with 'dir's possible values.

The cases if confusion are <? And ?>.  How can argus "not" know the
direction of the transaction "sort of?"

Thanks,

Matt

Re: Cisco non-flow data.

Carter Bullard — 2012-05-16T13:57:22

Hey Torbjorn,
Yes netflowV9 and full sflow support is a todo item for argus-3.0.8, which should start in June.
I restructured all the "foreign" flow support into the argus-import.c file in 3.0.6, and put a huge
push into flow-tools (100%) and sflow support (60%).  I actually have much of the netflowV9
stuff done, but the sample netflowV9 data that I was using had some errors in it, so I didn't
have it ready for the 3.0.6 release.

So, do you want to be a test site?

I'm hoping to have all of this done in a developers release by the end of June.

Carter


On May 16, 2012, at 8:01 AM, Torbjorn Wictorin wrote:

Cisco non-flow data.

Torbjorn Wictorin — 2012-05-16T12:01:46

Hello,

how are the plans regarding argus handling of cisco non-flow data?
In common/argus_import.c there is a routine ArgusParseCiscoRecordV9Data 
that I assume is supposed to do that, but it is a dummy in 3.0.6.
Any plans for the (reasonably near) future?

Torbjörn Wictorin,
Uppsala university

new client patches for cygwin compile problems

Carter Bullard — 2012-05-09T17:05:44

Gentle people,
We are now in a period where I'm fixing gotcha's after the official release of argus-3.0.6.
We now have two sets of fixes, a set of memory leaks and, now, a set of fixes for cygwin.
This looks to be the only issues right now.

The next round of released code will be argus-clients-3.0.6.1, and because
these are show stopper problems, I'll officially release this next version soon.

Until I hear from the list, my strategy for testing is to put up a single patch file
that represents the 3.0.6 to 3.0.6.1 upgrade, patch-argus-clients-3-0-6-1.diff, and
to provide an accompanying argus-clients-3.0.6.1.tar.gz file. These are only
on the developers side of the site. When all is well and good, I'll move these two
files to the stable side of the server.

As a result, I've uploaded patch-argus-clients-3-0-6-1-diff to the developers site,
and a new argus-clients-3.0.6.1.tar.gz. The patch file should be applied to the
original argus-clients-3.0.6 distribution. But, you can apply it to a previously
patch distribution, if that is your preference. Just type carriage return in response
to patch's complaints about a previously patched chunk.

Again I will release these final tarfiles and patches tomorrow being Thur if no other
issues come up.

http://qosient.com/argus/dev/argus-clients-3.0.6.1.tar.gz
http://qosient.com/argus/dev/patch-argus-clients-3-0-6-1-diff

Please send any comments / opinions / complaints / whatever to either me or the list !!!!
Hope all is most excellent !!!!!!

Carter

Re: problems using ARGUS_OUTPUT_FILE and 'filter'

Russell Fulton — 2012-05-09T04:48:50

This gets weirder -- my argus instance on other sensors are also ignoring the filter on the ARGUS_OUTPUT_FILE line too.  I just never noticed that I have a heap more flows than I should in all my argus files :(

Russell


On 9/05/2012, at 12:47 PM, Russell Fulton wrote:

problems using ARGUS_OUTPUT_FILE and 'filter'

Russell Fulton — 2012-05-09T00:47:05

When I use this filter on the command line it works as expected but when applied in the -F argus.conf I get all sorts of stuff that I does not match the filter?

argus-3.0.6.

Makes no difference if I put the filter on the FILTER = or the ARGUS_OUTPUT_FILE.

ARGUS_OUTPUT_FILE=/home/sensors/data/wireless/argus/user-data "tcp and src net ( 172.23.0.0/16 or 172.24.0.0/18 ) and not dst net ( 172.23.0.0/16 or 172.24.0.0/18 )  and dst port 80"

see attached file...

I've used this before often so I am very puzzled - on another sensor I have an identical config but with a ( 130.216.0.0/16 )  rather than the 172 addresses.

Clearly I'm doing something stupid, but what?

Any ideas?

Russell

PS:

sample of captured flows with -F argus-user-data.conf (attached)

   12:12:31.589938  e         tcp       207.46.61.90.80        ?>    130.216.181.194.57572         1         60   RST
   12:12:31.766180  e         tcp     74.125.237.102.443       ?>     130.216.215.70.1629          1         60   RST
   12:12:31.810594  e         tcp      203.97.30.168.80        ?>    130.216.211.104.64352         1         60   RST
   12:12:32.025105  e         tcp     67.195.186.237.80        ?>     130.216.180.70.63388         1         60   RST
   12:12:32.055496  e         tcp    108.166.124.101.8081      ?>     130.216.185.49.49962         1         60   RST
   12:12:32.284281  e         tcp    203.167.141.160.80        ?>     130.216.45.214.50546         1         60   RST
   12:12:32.128061  e         tcp    203.167.141.160.80        ?>    130.216.235.214.3909          1         60   RST
   12:12:32.313530  e         tcp     74.125.237.103.443       ?>     130.216.215.70.1630          2        120   RST
   12:12:32.407612  e         tcp       157.56.52.37.443       ?>    130.216.138.119.55450         1         60   RST
   12:12:32.419595  e         tcp      203.97.30.166.443       ?>    130.216.138.180.56112         2        120   RST
   12:12:33.058135  e         tcp    203.167.141.160.80        ?>     130.216.64.209.49393         1         60   RST
   12:12:33.125684  e         tcp       193.1.122.17.80        ?>     130.216.109.13.1509          1         60   RST
   12:12:33.216804  e         tcp       96.7.131.235.80        ?>     130.216.73.113.62942         1         60   RST
   12:12:32.840228  e         tcp      203.97.30.172.80        ?>    130.216.211.104.64353         1         60   RST
   
sample from
sudo /usr/sbin/argus -w  data/wireless/argus/user-data -i eth1 - tcp and src net \( 172.23.0.0/16 or 172.24.0.0/18 \) and not dst net \( 172.23.0.0/16 or 172.24.0.0/18 \)  and dst port 80

   12:30:11.657737  e         tcp      172.23.210.58.52203     ?>     205.251.203.89.80           16       1056   CON
   12:30:11.657816  e         tcp     172.23.200.226.50630     ?>    208.117.254.157.80           43       4484   CON
   12:30:11.658591  e         tcp      172.23.63.238.62230     ?>     119.31.253.194.80            6        396   CON
   12:30:11.658597  e         tcp     172.23.189.245.63528     ?>    203.167.141.154.80            4        288   CON
   12:30:11.658740  e         tcp      172.23.51.133.56828     ?>        91.121.5.64.80           72       5732   CON
   12:30:11.659003  e         tcp      172.23.191.23.49648     ?>     208.43.117.186.80          105     151586   CON
   12:30:11.659135  e         tcp      172.23.97.109.54534     ?>      183.60.157.73.80           36       2410   CON
   12:30:11.659852  e         tcp      172.24.12.211.52737     ?>    205.196.120.161.80           50       3054   CON
   12:30:11.659857  e         tcp      172.24.12.211.52681     ?>    205.196.122.242.80           51       3060   CON
   12:30:11.659868  e         tcp      172.24.12.211.52750     ?>     199.91.152.194.80           43       2670   CON
   12:30:11.659870  e         tcp      172.24.12.211.52761     ?>     199.91.152.194.80           38       2316   CON
   12:30:11.660339  e         tcp     172.23.151.151.38450     ?>     188.132.173.12.80           22       1452   FIN

updated patch for memory leaks on the server

Carter Bullard — 2012-05-08T22:11:41

Gentle people,
I have uploaded a new patch-all-memory-leak-clients-3-0-6.diff to the developers site,
which adds a memory leak fix for radium, when it is configured to generate geo labels
for flows its processing.

This fixes all known memory leak issues in argus-clients-3.0.6.
I've updated a new argus-clients-3.0.6.1.tar.gz with these new memory leak fixes.
If you could give these a try on your systems, that would be great.  

I will release these final tarfiles and patches tomorrow or Thur.

   http://qosient.com/argus/dev/argus-clients-3.0.6.1.tar.gz
   http://qosient.com/argus/dev/patch-all-memory-leak-clients-3-0-6.diff

Apply the patch by being in the parent directory that contains argus-clients-3.0.6,
and then:

   % patch -p0 < patch-memory-leak-clients-3-0-6.diff

For those that have a patch that is sensitive to RCS, SCCS, Perforce, whatever, try:

   % patch -g0 -p0 < patch-memory-leak-clients-3-0-6.diff

The you will just need to make, as there are no configure.ac changes.

   % cd argus-clients-3.0.6; make

The patch will upgrade argus-clients-3.0.6 to argus-clients-3.0.6.1.

If you had already patch your files, you should still be able to use the new patch
file to add the changes.  It may complain a bit about already having the patch, but
that would be fine.

Carter

patch file for rabins and rastream memory leak fix

Carter Bullard — 2012-05-07T16:45:31

Gentle people,
I have uploaded  argus-clients-3.0.6.1 to the developers site, and I
have uploaded a patch, for those that like patches, that fixes the reported
problems with rabins and rastream.  The fixes addresses the "-f racluster.conf"
issues with rabins.1, as well as the memory leak problems with long
term use of rabins and rastream, which occur primarily with ARGUS_EVENT
records.  

Apply the patch by being in the parent directory that contains argus-clients-3.0.6,
and then:

   % patch -p0 < patch-memory-leak-clients-3-0-6.diff

For those that have a patch that is sensitive to RCS, SCCS, Perforce, whatever, try:

   % patch -g0 -p0 < patch-memory-leak-clients-3-0-6.diff

The you will just need to make, as there are no configure.ac changes.

   % cd argus-clients-3.0.6; make

The patch will upgrade argus-clients-3.0.6 to argus-clients-3.0.6.1.

If you use rastream.1, you will want to upgrade to argus-3.0.6.1.
For those that want to use rabins.1 with the "-f racluster.conf", you will also
need to upgrade.

Please take a look at these, and if you have any issues, please holler !!!!
I'll be "releasing" argus-clients-3.0.6.1 after the patch has gone through some
testing, possibly at the end of the week.

   http://qosient.com/argus/dev/argus-clients-3.0.6.1.tar.gz

If you encounter any problems, don't hesitate to email to the list !!!!

Hope all is most excellent !!!!! 

Carter

Carter Bullard
CEO/President
QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax

argus-clients-3.0.6 rabins, rastream memory leak

Carter Bullard — 2012-05-02T22:57:16

Gentle people,
We've got a memory leak in the newest versions of rabins.1 and rastream.1.
I'm working on them now, but I have confirmed that long lived rastream processes
do collection the memory.  If you are using rastream.1, please consider testing
the patches when I make them available.

Should have a fix for you later tomorrow, which I will send to the list.

Hope all is most excellent,
Carter

Re: rabins and racluster.conf

Mark E. Mallett — 2012-04-30T19:58:03

That does seem to fix it for this test.  Thanks!

mm

Re: rabins and racluster.conf

Carter Bullard — 2012-04-30T19:50:27

Hey Mark,
May not be a complete solution still, but give this additional patch a try.
This moves the variable declarations for filter matching into the nested
aggregator loop.

==== //depot/argus/clients/clients/rabins.c#71 - /Volumes/Users/carter/argus/clients/clients/rabins.c ====
996d995
<    int retn = 0, fretn = -1, lretn = -1;
1001a1001


I get this kind of result from argus.simple.data.out and your configuration file.

thoth:clients carter$ ../bin/rabins -r ~/argus/data/argus.simple*out -f /tmp/racluster.conf \
      -M time 1d -s stime dur proto dport spkts dpkts sbytes dbytes state

                 StartTime        Dur  Proto  Dport  SrcPkts  DstPkts     SrcBytes     DstBytes State 
2012/02/13.17:48:36.592155  27.203648    tcp http         38       47         5922        44901   FIN
2012/02/13.17:48:36.589949   0.000608    udp *             2        2          148          291   CON
2012/02/13.17:48:36.589413 213.552917    arp               8        4          424          256   CON

Carter 




On Apr 30, 2012, at 2:50 PM, Mark E. Mallett wrote:

Re: rabins and racluster.conf

Carter Bullard — 2012-04-30T19:24:02

OK, well some progress is better news.   Yes, my data is generating the
same inconsistency, so I'm on it.
Carter



On Apr 30, 2012, at 2:50 PM, Mark E. Mallett wrote:

Re: rabins and racluster.conf

Mark E. Mallett — 2012-04-30T18:50:06

Not a problem.



Looks better but still not completely joyful. I applied your patches (and also
did a diff against your rabins.c to make sure that one was ok) and did a
complete remake and reinstall.

It now does the aggregation for the first few lines in my c3.conf, which
it did not do before, but still does not for the final "catch-all"
filter line.

Taking your insight that racluster should do the same thing as rabins with
one bin, here's what I find with your argus.simple.data.out file

$ racluster -n -f c3.conf -r argus.simple.data.out -s stime proto bytes dport
         StartTime  Proto   TotBytes  Dport 
   17:48:36.592155    tcp      50823 80
   17:48:36.589949    udp        439 0
   17:48:36.589413    arp        680

$ rabins -n -f c3.conf -r argus.simple.data.out -M time 1d -s stime proto bytes dport
         StartTime  Proto   TotBytes  Dport 
   17:48:36.592155    tcp      50823 80
   17:48:36.589949    udp        439 0

If you don't get the same results then maybe I've done something wrong
here.

This is a little clearer perhaps, using one of my own files with data
just within one day:

$ racluster -n -f c3.conf -r infile -s stime proto bytes dport
         StartTime  Proto   TotBytes  Dport 
   00:20:53.200000    tcp   70126187 80
   00:21:44.447000    tcp   71531614 443
   00:26:50.436603    tcp    3169863 22
   00:20:28.010000    udp   12408866 0
   00:20:53.221000    tcp  168594597 0
   00:25:22.955191    arp     120540
   00:33:44.486000   icmp     200541 0x0000
   00:57:58.110098    llc        300 0
   08:36:56.807000   igmp         28

$ rabins -n -f c3.conf -r infile -M time 1d -s stime proto bytes dport
         StartTime  Proto   TotBytes  Dport 
   00:20:53.200000    tcp   70126187 80
   00:21:44.447000    tcp   71531614 443
   00:26:50.436603    tcp    3169863 22
   00:20:28.010000    udp   12408866 0
   00:20:53.221000    tcp  168594597 0

You can see that the catch-all filter rule is not getting anything with
rabins, whereas it is with racluster.  Just for completeness, the c3.conf
file is:

$ cat c3.conf
# racluster.conf file, testing using this with rabins to
#  get some specific usage breakdowns.  Aggregation by proto and dport
#  may let us get protocol-based usage for some protocols.
RACLUSTER_PRESERVE_FIELDS=no
#
filter="tcp and dst port eq 80" model="proto dport"
filter="tcp and dst port eq 443" model="proto dport"
filter="tcp and dst port eq 22" model="proto dport"
filter="tcp or udp" model="proto"
filter="" model="proto"

As an aside: in the commands above I am displaying enough (proto and
dport) to try to show which filter rules matched. Is there a way to add
a tag of some sort, that would make it easier to extract aggregated data
for that rule later? (Perhaps even create a tag and aggregate around the
tag value?)

-mm-

Re: rabins and racluster.conf

Carter Bullard — 2012-04-30T13:38:53

Hey Mark,
Sorry for the delayed response !!!!!   OK, here are a few patches that should fix the problem.
Looked to be a "cut, copy and no paste" style of omission in the rabins.c source.

The patch to ./common/argus_client.c is not related to your problem, but can be included,
especially if you've gotten any filter error messages when running rabins with the cluster.conf
approach.

Please give these a test, and if all is well, I'll put out the first round of argus-client-3.0.6 fixes !!!
I've included a new copy of rabins.c if you aren't comfortable with patch files.

Hope all is most excellent,

Carter

==== //depot/argus/clients/clients/rabins.c#70 - /Volumes/Users/carter/argus/clients/clients/rabins.c ====
453a454,458

==== //depot/argus/clients/common/argus_client.c#268 - /Volumes/Users/carter/argus/clients/common/argus_client.c ====
11620,11621c11620,11621
<                   if (ArgusFilterCompile (&agg->filter, agg->filterstr, ArgusParser->Oflag) < 0)
<                      ArgusLog (LOG_ERR, "ArgusNewAggregator ArgusFilterCompile returned error");
---



On Apr 24, 2012, at 3:13 AM, Carter Bullard wrote:

Re: rabins and racluster.conf

Carter Bullard — 2012-04-24T07:13:39

Hey Mark,
So there is sample data available on the web site to test these types of issues.

   http://qosient.com/argus/data

If you grab the argus.simple.data.out and do a few tests, you'll see, like I did,
that your example demonstrates a bug in rabins(), I very much hate to say.
Using your c3.conf as racluster.conf in these examples, here you go:

thoth:data carter$ racount -r argus.simple.data.out
racount   records     total_pkts     src_pkts       dst_pkts       total_bytes        src_bytes          dst_bytes
    sum   10          101            48             53             51942              6494               45448     

thoth:data carter$ racluster -f racluster.conf -r argus.simple.data.out -w - | racount
racount   records     total_pkts     src_pkts       dst_pkts       total_bytes        src_bytes          dst_bytes
    sum   4           101            48             53             51942              6494               45448    

thoth:data carter$ rabins -f racluster.conf -M time 1h -r argus.simple.data.out -w - | racount
racount   records     total_pkts     src_pkts       dst_pkts       total_bytes        src_bytes          dst_bytes
    sum   5           85             38             47             50823              5922               44901  


So, rabins isn't generating correct data in this case.  Not good.  More detailed output is:

thoth:data carter$ ra -r argus.simple.data.out
                 StartTime        Dur          SrcId      Flgs  Proto            SrcAddr  Sport   Dir            DstAddr  Dport  SrcPkts  DstPkts     SrcBytes     DstBytes State 
2012/02/13.18:03:53.527324   0.000000   192.168.0.68              man                  0.     0                        0.     0        0        0            0            0   STA
2012/02/13.17:48:36.589413 213.551453   192.168.0.68  M           arp       192.168.0.68          who       192.168.0.66               3        3          126          192   CON
2012/02/13.17:48:36.589949   0.000000   192.168.0.68  e           udp       192.168.0.68.50251    <->       192.168.0.66.domain        1        1           77          155   CON
2012/02/13.17:48:36.590557   0.000000   192.168.0.68  e           udp       192.168.0.68.53404    <->       192.168.0.66.domain        1        1           71          136   CON
2012/02/13.17:48:36.590954   0.000000   192.168.0.68  M           arp       192.168.0.68          who        192.168.0.1               1        1           42           64   CON
2012/02/13.17:48:36.591391 213.550949   192.168.0.68  e           arp       192.168.0.66          who        192.168.0.1               4        0          256            0   INT
2012/02/13.17:48:36.592155  27.203621   192.168.0.68  e           tcp       192.168.0.68.60245     ->      128.2.129.188.http         12       15         2314        15894   FIN
2012/02/13.17:48:36.632662  27.163141   192.168.0.68  e           tcp       192.168.0.68.60246     ->      216.92.14.146.http         10       14         1001        14167   FIN
2012/02/13.17:48:36.705481  27.090235   192.168.0.68  e           tcp       192.168.0.68.60247     ->      128.2.129.188.http         10       13         1433        13292   FIN
2012/02/13.17:48:36.705669  27.090014   192.168.0.68  e i         tcp       192.168.0.68.60248     ->      128.2.129.188.http          6        5         1174         1548   FIN

thoth:data carter$ racluster -r argus.simple.data.out
                 StartTime        Dur          SrcId      Flgs  Proto            SrcAddr  Sport   Dir            DstAddr  Dport  SrcPkts  DstPkts     SrcBytes     DstBytes State 
2012/02/13.17:48:36.589413 213.551453   192.168.0.68  M           arp       192.168.0.68          who       192.168.0.66               3        3          126          192   CON
2012/02/13.17:48:36.589949   0.000000   192.168.0.68  e           udp       192.168.0.68.50251    <->       192.168.0.66.domain        1        1           77          155   CON
2012/02/13.17:48:36.590557   0.000000   192.168.0.68  e           udp       192.168.0.68.53404    <->       192.168.0.66.domain        1        1           71          136   CON
2012/02/13.17:48:36.590954   0.000000   192.168.0.68  M           arp       192.168.0.68          who        192.168.0.1               1        1           42           64   CON
2012/02/13.17:48:36.591391 213.550949   192.168.0.68  e           arp       192.168.0.66          who        192.168.0.1               4        0          256            0   INT
2012/02/13.17:48:36.592155  27.203621   192.168.0.68  e           tcp       192.168.0.68.60245     ->      128.2.129.188.http         12       15         2314        15894   FIN
2012/02/13.17:48:36.632662  27.163141   192.168.0.68  e           tcp       192.168.0.68.60246     ->      216.92.14.146.http         10       14         1001        14167   FIN
2012/02/13.17:48:36.705481  27.090235   192.168.0.68  e           tcp       192.168.0.68.60247     ->      128.2.129.188.http         10       13         1433        13292   FIN
2012/02/13.17:48:36.705669  27.090014   192.168.0.68  e i         tcp       192.168.0.68.60248     ->      128.2.129.188.http          6        5         1174         1548   FIN

rabins -M time 1h -r argus.simple.data.out
                 StartTime        Dur          SrcId      Flgs  Proto            SrcAddr  Sport   Dir            DstAddr  Dport  SrcPkts  DstPkts     SrcBytes     DstBytes State 
2012/02/13.17:48:36.589413 213.551453   192.168.0.68  M           arp       192.168.0.68          who       192.168.0.66               3        3          126          192   CON
2012/02/13.17:48:36.589949   0.000000   192.168.0.68  e           udp       192.168.0.68.50251    <->       192.168.0.66.domain        1        1           77          155   CON
2012/02/13.17:48:36.590557   0.000000   192.168.0.68  e           udp       192.168.0.68.53404    <->       192.168.0.66.domain        1        1           71          136   CON
2012/02/13.17:48:36.590954   0.000000   192.168.0.68  M           arp       192.168.0.68          who        192.168.0.1               1        1           42           64   CON
2012/02/13.17:48:36.591391 213.550949   192.168.0.68  e           arp       192.168.0.66          who        192.168.0.1               4        0          256            0   INT
2012/02/13.17:48:36.592155  27.203621   192.168.0.68  e           tcp       192.168.0.68.60245     ->      128.2.129.188.http         12       15         2314        15894   FIN
2012/02/13.17:48:36.632662  27.163141   192.168.0.68  e           tcp       192.168.0.68.60246     ->      216.92.14.146.http         10       14         1001        14167   FIN
2012/02/13.17:48:36.705481  27.090235   192.168.0.68  e           tcp       192.168.0.68.60247     ->      128.2.129.188.http         10       13         1433        13292   FIN
2012/02/13.17:48:36.705669  27.090014   192.168.0.68  e i         tcp       192.168.0.68.60248     ->      128.2.129.188.http          6        5         1174         1548   FIN

So, without a racluster.conf file, all the programs do the right thing.
With your racluster configuration, we get this using racluster.1, which is correct behavior:

racluster -f racluster.conf -M time 1h -r argus.simple.data.out
                 StartTime        Dur          SrcId      Flgs  Proto            SrcAddr  Sport   Dir            DstAddr  Dport  SrcPkts  DstPkts     SrcBytes     DstBytes State 
2012/02/13.17:48:36.592155  27.203648   192.168.0.68  e g         tcp       192.168.0.68.*         ->        128.0.0.0/1.http         38       47         5922        44901   FIN
2012/02/13.17:48:36.589413 213.552917   192.168.0.68  M           arp            0.0.0.0          who            0.0.0.0               8        4          424          256   CON
2012/02/13.17:48:36.589949   0.000608   192.168.0.68  e           udp            0.0.0.0.*        <->            0.0.0.0.*             2        2          148          291   CON

However, rabins, when run with the "-M time 1h" option, should generate the exact same output as racluster (should have only one bin) but:

thoth:data carter$ rabins -f racluster.conf -M time 1h -r argus.simple.data.out               
                 StartTime        Dur          SrcId      Flgs  Proto            SrcAddr  Sport   Dir            DstAddr  Dport  SrcPkts  DstPkts     SrcBytes     DstBytes State 
2012/02/13.17:48:36.592155  27.203621   192.168.0.68  e           tcp       192.168.0.68.60245     ->      128.2.129.188.http         12       15         2314        15894   FIN
2012/02/13.17:48:36.632662  27.163141   192.168.0.68  e           tcp       192.168.0.68.60246     ->      216.92.14.146.http         10       14         1001        14167   FIN
2012/02/13.17:48:36.705481  27.090235   192.168.0.68  e           tcp       192.168.0.68.60247     ->      128.2.129.188.http         10       13         1433        13292   FIN
2012/02/13.17:48:36.705669  27.090014   192.168.0.68  e i         tcp       192.168.0.68.60248     ->      128.2.129.188.http          6        5         1174         1548   FIN

So something happened to the arp and udp traffic.  I'll have to fix this, hopefully by Thurs   :O(

Carter

On Apr 23, 2012, at 5:03 PM, Mark E. Mallett wrote:

Re: rabins and racluster.conf

Mark E. Mallett — 2012-04-23T21:03:01

yes, 3.0.6, on various systems.

And fantastic, that's what I hoped. Let me describe my simple tests
and perhaps somebody can tell me what dumb thing I am doing.

It looked like the racluster.conf aggregation rules could be pretty
useful so I thought I'd try it out with some simple reports where the
details aren't too important but the method is. I figured if the method
works I can go onwards from there. But I got stuck early on.

The simple test is to get some per-hour totals broken down into
potentially high-interest categories in different ways.  I chose http
and https and ssh and "everything else" as categories for the test.
Here's my file c3.conf

=======
# I've tried this on or off
RACLUSTER_PRESERVE_FIELDS=no
#
filter="tcp and dst port eq 80" model="proto dport"
filter="tcp and dst port eq 443" model="proto dport"
filter="tcp and dst port eq 22" model="proto dport"
# filter="tcp or udp" model="proto"
filter="" model="proto"
=========

I processed this with rabins doing time binning (see command below),
expecting this to give me one record of aggregated data for http, one
for https, one for ssh per time bin, and then other records in each bin
aggregated only by protocol.

When I run rabins I either write the argus records to a file and look at
that with some other ra* utility or just look at the output directly;
the result seems to be the same.  This is the sort of command I use to
time bin and look at the output:

  rabins -n -f c3.conf -M zero -M time 1h -r input-file \
      -s stime proto dport pkts bytes dur:12 | less

With that command, I do not seem to get any aggregation at all. I get a
lot of individual flow records matching the tcp statements (http, https,
ssh), and no records for any other protocol or ports.  This last part
surprises me most because I thought the c3.conf file will catch
everything, even if the aggregation doesn't happen.

If I add a "model" set on the command line, i.e. an '-m' switch like this:

  rabins -n -f c3.conf -M zero -M time 1h -m proto dport -r input-file \
      -s stime proto dport pkts bytes dur:12 | less

then I get the per-bin aggregation I expected of the http, https, and ssh
traffic, but I still get no other output (except what's generated by the
"-M zero" for otherwise empty bins).  Here I would still expect the last
line in c3.conf to catch some things and produce things aggregated
by "proto".

If I uncomment the next-to-last filter line, then I do see some other
records show up; but for the most part they also appear unaggregated
except by flow.

I could go on for a while but this is already long enough and probably
quite enough to introduce my confusion.  One of the issues seems to be
that the "model" section of each line isn't being applied, and the only
model that gets applied is the one in '-m' on the command line.  But I
don't think that's the only issue.  There's probably some dots that I
haven't connected..

Yours,
-mm-

Re: rabins and racluster.conf

Carter Bullard — 2012-04-23T19:40:52

Hey Mark,
Yes, the "-f " is for a racluster.conf file.  It should aggregate within the bin, so its as if you ran racluster on each bin as the data arrives. 

Sorry about the man page, I'll take a look later today.  Any updates will go on the web server.  You're using 3.0.6 code?

Carter

On Apr 23, 2012, at 2:46 PM, "Mark E. Mallett" <mem< at >mv.mv.com> wrote:

rabins and racluster.conf

Mark E. Mallett — 2012-04-23T18:46:52

Hi,

Is rabins supposed to be able to work with a racluster.conf file?
The man page doesn't specifically say that it does, nor does rabins give
"-f" as a usage option when you ask for --help. However:

 - the man page does say that rabins incorporates functions of racluster,
 - experimentation shows that rabins does do *something* with an
   racluster.conf style file when '-f xx.conf' is used. It's just
   that the something that it does with it doesn't really match what
   I'd think it should do.
 - I do see some code in rabins referencing the flow model file, but
   I haven't tried to follow it very far.

If the answer is "yes" then I may have followup questions and examples,
probably produced by operator error. If "no," then never mind. :-)

Yours,
-mm-

BTW, the racluster.conf (racluster(5)) man page has some holes in it;
there are some paragraphs that just end mid-

argus[-clients]-3.0.6 officially released

Carter Bullard — 2012-04-18T18:49:30

Gentle people,
We've released argus-3.0.6 and its clients.  There were slight modifications
to the previous tarfiles that were uploaded in the last few days.  One significant
change regarding aggregation was inserted, making field preservation the
default, and minor tweaking to man pages.

While argus-3.0.6 is predominately a bug fix release, there are hundreds of improvements,
so please do consider upgrading.

Please give a round of applause to all involved in making this one of the best
efforts to improve argus.  Of course if you discover any issues, please send
email to the list !!!

Now on to argus-3.0.7.1.
Hope all is most excellent,

Carter