http://blog.gmane.org/gmane.network.argus hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://comments.gmane.org/gmane.network.argus/9355 <pre>Gentle people, I've uploaded argus-clients-3.0.7.10 to the development server. This fixes all the known bugs reported, except the report today regarding rastream(). This includes, raservices() signature limits, fixes to the " delayed " filter compiler reporting syntax errors, removal of the syslog error for "timestamps wayyyy out of range" error reports, radump() problems when there isn't any user data, fixes to flow key processing when using RMON aggregation, and CIDR address issues. http://qosient.com/argus/dev/argus-clients-3.0.7.10.tar.gz Please give this version a try. I'd like to make it are the next round the nex argus-clients-3.0.8 stable version. Hope all is most excellent, Carter </pre> Carter Bullard 2013-05-21T03:22:17 http://comments.gmane.org/gmane.network.argus/9354 <pre>Hello All: I am having a problem with rastream that's manifested itself when using the -f "shell script executor" argument to rotate files at 'time 1h&lt;x-apple-data-detectors://0&gt; '. If I run rastream as a daemon, then the script seems to run before the "hour" is over (and the "hour" is over at the incorrect time): # rastream -d -S 127.0.0.1:561 -B 15s -M time 1h&lt;x-apple-data-detectors://1&gt; -w /var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh A few hours' files look like: # ls --full-time /var/opt/argus/2013-05-18 total 3728 -rw-r--r--. 1 root 18752 2013-05-18 01:00:59.556839459 -0400 argus_00:00:00&lt;x-apple-data-detectors://3&gt; -rw-r--r--. 1 root 160607 2013-05-18 01:00:17.793286000 -0400 argus_00:00:00.gz -rw-r--r--. 1 root 12068 2013-05-18 02:00:59.619364943 -0400 argus_01:00:00&lt;x-apple-data-detectors://6&gt; -rw-r--r--. 1 root 163409 2013-05-18 02:00:17.943700000 -0400 argus_01:00:00.gz -rw-r--r--. 1 root 9032 2013-05-1803:01:00.579907536 -0400 argus_02:00:00&lt;x-apple-data-detectors://9&gt; -rw-r--r--. 1 root 122920 2013-05-18 03:00:17.834317000 -0400 argus_02:00:00.gz -rw-r--r--. 1 root 22092 2013-05-18 04:01:00.698357771 -0400 argus_03:00:00&lt;x-apple-data-detectors://12&gt; -rw-r--r--. 1 root 122002 2013-05-18 04:00:17.835675000 -0400 argus_03:00:00.gz -rw-r--r--. 1 root 17704 2013-05-18 05:01:00.450618851 -0400 argus_04:00:00&lt;x-apple-data-detectors://15&gt; -rw-r--r--. 1 root 133212 2013-05-18 05:00:17.742040000 -0400 argus_04:00:00.gz -rw-r--r--. 1 root 14592 2013-05-18 06:00:54.886285774 -0400 argus_05:00:00&lt;x-apple-data-detectors://18&gt; -rw-r--r--. 1 root 160523 2013-05-18 06:00:17.562776000 -0400 argus_05:00:00.gz It looks like the gzipped file is last modified before the hour file, which leads me to believe that rastream isn't finished writing to the argus file before -f[] is executed. If I run rastream as follows, I have no problem: # nohup rastream -S 127.0.0.1:561 -B 15s -M time 1h&lt;x-apple-data-detectors://20&gt; -w /var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh &amp; A few hours' files look like: # ls --full-time /var/opt/argus/2013-05-20 total 5372 -rw-r--r--. 1 root 217245 2013-05-20 10:00:17.573908000 -0400 argus_09:00:00.gz -rw-r--r--. 1 root 6377 2013-05-2011:00:17.762140000 -0400 argus_10:00:00.gz -rw-r--r--. 1 root 9269 2013-05-2011:38:08.879810000 -0400 argus_11:00:00.gz -rw-r--r--. 1 root 313 2013-05-2013:00:17.170958000 -0400 argus_12:00:00.gz -rw-r--r--. 1 root 8965 2013-05-2014:00:17.540889000 -0400 argus_13:00:00.gz (early day over there) I have verified that the system time is correct and ntpd is running properly. Clients are 3.0.7.9. Carter, do you have any ideas? Thanks, Matt </pre> Matt Brown 2013-05-20T18:42:28 http://comments.gmane.org/gmane.network.argus/9341 <pre>Carter, Thank you so much for your analysis of the APT1 threats. Those emails were extremely educational. I wanted to pick your brain about a couple of things related to anomaly detection... We backhaul all remote offices through a central network that Argus can monitor. Since those remote offices use DHCP, it's hard for Argus to build a reliable model of "normal" behavior by IP address. And it can't see the MAC addresses of flows from those remote offices. What's the best approach for anomaly detection in that kind of scenario? Do you look at the producer/consumer metrics of the whole DHCP subnet and then compare individual flows against that baseline? What kind of anomaly detection strategy do you use for environments where you have farms of different functional roles - web, MTA, database, etc.? Do you recommend building a behavioral model by individual host or would you compare individual hosts against a baseline for that class of system? Thanks. Craig </pre> Craig Merchant 2013-05-16T06:29:38 http://comments.gmane.org/gmane.network.argus/9339 <pre>Hello all, I took a day's worth of argus data and, as suggested on http://thread.gmane.org/gmane.network.argus/6228/focus=6234, I analyzed it with rauserdata as follows: #racluster -r * -w day.cache #rauserdata -r day.cache &gt; /tmp/raservices.conf I then inspected /tmp/raservices.conf and it's messy (lots of single lines with arbirary ports, likely sport maybe rpc?), but I figured why not give raservices a shot: #racluster -r * -w - | raservices -f raservices.conf I receive the following error: raservices[21315]: 16:51:00.727719 RaCreateSrvEntry: format error Service: http I straced the process, and I see no occurances of "http" in the output (other than the writev()); the data appears to be read correctly until a blank line is read [read(3, "", 4096) = 0]: read(3, "\" \n\nService: 48956 "..., 4096) = 4096 read(3, "...xxxxxx" dst ="..., 4096) = 4096 read(3, "xxxx"..., 4096) = 689 read(3, "", 4096) = 0 close(3) = 0 munmap(0xb766e000, 4096) = 0 gettimeofday({1368651683, 272271}, NULL) = 0 time(NULL) = 1368651683 writev(2, [{"raservices[21523]: 17:01:23.2722"..., 79}, {"\n", 1}], 2raservices[21523]: 17:01:23.272271 RaCreateSrvEntry: format error Service: http ) = 80 Any idea on why this would be? Is my data processing flow incorrect? Both clients are 3.0.7.8. Thanks, Matt </pre> Matt Brown 2013-05-15T21:55:32 http://comments.gmane.org/gmane.network.argus/9332 <pre>Hello all/Carter, I am using rastream to write argus data to files. When I query these files using ra or racluster, suser and duser are not returning any data. I'm guessing it isn't being written by rastream which has been started as follows: rastream -S 127.0.0.1:561 -B 15s -M time 1h -w /var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh How do I use rastream to record N bytes of suser and duser? Thanks, Matt </pre> Matt Brown 2013-05-14T14:50:57 http://comments.gmane.org/gmane.network.argus/9329 <pre>Carter, It looks like changing the CIDR notation parameter may be masking the problem rather than fixing it. I found a bunch of MySQL error messages in the system message log so I restarted rasqlinsert with -D 3. For some reason redirection of STDERR prevented the program from running so I just used cut and paste into a file. --Dave [root&lt; at &gt;monolith ~]# /usr/local/bin/rasqlinsert -M time 1d -M cache -S localhost:9603 -w mysql://argus:argus&lt; at &gt;localhost/argus/matrix_%Y_%m_%d -m srcid matrix proto -s ltime dur srcid saddr daddr proto bytes -D3 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.340 RaTopNewProcess(0x718dc010) returns 0x1d557d0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.398 RaMySQLInit () RaSource (null) RaArchive (null) RaFormat (null) rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 ArgusInitAddrtoname (0x7f41718dc010, 0x0, 0x0) rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 ArgusParseInit(0x7f41718dc010, NULL) rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 main: reading files completed rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 Trying 127.0.0.1 port 9603 Expecting Argus records rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 connected rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.405 ArgusGetServerSocket (0x7f41715ec010) returning 4 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.413 ArgusReadConnection() read 16 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.414 ArgusParseInit(0x7f41718dc010 0x7f41715ec010 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.414 ArgusWriteConnection(0x715ec010, 0xb9eb56e0, 7) returning 7 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.414 ArgusReadConnection(0x715ec010, 2) returning 1 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.414 ArgusReadStream(0x7f41718dc010) starting rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.674 RaProcessSplitOptions(matrix_2013_05_14, 4096, 0x715ec620): returns 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.685 ArgusCreateSQLSaveTable (matrix_2013_05_14) returning rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.686 ArgusProcessThisRecord () sql query SELECT record FROM matrix_2013_05_14 WHERE srcid="69.113.13.218" and saddr="10.1.1.45" and daddr="10.1.1.60" and proto="tcp" rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.686 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.686 ArgusClientTimeout ArgusTotalSearches 1 ArgusTotalSQLUpdates 1 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.686 ArgusSQLQuery (UPDATE matrix_2013_05_14 SET ltime="1368528902.272",dur="39244.605",bytes="1204124179",record="..." WHERE srcid="69.113.13.218" and saddr="0.0.0.0" and daddr="0.0.0.0" and proto="ip") rasqlinsert[12009]: 2013-05-14-10:55:02.687 mysql_real_query error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"69.113.13.21' at line 1 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.837 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.837 ArgusClientTimeout ArgusTotalSearches 1 ArgusTotalSQLUpdates 1 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.987 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:02.987 ArgusClientTimeout ArgusTotalSearches 1 ArgusTotalSQLUpdates 1 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.120 ArgusProcessThisRecord () sql query SELECT record FROM matrix_2013_05_14 WHERE srcid="69.113.13.218" and saddr="10.1.1.50" and daddr="209.177.156.55" and proto="tcp" rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.121 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.121 ArgusClientTimeout ArgusTotalSearches 2 ArgusTotalSQLUpdates 1 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.159 ArgusProcessThisRecord () sql query SELECT record FROM matrix_2013_05_14 WHERE srcid="69.113.13.218" and saddr="10.1.1.10" and daddr="10.1.1.50" and proto="tcp" rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.160 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.160 ArgusClientTimeout ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.160 ArgusSQLQuery (UPDATE matrix_2013_05_14 SET ltime="1368528898.093",dur="39240.430",bytes="1204115615",record="..." WHERE srcid="69.113.13.218" and saddr="0.0.0.0" and daddr="0.0.0.0" and proto="ip") rasqlinsert[12009]: 2013-05-14-10:55:03.160 mysql_real_query error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"69.113.13.218' at line 1 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.160 ArgusSQLQuery (UPDATE matrix_2013_05_14 SET ltime="1368528901.670",dur="39244.008",bytes="1204183953",record="..." WHERE srcid="69.113.13.218" and saddr="0.0.0.0" and daddr="0.0.0.0" and proto="ip") rasqlinsert[12009]: 2013-05-14-10:55:03.160 mysql_real_query error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"69.113.13.' at line 1 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.310 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.310 ArgusClientTimeout ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.460 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.460 ArgusClientTimeout ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.611 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.611 ArgusClientTimeout ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.761 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.761 ArgusClientTimeout ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.911 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:03.911 ArgusClientTimeout ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.061 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.061 ArgusClientTimeout ArgusTotalSearches 3 ArgusTotalSQLUpdates 3 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.138 ArgusProcessThisRecord () sql query SELECT record FROM matrix_2013_05_14 WHERE srcid="69.113.13.218" and saddr="10.1.1.45" and daddr="10.1.1.68" and proto="tcp" rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.139 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.139 ArgusClientTimeout ArgusTotalSearches 4 ArgusTotalSQLUpdates 3 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.289 ArgusRefreshDisplay (0x718dc010) screen 0 display 0 rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.289 ArgusClientTimeout ArgusTotalSearches 4 ArgusTotalSQLUpdates 4 written 0 bytes rasqlinsert[12009.40179871417f0000]: 2013-05-14-10:55:04.289 ArgusSQLQuery (UPDATE matrix_2013_05_14 SET ltime="1368528899.219",dur="39241.555",bytes="1204115753",record="..." WHERE srcid="69.113.13.218" and saddr="0.0.0.0" and daddr="0.0.0.0" and proto="ip") rasqlinsert[12009]: 2013-05-14-10:55:04.289 mysql_real_query error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"69.113.13.' at line 1 </pre> Dave Edelman 2013-05-14T11:08:44 http://comments.gmane.org/gmane.network.argus/9323 <pre>I have a single instance of argus that has been running for years and creating hourly files of the flow data. On a daily basis, I copy a day's worth of the flow files to a second system where I run rqasqlinsert in various flavors to create several different tables. I finally decided to use radium and rastream to do this the right way (but I didn't stop the local file creation, just to be safe.) The details follow but it looks like there is a toxic interaction between RA_CIDR_ADDRESS_FORMAT="yes" in ~/.rarc and rasqlinsert using -S from a radium instance in client version 3.0.7.8 and possibly earlier. --------- Danger beyond this point are the gory details -------------------------------- The original method worked very well: A typical MySQL table queried for all 10.1.1.10 activity gave reasonable results (I learned not to select the record blob :-) ) mysql&gt; select ltime, dur, saddr, daddr, proto, bytes from matrix_2013_04_09 where saddr = '10.1.1.10' or daddr = '10.1.1.10'; +-------------------+--------------+------------+-----------------+-------+- ------------+ | ltime | dur | saddr | daddr | proto | bytes | +-------------------+--------------+------------+-----------------+-------+- ------------+ | 1365551889.364000 | 86287.953000 | 10.1.1.10 | 10.1.1.60 | tcp | 843282 | | 1365551948.803000 | 86347.195000 | 10.1.1.10 | 10.1.1.46 | tcp | 783536 | | 1365551993.490000 | 86391.336000 | 10.1.1.10 | 10.1.1.45 | tcp | 16050742461 | | 1365551986.978000 | 86381.195000 | 10.1.1.10 | 10.1.1.50 | tcp | 3962458654 | | 1365551957.703000 | 86340.031000 | 10.1.1.10 | 255.255.255.255 | udp | 1346514 | | 1365551992.462000 | 86374.555000 | 10.1.1.10 | 10.1.1.50 | udp | 1667967 | | 1365551992.462000 | 86369.734000 | 10.1.1.10 | 10.1.1.50 | icmp | 1426656 | | 1365551938.461000 | 86280.000000 | 10.1.1.45 | 10.1.1.10 | arp | 157312 | | 1365551964.909000 | 86302.922000 | 10.1.1.50 | 10.1.1.10 | arp | 152960 | | 1365551953.810000 | 86274.445000 | 10.1.1.46 | 10.1.1.10 | arp | 107904 | | 1365551889.353000 | 86206.047000 | 10.1.1.60 | 10.1.1.10 | arp | 108544 | | 1365551868.423000 | 86055.141000 | 10.1.1.10 | 10.1.1.12 | udp | 30672 | | 1365551873.421000 | 86055.141000 | 10.1.1.10 | 10.1.1.12 | arp | 36352 | | 1365551639.982000 | 85736.523000 | 10.1.1.10 | 10.1.1.127 | udp | 61560 | | 1365551920.751000 | 85978.414000 | 10.1.1.10 | 10.1.1.50 | arp | 49408 | | 1365551560.220000 | 77317.164000 | 10.1.1.10 | 10.1.1.101 | udp | 115032 | | 1365551560.220000 | 77310.164000 | 10.1.1.10 | 10.1.1.101 | arp | 29696 | | 1365550944.567000 | 76398.391000 | 10.1.1.10 | 224.0.0.251 | udp | 3703 | | 1365529023.282000 | 54396.770000 | 10.1.1.10 | 10.1.1.101 | tcp | 1032139 | | 1365475483.795000 | 786.242000 | 10.1.1.101 | 10.1.1.10 | arp | 896 | | 1365501012.119000 | 24844.955000 | 10.1.1.10 | 10.1.1.45 | udp | 2130 | | 1365501012.119000 | 24844.953000 | 10.1.1.10 | 10.1.1.45 | icmp | 2718 | | 1365501007.827000 | 24840.254000 | 10.1.1.10 | 10.1.1.126 | arp | 384 | | 1365501012.354000 | 24844.787000 | 10.1.1.10 | 167.206.245.130 | udp | 5766 | | 1365501008.507000 | 24840.812000 | 10.1.1.10 | 113.37.91.61 | icmp | 612 | | 1365501012.895000 | 24844.477000 | 10.1.1.10 | 113.37.91.61 | tcp | 37293 | | 1365502197.518000 | 24840.180000 | 10.1.1.126 | 10.1.1.10 | arp | 384 | +-------------------+--------------+------------+-----------------+-------+- ------------+ 27 rows in set (0.00 sec) rasql gave happy results: rasql -u -r mysql://argus:argus&lt; at &gt;localhost/argus/matrix_2013_04_09 -M sql=" saddr = '10.1.1.10' or daddr = '10.1.1.10'" LastTime Dur SrcId SrcAddr DstAddr Proto TotBytes 1365551889.364 86287.953 69.113.13.218 10.1.1.60 10.1.1.10 tcp 843282 1365551948.803 86347.195 69.113.13.218 10.1.1.46 10.1.1.10 tcp 783536 1365551993.490 86391.336 69.113.13.218 10.1.1.45 10.1.1.10 tcp 160507424* 1365551986.978 86381.195 69.113.13.218 10.1.1.50 10.1.1.10 tcp 3962458654 1365551957.703 86340.031 69.113.13.218 10.1.1.10 255.255.255.255 udp 1346514 1365551992.462 86374.555 69.113.13.218 10.1.1.10 10.1.1.50 udp 1667967 1365551992.462 86369.734 69.113.13.218 10.1.1.10 10.1.1.50 icmp 1426656 1365551938.461 86280.000 69.113.13.218 10.1.1.45 10.1.1.10 arp 157312 1365551964.909 86302.922 69.113.13.218 10.1.1.50 10.1.1.10 arp 152960 1365551953.810 86274.445 69.113.13.218 10.1.1.46 10.1.1.10 arp 107904 1365551889.353 86206.047 69.113.13.218 10.1.1.60 10.1.1.10 arp 108544 1365551868.423 86055.141 69.113.13.218 10.1.1.10 10.1.1.12 udp 30672 1365551873.421 86055.141 69.113.13.218 10.1.1.10 10.1.1.12 arp 36352 1365551639.982 85736.523 69.113.13.218 10.1.1.10 10.1.1.127 udp 61560 1365551920.751 85978.414 69.113.13.218 10.1.1.10 10.1.1.50 arp 49408 1365551560.220 77317.164 69.113.13.218 10.1.1.10 10.1.1.101 udp 115032 1365551560.220 77310.164 69.113.13.218 10.1.1.10 10.1.1.101 arp 29696 1365550944.567 76398.391 69.113.13.218 10.1.1.10 224.0.0.251 udp 3703 1365529023.282 54396.770 69.113.13.218 10.1.1.101 10.1.1.10 tcp 1032139 1365475483.795 786.242 69.113.13.218 10.1.1.101 10.1.1.10 arp 896 1365501012.119 24844.955 69.113.13.218 10.1.1.10 10.1.1.45 udp 2130 1365501012.119 24844.953 69.113.13.218 10.1.1.45 10.1.1.10 icmp 2718 1365501007.827 24840.254 69.113.13.218 10.1.1.10 10.1.1.126 arp 384 1365501012.354 24844.787 69.113.13.218 10.1.1.10 167.206.245.130 udp 5766 1365501008.507 24840.812 69.113.13.218 10.1.1.10 113.37.91.61 icmp 612 1365501012.895 24844.477 69.113.13.218 10.1.1.10 113.37.91.61 tcp 37293 1365502197.518 24840.180 69.113.13.218 10.1.1.126 10.1.1.10 arp 384 And a confirmation from the original flow files checked out well racluster -m srcid matrix protocol -r * -u -p 3 -s ltime dur srcid saddr daddr proto bytes - host 10.1.1.10 LastTime Dur SrcId SrcAddr DstAddr Proto TotBytes 1365551992.462 86374.555 69.113.13.218 10.1.1.10 10.1.1.50 udp 1667967 1365551560.220 77317.164 69.113.13.218 10.1.1.10 10.1.1.101 udp 115032 1365551957.703 86340.031 69.113.13.218 10.1.1.10 255.255.255.255 udp 1346514 1365501012.354 24844.787 69.113.13.218 10.1.1.10 167.206.245.130 udp 5766 1365551868.423 86055.141 69.113.13.218 10.1.1.10 10.1.1.12 udp 30672 1365551639.982 85736.523 69.113.13.218 10.1.1.10 10.1.1.127 udp 61560 1365501012.119 24844.955 69.113.13.218 10.1.1.10 10.1.1.45 udp 2130 1365550944.567 76398.391 69.113.13.218 10.1.1.10 224.0.0.251 udp 3703 1365551986.978 86389.391 69.113.13.218 10.1.1.10 10.1.1.50 tcp 3962486779 1365529023.282 54396.770 69.113.13.218 10.1.1.10 10.1.1.101 tcp 1032139 1365501012.895 24844.477 69.113.13.218 10.1.1.10 113.37.91.61 tcp 37293 1365551993.490 86391.336 69.113.13.218 10.1.1.10 10.1.1.45 tcp 160507424* 1365551948.803 86347.195 69.113.13.218 10.1.1.10 10.1.1.46 tcp 783536 1365551889.364 86287.953 69.113.13.218 10.1.1.10 10.1.1.60 tcp 843282 1365501012.119 24844.953 69.113.13.218 10.1.1.10 10.1.1.45 icmp 2718 1365551992.462 86369.734 69.113.13.218 10.1.1.10 10.1.1.50 icmp 1426656 1365501008.507 24840.812 69.113.13.218 10.1.1.10 113.37.91.61 icmp 612 1365551873.421 86055.141 69.113.13.218 10.1.1.10 10.1.1.12 arp 36352 1365551920.751 85978.414 69.113.13.218 10.1.1.10 10.1.1.50 arp 49408 1365551560.220 77310.164 69.113.13.218 10.1.1.10 10.1.1.101 arp 29696 1365501007.827 24840.254 69.113.13.218 10.1.1.10 10.1.1.126 arp 384 1365551938.461 86340.000 69.113.13.218 10.1.1.45 10.1.1.10 arp 157440 1365551953.810 86274.445 69.113.13.218 10.1.1.46 10.1.1.10 arp 107904 1365551964.909 86302.922 69.113.13.218 10.1.1.50 10.1.1.10 arp 152960 1365551889.353 86206.047 69.113.13.218 10.1.1.60 10.1.1.10 arp 108544 1365475483.795 786.242 69.113.13.218 10.1.1.101 10.1.1.10 arp 896 1365502197.518 24840.180 69.113.13.218 10.1.1.126 10.1.1.10 arp 384 Then I set these three running on the machine with the database (argus-clients-3.0.7.8) /usr/local/bin/radium -f /usr/local/argus/SNKradium.conf -d /usr/local/bin/rastream -S localhost:9603 -f /usr/local/argus/SNKstream.sh -M time 1h -B 15 -w /data/argus/%Y/%m/%d/argus.%Y.%m.%d.%H -d /usr/local/bin/rasqlinsert -M time 1d -M cache -S localhost:9603 -w mysql://argus&lt; at &gt;localhost/argus/matrix_%Y_%m_%d -m srcid matrix proto -s ltime dur srcid saddr daddr proto bytes -d # cat /usr/local/argus/SNKradium.conf RADIUM_DAEMON=no RADIUM_CLASSIFIER_FILE=/usr/local/argus/SNKlabel.conf RADIUM_ACCESS_PORT=9603 RADIUM_ARGUS_SERVER=rodnel-new:561 The SNKstream.sh file doesn't do anything but gzip the file. Now I get these results: The MySQL table is a bit unusual but not absolutely awful: mysql&gt; select ltime,dur,srcid,saddr, daddr, proto, bytes from matrix_2013_05_09 where saddr = '10.1.1.10' or daddr = '10.1.1.10'; +-------------------+--------------+---------------+------------+----------- ------+-------+------------+ | ltime | dur | srcid | saddr | daddr | proto | bytes | +-------------------+--------------+---------------+------------+----------- ------+-------+------------+ | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50 | udp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50 | icmp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.50 | 10.1.1.10 | arp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50 | tcp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 255.255.255.255 | udp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.60 | tcp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45 | tcp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.60 | 10.1.1.10 | arp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.45 | 10.1.1.10 | arp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.50 | arp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.127 | udp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45 | udp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.45 | icmp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.126 | arp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 167.206.245.130 | udp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 113.37.91.61 | icmp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 113.37.91.61 | tcp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.126 | 10.1.1.10 | arp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.10 | 10.1.1.71 | udp | 4609938798 | | 1368144006.774000 | 86377.883000 | 69.113.13.218 | 10.1.1.71 | 10.1.1.10 | arp | 4609938798 | +-------------------+--------------+---------------+------------+----------- ------+-------+------------+ 20 rows in set (0.00 sec) The files created by rastream look correct: racluster -m srcid matrix protocol -r * -u -p 3 -s ltime dur srcid saddr daddr proto bytes - host 10.1.1.10 LastTime Dur SrcId SrcAddr DstAddr Proto TotBytes 1368143927.352 86340.555 69.113.13.218 10.1.1.10 255.255.255.255 udp 1346514 1368143957.087 86335.609 69.113.13.218 10.1.1.10 10.1.1.50 udp 1674059 1368106993.711 44648.406 69.113.13.218 10.1.1.10 167.206.245.130 udp 5766 1368143651.470 85679.195 69.113.13.218 10.1.1.10 10.1.1.127 udp 61560 1368106993.205 44648.055 69.113.13.218 10.1.1.10 10.1.1.45 udp 2130 1368110304.637 0.001 69.113.13.218 10.1.1.10 10.1.1.71 udp 188 1368106995.036 44648.059 69.113.13.218 10.1.1.10 113.37.91.61 tcp 35811 1368143944.782 86342.789 69.113.13.218 10.1.1.10 10.1.1.45 tcp 173423905* 1368143942.334 86354.383 69.113.13.218 10.1.1.10 10.1.1.50 tcp 3945119719 1368143931.073 86329.680 69.113.13.218 10.1.1.10 10.1.1.60 tcp 429295 1368106993.205 44648.055 69.113.13.218 10.1.1.10 10.1.1.45 icmp 2718 1368143957.087 86335.609 69.113.13.218 10.1.1.10 10.1.1.50 icmp 1439680 1368106987.571 44641.789 69.113.13.218 10.1.1.10 113.37.91.61 icmp 612 1368143870.011 85965.906 69.113.13.218 10.1.1.10 10.1.1.50 arp 60032 1368106986.534 44640.969 69.113.13.218 10.1.1.10 10.1.1.126 arp 384 1368143949.756 86280.000 69.113.13.218 10.1.1.45 10.1.1.10 arp 160128 1368143942.065 86316.039 69.113.13.218 10.1.1.50 10.1.1.10 arp 158848 1368143875.921 86209.273 69.113.13.218 10.1.1.60 10.1.1.10 arp 109440 1368110304.635 0.000 69.113.13.218 10.1.1.71 10.1.1.10 arp 128 1368108176.467 44641.297 69.113.13.218 10.1.1.126 10.1.1.10 arp 384 Then we come to the output of rasql which for some reason informs me way too many times that something on my network (10.1.1.0/25) sent a bunch of traffic to somewhere NB: this is the one place where I see CIDR notation and that might be is a clue. rasql -u -r mysql://argus&lt; at &gt;localhost/argus/matrix_2013_05_09 -M sql=" saddr = '10.1.1.10' or daddr = '10.1.1.10'" LastTime Dur SrcId SrcAddr DstAddr Proto TotBytes 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0/25 0.0.0.0/4 ip 4609938798 So I copied one day's files from the system running argus to a clean directory on the MySQL machine and ran the rasqlinsert incantation that I always used to use: rasqlinsert -M time 1d -r * -w mysql://argus&lt; at &gt;localhost/argus/testMatrix_%Y_%m_%d -m srcid matrix proto -s ltime dur srcid saddr daddr proto bytes and got results like the ones I used to get: rasql -u -r mysql://argus&lt; at &gt;localhost/argus/testMatrix_2013_05_09 -M sql=" saddr = '10.1.1.10' or daddr = '10.1.1.10'" LastTime Dur SrcId SrcAddr DstAddr Proto TotBytes 1368143991.233 86389.844 69.113.13.218 10.1.1.60 10.1.1.10 tcp 429589 1368143944.782 86342.789 69.113.13.218 10.1.1.45 10.1.1.10 tcp 173086923* 1368143957.087 86335.609 69.113.13.218 10.1.1.50 10.1.1.10 udp 1674059 1368143957.087 86335.609 69.113.13.218 10.1.1.10 10.1.1.50 icmp 1439680 1368143942.065 86316.039 69.113.13.218 10.1.1.50 10.1.1.10 arp 158848 1368143942.334 86313.445 69.113.13.218 10.1.1.50 10.1.1.10 tcp 3945115585 1368143987.364 86340.555 69.113.13.218 10.1.1.10 255.255.255.255 udp 1346514 1368143996.241 86329.594 69.113.13.218 10.1.1.60 10.1.1.10 arp 109568 1368143949.756 86280.000 69.113.13.218 10.1.1.45 10.1.1.10 arp 160128 1368143870.011 85965.906 69.113.13.218 10.1.1.10 10.1.1.50 arp 60032 1368143651.470 85679.195 69.113.13.218 10.1.1.10 10.1.1.127 udp 61560 1368106993.205 44648.055 69.113.13.218 10.1.1.10 10.1.1.45 udp 2130 1368106993.205 44648.055 69.113.13.218 10.1.1.45 10.1.1.10 icmp 2718 1368106986.534 44640.969 69.113.13.218 10.1.1.10 10.1.1.126 arp 384 1368106993.711 44648.406 69.113.13.218 10.1.1.10 167.206.245.130 udp 5766 1368106987.571 44641.789 69.113.13.218 10.1.1.10 113.37.91.61 icmp 612 1368106995.036 44648.059 69.113.13.218 10.1.1.10 113.37.91.61 tcp 35811 1368108176.467 44641.297 69.113.13.218 10.1.1.126 10.1.1.10 arp 384 1368110304.635 0.000 69.113.13.218 10.1.1.71 10.1.1.10 arp 128 1368110304.637 0.001 69.113.13.218 10.1.1.71 10.1.1.10 udp 188 Then I used the files created by rastream to do the same thing (remember that these files came from the same radium feed as fed the rasqlinsert that wasn't so good) cd /data/argus/2013/05/09 rasqlinsert -M time 1d -r * -w mysql://argus&lt; at &gt;localhost/argus/test2Matrix_%Y_%m_%d -m srcid matrix proto -s ltime dur srcid saddr daddr proto bytes and got the results that I expected: rasql -u -r mysql://argus&lt; at &gt;localhost/argus/test2Matrix_2013_05_09 -M sql=" saddr = '10.1.1.10' or daddr = '10.1.1.10'" LastTime Dur SrcId SrcAddr DstAddr Proto TotBytes 1368143931.073 86329.680 69.113.13.218 10.1.1.60 10.1.1.10 tcp 429295 1368143944.782 86342.789 69.113.13.218 10.1.1.45 10.1.1.10 tcp 173423905* 1368143957.087 86335.609 69.113.13.218 10.1.1.50 10.1.1.10 udp 1674059 1368143957.087 86335.609 69.113.13.218 10.1.1.10 10.1.1.50 icmp 1439680 1368143942.065 86316.039 69.113.13.218 10.1.1.50 10.1.1.10 arp 158848 1368143942.334 86313.445 69.113.13.218 10.1.1.50 10.1.1.10 tcp 3945115585 1368143927.352 86280.539 69.113.13.218 10.1.1.10 255.255.255.255 udp 1345782 1368143875.921 86209.273 69.113.13.218 10.1.1.60 10.1.1.10 arp 109440 1368143949.756 86280.000 69.113.13.218 10.1.1.45 10.1.1.10 arp 160128 1368143870.011 85965.906 69.113.13.218 10.1.1.10 10.1.1.50 arp 60032 1368143651.470 85679.195 69.113.13.218 10.1.1.10 10.1.1.127 udp 61560 1368106993.205 44648.055 69.113.13.218 10.1.1.10 10.1.1.45 udp 2130 1368106993.205 44648.055 69.113.13.218 10.1.1.45 10.1.1.10 icmp 2718 1368106986.534 44640.969 69.113.13.218 10.1.1.10 10.1.1.126 arp 384 1368106993.711 44648.406 69.113.13.218 10.1.1.10 167.206.245.130 udp 5766 1368106987.571 44641.789 69.113.13.218 10.1.1.10 113.37.91.61 icmp 612 1368106995.036 44648.059 69.113.13.218 10.1.1.10 113.37.91.61 tcp 35811 1368108176.467 44641.297 69.113.13.218 10.1.1.126 10.1.1.10 arp 384 1368110304.635 0.000 69.113.13.218 10.1.1.71 10.1.1.10 arp 128 1368110304.637 0.001 69.113.13.218 10.1.1.71 10.1.1.10 udp 188 Just in case the -M cache is making a difference, I included it in a test and it didn't break anything: rasqlinsert -M time 1d -r * -M cache -w mysql://argus&lt; at &gt;localhost/argus/test3Matrix_%Y_%m_%d -m srcid matrix proto -s ltime dur srcid saddr daddr proto bytes rasql -u -r mysql://argus&lt; at &gt;localhost/argus/test3Matrix_2013_05_09 -M sql=" saddr = '10.1.1.10' or daddr = '10.1.1.10'" LastTime Dur SrcId SrcAddr DstAddr Proto TotBytes 1368143991.233 86389.844 69.113.13.218 10.1.1.60 10.1.1.10 tcp 429589 1368143944.782 86342.789 69.113.13.218 10.1.1.45 10.1.1.10 tcp 173086923* 1368143957.087 86335.609 69.113.13.218 10.1.1.50 10.1.1.10 udp 1674059 1368143957.087 86335.609 69.113.13.218 10.1.1.10 10.1.1.50 icmp 1439680 1368143942.065 86316.039 69.113.13.218 10.1.1.50 10.1.1.10 arp 158848 1368143942.334 86313.445 69.113.13.218 10.1.1.50 10.1.1.10 tcp 3945115585 1368143987.364 86340.555 69.113.13.218 10.1.1.10 255.255.255.255 udp 1346514 1368143996.241 86329.594 69.113.13.218 10.1.1.60 10.1.1.10 arp 109568 1368143949.756 86280.000 69.113.13.218 10.1.1.45 10.1.1.10 arp 160128 1368143870.011 85965.906 69.113.13.218 10.1.1.10 10.1.1.50 arp 60032 1368143651.470 85679.195 69.113.13.218 10.1.1.10 10.1.1.127 udp 61560 1368106993.205 44648.055 69.113.13.218 10.1.1.10 10.1.1.45 udp 2130 1368106993.205 44648.055 69.113.13.218 10.1.1.45 10.1.1.10 icmp 2718 1368106986.534 44640.969 69.113.13.218 10.1.1.10 10.1.1.126 arp 384 1368106993.711 44648.406 69.113.13.218 10.1.1.10 167.206.245.130 udp 5766 1368106987.571 44641.789 69.113.13.218 10.1.1.10 113.37.91.61 icmp 612 1368106995.036 44648.059 69.113.13.218 10.1.1.10 113.37.91.61 tcp 35811 1368108176.467 44641.297 69.113.13.218 10.1.1.126 10.1.1.10 arp 384 1368110304.635 0.000 69.113.13.218 10.1.1.71 10.1.1.10 arp 128 1368110304.637 0.001 69.113.13.218 10.1.1.71 10.1.1.10 udp 188 I kill CIDR notation in my ~/.rarc file to see what happens (I dropped the current table and restarted the clients) and it is looking much better rasql -u -r mysql://argus:argus&lt; at &gt;localhost/argus/matrix_2013_05_10 -M sql=" saddr = '10.1.1.10' or daddr = '10.1.1.10'" LastTime Dur SrcId SrcAddr DstAddr Proto TotBytes 1368153616.837 60.160 69.113.13.218 10.1.1.60 10.1.1.10 tcp 588 1368154260.248 700.975 69.113.13.218 10.1.1.50 10.1.1.10 udp 13899 1368154260.248 700.975 69.113.13.218 10.1.1.10 10.1.1.50 icmp 10688 1368153868.734 307.102 69.113.13.218 10.1.1.50 10.1.1.10 tcp 425755 1368154283.605 721.920 69.113.13.218 10.1.1.60 10.1.1.10 arp 1408 1368154247.544 675.886 69.113.13.218 10.1.1.10 255.255.255.255 udp 12078 1368153964.784 360.043 69.113.13.218 10.1.1.45 10.1.1.10 tcp 10108 1368154172.306 566.982 69.113.13.218 10.1.1.50 10.1.1.10 arp 1280 1368154209.756 600.000 69.113.13.218 10.1.1.45 10.1.1.10 arp 1408 1368153742.552 0.000 69.113.13.218 10.1.1.10 10.1.1.127 udp 513 1368153908.043 67.891 69.113.13.218 10.1.1.10 10.1.1.50 arp 256 The fix is not retroactive, NB: the testMatrix, test2Matrix, and test3Matrix tables were all generated by rasqlinsert with the .rarc containing RA_CIDR_ADDRESS_FORMAT="yes" and they were fine so it looks like an interaction between CIDR notation and rasqlinsert -S from a radium source rasql -u -r mysql://argus:argus&lt; at &gt;localhost/argus/matrix_2013_05_09 -M sql=" saddr = '10.1.1.10' or daddr = '10.1.1.10'" LastTime Dur SrcId SrcAddr DstAddr Proto TotBytes 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 1368144006.774 86377.883 69.113.13.218 10.1.1.0 0.0.0.0 ip 4609938798 </pre> Dave Edelman 2013-05-10T03:14:12 http://comments.gmane.org/gmane.network.argus/9320 <pre>Hello, How do I stop the concatenation of daddr and dport (for instance)? How do I stop the resolution of service names when using dport (for instance)? Thanks, Matt </pre> Matt Brown 2013-05-09T18:39:02 http://comments.gmane.org/gmane.network.argus/9312 <pre>Hello all, With 3.0.6.2 I am seeing something odd with rastream's -w. It appears to not deal with %T %H %M or %S properly, not returning now(), but starting with 01:00:00 and 01 00 00 respectively. Why is this? Unfortunately gmane's search function seems to not be functioning. Any assistance is appreciated. Thanks, Matt Brown </pre> Matt Brown 2013-05-08T17:52:14 http://comments.gmane.org/gmane.network.argus/9308 <pre>Hi All, Running the latest (i think) argus-clients 3.0.7.7. I've got a setup where I have multiple src ids configured. I'm trying to just pull records for one src id but am getting a syntax error. [root&lt; at &gt;usher ~]# ra -nnr /var/log/argus/argus.out - srcid eth0 ra[31322]: 14:14:38.492897 srcid eth0 unknown ra[31321]: 14:14:38.890573 srcid eth0 filter syntax error I believe that the srcids are correct as ra can print them out. [root&lt; at &gt;usher ~]# ra -nnr /var/log/argus/argus.out -s saddr,sport,daddr,dport,srcid -N 3 SrcAddr Sport DstAddr Dport SrcId 172.16.255.170.123 24.124.0.251.123 eth1 72.94.xx.xxx.123 24.124.0.251.123 eth0 172.16.255.254 172.16.255.176 eth1 Is this a bug or am I doing something wrong? Cheers, Harry </pre> Harry Hoffman 2013-05-07T18:17:51 http://comments.gmane.org/gmane.network.argus/9307 <pre>I'm looking for suggestions of how to configure argus to accurately report traffic in a network configuration where my argus sensor is on my side of an upstream router. I'm in a University department with multiple subnets on virtual LANs switched internally, but with a tapped link to the University's router. Something like this, with taps on TX and RX to the argus sensor box. SW &lt;-&gt; +-------------------+ +---------------+ SW &lt;-&gt; |aggregrator switch | -&gt; TX -&gt; |router/firewall| -&gt; Internet SW &lt;-&gt; +-------------------+ &lt;- RX &lt;- +---------------+ My current configuration of argus with ARGUS_INTERFACE=dup:eth0,eth1/uplink results in double counting of local, routed traffic (once on TX and once on RX). Using bond results in the same thing. To correct this, I've been thinking of moving to independent interfaces, capturing all traffic on RX to get both local routed traffic and inbound Internet traffic, and capturing only Internet bound TX traffic. However, I'm not 100% positive this will work. Is this the right path, and what are the gotchas I should be aware of with respect to ensuring my Internet flows see both src and dst packets? Is there a better direction to be looking at? Thanks, Michael Sanderson </pre> Michael Sanderson 2013-05-07T18:11:47 http://comments.gmane.org/gmane.network.argus/9303 <pre>I have rastream processing on hard 5 minute boundaries and I would like to create summary data after it closes each file. flow-capture had a nice option that would let you call an external program after it finished spooling a file; do I have an option like this with argus? I can script it, just curious if there is something built-in. thanks. </pre> Paul Halliday 2013-05-07T13:52:12 http://comments.gmane.org/gmane.network.argus/9292 <pre> Hi, I used argus to transform and aggregate a set of pcap files into an argus file. The resulting argus file has timestamps that are 2 hours behind the original timestamps when the packets were captured. So a 4pm timestamp would be 2pm in the generated argus file. That's fine, but is there a way to shift the ragraph X-axis (add 2 hours to the lables)? I looked into ragraph options, and ra-options but couldn't find what I need. Thanks, -Manaf</pre> manaf gharaibeh 2013-05-05T09:24:39 http://comments.gmane.org/gmane.network.argus/9276 <pre>Hi, Again I'm trying to follow through some of the APT examples. I'm very interested in the IP address SQL backing tables to speed searches, as this is a common search for my purposes. I've built tables with two days of data like: rasqlinsert -M time 1d -M rmon -m saddr -R 2013/04/29 -R 2013/04/30 -w mysql://user&lt; at &gt;host/argus_ip/ipAddrs_%Y_%m_%d -M cache - ip This took a while (29 hours) but did build tables: mysql&gt; use argus_ip; Database changed mysql&gt; show tables; +--------------------+ | Tables_in_argus_ip | +--------------------+ | ipAddrs_2013_04_29 | | ipAddrs_2013_04_30 | +--------------------+ 2 rows in set (0.00 sec) mysql&gt; describe ipAddrs_2013_04_29; +--------+-----------------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------+-----------------------+------+-----+---------+-------+ | stime | double(18,6) unsigned | NO | | NULL | | | flgs | varchar(32) | YES | | NULL | | | proto | varchar(16) | NO | | NULL | | | saddr | varchar(64) | NO | PRI | NULL | | | sport | varchar(10) | NO | | NULL | | | dir | varchar(3) | YES | | NULL | | | daddr | varchar(64) | NO | | NULL | | | dport | varchar(10) | NO | | NULL | | | pkts | bigint(20) | YES | | NULL | | | bytes | bigint(20) | YES | | NULL | | | state | varchar(32) | YES | | NULL | | | record | blob | YES | | NULL | | +--------+-----------------------+------+-----+---------+-------+ 12 rows in set (0.00 sec) mysql&gt; select count(*) from ipAddrs_2013_04_29; +----------+ | count(*) | +----------+ | 715059 | +----------+ 1 row in set (0.02 sec) mysql&gt; select count(*) from ipAddrs_2013_04_30; +----------+ | count(*) | +----------+ | 1362479 | +----------+ 1 row in set (0.00 sec) mysql&gt; select stime,flgs,proto,saddr,sport,dir,daddr,dport,pkts,bytes,state from ipAddrs_2013_04_29 limit 1; +-------------------+------+-------+---------------+-------+------+---------+-------+------+---------+-------+ | stime | flgs | proto | saddr | sport | dir | daddr | dport | pkts | bytes | state | +-------------------+------+-------+---------------+-------+------+---------+-------+------+---------+-------+ | 1367208000.000000 | | ip | 66.249.76.186 | | &lt;-&gt; | 0.0.0.0 | | 2465 | 1964858 | CON | +-------------------+------+-------+---------------+-------+------+---------+-------+------+---------+-------+ 1 row in set (0.00 sec) ...but now I can't seem to get any data back out using the incantations I've seen: # time rasql -t -7d+7d -M time 1d -M sql="saddr=’66.249.76.186’" -r mysql://user&lt; at &gt;host/argus_ip/ipAddrs_%Y_%m_%d real 0m0.073s user 0m0.008s sys 0m0.021s # time rasql -t 2013/04/29 -M time 1d -M sql="saddr=’66.249.76.186’" -r mysql://user&lt; at &gt;host/argus_ip/ipAddrs_%Y_%m_%d real 0m0.045s user 0m0.010s sys 0m0.017s This does work, but much more slowly that I expected: # time rasql -t 2013/04/29 -M time 1d -r mysql://user&lt; at &gt;host/argus_ip/ipAddrs_%Y_%m_%d - host 66.249.76.186 StartTime Flgs Proto SrcAddr Sport Dir DstAddr Dport TotPkts TotBytes State 04/29/13 00:00:00.000000 0 66.249.76.186 &lt;-&gt; 0.0.0.0 2465 1964858 CON real 0m46.419s user 0m4.245s sys 0m0.991s # time rasql -t 2013/04/29+1d -M time 1d -r mysql://user&lt; at &gt;host/argus_ip/ipAddrs_%Y_%m_%d - host 66.249.76.186 ^C real 26m18.117s user 0m0.037s sys 0m0.165s I'm guessing that the second form (using a filter to rasql) is pulling back ALL records from mySQL and filtering that, whereas the first format should be using mySQL to do the filtering (the preferred method). So I suppose my questions are "Is this last statement correct?" and "What am I doing wrong with the first form of query?" Is the query speed for the working format inline with expectations? Seems a bit slow... Cheers, Jesse </pre> Jesse Bowling 2013-05-03T16:16:27 http://comments.gmane.org/gmane.network.argus/9274 <pre>Hi, I have been banging around with ra and racluster and while I can get close to what I want I am still not very comfortable with the commands and not entirely sure I am using them correctly. Getting really close to greping, awking, sorting and uniq -c'ing out the stuff I need which I doubt I need to do.. Anyway any help would be appreciated. I am trying to get: The Top 20 Sources and then Destinations (sum bytes) matching 10.1.0.0/16for the current day Count of distinct IP's matching 10.20.0.0/22 in the last hour Lastly, I am trying to produce a rate chart with ragraph that portrays inbound/outbound, something like this: http://www.pintumbler.org/ex.png possible? Thanks! </pre> Paul Halliday 2013-05-03T12:36:12 http://comments.gmane.org/gmane.network.argus/9265 <pre>Carter, The -s value of xxx looks like it might be useful for some things that I'm doing. I understand that it doesn't show up in the man page but when I try to use it, I get a SEGFAULT where other sets of random letters just get ignored. --Dave </pre> Dave Edelman 2013-04-22T12:42:28 http://comments.gmane.org/gmane.network.argus/9264 <pre>In general, I do have srcid set in my flow records. If I do not use srcid as a key for racluster, which srcid is retained if I have simultaneous flow records with the same keys (asymmetrical routing) that are then aggregated into a single record? What is aggregated for suser and duser in racluster output when flow records are aggregated and my normal setting of a 2048 byte limit is exceeded? --Dave </pre> Dave Edelman 2013-04-22T12:37:43 http://comments.gmane.org/gmane.network.argus/9261 <pre>Hi All, So, I just got back from Educause SPC where the bro guys reminded me about time machine (I hadn't looked at it in a really long time). I decided to go today and have a read over it and from the description I'm not sure that I can find the differences between time machine and having argus store packet payload for a given N bytes. "Since it is not feasible to capture the complete load of a fully utilized Gbps link to disk, the time machine utilizes a mechanism called "connection cutoff" to reduce the the amount of data to process. This "connection cutoff" only records the first X bytes of every monitored connection (identified via the 5-tupel of source and destination IP and Port and the transport protocol). Indeed this approach it does not impair the analysis capabilities (unless the cutoff is set to low) because most of the "interessting" data is located in the first few packets of a connection. The effiency of this approach comes from leveraging the heavy-tailed nature of network traffic: because the bulk of the traffic in high-volume streams comes from just a few connections." Anyone using time machine and/or argus to do this and care to comment? Cheers, Harry </pre> Harry Hoffman 2013-04-19T16:51:30 http://comments.gmane.org/gmane.network.argus/9258 <pre>Hi, I've got a bunch of pcap files and would like to generate argus data from them. It's important to keep things sane, so the flows are merged, etc. How can i achieve this? I guess it's some combination of the argus command in a for loop, feeding data to rasplit, but don't really know where to start. </pre> Michal Purzynski 2013-04-19T11:25:20 http://comments.gmane.org/gmane.network.argus/9241 <pre>Hi All, I've installed OpenWrt on my WiFi router and now I'm looking for a way to generate network statistics for the ADSL connection. The ADSL connection is through a separate router, so generating the statistics should be a simple matter of processing the packets passing through the WAN0 interface. Daily statistics I would like to generate include: - Total inbound and outbound data - Inbound and outbound data per local IP address/port/protocol (TCP/UDP) - Inbound and outbound data per remote IP address/port/protocol (TCP/UDP) - Inbound and outbound data per unique local IP/Remote IP/port/protocol (TCP/UDP) - Average inbound and outbound throughput per 5 minute interval (total bytes/second) Two questions - Is Argus the right solution? - How hard will it be to get Argus running in OpenWrt Attitude Adjustment? Thanks, Graeme </pre> Graeme Russ 2013-04-11T23:09:30 http://comments.gmane.org/gmane.network.argus/9233 <pre>Hi folks, Would someone here be able to point me at the manuals which discuss how argus determines 'dur' (record total duration)? There is an interesting looking example in the rahisto manual: % rahisto -H dur 10 -r ˜/argus/data/argus*out.gz - port http I'd have assumed that understanding how to compute 'dur' requires protocol specific knowledge? (I'm curious about measuring client/server request/response times for a memcached cluster) Jim - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - James A. Robinson jimr&lt; at &gt;highwire.stanford.edu HighWire | Stanford University http://highwire.stanford.edu/ +1 650 7237294 (Work) +1 650 7259335 (Fax) </pre> James A. Robinson 2013-04-09T00:09:19 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.network.argus