gmane.network.argus

Re: Rastream doesn't rotate properly when daemonied?

Matt Brown — 2013-05-21T17:31:36

Hello Carter,



Thanks for getting back to me quickly.



I've tried the following to attempt to troubleshoot the problem, but it
doesn't appear that anything's outstanding.



Apologies for the verbosity…





# cd ./argus-clients-3.0.7.10

# touch .devel .debug

# ./configure

# make clean && make

# kill -2 $(pgrep rastream)

# ./bin/rastream -d -F /etc/rastream.conf -S 127.0.0.1:561 -B 15s -M
time 1h<x-apple-data-detectors://1> -w
/var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh -D4

rastream[29500.c0567db7]: 2013-05-21 10:54:31.455541 ArgusNewQueue ()
returning 0x8609548

rastream[29500.c0567db7]: 2013-05-21 10:54:31.455612 ArgusNewHashTable
(65536) returning 0x8609bc0

rastream[29500.c0567db7]: 2013-05-21 10:54:31.455632 ArgusNewList ()
returning 0x860a010

rastream[29500.c0567db7]: 2013-05-21 10:54:31.455695 ArgusNewQueue ()
returning 0x8611878

rastream[29500]: 2013-05-21 10:54:31.455729 started



syslog reports:

May 21 10:54:31 ny-sentinel rastream[29500]: 2013-05-21 10:54:31.4557

Re: Rastream doesn't rotate properly when daemonied?

Carter Bullard — 2013-05-21T14:48:58

Yes you have to compile in debug support.  We should turn on symbols, etc... as well.  In the clients root directory, try this:

% touch .devel .debug
% ./configure
% make clean; make

And use the resulting binaries for the testing.
Carter

On May 21, 2013, at 10:08 AM, Matt Brown <matthewbrown< at >gmail.com> wrote:

Re: Rastream doesn't rotate properly when daemonied?

Matt Brown — 2013-05-21T14:08:52

Carter,

Do settings need to be changed at compile time in order to get debugging
features?  I have started with -D4 and only the start time and forked time
are logged.  I have confirmed that rastream is appending to the proper file.


Just to let you know *latest* is still 3.0.7.9, not 3.0.7.10.


Thanks very much!

Matt

On May 21, 2013, at 9:55 AM, Carter Bullard <carter< at >qosient.com> wrote:

Try the newest client distribution, just to see if any of the changes we
made to date affect the problem.  This may be a tough one to debug,
but if I can recreate the issue, then we're on our way to happiness.

We'll need you to turn on debug on your daemon, the output should
go to the syslog, so testing that briefly may help a bit.  Run your rastream
with the "-d -D4" options, and see if you get any debug info in your syslog.

Carter

On May 21, 2013, at 9:48 AM, Matt Brown <matthewbrown< at >gmail.com> wrote:

Good morning Carter!


I wrote an init script and had some issues when starting rastream with the
script targett

Re: Rastream doesn't rotate properly when daemonied?

Carter Bullard — 2013-05-21T13:55:09

Try the newest client distribution, just to see if any of the changes we
made to date affect the problem.  This may be a tough one to debug,
but if I can recreate the issue, then we're on our way to happiness.

We'll need you to turn on debug on your daemon, the output should
go to the syslog, so testing that briefly may help a bit.  Run your rastream
with the "-d -D4" options, and see if you get any debug info in your syslog.

Carter 

On May 21, 2013, at 9:48 AM, Matt Brown <matthewbrown< at >gmail.com> wrote:

Re: Rastream doesn't rotate properly when daemonied?

Matt Brown — 2013-05-21T13:48:45

Good morning Carter!




I wrote an init script and had some issues when starting rastream with the
script targetting ~/.rarc even if I set $ARGUSHOME or $HOME within the
script (on CentOS) (due to service's use of /bin/env).





So, the line I use within the init script is:



/usr/local/bin/rastream -d -F /etc/rastream.conf -S 127.0.0.1:561 -B 15s -M
time 1h <x-apple-data-detectors://0> -w /var/opt/argus/%Y-%m-%d/argus_%T -f
/usr/local/bin/rastream.sh



The same problem with timing occurs as had previously occur when relying on
the full ~/.rarc.





Here is an etherpad with relevant information:
https://etherpad.mozilla.org/dmoSdfQ9H4



# rastream --version

Rastream Version 3.0.7.9





Thanks very much Carter!



Matt

On May 21, 2013, at 6:44 AM, Carter Bullard <carter< at >qosient.com> wrote:


Hey Matt,
So.......any chance you are setting the timezine in a rarc file somewhere,
like /etc/rarc or in your yme directory ?
What version are you running ?

Carter

On May 20, 2013, at 2:42 PM, Matt Brown <matt

Re: Rastream doesn't rotate properly when daemonied?

Carter Bullard — 2013-05-21T10:44:23

Hey Matt,
So.......any chance you are setting the timezine in a rarc file somewhere, like /etc/rarc or in your yme directory ?
What version are you running ?

Carter

On May 20, 2013, at 2:42 PM, Matt Brown <matthewbrown< at >gmail.com> wrote:

argus-clients-3.0.7.10 on the dev server

Carter Bullard — 2013-05-21T03:22:17

Gentle people,
I've uploaded argus-clients-3.0.7.10 to the development server.
This fixes all the known bugs reported, except the report today regarding
rastream().  This includes, raservices() signature limits, fixes to the
" delayed " filter compiler reporting syntax errors, removal of the
syslog error for "timestamps wayyyy out of range" error reports,
radump() problems when there isn't any user data, fixes to flow key
processing when using RMON aggregation, and CIDR address issues.

   http://qosient.com/argus/dev/argus-clients-3.0.7.10.tar.gz

Please give this version a try.  I'd like to make it are the next round the
nex argus-clients-3.0.8 stable version.

Hope all is most excellent,

Carter

Rastream doesn't rotate properly when daemonied?

Matt Brown — 2013-05-20T18:42:28

Hello All:





I am having a problem with rastream that's manifested itself when using the
-f "shell script executor" argument to rotate files at 'time
1h<x-apple-data-detectors://0>
'.





If I run rastream as a daemon, then the script seems to run before the
"hour" is over (and the "hour" is over at the incorrect time):

# rastream -d -S 127.0.0.1:561 -B 15s -M time 1h<x-apple-data-detectors://1> -w
/var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh



A few hours' files look like:



# ls --full-time /var/opt/argus/2013-05-18

total 3728

-rw-r--r--. 1 root  18752 2013-05-18 01:00:59.556839459 -0400
argus_00:00:00<x-apple-data-detectors://3>

-rw-r--r--. 1 root 160607 2013-05-18 01:00:17.793286000 -0400
argus_00:00:00.gz

-rw-r--r--. 1 root  12068 2013-05-18 02:00:59.619364943 -0400
argus_01:00:00<x-apple-data-detectors://6>

-rw-r--r--. 1 root 163409 2013-05-18 02:00:17.943700000 -0400
argus_01:00:00.gz

-rw-r--r--. 1 root   9032 2013-05-1803:01:00.579907536 -0400
argus_02:00:00<x-apple-dat

Re: Ra - filter syntax error

Carter Bullard — 2013-05-20T14:01:49

Hey Elof,
Great, and thanks for the info.  While the clients will be blocked waiting for the filter to be compiled, because this is structured as a deadman timer, if the compiler returns quickly, there won't be any delay.

For remote accesses, we don't connect until we've compiled the filter.  The logic is if we can't compile the filter, the remote won't be able to either...so a long time wait for the local compiler won't over run us with data.

I'll make it 1sec in the code base until I hear otherwise.
Carter


On May 20, 2013, at 7:41 AM, elof2< at >sentor.se wrote:

Ra - filter syntax error

2013-05-20T11:41:56

Carter and I have discussed a very unusual error message; "filter syntax 
error", which may show up if the machine is HEAVILY burdened, like 
swapping a lot and/or receiving tons of interrupts, while feeding a long 
and complex filter expression to a ra* process.


I suggest that Carter add a specific error message for this 
particular scenario, logging the message "filter compilation timeout" 
instead of the generic "filter syntax error".



During the weekend I've had cron start ra with a long and complex filter 
string every 5 minutes.
With a timeout of 500ms, I had 9 "filter syntax error" in 34 hours.
(on an old Intel Xeon 3050 < at > 2.13GHz machine from 2008)
(with 200ms I suspect I would have had approx 300 warnings)


So to prevent this message to appear "too often" on choked machines, you 
could increase the wait.tv_usec to 750000  ...or even higher if a higher 
value doesn't introduce any overall negative impact on ra*.



I now upped the timeout to 900000 just to see if that is enough to quell 
even t

Re: raservices ((doesn't crash)) when processing

Matt Brown — 2013-05-17T18:25:58

With the 3.0.7.10 release, will you also change the ARGUSMAXSIGFILE in
../clients/include/argus_client.h ?


On May 17, 2013, at 12:32 PM, Carter Bullard <carter< at >qosient.com> wrote:

Hey Dave,
Its not ArgusMergeLabels() that has a problem.  raservices() is munging the
string that  ArgusMergeLabels() returns.  Copy this version of raservices.c,
to ./examples/raservices, and re-make.  Should fix things.

Carter

<raservices.c>



On May 17, 2013, at 11:42 AM, "Dave Edelman" <dedelman< at >iname.com> wrote:

Carter,

I have it working but I think that there is a problem with ArgusMergeLabels()
when it is set for ArgusUnion. If you look at the attached file, it seems
that the buffer is not being cleared correctly. I am running raservices
againstunclustered flow records that have been labeled by radium as they
arrive from the argus collector. I can provide the equivalent ra output if
you want, that’s why I included the offset.

--Dave


*From:* Carter Bullard [mailto:carter< at >qosient.com]
*Sent:* Thursday, May 16, 201

argus data labels and DNS names

Carter Bullard — 2013-05-17T17:51:22

Hey Dave,
I see that you're putting name resolutions in your flow data labels.  Good idea…
There are a lot of buttons and dials for name lookups in argus and the clients.
Now that someone is doing this on the list, we should turn this stuff on for labels.

In the new clients that I'll put up tonight/tomorrow, there are a few new
variables in the ./support/Config/ralabel.conf file to turn on/off various DNS
functions.

We have a non-blocking DNS resolver in the library, and clients like ratop() and
rasqlinsert() currently use it so that they are not blocking, waiting for the DNS query to
return. There is a new variable to turn that on or off.  If you MUST have a
DNS name at the time of labeling, then you would set this to "no".  If you
can handle lazy lookups, which keeps radium() going fast, then I would set
this to "yes".

There are a few new variables to specify what you want in the name,
host name only (truncate the domain name) or just the domain name
(snip off the leading chars upto the first ' . ').

Re: raservices crashes when processing

Carter Bullard — 2013-05-17T16:32:33

Hey Dave,
Its not ArgusMergeLabels() that has a problem.  raservices() is munging the 
string that  ArgusMergeLabels() returns.  Copy this version of raservices.c,
to ./examples/raservices, and re-make.  Should fix things.

Carter



On May 17, 2013, at 11:42 AM, "Dave Edelman" <dedelman< at >iname.com> wrote:

Re: raservices crashes when processing

Dave Edelman — 2013-05-17T15:42:16

Carter,
 
I have it working but I think that there is a problem with
ArgusMergeLabels() when it is set for ArgusUnion. If you look at the
attached file, it seems that the buffer is not being cleared correctly. I am
running raservices against unclustered flow records that have been labeled
by radium as they arrive from the argus collector. I can provide the
equivalent ra output if you want, that's why I included the offset.
 
--Dave
 
 
From: Carter Bullard [mailto:carter< at >qosient.com] 
Sent: Thursday, May 16, 2013 11:14 AM
To: Dave Edelman
Cc: 'Matt Brown'; argus-info< at >lists.andrew.cmu.edu
Subject: Re: [ARGUS] raservices crashes when processing
 
Hey Dave,
Of course, everything in the clients has a constant defined somewhere.
Change the value of ARGUSMAXSIGFILE in ./include/argus_client.h to
something like this:
 
==== //depot/argus/clients/include/argus_client.h#64 -
/Volumes/Users/carter/argus/clients/include/argus_client.h ====
142c142
< #define ARGUSMAXSIGFILE               2048
---
 
 
Carter
 
On May 16,

Re: raservices crashes when processing

Carter Bullard — 2013-05-16T16:53:59

Hey Matt,
Well 2K of signatures is too small, obviously, so, thanks for starting that fix.
But, if you don't mind, using the word crash is not good, so lets use the word
" fails ", unless, of course, it really does crash, then crash is the best term ;O)

So rauserdata() is designed to generate upto 16 signatures per application.
While it does want to try to leverage port numbers as application identifiers,
and since there are 64K ports, we probably should be ready for .5M of flows,
I suppose.  Just didn't want to allocate a chunk of memory, and not use it.

You don't need to aggregate the flows to build signatures, or to label
traffic.  I don't really recommend it, but it is a good starting point so, no
harm, no foul. 

Flows can change their character during the life of the flow, but if
you aggregate, you will only match on the " first X bytes " in the
flow.  The feature is really designed to allow you to continuously
monitor flows for application conformance, allow you to know if
the application is still w

Re: raservices crashes when processing

Matt Brown — 2013-05-16T15:58:51

Good Morning Carter,





As far as collecting user data, looks good to me:



# radump -r * -s suser duser | wc -l

195492

# radump -r * -s suser duser | grep 's\[0\]=""' | wc -l

36307

# radump -r * -s suser duser | grep 's\[[1-9].*' | wc -l

159184



I used the data file produced with rastream:

rastream -d -S 127.0.0.1:561 -B 15s -M time 1h <x-apple-data-detectors://1> -w
/var/opt/argus/%Y-%m-%d/argus_%T -f /usr/local/bin/rastream.sh



argus running as:

argus -d -i eth0 -P 561



argus.conf with ARGUS_CAPTURE_DATA_LEN set:

# cat /etc/argus.conf | egrep -v '^$|^[#]'

ARGUS_FLOW_TYPE="Bidirectional"

ARGUS_FLOW_KEY="CLASSIC_5_TUPLE"

ARGUS_MONITOR_ID="..." #         // String

ARGUS_SET_PID=yes

ARGUS_PID_PATH="/var/run"

ARGUS_FLOW_STATUS_INTERVAL=60

ARGUS_MAR_STATUS_INTERVAL=300

ARGUS_CAPTURE_DATA_LEN=256







Working off the contents of ../support/Config/sig.std and Dave's great
advice, I performed the following:

# racluster -r * -w day.cache

# rauserdata -r day.cache -M printer="encode32" >

Re: raservices crashes when processing

Carter Bullard — 2013-05-16T15:13:35

Hey Dave,
Of course, everything in the clients has a constant defined somewhere.
Change the value of ARGUSMAXSIGFILE in ./include/argus_client.h to
something like this:

==== //depot/argus/clients/include/argus_client.h#64 - /Volumes/Users/carter/argus/clients/include/argus_client.h ====
142c142
< #define ARGUSMAXSIGFILE2048
---


Carter

On May 16, 2013, at 8:51 AM, "Dave Edelman" <dedelman< at >iname.com> wrote:

Re: raservices crashes when processing

Dave Edelman — 2013-05-16T12:51:50

The std.sig is fine but it is 435 lines long. 
If I use rauserdata to create a filter file which is longer than 2048 lines (including empty lines) raservices segfaults. If I take the first middle or last 2048 lines of my filter file, raservices is fine. If I remove all of the blank lines from my filter file I can still use any 2048 lines with no problem but raservices segfaults on 2049 lines in the filter file.

--Dave

From: Carter Bullard [mailto:carter< at >qosient.com] 
Sent: Thursday, May 16, 2013 8:37 AM
To: Dave Edelman
Cc: Matt Brown; <argus-info< at >lists.andrew.cmu.edu>
Subject: Re: [ARGUS] raservices crashes when processing

Hey Dave,
Not sure that I follow your situation.  So you're having problems with the provided sig.std or one you created?

Carter

On May 15, 2013, at 8:59 PM, "Dave Edelman" <dedelman< at >iname.com> wrote:
I had the same results so I looked at an example in the argus-client distribution. /support/Config/std.sig has this header:

#  Services fingerprint file, generated by:
#      raus

Re: raservices crashes when processing

Carter Bullard — 2013-05-16T12:37:23

Hey Dave,
Not sure that I follow your situation.  So you're having problems with the provided sig.std or one you created?

Carter

On May 15, 2013, at 8:59 PM, "Dave Edelman" <dedelman< at >iname.com> wrote:

Re: raservices crashes when processing

Carter Bullard — 2013-05-16T12:30:19

Hey Matt,
This is not a crash, which is a programatic unrecoverable fault.  You just didn't generate a good raservices() configuration file.

Try using the provided ./support/Config/sig.std, as a starting point for raservices(), to see if you can get good labels?

Are you sucessfully generating user data yet?

Carter

On May 15, 2013, at 5:55 PM, Matt Brown <matthewbrown< at >gmail.com> wrote:

Anomaly detection

Craig Merchant — 2013-05-16T06:29:38

Carter,

Thank you so much for your analysis of the APT1 threats.  Those emails were extremely educational.

I wanted to pick your brain about a couple of things related to anomaly detection...

We backhaul all remote offices through a central network that Argus can monitor.  Since those remote offices use DHCP, it's hard for Argus to build a reliable model of "normal" behavior by IP address.   And it can't see the MAC addresses of flows from those remote offices.  What's the best approach for anomaly detection in that kind of scenario?  Do you look at the producer/consumer metrics of the whole DHCP subnet and then compare individual flows against that baseline?

What kind of anomaly detection strategy do you use for environments where you have farms of different functional roles - web, MTA, database, etc.?  Do you recommend building a behavioral model by individual host or would you compare individual hosts against a baseline for that class of system?

Thanks.

Craig