gmane.network.netflow.nfdump.general

Re: nfdump-1.6.10 released

Evgheni Dereveanchin — 2013-05-18T14:21:51

No worries Peter, thanks for fixing it so quickly! And for the great
software, of course :)
I've built the reuploaded version and the issue is gone. Everything works
like a charm.

Regards,
Evgheni

-----Original Message-----
From: Peter Haag [mailto:phaag-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org] 
Sent: 18 May 2013 16:30
To: Evgheni Dereveanchin
Cc: 'nfdump-discuss-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org'
Subject: Re: [Nfdump-discuss] nfdump-1.6.10 released

I'm sorry for this problem! When I put 1.6.10 on sourceforge, I got the
report already from another user. I immediately withdrawed the package and
fixed the double free bug. Please pull a clean copy from sourceforge now.
It's fixed.

Sorry for the inconvenience.

- Peter

On 18/5/13 9:02 AM, Evgheni Dereveanchin wrote:
Dst
Download a free trial.

--
Be nice to your netflow data. Use NfSen and nfdump :)
------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform d

Re: nfdump-1.6.10 released

Peter Haag — 2013-05-18T13:30:02

I'm sorry for this problem! When I put 1.6.10 on sourceforge, I got the report already from another user. I immediately
withdrawed the package and fixed the double free bug. Please pull a clean copy from sourceforge now. It's fixed.

Sorry for the inconvenience.

- Peter

On 18/5/13 9:02 AM, Evgheni Dereveanchin wrote:

Re: nfdump-1.6.10 released

Evgheni Dereveanchin — 2013-05-18T07:02:53

Hi Peter,

Thanks for the new release!

Compiled it right away on CentOS 6.4 - it seems to collect deta OK, but I
receive segfaults when trying to search the data via nfsen:

----------------------------------------------------------------------------
---------
root< at >mon:~# nfdump -M /flow/nfsen/profiles-data/live/FW01  -T  -r
2013/05/17/nfcapd.201305171950 -a  -B -c 20
Date first seen          Duration Proto      Src IP Addr:Port           Dst
IP Addr:Port   Out Pkt   In Pkt Out Byte  In Byte Flows
<IP list displayed here, all looks OK>
Summary: total flows: 719998, total bytes: 215.9 M, total packets: 0, avg
bps: 5.8 M, avg pps: 0, avg bpp: 0
Time window: 2013-05-17 19:50:02 - 2013-05-17 19:55:02
Total flows processed: 719998, Blocks skipped: 0, Bytes read: 82083524
Sys: 0.262s flows/second: 2738061.8  Wall: 0.264s flows/second: 2721698.0
*** glibc detected *** nfdump: free(): invalid pointer: 0x00007f507c32e468
***
======= Backtrace: =========
/lib64/libc.so.6(+0x760e6)[0x7f507c0150e6]
nfdump[0x417ddf]
nfd

nfdump-1.6.10 released

Peter Haag — 2013-05-17T10:10:47

Hi List,
Just to let you know - 1.6.10 is out:

Maintainance/bugfix release.
You should update, if you use IPFIX, or ASA/NSEL

- Fix SPARC compile/optimise bug
- Add output packet/bytes counter to global stat for NSEL flows ASA > 8.5
  Fixes stat problems in NfSen
- Add NSEL filter options xnet
- Modify extension descriptor code for nfdump1.7.
  Still use 1.6 extension map layout for compatibility
- Add prototype for nfpcapd - pcap -> nfdump collector.
  Converts traffoc directly to nfdump files. - experimental - not enabled
- Fix bug in ipfix module: uninitialised variable
- Cleanup syslog/LogError calls
- Fix minor non critical bugs and compile issues

- Peter

Re: [nfdump 1.6.9] -- Dst IP option remove IPSOURCE

Peter Haag — 2013-05-16T16:43:29

If you aggregate dstip then only dst IP is relevant. nfdump-1.6.3 did not mask out zero other fields, which was a bug.

- Peter

On 16/5/13 4:31 PM, marcello pisano wrote:

Re: [nfdump 1.6.9] -- Dst IP option remove IPSOURCE

marcello pisano — 2013-05-16T14:31:26

Hello Mr Haag,

thank you for you answer. Sorry I didn't see that difference on command. My
target was only the option -A. Here another Example:



*nfdump: Version: 1.6.3 $LastChangedDate: 2011-01-09 12:28:32 +0100 (Sun,
09 Jan 2011) $*
*$Id: nfdump.c 69 2010-09-09 07:17:43Z haag $*

*  nfdump -T  -r nfcapd.201305161546  -A dstip -o extended -c 20*
*Date flow start          Duration Proto      Src IP Addr:Port          Dst
IP Addr:Port   Flags Tos  Packets    Bytes      pps      bps    Bpp Flows*
*Skip unknown record type 7*
*Skip unknown record type 9*
*Skip unknown record type 8*
*2013-05-16 15:45:59.952    58.997 UDP     172.16.162.137:58175 ->
239.255.255.250:1900  ......   0       51     7035        0      953    137
   51*
*2013-05-16 15:45:59.900    59.099 UDP      172.28.129.60:3000  ->
225.0.48.12:3000  ......  28   159575  216.4 M     2700   29.3 M   1355
603*
*2013-05-16 15:45:59.900    59.099 UDP      172.16.130.50:2000  ->
239.0.1.2:3000  ......   0   216648  291.2 M     3665   39.4 M   1343

Re: [nfdump 1.6.9] -- Dst IP option remove IPSOURCE

Peter Haag — 2013-05-16T13:27:04

Hi,
I don't see anything wrong. The two output listings represent, what you were nfdump asking for.

-a is equiv to -a -A proto,srcip,dstip,srcport,dstport

so you compare -a -A proto,srcip,dstip,srcport,dstport wiht -a -A dstip which obviously results in two different output
listings. Unused elements in a flow are zeroed out.

Hope, that helps.

- Peter

On 5/15/13 W20 12:57, marcello pisano wrote:

[nfdump 1.6.9] -- Dst IP option remove IP SOURCE

marcello pisano — 2013-05-15T10:57:12

Hello to all,

I did an upgrade from nfdump 1.6.3 to 1.6.9. I use often option "-A" to
aggregate flows, but after upgrade I have that if I use that option source
address of all flow becomes this:


*[root< at >test2 15]# nfdump  -r nfcapd.201305151054  -a  -A dstip -o extended
-c 2

Date first seen          Duration Proto           Src IP Addr:Port
Dst IP Addr:Port   Flags Tos  Packets    Bytes      pps      bps    Bpp
Flows**
**2013-05-15 10:53:59.903    59.077     0         ** 0.0.0.0:0**
->        224.0.0.1:0     ......   0      250    71370        4     9664
285   176**
**2013-05-15 10:54:00.900    58.000     0          **0.0.0.0:0**
->    172.16.50.212:0     ......   0       59     7744        1     1068
131    59**
*
If I don't use that option results is:

*[root< at >test2 15]# nfdump  -r nfcapd.201305151054  -a  -o extended -c 2*

*Date first seen          Duration Proto      Src IP Addr:Port          Dst
IP Addr:Port   Flags Tos  Packets    Bytes      pps      bps    Bpp Flows**
**2013-05-15 10:53:59.928    4

Perl module for working with nfdump files

Tomas Podermanski — 2013-05-14T12:37:16

Hi nfdump community,

    as the nfdump is a great tool we desperately missed the additional
functionality allowed us either modify or create the records in those
files. So we decided to develop own perl module. After using it in our
several projects we decided to publish the module to whole community.
The module is available via CPAN site 
(http://search.cpan.org/~tpoder/Net-NfDump/lib/Net/NfDump.pm). It uses
standard DBI-like conventions, so it should be quite easy to use.  Maybe
someone will find it useful for yourself.

Best regards
    Tomas



------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d

Re: vSphere 5.1 distributed switch to nfcapd withIPFIX

David Walsh — 2013-05-14T00:59:11

 FYI

I have finally got VMware looking at this for me.   I'll reply to the list when I get more information.   I am providing them with the logs of my vDS.

Cheers,
               David

On 07/05/2013, at 10:44 AM, David Walsh <davow-M13coBo2xU80n/F98K4Iww< at >public.gmane.org> wrote:



------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d

Re: vSphere 5.1 distributed switch to nfcapd with IPFIX

Peter Haag — 2013-05-07T07:08:26

Hmm .. obviously your VDS's don't send the date, as other IPFIX exporters do. I would need a packet dump of your VDS's
traffic to the nfcapd collector. If you can send me the data off list, I'll have a look.

- Peter


On 7/5/13 2:44 AM, David Walsh wrote:

vSphere 5.1 distributed switch to nfcapd with IPFIX

David Walsh — 2013-05-07T00:44:20

Hi,
     I have some vSphere 5.1 VDS's sending IPFIX net flow to our nfsen server.  (nfsen v 1.3.5)

I am running nfdump Version: 1.6.9 with the IPFIX patch posted on this list on the 13/4/2013 by Peter.

I am receiving the net flow data and below is the output in raw form after I applied the patch. You will notice that "first" and "last" are set on 1970-01-01 10:00:00. There is an up to date time in the last variable of the packet in "received at".

NFsen can read the data and it is correct (I compare it to data we pull via snmp) however NFsen /ndump are formatting the data with timestamps of 1970-01-01 10:00:00 instead of the actual time.

I notice this has been raised on various sites but I have not seen a fix.  I don't mind testing some patches if they become available to fix up this timestamp issue.



# nfdump -M /opt/data/nfsen/profiles-data/live/netflow-vds-vsh -R 2013/05/03/nfcapd.201305031040 -c 100 -o raw


Flow Record: 
  Flags        =              0x06 FLOW, Unsampled
  export sysid =

UNIX timestamps without using '-o pipe'

2013-05-06T05:37:35

Dear Peter,

It would be really useful to have an nfdump output format (e.g. to use with '-o fmt:') that allows to output UNIX timestamps for flow record start and end times. To the best of my knowledge, outputting UNIX timestamps is currently solely possible using '-o pipe'. However, this output format provides way more information than we need.

We plan to use this in an NfSen plugin, where we would like to parse a few fields from the nfdump files, including start and end times as UNIX timestamps.

Would it be possible to include something like this in an upcoming version of nfdump?

Kind regards,

--
Rick Hofstede
University of Twente, The Netherlands
------------------------------------------------------------------------------
Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET
Get 100% visibility into your production application - at no cost.
Code-level diagnostics for performance bottlenecks with <2% overhead
Download for free and get started troubleshooting in minutes.
http://p.sf

Re: nfdump extension problem

2013-04-22T15:47:04

Hi Peter,

Thanks a lot, this solved the problem right away!

Best regards,

--
Rick Hofstede

On Apr 22, 2013, at 4:16 PM, Peter Haag <phaag-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org> wrote:



------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter

Re: nfdump extension problem

Peter Haag — 2013-04-22T14:16:03

Hi Rick,

On 4/21/13 W16 16:42, R.J.Hofstede-HVpl4/oQiP/z+pZb47iToQ< at >public.gmane.org wrote:

If you have your .c and .h snippest, it would be a bit easier to diagnose the problem. However, I assume the problem is
the unpropper alignment of the second uint32_t. Which means your extension should to look like:

struct extension {
uint8_t  1;
        uint8_t  fill1;
        uint16_t fill2;
uint32_t 2;
uint32_t 3;
uint32_t 4;
uint32_t 5;
uint32_t 6;
}

not properly aligned values ask for trouble. Therefore make sure 16, 32 and 64 bit values are aligned.

- Peter

nfdump extension problem

2013-04-21T14:42:56

Dear all,

I'm trying to implement an extension for nfdump that stores and processes data from (enterprise-specific) NetFlow fields. In total, I've defined 6 fields and all but one (i.e. no. 2) are working fine. The fields have the following properties:

1: unint8
2: unint32 <== problem
3-6: uint32

Since the extension has to be 64-bit-aligned, I've defined the corresponding masks:

1: 0xFF00000000000000LL
2: 0x00FFFFFFFF000000LL
3: 0xFFFFFFFF00000000LL
4: 0x00000000FFFFFFFFLL
5: 0xFFFFFFFF00000000LL
6: 0x00000000FFFFFFFFLL

Again, field 1, 3, 4, 5 and 6 are working perfectly fine. The only difference between field no. 2 and the others is that its shifted by another value than 32 bits. It needs to carry a UNIX timestamp, which is correctly 'received' by nfcapd (I've verified that). However, after reading the file by nfdump, the value is wrong. For example, the HEX value '516C5DE4' (1366056420) results in the decimal value after processing '81'.

Does anyone of you have a clue what may be the (direction of th

Re: problems with nfcapd and openbsd's pflow interface for netflow > 5

Peter Haag — 2013-04-19T16:56:00

Which version of nfdump are you using?  1.6.9 should work find with IPFIX.
Otherwise I need to check.

- Peter


On 4/18/13 W16 15:12, Tor Houghton wrote:

problems with nfcapd and openbsd's pflow interfacefor netflow > 5

Tor Houghton — 2013-04-18T13:12:43

Hi,

I'm capturing flows on an OpenBSD 5.2 system using the pflow interface. When
I export data using version 5, nfcapd behaves as expected.

However, if I export the flows as version 9, or IPFIX, nfcapd has a problem
with the "first" and "last" fields of the flow record:

Flow Record: 
  Flags        =              0x06 FLOW, Unsampled
  export sysid =                 1
  size         =               564
  first        =                 0 [1970-01-01 01:00:00]
  last         =                 0 [1970-01-01 01:00:00]
  msec_first   =                 0
  msec_last    =                 0
..
..
..
  (src)tos     =                 0
  (in)packets  =                 6
  (in)bytes    =               598
  ip router    =      192.168.16.1
  received at  =     1366290086402 [2013-04-18 15:01:26.402]

Wireshark has no trouble decoding the packet. Has anyone else experienced
this? 

Tor

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capabl

Patch for nfdump IPFIX users

Peter Haag — 2013-04-13T12:50:58

Hi list,
If you are using IPFIX, you should apply the patch below. It fixes an uninitialised variable, when building up the
internal translation table:

--- nfdump-1.6.9/bin/ipfix.c    2012-11-10 12:40:54.000000000 +0100
+++ nfdump-current-8/bin/ipfix.c    2013-04-13 12:15:41.000000000 +0200
< at >< at > -615,6 +615,7 < at >< at >

    table->updated      = time(NULL);
    // IPFIX only has 64bit counters
+   table->flags            = 0;
    SetFlag(table->flags, FLAG_PKG_64);
    SetFlag(table->flags, FLAG_BYTES_64);
    table->ICMP_offset  = 0;

Otherwise, your flows may potentially lock corrupt for IP addresses v4/v6 and/or bytes/packet counters.

Cheers

- Peter

[PATCH] nfanon: add -p option to preserve ASnumbers

Tillmann Karras — 2013-04-09T19:42:35

Hello Peter,

I've made a patch that adds a command-line option to nfanon to preserve 
AS numbers. See [0] or attached. Feel free to merge, ignore, ...

Best regards
Tillmann

[0] 
https://github.com/Tilka/nfdump/commit/417c9281fab1a7ac4e23cc8d7f3d952b98c1cafe------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter_______________________________________________
Nfdump-discuss mailing list
Nfdump-discuss-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
https://lists.sourceforge.net/lists/listinfo/nfdump-discuss

Re: using nfdump for Cisco ASA

Evgheni Dereveanchin — 2013-04-02T06:25:04

Hi Monloi, also no packet count on 8.5.4

By the way, the traffic counts look odd as well. I was even able to download a 1 gig file over SFTP without this volume being noticed by nfdump. I'm using latest nfdump 1.6.9 ( nfdump-1.5.8-4-NSEL shows similar behavior )

Here are a few graphs: the flows graph looks normal while the Bytes one has spikes day and night, probably because not all traffic is counted.

[cid:image002.png-s1Uf9mtxXpbCYx1p5KPnZjRmByFHzeGd< at >public.gmane.org][cid:image004.png-s1Uf9mtxXpbCYx1p5KPnZjRmByFHzeGd< at >public.gmane.org]

The ASA configuration looks like this:

flow-export destination INTERNAL 192.168.1.2 9995
flow-export template timeout-rate 5
flow-export active refresh-interval 60
class-map netflow_export_class
match any
policy-map global_policy
class netflow_export_class
  flow-export event-type flow-create destination 192.168.1.2
  flow-export event-type flow-teardown destination 192.168.1.2

So just flow create and teardown events are sent (no flow-updates), this should transmit all