Home > Big Data and Hadoop > Using NetFlow/IPFIX to detect cyber threats

Using NetFlow/IPFIX to detect cyber threats

Most commercial switches, routers, firewalls in your datacenter support NetFlow (network logs) and in some cases IPFIX (next-gen replacement for NetFlow version 9).  Why not use NetFlow to alert you to cyber threats like worms, botnets, Advanced Persistent Threats (APT)?                         NetFlow solution architecture

The idea of using NetFlow is not a new one.  Argonne National Labs have been using NetFlow to detect zero-day attacks since 2006.

An Advanced Persistent Threat (APT) starts by mining employee data from Facebook, LinkedIn and other social media sites and focuses on stealing a corporation’s intellectual property using innocuous applications like Skype to move the content around.   APTs fly under the radar of signature-based perimeter security appliances like firewalls and Intrusion Detection Systems (IDS).  However, you can use NetFlow/IPFIX to identify APTs by comparing flows in the NetFlow/IPFIX collector with a host reputation database offered by cloud services like McAfee GTI.  The actual ingest of the host reputation database and comparison with flows would involve a tool like Plixer Scrutinizer™.  By blocking traffic going to the known compromised hosts (which hosts the APT command and control malware) you are neutralizing the goal of the adversary who sent the APT into your network.

Why bother with IPFIX (next-gen NetFlow) when there are older versions of NetFlow?

You can export URL information via IPFIX (using vendor extensions supported by IPFIX).  This allows you to determine what URL a user clicked on before succumbing to malware.  How many other people clicked on the same bad URL?  Products which export URL information via IPFIX include Ntop nProbe, Dell SonicWALL, Citrix AppFlow.

For Voice-over-IP (VoIP) traffic you can export details like caller-id, codec, jitter and packet loss.

Why use dedicated NetFlow/IPFIX sensors when routers/switches/firewalls may suffice? 

Even router vendors like Cisco recognize that customers who buy high end routers may not want to expend expensive CPU cycles on NetFlow/IPFIX generation nor rely on sampling NetFlow which makes it unusable for cyber-security applications.  The need is for offloaded appliances that product packet-accurate non-sample NetFlow/IPFIX.  Cisco’s own NetFlow Generation Appliance (NGA) is an option.  The older NGA 3140 tops out at 120,000 frames per second (fps).   Higher end offload appliances from some  vendors can sustain 250,000 to 500,000 fps to keep up with busy 10 Gb network pipes.

So we have a way to generate NetFlow/IPFIX but what about the analytics needed to actually detect cyber-attacks?  While you may have a traditional SIEM (HP ArcSight ESM, McAfee ESM, IBM QRadar) or a tool like Splunk it is unrealistic to send NetFlow/IPFIX data at very high rates into these systems.  A better way would be to trim down the traffic and analyze it on the wire before sending it to the SIEM.

NetFlow Logic a bay area startup has a high volume NetFlow processing product “NetFlow Integrator” which can ingest NetFlow/IPFIX records and process the stream in flight using an in-memory database. The product scales its throughput based on the number of underlying server cores.  For instance a 16 core server would allow it to scale throughput to over 500,000 fps.

The product is not a NetFlow collector but is categorized as a NetFlow/IPFIX Mediator (see RFC 5982).  NetFlow Integrator reduces NetFlow data by consolidating information into “conversations” rather than flows within a conversation.  Flow records are processed by one or more rules (canned or custom – creating using a GUI/SDK) which have their own logic to apply to each flow record.  These rules can aid in the following types of detection:

Detection of Botnet:

Botnet detection using NetFlow IntegratorA user would load a list of known Command & Control servers (possibly obtained from sites like Emerging Threats or from your own private source) into the rule.  Every incoming NetFlow record is examined to determine if the source or destination IP address matches this list.  If there is a match this matched information is forwarded to the SIEM.  The SIEM in turn will alert a security analyst if any botnet slaves are detected on the network.

APT detection:

NetFlow Integrator has rules to identify scanners who are doing “port sweeps” of your network.  It can also look for data exfiltration by examining an infected host that starts proliferating on the internal network.  A custom algorithm detects when a client suddenly starts behaving like a server.  This is something that can’t be done by signature based firewalls. 

In conclusion, to detect malware/botnets/APT use NetFlow/IPFIX which is something your routers, switches, firewalls support today.  Keep your existing SIEM in place but introduce an IPFIX offload appliance especially if you have large 10 Gb network pipes and don’t want to burden your routers.  Use a tool like that from NetFlow Logic to analyze NetFlow/IPFIX records on the wire and use your SIEM for the alerting and remediation.

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: