Home > Big Data and Hadoop > Detecting zero day attacks using big data

Detecting zero day attacks using big data

Snort and Intrusion Detection: 

Snort is a widely used open-source network Intrusion Detection System (NIDS) capable of both traffic analyses in real time as well as packet logging.  The reason for its popularity is that it is open source and effective at detecting everything from port scans, buffer overflows to OS fingerprinting attempts.  Known attacks follow a certain activity pattern and these are captured in “signatures” available from the open source community as well as from SOURCEfire.

Snort architectureIn the Snort architecture, the packet sniffer as the name suggests eavesdrops on network traffic, the preprocessor checks packets against plug-ins to determine if the packets exhibit a certain behavior, the detection engine takes incoming packets and runs them through a set of rules.  If the rules match what is in the packet then an alert is generated.  The alert may go to a log file or a MySQL or PostGres database.

Using Snort on big data stored in Hadoop

What if today you received new Snort signatures you didn’t have 3 months ago but want to use the new signatures to detect zero-day attacks (unknown exploits) in historical packet capture data?  This historical packet capture data may be in archive storage within your corporate data center or located on cloud storage like Amazon S3.

One solution is to analyze full packet captures using Apache Pig (a tool that abstracts a user from the complexities of MapReduce).  If you aren’t comfortable using MapReduce but have a few days of packet capture data on your laptop and know how to write some queries to query this local capture data, you can then transition your queries to a Hadoop cluster containing weeks or months of packet capture data using an open-source tool called PacketPig.

PacketPig  (an open source project located on github) offers many loaders (Java programs which provide access to specific info in a packet capture) one of which is SnortLoader () which allows you to analyze months of packet capture data dispersed across Hadoop nodes. The way you detect a zero day attack using PacketPig is by using SnortLoader() to scan archived packet capture data using old Snort signatures (from an old snort.conf file) then scanning it a second time using the latest Snort signatures.  After filtering out signatures that appear in both scans what you have left are zero day attacks.  More details of how this is done may be found here.  This is yet another example of how Hadoop running on commodity servers with direct attached storage can help provide a cyber-security solution for zero day attacks.

  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: