Vortex IDS - Get Super Snagadocious on Ubuntu

I got bored this weekend and decided that I should play with something new so I chose to mess with Vortex. From the README file:

Vortex is designed to facilitate agile creation of advanced network intrusion detection or network surveillance systems. It performs the hard work of packet capture, filtering, and reassembly then provides that data to external programs for analysis. Stream data is stored in files and stream meta data is output via STDOUT (meta data is encoded in file name). Vortex provides a platform for highly flexible, multithreaded, near real time deep analysis of network payload data.

The first thing I noticed when trying to mess with this stuff was there was hardly any documentation. The README file had enough info though to get started. I decided I wanted to try another method for grabbing streams to analyze the email from the SANS challenge #2. Although Xplico is pretty pimp I don't see it scaling to a large environment. Also I want more flexibility in case I am looking for specific threats etc. So let's get down to business.

Installing Vortex

I decided to use Ubuntu for this test so lets get the source:

$ wget http://downloads.sourceforge.net/project/vortex-ids/vortex/2.8.1/vortex-2.8.1.tgz?use_mirror=cdnetworks-us-1

$ tar zxvf vortex-2.8.1.tgz

$ cd vortex/

$ more README

You should really check out the README file as there is a lot in there of value. So in order to get this mug to compile properly we need to solve some dependancies. Run the following command:

$ sudo apt-get install libnids-dev libnet-dev libpcap-dev

Now let's compile the source:

$ gcc vortex.c -lnids -lpthread -Wall -o vortex

Run it to see if it works:

$ ./vortex -h

Usage: ./vortex [ -lpmheI ] [ -c count ] [ -i device ] [ -r file ] [ -u user ] [ -S bytes ] [ -C bytes ] [-t dir ] [ -s count ] [ -H count ] [ -q limit ] [ -D level ] [-F file | -f filter ] [-M MTU (snaplen)] [-P poll rate] [ -TEK time ] [ -Q size ] [ -R usecs ] [ -Nn prio ] [ -Oo cpu ] [ -L name ]

-h print this help message and exit
-c count set number to connections to follow
-i device listen on device
-r file read capture from pcap file
-l set output to line buffering
-p don't put interface(s) in promiscuous mode
-u user after initialization, setuid to user
-S bytes number of bytes to collect from client to server Default: 104857600 (100MB)
-C bytes number of bytes to collect from server to client Default: 104857600 (100MB)
-t dir directory for storage of stream data (defaut: currend working dir)
-s count Size of connection hash table--Maximum number of streams to follow simultaneously = 3/4 * count. Default: 1048576
This affects memory consumption significantly. If you have problems with TCP_LIMIT, increase this value. (See n_tcp_streams in libNIDS)
-H count size of IP defrag has table. Default: 65536 (See n_hosts in libNIDS)
-m enable libNIDS multiprocess mode DEPRICATED--don't use this (See multiproc in libNIDS)
-q limit set libNIDS packetqueue limit. DEPRICATED--only applies in multiproc mode
-D level set debug level Default: 0
-f filter tcpdump-style capture filter expression (don't forget quotes/shell escapes)
-F file file containing packet filter expression
-M MTU MTU or snaplen--maximum packet size to capture. default: 1560
-w enable libNIDS TCP/IP stack workaround mode (See TCP_workarounds in libNIDS)
-k disable libNIDS TCP/IP checksum processing (See TCP_checksums in libNIDS)
-P rate Only reassemble and collect every poll rate connections. default: 1
-T time Report Performance Statistics every time seconds (approx) default: 0
-E time Report Error counts every time seconds (approx) default: 0
-L name Logging name for syslog. Default: vortex
-Q size Size of output ring queue. Sets limit for number of finished streams waiting to be written. Default: 10000
-R usec Wait period in us (inverse of poll rate) for stream output thread in microseconds. Default: 10000
-n prio Priority (niceness) for capture thread. Can be from -20 to 19 on most systems. Default: -15
-N prio Priority (niceness) for other threads. Can be from -20 to 19 on most systems. Default: 10
-o cpu CPU to bind capture thread to. Default: 1
-O cpu CPU to bind other threads to. Default: 0
-I Lock threads to specific cores. (see o and O above). Default is to not lock so specific cores (Expiramental--still not working properly!).
-e enable extended output (more metadata in file name).
-K TCP Idle connection timeout in seconds Default: -1 (disabled). This timeout ignores empty keepalives.
-v Output empty streams (create files with 0 bytes).

This thing has some crazy options in it. There is also something in there called xpipes. Not going to use this but we should make it work anyway.

$ gcc xpipes.c -lpthread -Wall -o xpipes

Ok now we are ready to analyze some data. Lets grab the pcap from the SANS example:

$ wget http://forensicscontest.com/contest02/evidence02.pcap

Now let's analyze it with vortex:

$ ./vortex -r evidence02.pcap -t /tmp/vortexout
Couldn't set capture thread priority!
/tmp/vortexout/192.168.1.159:1036s64.12.102.142:587
/tmp/vortexout/192.168.1.159:1036c64.12.102.142:587
/tmp/vortexout/192.168.1.159:1038s64.12.102.142:587
/tmp/vortexout/192.168.1.159:1038c64.12.102.142:587
VORTEX_ERRORS TOTAL: 0 IP_SIZE: 0 IP_FRAG: 0 IP_HDR: 0 IP_SRCRT: 0 TCP_LIMIT: 0 TCP_HDR: 0 TCP_QUE: 0 TCP_FLAGS: 0 UDP_ALL: 0 SCAN_ALL: 0 VTX_RING: 0 OTHER: 0
VORTEX_STATS PCAP_RECV: 0 PCAP_DROP: 0 VTX_BYTES: 288287 VTX_EST: 2 VTX_WAIT: 0 VTX_CLOSE_TOT: 2 VTX_CLOSE: 2 VTX_LIMIT: 0 VTX_POLL: 0 VTX_TIMOUT: 0 VTX_IDLE: 0 VTX_RST: 0 VTX_EXIT: 0 VTX_BSF: 0

We end up with this:

-rw-r--r-- 1 507 2010-02-28 14:20 192.168.1.159:1036c64.12.102.142:587
-rw-r--r-- 1 1495 2010-02-28 14:20 192.168.1.159:1036s64.12.102.142:587
-rw-r--r-- 1 507 2010-02-28 14:20 192.168.1.159:1038c64.12.102.142:587
-rw-r--r-- 1 285778 2010-02-28 14:20 192.168.1.159:1038s64.12.102.142:587

Let's take a looksie at the contents of one of the files:

EHLO annlaptop
AUTH LOGIN
c25lYWt5ZzMza0Bhb2wuY29t
NTU4cjAwbHo=
MAIL FROM:
RCPT TO:
DATA
Message-ID: <000901ca49ae$89d698c0$9f01a8c0@annlaptop>
From: "Ann Dercover"
To:
Subject: lunch next week
Date: Sat, 10 Oct 2009 07:35:30 -0600
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0006_01CA497C.3E4B6020"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.2180
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180

--SNIP--

So there are all kinds of goodies in there. We see the entire email content in this easily searchable stream.

Possibilities?

There are a ton of different things you could do at this point. Let's say you just want to capture all email from an smtp gateway you could run the following:

$ vortex -i eth1 -t /data/vortexstreams -f "tcp port 25" -e -K 300

That would give you an archive of all your emails sent and received from your gateways. (assuming you tapped the interface of said gateways and plugged eth1 into the tap) You want to know all emails sent to sec558@gmail.com? Try this:

$ grep "RCPT TO: sec558@gmail.com" *
192.168.1.159:1036s64.12.102.142:587:RCPT TO: sec558@gmail.com

note: Blogger doesn't like <> so you need to wrap the email in them to actually make it work.

Some new worm send a document called secretrendezvous.docx and you want to know who got it?

$ grep "filename=\"secretrendezvous.docx\"" *
192.168.1.159:1038s64.12.102.142:587: filename="secretrendezvous.docx"

$ grep "RCPT TO" 192.168.1.159\:1038s64.12.102.142\:587
RCPT TO: mistersecretx@aol.com

Time to rebuild mistersecretx's machine. So again the sky is the limit here on stuff you can do. You can easily write a script that scans the directory that the streams are in or you can squirt it directly into a custom analyzer.

In the README there is an example to analyze on the fly:

vortex -i eth0 -s 1000000 -C 0 -S 1000 -l -t /dev/shm/ -f "tcp port 21" -e -K 300 | while read file
#Loop on each stream file. file contains metadata that looks something like "/dev/shm/tcp-38-1261503711-1261503711-c-174-172.16.17
.18:3787s10.20.30.40:21"
do
#extract credentials from stream
CREDS=`cat $file | tr '\r\n' ' ' | sed -r 's/.*USER (\S+) PASS (\S+) .+$/\1 \2/g'`
#CREDS is something like "username password"

RESULT=`echo "$CREDS" | cut -f 1 -d " " --complement | /usr/sbin/cracklib-check`
#RESULTS is either "password: warning message" or "password: OK"

#If password isn't OK, the log then username, warning message, and connection info
if ! echo $RESULT | grep "OK$" > /dev/null
then
USER=`echo $CREDS | cut -f 1 -d " "`
MESSAGE=`echo $RESULT | awk -F: '{ print $NF }'`
METADATA=`basename $file`

echo "Weak password ($MESSAGE ) for $USER in connection $METADATA" | logger -t "ftp_pass_check" -s
fi

#Purge the file
rm $file
done

So far I am really impressed by Vortex. This thing looks really cool as to what you can do with it. I really want to play with it some more and will definately do a part 2 to this post where I take this example to the next level. There are a few answers I need to come up with first though. Do I want an entire archive of sent and received internet emails? What are the legal ramifications of said archive? Do I want to only save streams of emails with links embedded in them or only emails with attachments? Do I want to extract the actual attachment and run them through something else? Do I want to index these badboys into a database?

Before you answer these questions consider this scenario. Your organization is targeted for a 0-day attack via a phishing email using a malicious pdf file. Let's say this malware creates a conduit for a badguy to take control of the users PC. Then the badguy deletes the email and the pdf and now has full control of this machine. You could say wait.. I have AV running on my mail servers and on the PC. But when it came in AV never detected it and now since they deleted it its gone and will never be detected again. This is where an external email archive would come in handy. You track down a user who noticed this email seemed weird with an attachment named "Marketing Report.pdf". You could now go to your streams directory and grep the streams for "filename=MarketingReport.pdf" and get a list of all the emails that came in with that attachment and the users whom received it.

Keep an eye out for part 2 :)