Vortex IDS - Get Super Snagadocious on Ubuntu

I got bored this weekend and decided that I should play with something new so I chose to mess with Vortex. From the README file:

Vortex is designed to facilitate agile creation of advanced network intrusion detection or network surveillance systems. It performs the hard work of packet capture, filtering, and reassembly then provides that data to external programs for analysis. Stream data is stored in files and stream meta data is output via STDOUT (meta data is encoded in file name). Vortex provides a platform for highly flexible, multithreaded, near real time deep analysis of network payload data.

The first thing I noticed when trying to mess with this stuff was there was hardly any documentation. The README file had enough info though to get started. I decided I wanted to try another method for grabbing streams to analyze the email from the SANS challenge #2. Although Xplico is pretty pimp I don't see it scaling to a large environment. Also I want more flexibility in case I am looking for specific threats etc. So let's get down to business.

Installing Vortex

I decided to use Ubuntu for this test so lets get the source:

$ wget http://downloads.sourceforge.net/project/vortex-ids/vortex/2.8.1/vortex-2.8.1.tgz?use_mirror=cdnetworks-us-1

$ tar zxvf vortex-2.8.1.tgz

$ cd vortex/

$ more README

You should really check out the README file as there is a lot in there of value. So in order to get this mug to compile properly we need to solve some dependancies. Run the following command:

$ sudo apt-get install libnids-dev libnet-dev libpcap-dev

Now let's compile the source:

$ gcc vortex.c -lnids -lpthread -Wall -o vortex

Run it to see if it works:

$ ./vortex -h

Usage: ./vortex [ -lpmheI ] [ -c count ] [ -i device ] [ -r file ] [ -u user ] [ -S bytes ] [ -C bytes ] [-t dir ] [ -s count ] [ -H count ] [ -q limit ] [ -D level ] [-F file | -f filter ] [-M MTU (snaplen)] [-P poll rate] [ -TEK time ] [ -Q size ] [ -R usecs ] [ -Nn prio ] [ -Oo cpu ] [ -L name ]

-h print this help message and exit
-c count set number to connections to follow
-i device listen on device
-r file read capture from pcap file
-l set output to line buffering
-p don't put interface(s) in promiscuous mode
-u user after initialization, setuid to user
-S bytes number of bytes to collect from client to server Default: 104857600 (100MB)
-C bytes number of bytes to collect from server to client Default: 104857600 (100MB)
-t dir directory for storage of stream data (defaut: currend working dir)
-s count Size of connection hash table--Maximum number of streams to follow simultaneously = 3/4 * count. Default: 1048576
This affects memory consumption significantly. If you have problems with TCP_LIMIT, increase this value. (See n_tcp_streams in libNIDS)
-H count size of IP defrag has table. Default: 65536 (See n_hosts in libNIDS)
-m enable libNIDS multiprocess mode DEPRICATED--don't use this (See multiproc in libNIDS)
-q limit set libNIDS packetqueue limit. DEPRICATED--only applies in multiproc mode
-D level set debug level Default: 0
-f filter tcpdump-style capture filter expression (don't forget quotes/shell escapes)
-F file file containing packet filter expression
-M MTU MTU or snaplen--maximum packet size to capture. default: 1560
-w enable libNIDS TCP/IP stack workaround mode (See TCP_workarounds in libNIDS)
-k disable libNIDS TCP/IP checksum processing (See TCP_checksums in libNIDS)
-P rate Only reassemble and collect every poll rate connections. default: 1
-T time Report Performance Statistics every time seconds (approx) default: 0
-E time Report Error counts every time seconds (approx) default: 0
-L name Logging name for syslog. Default: vortex
-Q size Size of output ring queue. Sets limit for number of finished streams waiting to be written. Default: 10000
-R usec Wait period in us (inverse of poll rate) for stream output thread in microseconds. Default: 10000
-n prio Priority (niceness) for capture thread. Can be from -20 to 19 on most systems. Default: -15
-N prio Priority (niceness) for other threads. Can be from -20 to 19 on most systems. Default: 10
-o cpu CPU to bind capture thread to. Default: 1
-O cpu CPU to bind other threads to. Default: 0
-I Lock threads to specific cores. (see o and O above). Default is to not lock so specific cores (Expiramental--still not working properly!).
-e enable extended output (more metadata in file name).
-K TCP Idle connection timeout in seconds Default: -1 (disabled). This timeout ignores empty keepalives.
-v Output empty streams (create files with 0 bytes).

This thing has some crazy options in it. There is also something in there called xpipes. Not going to use this but we should make it work anyway.

$ gcc xpipes.c -lpthread -Wall -o xpipes

Ok now we are ready to analyze some data. Lets grab the pcap from the SANS example:

$ wget http://forensicscontest.com/contest02/evidence02.pcap

Now let's analyze it with vortex:

$ ./vortex -r evidence02.pcap -t /tmp/vortexout
Couldn't set capture thread priority!
/tmp/vortexout/192.168.1.159:1036s64.12.102.142:587
/tmp/vortexout/192.168.1.159:1036c64.12.102.142:587
/tmp/vortexout/192.168.1.159:1038s64.12.102.142:587
/tmp/vortexout/192.168.1.159:1038c64.12.102.142:587
VORTEX_ERRORS TOTAL: 0 IP_SIZE: 0 IP_FRAG: 0 IP_HDR: 0 IP_SRCRT: 0 TCP_LIMIT: 0 TCP_HDR: 0 TCP_QUE: 0 TCP_FLAGS: 0 UDP_ALL: 0 SCAN_ALL: 0 VTX_RING: 0 OTHER: 0
VORTEX_STATS PCAP_RECV: 0 PCAP_DROP: 0 VTX_BYTES: 288287 VTX_EST: 2 VTX_WAIT: 0 VTX_CLOSE_TOT: 2 VTX_CLOSE: 2 VTX_LIMIT: 0 VTX_POLL: 0 VTX_TIMOUT: 0 VTX_IDLE: 0 VTX_RST: 0 VTX_EXIT: 0 VTX_BSF: 0

We end up with this:

-rw-r--r-- 1 507 2010-02-28 14:20 192.168.1.159:1036c64.12.102.142:587
-rw-r--r-- 1 1495 2010-02-28 14:20 192.168.1.159:1036s64.12.102.142:587
-rw-r--r-- 1 507 2010-02-28 14:20 192.168.1.159:1038c64.12.102.142:587
-rw-r--r-- 1 285778 2010-02-28 14:20 192.168.1.159:1038s64.12.102.142:587

Let's take a looksie at the contents of one of the files:

EHLO annlaptop
AUTH LOGIN
c25lYWt5ZzMza0Bhb2wuY29t
NTU4cjAwbHo=
MAIL FROM:
RCPT TO:
DATA
Message-ID: <000901ca49ae$89d698c0$9f01a8c0@annlaptop>
From: "Ann Dercover"
To:
Subject: lunch next week
Date: Sat, 10 Oct 2009 07:35:30 -0600
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_NextPart_000_0006_01CA497C.3E4B6020"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.2180
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180

--SNIP--

So there are all kinds of goodies in there. We see the entire email content in this easily searchable stream.

Possibilities?

There are a ton of different things you could do at this point. Let's say you just want to capture all email from an smtp gateway you could run the following:

$ vortex -i eth1 -t /data/vortexstreams -f "tcp port 25" -e -K 300

That would give you an archive of all your emails sent and received from your gateways. (assuming you tapped the interface of said gateways and plugged eth1 into the tap) You want to know all emails sent to sec558@gmail.com? Try this:

$ grep "RCPT TO: sec558@gmail.com" *
192.168.1.159:1036s64.12.102.142:587:RCPT TO: sec558@gmail.com

note: Blogger doesn't like <> so you need to wrap the email in them to actually make it work.

Some new worm send a document called secretrendezvous.docx and you want to know who got it?

$ grep "filename=\"secretrendezvous.docx\"" *
192.168.1.159:1038s64.12.102.142:587: filename="secretrendezvous.docx"

$ grep "RCPT TO" 192.168.1.159\:1038s64.12.102.142\:587
RCPT TO: mistersecretx@aol.com

Time to rebuild mistersecretx's machine. So again the sky is the limit here on stuff you can do. You can easily write a script that scans the directory that the streams are in or you can squirt it directly into a custom analyzer.

In the README there is an example to analyze on the fly:

vortex -i eth0 -s 1000000 -C 0 -S 1000 -l -t /dev/shm/ -f "tcp port 21" -e -K 300 | while read file
#Loop on each stream file. file contains metadata that looks something like "/dev/shm/tcp-38-1261503711-1261503711-c-174-172.16.17
.18:3787s10.20.30.40:21"
do
#extract credentials from stream
CREDS=`cat $file | tr '\r\n' ' ' | sed -r 's/.*USER (\S+) PASS (\S+) .+$/\1 \2/g'`
#CREDS is something like "username password"

RESULT=`echo "$CREDS" | cut -f 1 -d " " --complement | /usr/sbin/cracklib-check`
#RESULTS is either "password: warning message" or "password: OK"

#If password isn't OK, the log then username, warning message, and connection info
if ! echo $RESULT | grep "OK$" > /dev/null
then
USER=`echo $CREDS | cut -f 1 -d " "`
MESSAGE=`echo $RESULT | awk -F: '{ print $NF }'`
METADATA=`basename $file`

echo "Weak password ($MESSAGE ) for $USER in connection $METADATA" | logger -t "ftp_pass_check" -s
fi

#Purge the file
rm $file
done

So far I am really impressed by Vortex. This thing looks really cool as to what you can do with it. I really want to play with it some more and will definately do a part 2 to this post where I take this example to the next level. There are a few answers I need to come up with first though. Do I want an entire archive of sent and received internet emails? What are the legal ramifications of said archive? Do I want to only save streams of emails with links embedded in them or only emails with attachments? Do I want to extract the actual attachment and run them through something else? Do I want to index these badboys into a database?

Before you answer these questions consider this scenario. Your organization is targeted for a 0-day attack via a phishing email using a malicious pdf file. Let's say this malware creates a conduit for a badguy to take control of the users PC. Then the badguy deletes the email and the pdf and now has full control of this machine. You could say wait.. I have AV running on my mail servers and on the PC. But when it came in AV never detected it and now since they deleted it its gone and will never be detected again. This is where an external email archive would come in handy. You track down a user who noticed this email seemed weird with an attachment named "Marketing Report.pdf". You could now go to your streams directory and grep the streams for "filename=MarketingReport.pdf" and get a list of all the emails that came in with that attachment and the users whom received it.

Keep an eye out for part 2 :)

Xplico and SANS Forensic Challenge #2

So after installing Xplico I wanted to give it a shot and do a forensic investigation using it. I chose this puzzle since at the time I was playing with this it had just wrapped up and the answers were available to check against. Now in order to complete this thing I will have to use more tools than Xplico but I will try and use it wherever possible.

So let's log in and create a new case. We shall call it SANS Puzzle 2:



Now click on the case so we can create a new session. We shall call it "Let's Rock This":



OK... Click on the session. You should get a purty screen that looks like this:



Time to upload the pcap file. All the cool kids md5sum the evidence file so we know we are working with the right stuff here. It should be:

cfac149a49175ac8e89d5b5b5d69bad3 evidence02.pcap

Give it a few moments (depending on your puter) and let it do it's thing. You should get a screen that looks like this:



Now let's look at the questions from the contest. Question #1 What is Ann's email?

So normally you could fire up Wireshark on this mug and find the SMTP packets and get the info. Who wants to do that though when you can click email on the left hand side and see all the emails contained in the pcap? :)



We click the first email listed and what can one say? Scandalous! Here is the email:



So we can answer question #1. Ann's email is Ann Dercover sneakyg33k@aol.com

Now we need her password which is question #2. This is a pretty nice feature with Xplico. Hover over the info.xml and save the pcap.

Open the pcap in Wireshark and cruise down to packet #14 to grab the Base64 encoded password.



Ann really should think about using an email provider that doesn't pass credentials in the clear. Cruise over to here and paste the password in there (NTU4cjAwbHo= is what you paste but you already knew that) and you hit decode and PLOW!!!! Ann's password is 558r00lz

Question #3: What is Ann's "Lover's" email? mistersecretx@aol.com is in the email we looked at above.

Question #4: What items is mistersecretx supposed to bring with him? Again look in the email above and we can see that he is supposed to bring his bootleg passport and a speedo. (Technically the answer is a fake passport and a bathing suit)

Question #5: What is the name of the attachment? Easy one again with Xplico. Look at the email and you see it right there. secretrendezvous.docx

Question #6: What is the MD5 sum of the attachment? On this one you have to do a little massaging for ease of use later. When I save the attachment in the email it saves it as "3". I renamed it so that the name was right then pumped it through md5sum:

9e423e11db88f01bbff81172839e1923 secretrendezvous.docx

Question #7: In what CITY and COUNTRY is their rendez-vous point? Since we renamed it to the correct extension openoffice opened it right up and here is what it looks like:


VIVA LA MEXICO!!!!! The answer is: Playa del Carmen, Mexico

Question #8: What is the MD5 sum of the embedded picture? Xplico doesn't really help here except for getting us the attachment to work with. So in a terminal run the following commands:

$unzip secretrendezvous.docx
$cd word
$cd media
$md5sum image1.png

aadeace50997b1ba24b09ac2ef1940b7 image1.png

That's it as far as the contest is concerned. This thing has a ton of capabilities and I plan on exploring them further by doing some of the other challenges so check back. When I originally tried this challenge I used tcpdump and wireshark to get a lot of these answers but it was very time consuming. When time is of the essence it is nice to have tools that make things easy. Looking at the forums I see that this is actively being improved and I am very impressed overall by the tool.

There are some gaps with this tool though. I recently did the Honeynet Challenge #1 and Xplico doesn't help at all really. You can see some of the FTP information but other than that there really isn't anything there. It does look promising for the Honeynet Challenge #2 though :)

Installing Xplico

I have been scouring the internet for some useful forensic tools when I cam across a tool call Xplico. I wanted to do some of the forensic challenges offered by the folks over at SANS and it looks like to win anything you have to write tools. Realistically in a high paced environment the luxury of having time to write your own tools sometimes doesn't exist.

Xplico has a couple of install options. First they have a virtualbox VM that you can download. If you are running Ubuntu 9.10 they have a nice .deb package for you. And for the brave ones they have the source that you can build yourself. I am running Ubuntu 9.10 64bit style so I need to do a little extra to get the prepackage to work.

So let's get our download on:

$ wget http://downloads.sourceforge.net/project/xplico/Xplico%20versions/version%200.5.4/xplico_0.5.4_i386.deb?use_mirror=softlayer

When I tried to force install it on a 64bit OS I got all this stuff:

xplico depends on libice6 (>= 1:1.0.0); however:
Package libice6 is not installed.
xplico depends on libmysqlclient16 (>= 5.1.21-1); however:
Package libmysqlclient16 is not installed.
xplico depends on libsm6; however:
Package libsm6 is not installed.
xplico depends on libsqlite0 (>= 2.8.17); however:
Package libsqlite0 is not installed.
xplico depends on libxt6; however:
Package libxt6 is not installed.
xplico depends on tshark; however:
Package tshark is not installed.
xplico depends on python-all; however:
Package python-all is not installed.
xplico depends on apache2.2-common; however:
Package apache2.2-common is not installed.
xplico depends on php5-common; however:
Package php5-common is not installed.
xplico depends on libapache2-mod-php5; however:
Package libapache2-mod-php5 is not installed.
xplico depends on php5-sqlite; however:
Package php5-sqlite is not installed.
xplico depends on php5-cli; however:
Package php5-cli is not installed.

Ubuntu is supossed to be cool enough to go download all this crap but somehow it was stuck. So I did the following:

$ sudo apt-get install libice6 tshark libsm6 build-essential

After that I was able to give it a:

$ sudo apt-get -f install

This basically fixed all my jacked up packages and looked like everything installed properly.

So at this point lets see if it worked. Give your terminal one of these

$ xplico -h

It should print out the help. So now lets fire up our interwebs and see what this looks like. Connect to the host you installed it on on port 9876.


So that is how I got mine working. Coming up next is using this tool :)