Tuesday, December 21, 2010

Breaking up with VMware at home. Hello Virtualbox..

For several years I have run many VMs at the crib using VMWare server. It is free and easy to use so it was a great choice for the house. I could see past all the bloat of the tomcat web interface stuff since it was easy to use and maintain. Well recently a security vulnerability came out for VMWare Server and if you run it on linux you are SOL. I finally found the motivation to move to VirtualBox.

First thing I did was shut down all of my existing VMWare VMs. I then issued the command:
sudo /etc/init.d/vmware stop

In my experience this is not enough to kill all of the tomcat crap so find those lingering processes and knock them down kill -9 style.

The next thing I did was make a copy of my current virtual machine so if I jacked something up I could always fall back to VMWare. Since VMWare has root owning the Virtual Machine you need to own it like pow.
sudo cp -R (vmwarevmpath)/(vmname) /whatever/vmname.vbox
sudo chown -R me:me /whatever/vmname.vbox

Ok so now we got the hard part done so lets go ahead and install VirtualBox. This is an Ubuntu 9.04 jaunty box so I did the added following to my /etc/apt/sources.list. If you are using a different Ubuntu version substitute your version:
deb http://download.virtualbox.org/virtualbox/debian jaunty non-free

If you are using something other than Ubuntu install VirtualBox following the instructions for that distro. Let's continue on for those using Ubuntu:
wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
sudo apt-get update
sudo apt-get install virtualbox-3.2

Ok now we need to do a little more prep work so we can get this rocking. Add your userid to the vbox user group:
sudo useradd -G vboxusers me

Log out then back in and you should be cool to continue. Ok time to make the donuts. First lets find the correct OS type.You do this with issuing the following:
$ VBoxManage list ostypes

This will give you a big list of OS types. Find the one that applies to your old VM. I was running Ubuntu 64 bit on this particular VM. So lets register our new VM.
VBoxManage createvm --name "SweetVM" --ostype Ubuntu_64 --register

VBoxManage storagectl Ubuntu --name "IDE Controller" --add ide
VBoxManage modifyvm "SweetVM" --hda "/whatever/vmname.vbox/SweetVM.vmdk"

When I ran the modifyvm command originally it complained about the IDE controller so I did some googling and found the IDE controller stuff. I had my vmware disk busted into 2 gig files and Virtualbox didn't hate on me. I read some stuff about converting stuff to the virtualbox native format but I figured what the heck and pointed it at the VMWare virtual disk file and it was cool and the gang.

At this point I started it but found I had no network card listed. So let's add a NIC:
VBoxManage modifyvm "SweetVM" --nic1 bridged
VBoxManage modifyvm "SweetVM" --bridgeadapter1 eth0

So I added an interface to the VM that is a bridge adapter as well as pointed the VirtualBox secret sauce to bridge that traffic on eth0 on my host box. There are a lot of other things you can add to your vm via the modifyvm command depending on your hardware needs inside your vm. For a full list check out this link.
Now for the purpose of testing lets fire it up with the remote console enabled so we can make sure everything is working.
VBoxManage modifyvm "SweetVM" --vrdp on --vrdpport 6464 --vrdpauthtype null --vrdpmulticon on

Now kick the tires and light the fires. I brought it up in the foreground so I could watch any errors:
VBoxHeadless --startvm SweetVM

You should see:
Oracle VM VirtualBox Headless Interface 3.2.12
(C) 2008-2010 Oracle Corporation
All rights reserved.

Listening on port 6464.

Now go ahead and connect via a rdp client to your host ip and the port you specified. I specified port 6464. When I connected to the box for some reason eth0 was missing. When I did an lspci it was finding an Intel nic. I found that my NIC was now eth1 so I modified my netowrk config in /etc/network/interfaces and changed everything to eth1. After I did that I did a:
sudo ifconfig eth1 up

Make sure you can ping out and all that jazz now shutdown your VM. Now lets shut off console access to the VM:
VBoxManage modifyvm "SweetVM" --vrdp off

Now we are ready to start this thing and toss it in the background so we don't need to see it any more.
nohup VBoxHeadless -s SweetVM&

That's it.. I am now Virtualbox at the crib.

Friday, March 5, 2010

My Computer Smells Like Feet! Stories From My Early Days

Sitting here at lunch today for some odd reason I kept thinking about some of the more entertaining experiences I have had when it comes to IT. There is one incident that stands out as stranger than fiction. I remember it plain as day and if I had to give it a name I would call it "My computer smells like feet"

So I know what you are thinking this is a complete joke.. So did I when the ticket came in. We had what you could call a handler that all tickets came to and he would then distribute it to the techs. I was probably 19 years old and this was my first real IT job. I had only been on the job for a few months so I was the low man on the totem pole. I got all kinds of crappy tickets for this as sort of a right of passage. Anyways I get back from working on someone else's problem when there is a bunch of people standing around my desk.

At first I was like oh no... what did I do? Did I mess something up and everyone is going to give me the business about it? (A few days before that I messed up an executive assistants machine but that is another story) So I was nervous when the handler came up to me and said we got a good one for you. The ticket says something about her computer smelling like feet. Immediately everyone else was laughing and I knew this was gunna be a shit job.

Reluctantly I took the ticket and headed to the building where the user sat. I get there and the cube has a nice view with a window. Thinking this was a joke I was like "I am here for the smelly computer". That got a few laughs but unfortunately for me she was serious. So the user explains to me that she will be working on her PC and it will start smelling like funky feet. Again I probe her to see if it is a joke but she keeps reassuring me that this is serious.

This was probably 1996 or 1997 and the place I was doing work used AST computers that were in a midtower case that was a pain to take apart. Also keep in mind we had to dress up for work sort of like the geek squad does now. You had to wear slacks and a tie. At this point I am under the desk on my hands and knees smelling the inside of this users PC. Just to make it more humiliating a couple of the guys came over to observe about the time I started smelling it. Everyone was sorta laughing and making fun so I was done. I thought to myself this user is playing a joke and the other techs are in on it and it is just rookie hazing. I tell the user I can't smell anything and I was tired of snorting dust bunnys.

I start to put the computer back together and was pretty pissed. At this point I was quite the spectacle in this part of the office. Just when I was going to say something it happened. All of a sudden I get this huge wiff of someone's stank ass feet. It was some serious foot funk like something you would expect on the set of a tough actin tinactin commercial. It was so funky I had to get out from under the desk. I was able to confirm the users complaint that her computer smelled like feet.

Here was the weird part though. The computer was turned off. After a few moments the smell dissipated so I finished assembling the PC and started looking around under the desk. Was someone using this for shoe storage or something? Then a couple of feet away under the desk was an AC vent. When the air kicked back on the smell came back. Turns out a squirrel had decided to get into the ventilation and die and the smell wasn't someone with a foot odor problem but a squirrel feeding some bacteria.

Although this is a funny story and I caught a lot of shit from my peers I learned a valuable lesson that day. When troubleshooting always keep an open mind no matter how weird the problem might be. A lot of times we approach a situation where we think we already know the answer and are blind to the real source of the problem. I knew that people were pullin a fast one on me until some feet stank odor popped up and slapped me in the face to teach me this lesson. Sometimes the answer is not in front of your face.. its in the AC vent :)

Sunday, February 28, 2010

Vortex IDS - Get Super Snagadocious on Ubuntu

I got bored this weekend and decided that I should play with something new so I chose to mess with Vortex. From the README file:

Vortex is designed to facilitate agile creation of advanced network intrusion detection or network surveillance systems. It performs the hard work of packet capture, filtering, and reassembly then provides that data to external programs for analysis. Stream data is stored in files and stream meta data is output via STDOUT (meta data is encoded in file name). Vortex provides a platform for highly flexible, multithreaded, near real time deep analysis of network payload data.

The first thing I noticed when trying to mess with this stuff was there was hardly any documentation. The README file had enough info though to get started. I decided I wanted to try another method for grabbing streams to analyze the email from the SANS challenge #2. Although Xplico is pretty pimp I don't see it scaling to a large environment. Also I want more flexibility in case I am looking for specific threats etc. So let's get down to business.

Installing Vortex

I decided to use Ubuntu for this test so lets get the source:

$ wget http://downloads.sourceforge.net/project/vortex-ids/vortex/2.8.1/vortex-2.8.1.tgz?use_mirror=cdnetworks-us-1

$ tar zxvf vortex-2.8.1.tgz

$ cd vortex/

$ more README

You should really check out the README file as there is a lot in there of value. So in order to get this mug to compile properly we need to solve some dependancies. Run the following command:

$ sudo apt-get install libnids-dev libnet-dev libpcap-dev

Now let's compile the source:

$ gcc vortex.c -lnids -lpthread -Wall -o vortex

Run it to see if it works:

$ ./vortex -h

Usage: ./vortex [ -lpmheI ] [ -c count ] [ -i device ] [ -r file ] [ -u user ] [ -S bytes ] [ -C bytes ] [-t dir ] [ -s count ] [ -H count ] [ -q limit ] [ -D level ] [-F file | -f filter ] [-M MTU (snaplen)] [-P poll rate] [ -TEK time ] [ -Q size ] [ -R usecs ] [ -Nn prio ] [ -Oo cpu ] [ -L name ]

-h print this help message and exit
-c count set number to connections to follow
-i device listen on device
-r file read capture from pcap file
-l set output to line buffering
-p don't put interface(s) in promiscuous mode
-u user after initialization, setuid to user
-S bytes number of bytes to collect from client to server Default: 104857600 (100MB)
-C bytes number of bytes to collect from server to client Default: 104857600 (100MB)
-t dir directory for storage of stream data (defaut: currend working dir)
-s count Size of connection hash table--Maximum number of streams to follow simultaneously = 3/4 * count. Default: 1048576
This affects memory consumption significantly. If you have problems with TCP_LIMIT, increase this value. (See n_tcp_streams in libNIDS)
-H count size of IP defrag has table. Default: 65536 (See n_hosts in libNIDS)
-m enable libNIDS multiprocess mode DEPRICATED--don't use this (See multiproc in libNIDS)
-q limit set libNIDS packetqueue limit. DEPRICATED--only applies in multiproc mode
-D level set debug level Default: 0
-f filter tcpdump-style capture filter expression (don't forget quotes/shell escapes)
-F file file containing packet filter expression
-M MTU MTU or snaplen--maximum packet size to capture. default: 1560
-w enable libNIDS TCP/IP stack workaround mode (See TCP_workarounds in libNIDS)
-k disable libNIDS TCP/IP checksum processing (See TCP_checksums in libNIDS)
-P rate Only reassemble and collect every poll rate connections. default: 1
-T time Report Performance Statistics every time seconds (approx) default: 0
-E time Report Error counts every time seconds (approx) default: 0
-L name Logging name for syslog. Default: vortex
-Q size Size of output ring queue. Sets limit for number of finished streams waiting to be written. Default: 10000
-R usec Wait period in us (inverse of poll rate) for stream output thread in microseconds. Default: 10000
-n prio Priority (niceness) for capture thread. Can be from -20 to 19 on most systems. Default: -15
-N prio Priority (niceness) for other threads. Can be from -20 to 19 on most systems. Default: 10
-o cpu CPU to bind capture thread to. Default: 1
-O cpu CPU to bind other threads to. Default: 0
-I Lock threads to specific cores. (see o and O above). Default is to not lock so specific cores (Expiramental--still not working properly!).
-e enable extended output (more metadata in file name).
-K TCP Idle connection timeout in seconds Default: -1 (disabled). This timeout ignores empty keepalives.
-v Output empty streams (create files with 0 bytes).

This thing has some crazy options in it. There is also something in there called xpipes. Not going to use this but we should make it work anyway.

$ gcc xpipes.c -lpthread -Wall -o xpipes

Ok now we are ready to analyze some data. Lets grab the pcap from the SANS example:

$ wget http://forensicscontest.com/contest02/evidence02.pcap

Now let's analyze it with vortex:

$ ./vortex -r evidence02.pcap -t /tmp/vortexout
Couldn't set capture thread priority!

We end up with this:

-rw-r--r-- 1 507 2010-02-28 14:20
-rw-r--r-- 1 1495 2010-02-28 14:20
-rw-r--r-- 1 507 2010-02-28 14:20
-rw-r--r-- 1 285778 2010-02-28 14:20

Let's take a looksie at the contents of one of the files:

EHLO annlaptop
Message-ID: <000901ca49ae$89d698c0$9f01a8c0@annlaptop>
From: "Ann Dercover"
Subject: lunch next week
Date: Sat, 10 Oct 2009 07:35:30 -0600
MIME-Version: 1.0
Content-Type: multipart/alternative;
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.2180
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180


So there are all kinds of goodies in there. We see the entire email content in this easily searchable stream.


There are a ton of different things you could do at this point. Let's say you just want to capture all email from an smtp gateway you could run the following:

$ vortex -i eth1 -t /data/vortexstreams -f "tcp port 25" -e -K 300

That would give you an archive of all your emails sent and received from your gateways. (assuming you tapped the interface of said gateways and plugged eth1 into the tap) You want to know all emails sent to sec558@gmail.com? Try this:

$ grep "RCPT TO: sec558@gmail.com" * TO: sec558@gmail.com

note: Blogger doesn't like <> so you need to wrap the email in them to actually make it work.

Some new worm send a document called secretrendezvous.docx and you want to know who got it?

$ grep "filename=\"secretrendezvous.docx\"" * filename="secretrendezvous.docx"

$ grep "RCPT TO"\:1038s64.12.102.142\:587
RCPT TO: mistersecretx@aol.com

Time to rebuild mistersecretx's machine. So again the sky is the limit here on stuff you can do. You can easily write a script that scans the directory that the streams are in or you can squirt it directly into a custom analyzer.

In the README there is an example to analyze on the fly:

vortex -i eth0 -s 1000000 -C 0 -S 1000 -l -t /dev/shm/ -f "tcp port 21" -e -K 300 | while read file
#Loop on each stream file. file contains metadata that looks something like "/dev/shm/tcp-38-1261503711-1261503711-c-174-172.16.17
#extract credentials from stream
CREDS=`cat $file | tr '\r\n' ' ' | sed -r 's/.*USER (\S+) PASS (\S+) .+$/\1 \2/g'`
#CREDS is something like "username password"

RESULT=`echo "$CREDS" | cut -f 1 -d " " --complement | /usr/sbin/cracklib-check`
#RESULTS is either "password: warning message" or "password: OK"

#If password isn't OK, the log then username, warning message, and connection info
if ! echo $RESULT | grep "OK$" > /dev/null
USER=`echo $CREDS | cut -f 1 -d " "`
MESSAGE=`echo $RESULT | awk -F: '{ print $NF }'`
METADATA=`basename $file`

echo "Weak password ($MESSAGE ) for $USER in connection $METADATA" | logger -t "ftp_pass_check" -s

#Purge the file
rm $file

So far I am really impressed by Vortex. This thing looks really cool as to what you can do with it. I really want to play with it some more and will definately do a part 2 to this post where I take this example to the next level. There are a few answers I need to come up with first though. Do I want an entire archive of sent and received internet emails? What are the legal ramifications of said archive? Do I want to only save streams of emails with links embedded in them or only emails with attachments? Do I want to extract the actual attachment and run them through something else? Do I want to index these badboys into a database?

Before you answer these questions consider this scenario. Your organization is targeted for a 0-day attack via a phishing email using a malicious pdf file. Let's say this malware creates a conduit for a badguy to take control of the users PC. Then the badguy deletes the email and the pdf and now has full control of this machine. You could say wait.. I have AV running on my mail servers and on the PC. But when it came in AV never detected it and now since they deleted it its gone and will never be detected again. This is where an external email archive would come in handy. You track down a user who noticed this email seemed weird with an attachment named "Marketing Report.pdf". You could now go to your streams directory and grep the streams for "filename=MarketingReport.pdf" and get a list of all the emails that came in with that attachment and the users whom received it.

Keep an eye out for part 2 :)

Saturday, February 20, 2010

Xplico and SANS Forensic Challenge #2

So after installing Xplico I wanted to give it a shot and do a forensic investigation using it. I chose this puzzle since at the time I was playing with this it had just wrapped up and the answers were available to check against. Now in order to complete this thing I will have to use more tools than Xplico but I will try and use it wherever possible.

So let's log in and create a new case. We shall call it SANS Puzzle 2:

Now click on the case so we can create a new session. We shall call it "Let's Rock This":

OK... Click on the session. You should get a purty screen that looks like this:

Time to upload the pcap file. All the cool kids md5sum the evidence file so we know we are working with the right stuff here. It should be:

cfac149a49175ac8e89d5b5b5d69bad3 evidence02.pcap

Give it a few moments (depending on your puter) and let it do it's thing. You should get a screen that looks like this:

Now let's look at the questions from the contest. Question #1 What is Ann's email?

So normally you could fire up Wireshark on this mug and find the SMTP packets and get the info. Who wants to do that though when you can click email on the left hand side and see all the emails contained in the pcap? :)

We click the first email listed and what can one say? Scandalous! Here is the email:

So we can answer question #1. Ann's email is Ann Dercover sneakyg33k@aol.com

Now we need her password which is question #2. This is a pretty nice feature with Xplico. Hover over the info.xml and save the pcap.

Open the pcap in Wireshark and cruise down to packet #14 to grab the Base64 encoded password.

Ann really should think about using an email provider that doesn't pass credentials in the clear. Cruise over to here and paste the password in there (NTU4cjAwbHo= is what you paste but you already knew that) and you hit decode and PLOW!!!! Ann's password is 558r00lz

Question #3: What is Ann's "Lover's" email? mistersecretx@aol.com is in the email we looked at above.

Question #4: What items is mistersecretx supposed to bring with him? Again look in the email above and we can see that he is supposed to bring his bootleg passport and a speedo. (Technically the answer is a fake passport and a bathing suit)

Question #5: What is the name of the attachment? Easy one again with Xplico. Look at the email and you see it right there. secretrendezvous.docx

Question #6: What is the MD5 sum of the attachment? On this one you have to do a little massaging for ease of use later. When I save the attachment in the email it saves it as "3". I renamed it so that the name was right then pumped it through md5sum:

9e423e11db88f01bbff81172839e1923 secretrendezvous.docx

Question #7: In what CITY and COUNTRY is their rendez-vous point? Since we renamed it to the correct extension openoffice opened it right up and here is what it looks like:

VIVA LA MEXICO!!!!! The answer is: Playa del Carmen, Mexico

Question #8: What is the MD5 sum of the embedded picture? Xplico doesn't really help here except for getting us the attachment to work with. So in a terminal run the following commands:

$unzip secretrendezvous.docx
$cd word
$cd media
$md5sum image1.png

aadeace50997b1ba24b09ac2ef1940b7 image1.png

That's it as far as the contest is concerned. This thing has a ton of capabilities and I plan on exploring them further by doing some of the other challenges so check back. When I originally tried this challenge I used tcpdump and wireshark to get a lot of these answers but it was very time consuming. When time is of the essence it is nice to have tools that make things easy. Looking at the forums I see that this is actively being improved and I am very impressed overall by the tool.

There are some gaps with this tool though. I recently did the Honeynet Challenge #1 and Xplico doesn't help at all really. You can see some of the FTP information but other than that there really isn't anything there. It does look promising for the Honeynet Challenge #2 though :)

Wednesday, February 17, 2010

Installing Xplico

I have been scouring the internet for some useful forensic tools when I cam across a tool call Xplico. I wanted to do some of the forensic challenges offered by the folks over at SANS and it looks like to win anything you have to write tools. Realistically in a high paced environment the luxury of having time to write your own tools sometimes doesn't exist.

Xplico has a couple of install options. First they have a virtualbox VM that you can download. If you are running Ubuntu 9.10 they have a nice .deb package for you. And for the brave ones they have the source that you can build yourself. I am running Ubuntu 9.10 64bit style so I need to do a little extra to get the prepackage to work.

So let's get our download on:

$ wget http://downloads.sourceforge.net/project/xplico/Xplico%20versions/version%200.5.4/xplico_0.5.4_i386.deb?use_mirror=softlayer

When I tried to force install it on a 64bit OS I got all this stuff:

xplico depends on libice6 (>= 1:1.0.0); however:
Package libice6 is not installed.
xplico depends on libmysqlclient16 (>= 5.1.21-1); however:
Package libmysqlclient16 is not installed.
xplico depends on libsm6; however:
Package libsm6 is not installed.
xplico depends on libsqlite0 (>= 2.8.17); however:
Package libsqlite0 is not installed.
xplico depends on libxt6; however:
Package libxt6 is not installed.
xplico depends on tshark; however:
Package tshark is not installed.
xplico depends on python-all; however:
Package python-all is not installed.
xplico depends on apache2.2-common; however:
Package apache2.2-common is not installed.
xplico depends on php5-common; however:
Package php5-common is not installed.
xplico depends on libapache2-mod-php5; however:
Package libapache2-mod-php5 is not installed.
xplico depends on php5-sqlite; however:
Package php5-sqlite is not installed.
xplico depends on php5-cli; however:
Package php5-cli is not installed.

Ubuntu is supossed to be cool enough to go download all this crap but somehow it was stuck. So I did the following:
$ sudo apt-get install libice6 tshark libsm6 build-essential
After that I was able to give it a:
$ sudo apt-get -f install
This basically fixed all my jacked up packages and looked like everything installed properly.

So at this point lets see if it worked. Give your terminal one of these
$ xplico -h

It should print out the help. So now lets fire up our interwebs and see what this looks like. Connect to the host you installed it on on port 9876.

So that is how I got mine working. Coming up next is using this tool :)

Thursday, January 21, 2010

OMG It's Operation Aurora!!!

So lately the interwebs are abuzz with this whole China vs Google Operation Aurora stuff and it got me thinking of a couple of things. One of my flaws when it comes to InfoSec is that I always try and think how can I prevent something like this happening? So the more I think about APT and how to defend against it the more depressed I get.

Realistically most organizations can't kick Microsoft to the curb and use a real OS like Linux. :) They have years and years of entrenched VB and access applications that run the show. No matter how much you communicate and try and train users about phishing they will continue to click on it. Basically the deck is stacked against security professionals. Windows + Users = FAIL

No one said security is easy and I am not looking for someone to jump in with a magic wand and fix all the problems. (even though that would be nice) I am simply saying that if companies want to get serious about keeping their IP and for some companies our national security in tact its time to make serious changes.

I know I have changed my outlook on securing an organization due to current threats. No matter how well you harden your systems and patch your vulnerabilities it really amounts to nothing if you don't get medieval on your clients. The "baddies" are going after your clients. Your weakest link is your clients. The thing that should be keeping you up at night is your clients.

So what is the answer? Concentrate on detection and containment. It's no longer if you will get pwned it is when. You need to know when they are stealing your stuff and how much they got.

Tuesday, January 19, 2010


This is my first post.. Here is to more interesting posts in the future!