Monday, December 17, 2012

Starting An Information Security Program

Many people have to face the daunting task of starting a security program in their company and in many cases they are not security professionals but rather the guys  doing operations because the small and medium businesses cannot afford a dedicated position. Also, it is probably the case that many of them are sent to a boot camp or training course and then they have to figure out the next steps.

I have to say that I am not a security professional neither, but I have been doing operations for many years and I would like to share my experiences.  Of course you do not have to trust me because I do not have a paper hanging on the wall saying that I am CISSP or something similar. On the other hand, working in the trenches teaches you more than any book or certification.

 The first thing you have to take into account is that buying expensive devices will not make you more secure. They are helpful to mitigate common non-targeted attacks and they will eliminate the standard noise coming from Internet, but a clever attacker will always find ways around. It you want to make an impact, you have to start from the ground up and build a solid foundation. These are my recommendations:

  • Create policies, procedures and standards. If  all the systems are built in the same way, all the changes are documented and everybody in operations work under the same rules, it will be more difficult to leave  systems running in the network with a configuration that may not be appropriate or with security holes . On the long run, probably they will be an easy target.
  • Use a tool to verify that all the systems comply with the policies. For example,  all the systems are at the same patch level, they have the same exact configuration, etc,...
  • Log all the configuration changes. Logging all the changes will allow you to be notified when an unexpected change in the configuration takes place and by whom. On top, it can be automatically reverted.
  • Catalogue your data according its value/confidentiality. Never store in the same place confidential information and the publicly accessible one. This applies for storage systems and also for servers. i.e. The same servers are hosting the public website, the Intranet and a document management system. Also applies to a storage system that keeps general purpose documents and also confidential information from management, accounting, etc... 
  • Access controls. Once the data is catalogued and segregated, apply all the needed access controls and network segmentation. Then, keep a log of all the access attempts (successful and failed),  network activity (.i.e. firewall logs and Netflow data from your routers/switches, full packet captures),  application proxies, etc..
  • Get to know what is normal in your environment. If you know what is normal in your organization,  everything else will be suspicious and you will have all the information to investigate and trace the source.

As you may notice,  applying all these ideas to an organization that has been running for many years without proper security policies is a long journey, but it will be worth the effort.

Friday, November 23, 2012

Enabling UTF-8 in FreeBSD

Note: this post is just a note to myself so I do not forget how to enable UTF-8 in FreeBSD.

I have not realized until today that FreeBSD does not enable UTF-8 by default, even though it is capable.   I  joined some IRC channels with German speakers some time ago and I always knew there was something wrong in the encoding but I was too lazy to fix it.

Today I had a few minutes to play with the settings in my Screen/Irssi configuration and I could not manage to make it work It is really frustrating... In fact, it was only working when UTF-8 was disabled in Screen, that is a good symptom.

After wondering for a while, a thought it could be the case that the problem comes from the locales, because (in theory) UTF-8 should be enabled everywhere. No!! It was not enabled in my FreeBSD box at home. Searching in Google I found this post explaining the steps.

The charset has to be enabled during the login process. This means we have to change /etc/login.conf or ~/.login_conf . Since I am only interested in having UTF support in the account running Irssi, I made the change there instead of making a global change.

The config is so easy. Edit  .login_conf in your Home directory.


Once this is done, it will only apply to new login sessions, so you have to exit Irssi and also close the Screen session.

Friday, October 12, 2012

Analyzing the Phalanx 2 Rootkit with Volatility

Andrew Case has written a great blog post in which he analyzes  the Linux rootkit Phalanx2 with Volatility.

He explains all the common steps:
  • Dump the memory with lime .
  • List hidden processes.
  • File descriptors opened by the hidden process ( open sockets!).
  • Network connections.
  • Hooked system callas
  • Recover

At the end of the post, Andre Case makes a complete forensic analysis of the kernel modules and the binaries used to inject the rootkit

Monday, October 8, 2012

Attacking XMPP connections

XMPP is a well known protocol used for real-time chat. There are many companies using it, but you already know Facebook and Google because they are the bigger ones.

This protocol has been there for many years, but it seems that just lately (or I was not aware of) some people has started coding tools to perform MITM attacks.

Few weeks ago I came across this nice tool called XMPPloit that is specially written for Google Talk, even though it is not platform specific.  Its main purpose is to proxy the connection between the user and the legitimate server (once we have already performed the MITM e.g. DNS poisoning)  and force the use of a non-encrypted channel with also the option to force plain-text authentication.

Below you can find a nice demonstration in Youtube:

Last week I also came across a similar tool called xmppmitm (via /r/netsec ), but there is not a lot of information and I have not tested it yet.

Monday, August 20, 2012

Recovering Memory Filesystems With Volatility

Via ,  I have learned about a Volatility plugin that can list and extract tmpfs  filesystems from Linux memory images.

As the author of the blog post comments, many distributions are using this filesystem for different tasks (e.g. /tmp /dev/shm) with the benefit of not having to delete files  and also the added performance compared with traditional filesystems that are stored in hard drives.

Of course, this has some implications when performing a forensic analysis, because all the information contained will be lost if the investigator only make a disk image.  On top of this, /tmp and /dev/shm are one of the few directories that are world writable and, therefore, preferred by the attackers to store information.

Friday, June 29, 2012

Simple IRC/Twitter Irssi Bot

Yesterday I had some free time and I wanted to play a bit with the Perl API for Irssi, that is a console IRC client if not *THE* client.

I don't like like Perl, but it is always nice to learn something new because I always use Python for scripting, that is my preferred language :)

The Perl script is a very basic bot that allows the members of a channel to send twits and also to fetch the timeline.  For being on the safe side, sending twits is only allowed for a list of nicks within an specific channel in order to avoid people sending shit or flooding the public chat.

If you want to play a bit,  the API is focused on signals, that are sent when an event happens (e.g. you receive a message). The events can be hooked, stopped, forwarded, etc.. so it is pretty straightforward. But you have to find the right event and it may be not that easy.

For interacting with Twitter, it uses the module Net::Twitter:Lite and oAuth authentication.

Monday, June 11, 2012

When The Comparison Functions Do Not Compare: The MySQL Fail

I have learned via Mikko Hypponen's twitter account about an epic bug in MySQL. 

This bug  demonstrates why I hate C so much. It is not enough that you have to be really careful when writing code that you also have to be aware of tricks and optimizations applied by the compilers. I cannot point to details but it is not the first time that something like this happens.

In a nutshell (read this post from Rapid7) the memcmp() function does not behave as expected when gcc uses the SSE optimization. As you may have noticed, I am not an expert in C so do not expect a long and detailed explanation.

The documentation states:

Compares the first num bytes of the block of memory pointed by ptr1 to the first num bytes pointed by ptr2, returning zero if they all match or a value different from zero representing which is greater if they do not.

There is no warning whatsoever, but it seems that it is not always the case. As H.D. Moore explains  in his post:

On some platforms and with certain optimizations enabled, this routine can return values outside of this range, eventually causing the code that compares a hashed password to sometimes return true even when the wrong password is specified. Since the authentication protocol generates a different hash each time this comparison is done, there is a 1 in 256 chance that ANY password would be accepted for authentication.

The exploit? Pretty simple: we try to authenticate in  a for loop until the bug appears.

$ for i in `seq 1 1000`; do mysql -u root --password=bad -h 2>/dev/null; done

Update: this post from Hispasec (Spanish) has more information regarding the vulnerability.

It seems the vulnerability would be a truncation from int to char, put together with a compiler optimization.

The function that compares the hash given with the stored in the database can be found in the file  'sql/password.c':
check_scramble(const char *scramble_arg, const char *message, const uint8 *hash_stage2)

return memcmp(hash_stage2, hash_stage2_reassured, SHA1_HASH_SIZE);


As we can see, memcmp() will return an  int, that will be truncated to a char. Therefore, the hash is valid if check_scramble returns the char 0 (non-zero is a wrong hash). As a side note: the type my_bool is defined as char in 'include/my_global.h'.

The patch changes this behaviour to ensure the result. Now, the function check_scramble returns:

return test(memcmp(hash_stage2, hash_stage2_reassured, SHA1_HASH_SIZE));

being test() a macro defined in include/my_global.h

#define test(a) ((a) ? 1 : 0)

This time, we have a boolean operation: return 1 if the value of 'a' is different to 0 and return 0 in other case.

Cracking Hashes Online

Perhaps this tool is well known within the community, but I was not aware of it until now. findmyhash is a Python script that searches for hashes in different online services, supporting the most common ones: MD5, SHA1, LM, NTLM, etc. (check the website for more details) .

Looking at the source code, it uses many  websites I was not aware of.

It is kinda neat that you may not need to use brute force to crack a hash, but the downside ,of course, is that you are sending your hashes to services you may not trust 100%.  Would it be legit to use it during a pentest since we are sending data from our costumers to third party services?

Monday, March 19, 2012

Want security? Go back to the basics

Today Mubix published this presentation in Google Docs. I post it again in my blog as it is a shared resource and the contents are really interesting.

In my opinion, the CCDC is really similar to what you can find in a real world company, when there is no clear security posture and they are getting badly owned (sadly it is more common than what we may imagine). Therefore, it is a great scenario to implement the very basic security techniques to prevent your boat from sinking :)

I would resume the slides with a few simple ideas:
  • Keep it organized. The chaos is your worst enemy.
  • Define specific rolls  for each area. Nobody is perfect and knowledgeable in all the fields: physical security, networks, Windows, Unix, etc...
  • Risk prioritization. Close the blatant holes and change the default passwords. As he says, nobody is going to drop an 0day. They will first try the common exploits against your servers.
  • Keep sense of the logs and the running processes!

Monday, March 5, 2012

Auditing logs: tracing e-mail transactions III

Previous posts:

Auditing logs: tracing e-mail transactions I
Auditing logs: tracing e-mail transactions II

In the two previous posts I described how I made sense of my e-mail logs, by parsing and storing them in a search engine called elasticsearch. This transformation helped me to quickly trace problems in a distributed environment with many servers. Of course, the alternative was  to manually grep the logfiles, which is a waste of time. In this closing post, I would like to present the last tweak I have made in the script and the conclusions after some months of tests.

The previous version was reading all the logs from a file (ie, the syslog file) to store the transactions in the elasticsearch index.  It is handy if we have to start indexing from scratch, but it is quite inefficient if we need to keep the index up-to-date because, when executing the script with Cron, we keep reading/storing the same transactions again and again.

At the same time I was writing this script, I started using graylog2 to keep a short term cache of my logs in order to speed up searches and also to use their nifty GUI as well:)  As a side effect, they were using elasticsearch, which helped me to continue using my scripts, which is nice.  As you may already notice, this offers new possibilities, like querying the graylog2 index with the REST API, that permits to search by keywords and time frames.

As a result, I updated the script to query the graylog2 index and extract all the postfix/sendmail logs given a time frame (60 minutes by default). This change helps a lot to reduce the load when it is executed by Cron and, at the same time, it permits to call it in smaller periods of time (ie, every 15 minutes).

By using available tools, we can greatly improve our capacity to resolve incidents, while, at the same time, doing it on a budget. The alternative would be spending hours grepping log files and trying to understand what happened, which is not an easy task.

This post is just using e-mail logs as an example, but this can also be applied to other raw data.  One possible option would be parsing the logs of the firewalls, netflow, IDS, HIDS,etc. running in our infrastructure and storing them in a structured way, so we can do searches to investigate an ongoing incident. Buy doing so, it would help us to correlate the alerts and to find until which extent and attacker has penetrated in our network.

This concept is similar to the way Sguil handles incidents happening in our network.  I like the idea of being able to jump between IDS alerts, netflow and Pcap files so easily, because it gives lots of visibility to the analyst and helps to make an informed decision.

Before I finish, I would like to talk a bit about the drawbacks.  Parsing the data and doing some automation is really useful and it greatly helps to solve incidents faster (it is really difficult to find somebody that disagrees), but the picture you have of what is happening is as good as your regular expressions. Yes, the more complex the data is, the more pain you will suffer and you may miss some parts of the picture.  You have to be aware and you may find yourself checking the logs from time to time :)

Tuesday, January 17, 2012

Setting up Snorby in Security-Onion

This video has been created by IronGeek and it is originally available on his website.

I wanted to post it on the blog because setting up a Network Security Monitoring system with Security-Onion is rather easy compared with the days I had to deploy Sguil by hand. Doug Burks is doing a great job here:)

I have to say that I prefer using Sguil in a NSM system as Richard Bejtlich has explained many times, but Snorby may be useful for many people.

Here you can find Richard Bejtlich talking about NSM and Sguil.

Wednesday, January 4, 2012

Auditing logs: tracing e-mail transactions II

In the previous post Auditing logs: tracing e-mail transactions I talked about a script I wrote to make sense of the e-mail logs when doing troubleshooting or just investigating an incident. The downside, of course, is that we have to process all the logs every time the script is ran and this can be time consuming.

To solve this problem I decided to index the 'reduced' data with elasticsearch, that is a search engine based on lucene. As a result, I can make quick searches upon millions of unique e-mail transactions and get a result in a matter of seconds.

Structuring the e-mail transactions
First of all, I have to clarify that each e-mail transaction is a unique operation in a single e-mail server. Therefore, sending an e-mail may involve several e-mail transactions because it is traversing several e-mail servers throughout our infrastructure. At the end, each of these unique transactions are represented by the following data structure:

  • from: A string representing the sender's e-mail address.
  • to: Array of recipients. Each element in the array is a dictionary with the following fields:
  1. recipent: one of the recipient's e-mail address
  2. relay: the next hop in the delivery chain. It can be a server or local delivery.
  3. status: A string that informs about the result of the operation.
 Since a particular e-mail can be stuck in the queue and re-delivered later, we may see the same recipient appearing several times, but with different status messages: ie. deferred and successfully sent.
  • msgid: unique message identifier that represents a particular sent e-mail. Not to confuse with an e-mail transaction that is unique within an e-mail server.
  • client: The previous hop in the e-mail chain. 
  • date: String representing the time in which the transaction took place.
  • host: the server that is executing the e-mail transaction.

Data retention policy
I have configured the index to automatically drop entries older than 90 days. In my opinion, we do not need to keep this data for a longer period of time, because the troubleshooting tends to involve transactions that took place in the previous days. Of course, In case it is needed, we still keep the raw logs for a longer period and they can be manually parsed or, simply, re-indexed.

Reducing and indexing the raw logs
The script parses the raw logs and indexes them by using the REST API.

There is a very dirty hack here, since many syslog servers do not send the year in the log messages. The script assumes we are indexing logs from the current year and makes a dirty trick on January 1st. In case you want to index a log file from the previous year you can use the -y flag to force this case. Yes, it is as dirty as I said :)   Modern syslog servers like rsyslog or syslog-ng can be configured to output the year, but it may not be the case with older installations.

The script connects to the elasticsearch server and sends the queries.

A query example would be the following one:

./ -f -d '2011-12-15 00:00:00' -D '2011-12-17 00:00:00' -h smtp -r 10
transaction: pBFERI8l019936
client: bob.desktop []
date:   2011-12-15 15:27:19
to:     recipent='' , relay='', status='Sent (Ok: queued as 5298B228B8B)'
host:   smtp

transaction: pBFDdOFC012829
client: bob.desktop []
date:   2011-12-15 14:39:25
to:     recipent='' , relay='', status='Sent (Ok: queued as 16F49228B8B)'
host:   smtp
The above query searches all the e-mail transactions that match:
  • The sender is
  • Between 2011-12-15 00:00:00 and 2011-12-17 00:00:00
  • The transactions occur in the server called smtp
  • Only output the 10 most recent transactions.

All the scripts can be found here.