Wednesday, October 23, 2013

A Ghost In Your Network

It is common that, in a penetration testing,  the red team may also be in charge of assessing the physical security in a company. In such circumstances, many  pentesters try to drop a device on the costumer's network to gain full access.

The effectiveness may be quite high, but it depends on how skilled the defender is. For example, a diligent sysadmin may scan the network every other day, spotting the device soon or later.  What can we do to change this?  How can we hide from the defenders?

It turns out we can use an old trick available in almost all PC devices, that is barely used nowadays and it has been there for ages. All PCs have a real time clock that can be programmed to automatically wake-up/boot our computer at a given time in the future.  What if we could drop a device that could sleep for days, waiting for the right moment to wake-up and drop a remote shell? The sysadmins can scan and scan and scan... They will not find it because it only runs for a couple of hours at night!

Imagine the following scenarios:
  • A rogue Access Point appearing every night at 10pm from nowhere, that gives direct access to the internal network for 30 min (unless you prevent it from going back to sleep).
  • A remote shell to your Meterpreter instance every night to run some scheduled tasks, before disappearing again from the network.
  • A rogue device that wakes up to exfiltrate data for a short period of time and then it goes back to sleep for days.

How can we configure our system to behave this way? It turns out its is fairly easy, because we only need to change a  file in our Linux installation. Every time our PC goes to sleep or it shuts down, the system writes the current time into /dev/rtc0 to be sure we are not waking up by mistake. This is a pretty standard behaviour in all Linux distributions. In my case, the tests were done in Ubuntu.

On /etc/init/hwclock save.conf , you can find the script that resets the hardware clock. You can see in bold the lines I added to keep the time I had set previously.

# hwclock-save - save system clock to hardware clock
# This task saves the time from the system clock back to the hardware
# clock on shutdown.

description     "save system clock to hardware clock"

start on runlevel [06]


    . /etc/default/rcS
    [ "$UTC" = "yes" ] && tz="--utc" || tz="--localtime"
    [ "$BADYEAR" = "yes" ] && badyear="--badyear"
    ACPITIME=`cat /sys/class/rtc/rtc0/wakealarm`
    exec hwclock --rtc=/dev/rtc0 --systohc $tz --noadjfile $badyear
    echo "$ACPITIME" > /sys/class/rtc/rtc0/wakealarm
end script

How can I hook a script to be executed when my device wakes up? To do so, you have to  drop an executable script on /etc/pm/sleep.d/. This script will be executed every time the device wakes up or sleeps and it receives a single parameter to differentiate this circumstance.


case $1 in
                #this code will be executed before we go sleep.
                #this code will be executed when we wake-up.
                /usr/local/bin/ &
return 0

At some point you have to write a UTC Unix Timestamp  into /sys/class/rtc/rtc0/wakealarm. In the example above we assume it is done by the script /usr/local/bin/

At the end, when the task is finished, we have to call the pm-suspend command, that will start the suspend to RAM process,. This process will execute again the script above but with suspend as the single parameter.

Friday, October 4, 2013

Efficiently Managing Your Linux Systems With Spacewalk And Puppet

Some months ago, I had been given the task to upgrade our home-grown linux management platform. This platform was written around Kickstart and worked pretty well for many years but it fell short and now we need something most robust and scalable.

This post describes the tools we have decided to use and why. As always, there isn't a single tool/platform that offers all we needed and we had to combine several of them to achieve the results. The text has been structured based in the needs we had.

Installation of new systems
Ideally it should be based in Kickstart to not rewrite again all the code done in the last years. This shouldn't be a problem because almost all the tools in the market are based in PXE and Kickstart. To do this, we use the Cobbler tools that come installed with Spacewalk.

All the inventory, installation, etc.. is done with  Cobbler because it is easy to script and more powerful compared with a simple web GUI.

Patch management and auditing
Spacewalk is probably the 'de facto' tool if you are running RedHat based Linux systems. This tool permits to create groups that is very handy if you want to carry a single action in many systems at one. An example would be, applying a patch in all the integration systems for testing before doing the same in the production systems.

Alternatively, you can use open-scap to verify that all your systems comply with your regulations: mainly patch level and configuration. See also the OpenSCAP page for RHEL6.

Configuration management
We decided to use Puppet instead of Spacewalk. This is the case because its configuration management tool was too simple and inefficient for our needs. In a nutshell,  you have to push all the configuration files with a web GUI and then add/edit the file permissions (too much manual work), it didn't have template engine, inheritance and other advanced features that are indispensable if you are managing a big data centre.

On the other hand, Puppet has a robust template language and inheritance, with many documentation/examples on the web. Many sysadmins are scared when they hear about this tool, because it is known to be complex and difficult to use, but it is quite the opposite unless you are trying to build something extremely big like a SaaS system.

In our setup, we moved all the complexity out off Kickstart, leaving a 50 lines kickstart file, and we decided to rewrite it with Puppet even though it required some extra time to have the new platform ready. In a few weeks of  work, we ported all the code and now we can ensure that all servers in a group are configured  the same way and no unexpected changes are made, changing/updating the configuration of all servers is a matter of a few key strokes and it is really easy to deploy/extend new systems.

Another good point is that Puppet is configured with text files and, therefore, it is really easy to use combined with any SCM like Git or Subversion to track back all the changes. In case something goes wrong, it is possible to revert the changes to the previous working version in a matter of seconds and without a big hassle.

Monitoring and alerting
For monitoring and alerting we decided to keep using our old Nagios installation, because it has proven to be reliable. Spacewalk can do some alerting and monitoring but we didn't trust it for such a critical task.