Cheap VPS & Xen Server

Residential Proxy Network - Hourly & Monthly Packages

Host Based Intrusion Detection – Samhain


This article describes in some detail how to install Samhain, the host based intrusion detection system. For further information regarding Samhain, please see

I am not going to ramble on about what host based intrusion detection is or why to use it, as there are plenty of articles already covering those subjects. This article is just to show you how to get Samhain up and running in a client / server configuration with a couple bells and whistles thrown in for fun.

I highly recommend you read the entire guide before you start, it will most certainly help.

There is a lot of swapping between client and server as I try my best to confuse you, so stay sharp!


You will need all the required build tools installed as we are going to compile Samhain. Here is a quick refresher:

Red Hat

yum groupinstall “Development Tools”


apt-get install build-essential

NOTE: Please keep in mind that development tools on production servers is perhaps not the best of ideas. These packages may further assist the wannebe hacker, fill up precious megabyte or eat your cat. It is recommended to build the required packages on your build server, test them, create rpm / deb package and then deploy said packages on your production environment.

Here is a short check list to follow:

  1. You will need MySQL and Apache running on your server. This guide will assume a vanilla MySQL and Apache configuration. I leave it up to the reader to figure out how to install and configure these services on your favourite distribution. (Hint : and
  2. You will need the MySQL development package (generaly mysql-devel) installed for the server side of things.
  3. MySQL must have a root password set. If the MySQL root password is not set, go and do that first. While your at MySQL, you may want to look at this : /usr/bin/mysql_secure_installation
  4. The server and client(s) host name must be fully qualified.
  5. The server and client(s) /etc/host file must be correct (really correct, not Red Hat default correct), and DNS must be working for both forward and reverse lookups.
  6. Port 50888 TCP should be open, or whatever port you set when building.
  7. ImageMagick is required on the client.


Download And Install

The above page has a full description of where to download the latest version of Samhain, and how to verify the integrity of the package. It is critical that the integrity of the package is checked. If you do not have a good foundation to build on, your house will surely crumble 🙂


Server Setup

Yule is the server side component of Samhain.

After you have extracted and checked the package, make sure you are the root user, in the top level directory of the unpacked source files.

We start by creating a user for the service, and generating a gpg key as that user:

adduser yule
su – yule
gpg –gen-key

You will be asked the following questions:

gpg (GnuPG) 1.4.5; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.
gpg: directory `/home/mytest/.gnupg’ created
gpg: new configuration file `/home/yule/.gnupg/gpg.conf’ created
gpg: WARNING: options in `/home/yule/.gnupg/gpg.conf’ are not yet active during this run
gpg: keyring `/home/yule/.gnupg/secring.gpg’ created
gpg: keyring `/home/yule/.gnupg/pubring.gpg’ created
Please select what kind of key you want:
(1) DSA and Elgamal (default)
(2) DSA (sign only)
(5) RSA (sign only)
Your selection? <– The default is fine, just press ENTER
DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096 <– 4096 For the paranoid
Requested keysize is 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0) 2y <– Some may feel 2 years is to long, it’s up to you …
Key expires at Sat 15 Dec 2012 22:24:38 GMT
Is this correct? (y/N) y <– If you are happy and you know it clap your hands
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
“Heinrich Heine (Der Dichter)<>”

Real name: yules <– Whatever name you want to use
Email address: <– Some e-mail address
Comment: 20 questions is a fun game
You selected this USER-ID:
“yules (20 questions) <>”

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O <– If you are happy, OK it
You need a Passphrase to protect your secret key.

Enter passphrase: This is a long passphrase ! <– Enter a strong passphrase
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

Not enough random bytes available. Please do some other work to give
the OS a chance to collect more entropy! (Need 284 more bytes)

Fed up waiting for this ? Click here :

gpg: /home/yule/.gnupg/trustdb.gpg: trustdb created
gpg: key B7043C9A marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2012-12-15
pub 1024D/B7043C9A 2010-12-16 [expires: 2012-12-15]
Key fingerprint = 421E CFE8 533E 017F 95C8 170A DB54 28E7 B704 3C9A
uid yules (20 questions) <>
sub 4096g/EB230E29 2010-12-16 [expires: 2012-12-15]

Quit this shell, so that we are back to the root user.


So now we have a gpg key, lets get on with building the packages.

The default gpg binary does not support the TIGER192 checksum. As such, we first build a vanilla Samhain binary so that we can get that capability from the Samhain binary.


Right, now we build the real thing …

./configure –with-gpg=/usr/bin/gpg –enable-network=server –with-database=mysql –enable-xml-log –with-port=50888 –enable-identity=yule
make install

At this point, the following should come up:

You need to sign the configuration file now
/usr/bin/gpg -a –clearsign yulerc
using –homedir /home/yule/.gnupg
gpg: WARNING: unsafe ownership on homedir `/home/yule/.gnupg’
You need a passphrase to unlock the secret key for
user: “yules (20 questions) <>”
1024-bit DSA key, ID BAFB6B91, created 2010-12-21
Enter passphrase: This is a long passphrase ! <– This is the passphrase we set earlier.

Side note: I am unsure why gpg is complaining about the ownership, as the permissions is just fine.

Now install the initialization script, set up MySQL user / permission and fix some file permissions.

make install-boot
mysql -p < sql_init/samhain.mysql.init
echo “grant select, insert on samhain.log to samhain@localhost IDENTIFIED BY ‘samhain’;” | mysql -p <– This will ask for your root MySQL password.
echo “FLUSH PRIVILEGES;” | mysql -p <– This will ask for your root MySQL password.
chown yule:yule /var/log/yule
chown yule:yule /etc/yulerc
chown yule:yule /var/lib/yule

Set yule to start at boot.

Red Hat

chkconfig –add yule
chkconfig yule on


update-rc.d yule defaults

Start yule with:

/etc/init.d/yule start

Yule may complain with something like :

<log sev=”WARN” tstamp=”2010-12-21T11:46:42+0000″ msg=”Invalid line 102 in configuration file: incorrect format, unrecognized option, or missing section header” />
<log sev=”WARN” tstamp=”2010-12-21T11:46:42+0000″ msg=”Invalid line 106 in configuration file: incorrect format, unrecognized option, or missing section header” />

However, the service should start fine. These two warnings are due to the [Database] header being commented out. Either uncomment it, or comment said two lines out. They are true by default.

For a list of configuration options with full explanations, see


Apache Configuration

Add the following in:

Red Hat




<Directory "/var/log/yule/">
   Options ExecCGI
   AllowOverride None
   Order allow,deny
   Allow from all
Alias /yule.html "/var/log/yule/yule.html"

Then reload Apache with:

Red Hat

service httpd restart


/etc/init.d/apache2 restart

Now visit http://yourserver/yule.hml

Client Setup

Log in on the server you wish to install the Samhain client on and make sure you are the root user. Also make sure you have all the essential build packages installed, refer to the overview for installation of these essential build packages.

First, we need a gpg key for root.

See previous example for detailed steps.

gpg –gen-key

Now we need to pull out the fingerprint for this key, so that we can use it when building the Samhain binary.

MY_FP=`gpg –fingerprint root | grep fingerpr | sed ‘s/ //g’ | awk ‘BEGIN { FS = “=” } ; {print $2}’`

As before, we need TIGER192 checksum capability first.


Now, since we are having a bit of fun, we are going to change the name of the binary and process. Classical security by obscurity. I’m picking the name john, a general purpose password cracker. Pick a name that will not stand out in a process listing and shout out “THIS IS A HIDS PROCESS !!!!11”. Then again, know how much/little a name change actually hides what this binary does before you rely on it to hide you’re HIDS from a l33t haxor.

We further specify that the configuration and data files should be pulled from the server. If you want to take this one step further, look into the following compile options : –enable-khide,–enable-suidcheck and –with-kcheck=/path/to/

Make sure to change IP_OF_YOUR_SERVER to the actual IP address of your Yule server.

./configure –with-gpg=/usr/bin/gpg –enable-network=client –with-config-file=REQ_FROM_SERVER –with-data-file=REQ_FROM_SERVER/var/lib/john/john \
–enable-stealth=129 –enable-install-name=john –enable-srp –with-fp=$MY_FP –with-port=50888 \
–with-logserver=IP_OF_YOUR_SERVER –with-sender=john

Make the required directories, copy the binary over (with the correct name) and put the initialization script in place.

mkdir /var/lib/john/
cp init/samhain.startLinux /etc/init.d/john
chmod 744 /etc/init.d/john
cp samhain /usr/local/sbin/john
cp samhain_setpwd /usr/local/sbin/john_setpwd
cp samhain_stealth /usr/local/sbin/john_stealth
cd /usr/local/sbin

Set the password and overwrite the binary.

/usr/local/sbin/john_setpwd john jingle 161718abcd212324
mv john.jingle john

“jingle” Does not matter, it’s just the append and the number is what you want in 16 bit 0-9, A-F (A.K.A HEX). You can use yule -G on the server to generate a random number for you.

The output should look something like:

INFO old password found
INFO replaced: f7c312aaaa12c3f7 by: 161718abcd212324
INFO finished

Change the description in the initialization script.

sed -i ‘s/File Integrity Checking/Password Cracking/’ /etc/init.d/john

Make sure the daemon starts at boot.

Red Hat

chkconfig –add john
chkconfig john on


update-rc.d john defaults


A Little Work On The Server

The HEX key we just embedded in the client binary, we need it now to tell the server about that client.

/usr/local/sbin/yule -P 161718abcd212324 | sed ‘s/HOSTNAME/CliENT_HOSTNAME_HERE/’ >> /etc/yulerc <– Make sure to put the client host name (FQDN) in.

Edit /etc/yulerc and move the key above the GPG signature.

For example, the last couple of lines of /etc/yulerc mihgt look like this:

# Client=HOSTNAME@00000000@C39F0EEFBC64E4A8BBF72349637CC07577F714B420B62882
# Client=HOSTNAME@8F81BA58956F8F42@8932D08C49CA76BD843C51EDD1D6640510FA032A7A2403E572BBDA2E5C6B753991CF7E091141D20A2499C5CD3E14C1639D17482E14E1548E5246ACF4E7193D524CDDAC9C9D6A9A36C596B4ECC68BEB0C5BB7082224946FC98E3ADE214EA1343E2DA8DF4229D4D8572AD8679228928A787B6E5390D3A713102FFCC9D0B2188C92
Version: GnuPG v1.4.5 (GNU/Linux)

We need to change that to look like this:

# Client=HOSTNAME@00000000@C39F0EEFBC64E4A8BBF72349637CC07577F714B420B62882
# Client=HOSTNAME@8F81BA58956F8F42@8932D08C49CA76BD843C51EDD1D6640510FA032A7A2403E572BBDA2E5C6B753991CF7E091141D20A2499C5CD3E14C1639D17482E14E1548E5246ACF4E7193D524CDDAC9C9D6A9A36C596B4ECC68BEB0C5BB7082224946FC98E3ADE214EA1343E2DA8DF4229D4D8572AD8679228928A787B6E5390D3A713102FFCC9D0B2188C92
Version: GnuPG v1.4.5 (GNU/Linux)

The following steps are always required when you’ve made changes to the configuration files.

Edit /etc/yulerc and remove the first 3 and last 7 lines, this is the GPG/PGP signature.


Hash: SHA1
There will be another line here later on.
# Configuration file template for yule.

Lots of Yule configuration removed ...

Version: GnuPG v1.4.5 (GNU/Linux)

Then sign the configuration file again with the user yule and copy the file in place as the root user:

su – yule
gpg -o yulerc.asc -a –clearsign –not-dash-escaped /etc/yulerc <– Type in the passphrase we set earlier.
/bin/mv /home/yule/yulerc.asc /etc/yulerc
service yule reload


A Little More Work On The Client

We will need to create the configuration file and embed it into a postscript file. Make sure you have Imagemagick installed, as you will need convert.

Go and download a good looking picture like You will want at least a 200K size image, if not larger, to hide the configuration file in it. Also, it is handy to have an original configuration file as backup.

NOTE: The following steps has to be done each time you wish to modify the configuration file of the client.

convert tycho_chandra_big.jpg <– Convert the JPG to a postscript file.
cp samhainrc.linux rc.`hostname` <– Get a default configuration.
gpg -a –clearsign –not-dash-escaped rc.`hostname` <– Clear sign the configuration.
mv rc.`hostname`.asc rc.`hostname` <– Move the signed file to the normal file name for the configuration file.
/usr/local/sbin/john_stealth -s rc.`hostname`<– Steganographically hide the configuration file inside the postscript file.
rm rc.`hostname` tycho_chandra_big.* <– Remove the “clean” files.

Make sure that the resulting postscript file is not very large, or Samhain will fail to download it. I do not have exact numbers, but from experience 66Mb is too large 🙂

Now copy the file over to your server:

scp YULE_SERVER:~/rc.`hostname` <– Assuming scp with root. In real life, please do not open ssh for root.

Back to the server:

cp ~/rc.CliENT_FQDN /var/lib/yule/ <– Make sure to fill in the clients FQDN.
chown yule:yule /var/lib/yule/rc.CliENT_FQDN <– Make sure to fill in the clients FQDN.

Back to the client:

/usr/local/sbin/john -t init -p info

This will now build the database in /var/lib/john. Don’t worry about all the output at this stage, we are just getting things up and running now.

After we have a database, we have to sign it and copy it over to the server.

gpg -a –clearsign –not-dash-escaped /var/lib/john/john
scp /var/lib/john/john.asc YULE_SERVER:~/file.`hostname` <– Assuming scp with root. In real life, please do not open ssh for root.
rm /var/lib/john/john*

Back to the server, we move this file to the correct place:

mv ~/file.CliENT_FQDN /var/lib/yule
chown yule:yule /var/lib/yule/*

It is important that all configuration files start with rc and all database files start with file.



Trouble, what trouble?

  1. Start with tailing the log file on the server : tail -f /var/log/yule/yule_log
  2. Change the log level in /etc/yulerc to info or above (always remember to re-sign the configuration file as described).
  3. Recompile without some of the options to test.
  4. Have a look at this link :

Clean Up

Now we don’t want to be leaving breadcrumbs behind us, some clean up is required.

  1. Delete all the source files and any tarballs that was downloaded if you built directly on a production server.
  2. Delete all entries from your shell history.
  3. Remove all the development packages that was installed if you built directly on a production server.
  4. Remove /usr/local/sbin/john_stealth and /usr/local/sbin/john_setpwd.

Basically, get rid of any evidence of what you just did.



Arguably, this is where the guide should start. Samhain does not understand what is right and what is wrong for this particular server. As such, you need to tune it. The simplest way is to build Samhain without any options what so ever like:

mkdir /var/lib/samhain/

Put the configuration file in /etc/samhainrc, and run

samhain -t init -p info > my_output 2>&1

You can then examine the output file and make the appropriate changes to the Samhain configuration file. The database will be created in /var/lib/samhain. Do not run samhain -t init more than once without deleting the database.

Once you are happy with the configuration, build Samhain in server / client mode.

NOTE: It is however rather important that you profile your server and tune Samhain before it is connected to the Internet.


Honey Pot!

Now for a bit of fun. We really do want intruders to let us know they are on our system. So, we create 2 (or more) files with catchy names and tell Samhain to monitor those files for any changes (that includes access times).

cp /etc/passwd /home/cracked_passwords
cp /etc/hosts /home/customers/credit_cards_2008.xls

Now, in Samhain’s configuration file, there is a section called [IgnoreNone], add these files in that section. You can test this by simply catting those files and then run the check. The output should be something like:

CRIT : [2009-04-27T21:33:11+0100] msg=<POliCY [User1] ——–T->, path=</home/cracked_passwords>, atime_old=<[2009-04-27T20:25:39]>,


Nagios Integration

I have not tested this yet, this is just on top of my head, so it may well be very wrong.

So now we have alerts for when things go wrong. By default, the standard Nagios plugin pack ships with check_log. Our Nagios check command will look something like:

check_log -F /var/log/yule_log -O /var/log/yule/yule_nagios_diff_log -q "ERROR|CRIT|ALERT"

You will need to modify how to alert on this particular service. By default Nagios will check 3 times before alerting, but with check_log you will never get an alert. The reason is as follows:

  1. check 1: The check returns an error, as it spotted your query (lets say CRIT) in the difference from the old stored log file and the current running log file. The check command now updates the old stored log file.
  2. check 2: There is no longer a difference between the old stored log file and the current running one, thus the check passes OK.

Either modify Nagios to alert after a single failure, or write a wrapper specifically for this check to create a lock file somewhere. You then check for this lock file and alert if it exists. Both approaches have some down sides. If we alert on a single check, be prepared for false alerts due to packet loss or a shift in the force. If we create a lock file, you will have to manually remove it.

Now that we are monitoring the log file for changes detected, we also need to monitor that the client process is still up and running. Of course, you will also want to monitor that the server process is running all the time.

I am sure someone will come up with a better way of Nagios integration, like I said, this is just thinking out loud.


What It All Means

At the end of the day, the clear text configuration of each machine being monitored, is neither kept on the client nor on the server. The clear text configuration files should be kept on a different machine inside an encrypted partition.

Nagios makes sure we are alerted of anything (via e-mail or SMS) and hopefully, an intruder will bite on the honey so that we can see him, potentially, even quicker.

Further more, you can not access any help files (such as ./samhain –help or man pages) to indicate that there is a HIDS running on the client.

Of course, if you get access to the server, you can see all the clients who logs in. There are further compiler options so that the logs are also encrypted.



In the voice of Shrek: “Security is like an onion, it has many layers.” Remember that host based intrusion detection is just one more layer in this onion. You also need a good firewall, network intrusion detection, monitoring, centralised logging, log analysis, TCP wrappers, SELinux (or some other mandatory access control mechanism), brute force blockers like fail2ban and much more.

As an example of this, the entire host based intrusion detection is rendered moot if the hacker just kills the process and you are not using monitoring to make sure that the service is running.

Please do not hesitate to contact me with any corrections or improvements or even some constructive criticism. 🙂