Cheap VPS & Xen Server

Residential Proxy Network - Hourly & Monthly Packages

Network Monitoring Appliance


My ambition was to implement a small (better tiny) appliance for monitoring network health and network resources, short and longtime trends, running under VMware Server or VMware ESX. So I had an eye upon all components which are implemented on the system, to be as leightweight as possible. This was also the reason why no SQL DBMS based software was used.

The appliance is based on Ubuntu Jeos LTS (8.04.3 at the time of this writing). Almost all used components are from the related repositories. This tutorial shows how the appliance was implemented. I do not issue any guarantee that this will work for you!

Used components:

* Ubuntu 8.04.3 JeOS as OS

* Nagios 2.11 for monitoring and alarming

* Smokeping 2.3 to observe latencies and paketloss

* MRTG 2.14.7 to observe networktraffic’s tendencies

* RRDTool 1.2.19 as Round-Robin Database for storing all measurement data

* Lighttpd 1.4.19 as fast, lightweight webserver frontend

* weathermap4rrd for illustrating the networkweather

* ssmtp as extremely lightweight MTA for maildelivery

 

Preliminary Note

In this tutorial I use the hostname jeos.example.com with an IP address allocated with DHCP. These settings might differ for you, so you have to replace them where appropriate. As this whole installation is not GUI based, you should be familiar using bash, vi and similar programs. Also all work should be done with root privilegs or with sudo prepending each command.

 

1. Ubuntu Server JeOS 8.04 LTS

The idea behind Ubuntu Server JeOS is to use it as lightweight, performant base to build appliances under VMware ESX/Server or KVM.

You can download an ISO image from http://cdimage.ubuntu.com/jeos/releases/8.04.3/release/jeos-8.04.3-jeos-i386.iso.

As we want the whole VM to be lightweight, we allocate 1 CPU, 192MR RAM (both easily changeable in VMware) and 1GB diskspace.

Installation is straightforward and some screenshots are shown subsequent. There is also a tutorial regarding installation on https://help.ubuntu.com/community/JeOS.

juice1

Hostname:

juice8

Guided disk partitioning:

juice9

We use the entire disk. swap makes no real sense for this VM, but the swapspace is automatically configured, so we leave it.

juice10

Create an account for further logins:

juice15

After a while base installation is finished:

juice22

Now the system is base-installed and ready for further activities. First you should do an update/upgrade of all installed packages to the newest versions:

apt-get update && apt-get upgrade

Maybe we have to do another reboot and log in with the account created while installing the system:

juice25

As your system is only reachable inside the VMware console, another good idea might be to install ssh:

apt-get install ssh

Now we are going to install all software we need for building the appliance. As this system should be used for solving network problems, we also install some more packages which may be helpful. Feel free to extend this list according to your needs:

apt-get install lighttpd ssmtp mailx anacron build-essential linux-headers-$(uname -r) psmisc nmap rsync snmp openssh-server sshfs ntop smokeping xinetd mrtg mrtg-rrd nagios2 nagios2-doc localepurge lynx dnsutils bzip2 traceroute tcptraceroute iptables wget lsof pv telnet time whois alien

 

VMware Tools

The VMware Tools may not be of great help, as this system is without X11, but nevertheless you can install them in two ways:

In the VMware Virtual-Center Web-Access (or whichever VMware administrative console you have) mount the virtual CD for this VM, and mount it also from inside the VM by doing a

mount /media/cdrom

Either convert the VmwareTools .rpm package by using alien into a .deb package and install them by doing a dpkg -i vmwaretools*.debjuice25

or

unpack the archive VmwareTools-*.tar.gz via tar xvzf vmwaretools_*.tar.gz and manually install them (see in detail on https://help.ubuntu.com/community/JeOS#Installing%20VMware%20Tools)

Subsequently a

apt-get remove build-essential linux-headers-$(uname -r) && apt-get clean && apt-get autoremove

could be done to remove unnecessary packages and to free some disk space.

Now the base system is really ready installed. Let’s go on with the server applications.

2. Lighttpd

As Nagios, MRTG, Smokeping and Weathermap4rrd use Lighttpd as common component to show their measurement results to the outside, in this step Lighttpd will be basically configured. All applications will be called via CGI scripts, so at least the standardmodule mod_cgi of lighty is necessary.

Therefore add the following to the section server.modules of /etc/lighttpd/lighttpd.conf:

            "mod_auth", 
            "mod_setenv", 
            "mod_cgi"

As Nagios also needs the modules mod_auth and mod_setenv, they should also be added.

For the mod_cgi should be checked, if there is a symlink inside of /etc/lighttpd/conf-enabled pointing to /etc/lighttpd/conf-available/10-cgi.conf:

ls /etc/lighttpd/conf-enabled

total 0 
lrwxrwxrwx 1 root root 40 2009-08-13 11:06 10-cgi.conf -> /etc/lighttpd/conf-available/10-cgi.conf

After that have a look into /etc/lighttpd/conf-available/10-cgi.conf and check, that an alias for cgi-bin exists, which points to the location in the filesystem, where the CGI-scripts really are:

alias.url       += ( "/cgi-bin/" => "/usr/lib/cgi-bin/" )

The CGI-scripts are located outside the documentroot of Lighttpd.

Logging of Lighttpd is done under /var/log/lighttpd/.

 

3. Smokeping

Smokeping realizes a simple method to estimate latency times, paketloss and the like in a networked environment. In the simplest case a destination is pinged every 5 minutes with 20 ICMP_ECHO_REQUEST’s, and the timely diversion (meanvalue and standarddeviation) of the replytimes of the ICMP_ECHO_REPLY’s are visualized. History over time is realized with the help of RRDTool. Therefore Smokeping runs as service/daemon, and is automatically started on system boot. Configuration of destinations and probes is done in the config file /etc/smokeping/config. The visualized measurements are viewed in the webbrowser, and delivered via Lighttpd to the client running the browser.

For a complete documentation of all possibilities of smokeping please have a look at the site of its maintainer, where you can also find some examples: http://oss.oetiker.ch/smokeping/doc/index.en.html.

Start/Stop/Restart of the service:

/etc/init.d/smokeping start|stop|restart

Check of the syntax of the configfile:

smokeping -check

 

4. MRTG

Basically MRTG is used to visualize the workload of routers in transmission networks. The visualization is done in a way that both short periods (several hours) of time are viewed in more detail, and longer periods (months and years) are visualized with less details (for instance to see trends for capacity planning). This is realized by reading counters via SNMP from the routers in (default) intervals of 5 minutes, stored and visualized in form of .png files. These .png files are typically delivered to the outside world via a webserver and viewed in a webbrowser. The values of the counters are stored either inside of ASCII Files (default), or better inside of Round-Robin Databases. The possibilities of MRTG regarding such measurements and visualizations are nearly infinite, as not only SNMP could be used as source of data, but also self written scripts or programs. Some examples can be seen on the site of its maintainer: http://oss.oetiker.ch/mrtg/ where also the complete documentation could be found under http://oss.oetiker.ch/mrtg/doc/index.en.html.

Configuration is done in the ASCII File /etc/mrtg.cfg. Differing from the default “LogFormat: rrdtool” is used (see http://oss.oetiker.ch/mrtg/doc/mrtg-rrd.en.html). This has 2 objectives, performance and reusability.

In a defaultinstallation all destinations are polled in 5 minute intervals, measured values are stored and .png files will be created. Through the use of “Logformat: rrdtool” the generation of the .png files is omitted, the .png files will only be generated on-demand. This is done with the help of the CGI-script mrtg-rrd.cgi, which is already installed in /usr/lib/cgi-bin and ready to be used.

As it is a perlscript you can check for the line

my $conffile = "/etc/mrtg-rrd.conf";

somewhere near the begin of it.

Permission/ownership should be

-rwxr-xr-x 1 www-data www-data 28160 2009-08-13 09:57 /usr/lib/cgi-bin/mrtg-rrd.cgi

More information can be found on http://www.fi.muni.cz/~kas/mrtg-rrd/.

Creating entries for systems which should be observed with MRTG is done with the help of cfgmaker (details are in its manpage). The syntax of the configuration could be checked with mrtg -check

MRTG is scheduled with cron, so no service/daemon can be started, stopped, restarted etc. In this case there is an entry in /etc/cron.d/mrtg.

Logging goes to /var/log/mrtg/.

 

5. SSMTP

SSMTP is an extremely simple, lightweigt, no-daemon send-only MTA to a smarthost (which should be enough for this kind of setup).

Configuration is done in /etc/ssmtp/ssmtp.conf

See more details on its manpage.

 

6. RRDTool

Round-Robin database for Smokeping, MRTG, Weathermap4rrd und Nagios. No configuration necessary. Documentation on http://oss.oetiker.ch/rrdtool/.

 

7. Nagios

Nagios is a powerful, highly configurable Monitoring- and Alarmingsystem, which can monitor a wide variety of systems (network, server, daemons, applications). Monitoring could be done for instance for availability or utilization. The monitoring could be restricted to services which are connectable from the outside (e.g. a webserver on port 80/tcp), or with the help of NRPE (Nagios Remote Plugin Executor) plugins for testing could also be executed remote.

There is a lot of info about nagios in the net, so feel free to use your preferred search engine to find whatever you need about it.

 

Lighttpd

Nagios uses a webserver, to communicate with its users, again in our case Lighttpd. There have to be added some more details to Lighty to run with Nagios:

To /etc/lighttpd/lighttpd.conf append:

alias.url +=     ( "/nagios2/cgi-bin" => "/usr/lib/cgi-bin/nagios2" ) 
alias.url +=     ( "/nagios2/stylesheets" => "/etc/nagios2/stylesheets" )
alias.url +=     ( "/nagios2" => "/usr/share/nagios2/htdocs" ) 

$HTTP["url"] =~ "^/nagios2/cgi-bin" { 
        cgi.assign = ( "" => "" ) 
} 
$HTTP["url"] =~ "nagios2" { 
        auth.backend = "htpasswd" 
        auth.backend.htpasswd.userfile = "/etc/nagios2/htpasswd.users" 
        auth.require = ( "" => ( 
                "method" => "basic", 
                "realm" => "Nagios Access", 
                "require" => "user=nagiosadmin|user=helpdesk" 
                ) 
        ) 
        setenv.add-environment = ( "REMOTE_USER" => "www-data" ) 
}

These entries bridge the gap between Nagios (which is most often running under Apache) and Lighttpd, and implement 2 accounts (nagiosadmin, helpdesk) for logging into Nagios.

The passwords for these accounts are created with htpasswd from the package apache2-utils:

htpasswd /etc/nagios2/htpasswd.users nagiosadmin

 

Nagios

Configuration of Nagios itself is done under /etc/nagios2. The main configfile is nagios.cfg. After modifications in the nagios configuration it would be wise to syntax-check it before you load it (pre-flight check):

In the directory /etc/nagios2 do a

nagios2 -v nagios.cfg

As Nagios also runs as a service/daemon (like Lighthttpd and Smokeping), it has to be reloaded after modifications in its configuration. This is done by

/etc/init.d/nagios2 reload

 

8. Weathermap4rrd

Weathermap4rrd is an application which creates a map of the networkweather, which is another form of visualization of the same data MRTG also visualizes. In a weathermap commonly there is a visualization of the most vital networknodes (routers) and the transmission paths between the nodes. The paths between the nodes are coloured in a way to show the load they are transporting, the nodes could also be colored in a way to show their health (maybe green for ok, red for down). So its possible to get a quick but nevertheless representive overview in one picture over a complex structured net and the datastreams in it. In contrast to MRTG only the newest data is visualized, there is no history.

Weathermap4rrd simply reads the data MRTG collects for the configured routers in the dedicated .rrd files. As MRTG polls its destination in 5 minute intervals, actualization of the weathermap also makes sense in 5 minute intervals. MRTG is scheduled with cron, in the easiest case this is also done with weathermap4rrd.

The cron entry for mrtg is in /etc/cron.d/mrtg and looks like

*/5 *   * * *   root    if [ -d /var/lock/mrtg ]; then if [ -x /usr/bin/mrtg ] && [ -r /etc/mrtg.cfg ]; then env LANG=C /usr/bin/mrtg /etc/mrtg.cfg >> /var/log/mrtg/mrtg.log 2>&1; fi else mkdir /var/lock/mrtg; fi

By appending a line like

*/5 *   * * *   root if [ -x /usr/bin/weathermap4rrd ]; then /usr/bin/weathermap4rrd 2>&1; fi

/usr/bin/weathermap4rrd is the binary which produces the weathermap according to its configfile in /etc/weathermap4rrd/weathermap.conf. In the configfile all configuration for weathermap4rrd is done, what kind of grapicfile is produced, where in the filesystem it is stored, which nodes and communication paths should be on the map, where the .rrd files are located whih should be evaluated, and so on. For details please have a look into the documentation of it on http://weathermap4rrd.tropicalex.net/documentation.php.

 

Weathermap4rrd PHP Version

When you decide to use the php version of weathermap4rrd (not in the Ubuntu repos), you first have to install some packages:

For Lighttpd: apt-get install php5-cgi php

For Weathermap4rrd-php: apt-get install php5-gd

In /etc/lighttpd/lighttpd.conf add:

   "mod_rewrite",
        "mod_redirect",
        "mod_fastcgi",
        "mod_access",

to the server.modules section of the configfile.

In /etc/lighttpd/conf-enabled a new symlink has to be created:

lrwxrwxrwx 1 root root 40 2009-08-13 11:06 10-cgi.conf -> /etc/lighttpd/conf-available/10-cgi.conf

and Lighty eventually has to be restarted.

The archive of the php-version of weathermap4rrd has to be downloaded from http://weathermap4rrd.tropicalex.net/ and to be unpacked directly under the documentroot of Lighty.

Configuration is in the directory documentroot/weathermap*/weathermap.conf and is syntactically very similar to the configuration of the perlversion (but has some additional capabilities).

The php version runs on demand (when the index.php script is loaded into the browser), the weathermap is automatically refreshed (so we do not need any crontab entries) and has some enhancements compared against the perl version.

http://www.ubuntu.com/products/whatisubuntu/serveredition/jeos

https://help.ubuntu.com/community/JeOS

https://help.ubuntu.com/community/JeOS#Installing%20VMware%20Tools)

Nagios

http://oss.oetiker.ch/smokeping/

http://oss.oetiker.ch/mrtg/

http://www.fi.muni.cz/~kas/mrtg-rrd/

http://oss.oetiker.ch/rrdtool/

http://www.lighttpd.net/

http://redmine.lighttpd.net/projects/1/wiki/NagiosRecipe

http://weathermap4rrd.tropicalex.net/

ftp://ftp.debian.org/debian/pool/main/s/ssmtp/

Comments

comments