Cheap VPS & Xen Server

Residential Proxy Network - Hourly & Monthly Packages

Server Monitoring With munin And monit On Debian Squeeze


In this article I will describe how you can monitor your Debian Squeeze server with

munin and monit. munin produces nifty little graphics about nearly every aspect of your server (load average, memory usage, CPU usage, MySQL throughput, eth0 traffic, etc.) without much configuration, whereas monit checks the availability of services like Apache, MySQL, Postfix and takes the appropriate action such as a restart if it finds a service is not behaving as expected. The combination of the two gives you full monitoring: graphics that lets you recognize current or upcoming problems (like “We need a bigger server soon, our load average is increasing rapidly.”), and a watchdog that ensures the availability of the monitored services.

Although munin lets you monitor more than one server, we will only discuss the monitoring of the system where it is installed here.

This tutorial was written for Debian Squeeze, but the configuration should apply to other distributions with little changes as well.

I want to say first that this is not the only way of setting up such a system. There are many ways of achieving this goal but this is the way I take. I do not issue any guarantee that this will work for you!

 

1 Preliminary Note

Our system’s hostname is server1.example.com, and we have a web site www.example.com on it with the document root /var/www/www.example.com/web.

 

2 Install And Configure munin

To install munin on Debian Squeeze, we do this:

apt-get install munin munin-node munin-plugins-extra

Next, we must edit the munin configuration file /etc/munin/munin.conf. Uncomment the dbdir, htmldir, logdir, rundir, and tmpldir lines (the default values are fine). We want munin to use the name server1.example.com instead of localhost.localdomain in the HTML output, therefore we replace localhost.localdomain with server1.example.com in the simple host tree section. Without the comments, the changed file looks like this:

vi /etc/munin/munin.conf

# Example configuration file for Munin, generated by 'make build'

# The next three variables specifies where the location of the RRD
# databases, the HTML output, logs and the lock/pid files.  They all
# must be writable by the user running munin-cron.  They are all
# defaulted to the values you see here.
#
dbdir   /var/lib/munin
htmldir /var/cache/munin/www
logdir /var/log/munin
rundir  /var/run/munin
#
# Where to look for the HTML templates
tmpldir /etc/munin/templates

# (Exactly one) directory to include all files from.
#
includedir /etc/munin/munin-conf.d
[...]
# a simple host tree
[server1.example.com]
    address 127.0.0.1
    use_node_name yes
[...]

We should find the Apache configuration file for munin /etc/apache2/conf.d/munin (which actually is a symlink to /etc/munin/apache.conf) – it defines an alias called munin to munin’s HTML output directory /var/cache/munin/www which means we can access munin from all web sites on this server by using the relative path /munin (e.g. http://www.example.com/munin).

Make sure you comment out the line Allow from localhost 127.0.0.0/8 ::1 and all Allow from all instead (otherwise you will only be able to access the munin output from localhost):

vi /etc/apache2/conf.d/munin

Alias /munin /var/cache/munin/www
<Directory /var/cache/munin/www>
        Order allow,deny
        Allow from all
        #Allow from localhost 127.0.0.0/8 ::1
        Options None

        # This file can be used as a .htaccess file, or a part of your apache
        # config file.
        #
        # For the .htaccess file option to work the munin www directory
        # (/var/cache/munin/www) must have "AllowOverride all" or something
        # close to that set.
        #

        # AuthUserFile /etc/munin/munin-htpasswd
        # AuthName "Munin"
        # AuthType Basic
        # require valid-user

        # This next part requires mod_expires to be enabled.
        #

        # Set the default expiration time for files to 5 minutes 10 seconds from
        # their creation (modification) time.  There are probably new files by
        # that time.
        #

    <IfModule mod_expires.c>
        ExpiresActive On
        ExpiresDefault M310
    </IfModule>

</Directory>

Restart Apache:

/etc/init.d/apache2 restart

Then restart munin:

/etc/init.d/munin-node restart

Now wait a few minutes so that munin can produce its first output, and then go to http://www.example.com/munin/ in your browser, and you see the first statistics. After a few days this could look like this:

1

(This is just a small excerpt of the many graphics that munin produces…)

 

3 Password-Protect The munin Output Directory (Optional)

Now it is a good idea to password-protect the munin output directory unless you want everybody to be able to see every little statistic about your server.

To do this, we must create the password file /etc/munin/munin-htpasswd. We want to log in with the username admin, so we do this:

htpasswd -c /etc/munin/munin-htpasswd admin

Enter a password for admin. Then open /etc/apache2/conf.d/munin again…

vi /etc/apache2/conf.d/munin

… and uncomment the following section:

[...]
        AuthUserFile /etc/munin/munin-htpasswd
        AuthName "Munin"
        AuthType Basic
        require valid-user
[...]

Then restart Apache:

/etc/init.d/apache2 restart

4 Install And Configure monit

To install monit, we do this:

apt-get install monit

Now we must edit /etc/monit/monitrc. The default /etc/monit/monitrc has lots of examples, and you can find more configuration examples on http://mmonit.com/monit/documentation/. However, in my case I want to monitor proftpd, sshd, mysql, apache, and postfix, I want to enable the monit web interface on port 2812, I want a https web interface, I want to log in to the web interface with the username admin and the password test, and I want monit to send email alerts to root@localhost, so my file looks like this (I’ve added examples for other daemons to the configuration so that you can adjust the file to your needs):

cp /etc/monit/monitrc /etc/monit/monitrc_orig
cat /dev/null > /etc/monit/monitrc
vi /etc/monit/monitrc

set daemon  60
set logfile syslog facility log_daemon
set mailserver localhost
set mail-format { from: monit@server1.example.com }
set alert root@localhost
set httpd port 2812 and
     SSL ENABLE
     PEMFILE  /var/certs/monit.pem
     allow admin:test

check process proftpd with pidfile /var/run/proftpd.pid
   start program = "/etc/init.d/proftpd start"
   stop program  = "/etc/init.d/proftpd stop"
   if failed port 21 protocol ftp then restart
   if 5 restarts within 5 cycles then timeout

check process sshd with pidfile /var/run/sshd.pid
   start program  "/etc/init.d/ssh start"
   stop program  "/etc/init.d/ssh stop"
   if failed port 22 protocol ssh then restart
   if 5 restarts within 5 cycles then timeout

check process mysql with pidfile /var/run/mysqld/mysqld.pid
   group database
   start program = "/etc/init.d/mysql start"
   stop program = "/etc/init.d/mysql stop"
   if failed host 127.0.0.1 port 3306 then restart
   if 5 restarts within 5 cycles then timeout

check process apache with pidfile /var/run/apache2.pid
   group www
   start program = "/etc/init.d/apache2 start"
   stop program  = "/etc/init.d/apache2 stop"
   if failed host www.example.com port 80 protocol http
      and request "/monit/token" then restart
   if cpu is greater than 60% for 2 cycles then alert
   if cpu > 80% for 5 cycles then restart
   if totalmem > 500 MB for 5 cycles then restart
   if children > 250 then restart
   if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout

check process postfix with pidfile /var/spool/postfix/pid/master.pid
   group mail
   start program = "/etc/init.d/postfix start"
   stop  program = "/etc/init.d/postfix stop"
   if failed port 25 protocol smtp then restart
   if 5 restarts within 5 cycles then timeout

#check process nginx with pidfile /var/run/nginx.pid
#   start program = "/etc/init.d/nginx start"
#   stop  program = "/etc/init.d/nginx stop"
#   if failed host 127.0.0.1 port 80 then restart
#
#check process memcached with pidfile /var/run/memcached.pid
#   start program = "/etc/init.d/memcached start"
#   stop  program = "/etc/init.d/memcached stop"
#   if failed host 127.0.0.1 port 11211  then restart
#
#check process pureftpd with pidfile /var/run/pure-ftpd/pure-ftpd.pid
#   start program = "/etc/init.d/pure-ftpd-mysql start"
#   stop program  = "/etc/init.d/pure-ftpd-mysql stop"
#   if failed port 21 protocol ftp then restart
#   if 5 restarts within 5 cycles then timeout
#
#check process named with pidfile /var/run/named/named.pid
#   start program = "/etc/init.d/bind9 start"
#   stop program = "/etc/init.d/bind9 stop"
#   if failed host 127.0.0.1 port 53 type tcp protocol dns then restart
#   if failed host 127.0.0.1 port 53 type udp protocol dns then restart
#   if 5 restarts within 5 cycles then timeout
#
#check process ntpd with pidfile /var/run/ntpd.pid
#   start program = "/etc/init.d/ntp start"
#   stop  program = "/etc/init.d/ntp stop"
#   if failed host 127.0.0.1 port 123 type udp then restart
#   if 5 restarts within 5 cycles then timeout
#
#check process mailman with pidfile /var/run/mailman/mailman.pid
#   group mail
#   start program = "/etc/init.d/mailman start"
#   stop  program = "/etc/init.d/mailman stop"
#
#check process amavisd with pidfile /var/run/amavis/amavisd.pid
#   group mail
#   start program = "/etc/init.d/amavis start"
#   stop  program = "/etc/init.d/amavis stop"
#   if failed port 10024 protocol smtp then restart
#   if 5 restarts within 5 cycles then timeout
#
#check process courier-imap with pidfile /var/run/courier/imapd.pid
#   group mail
#   start program = "/etc/init.d/courier-imap start"
#   stop program = "/etc/init.d/courier-imap stop"
#   if failed host localhost port 143 type tcp protocol imap then restart
#   if 5 restarts within 5 cycles then timeout
#
#check process courier-imap-ssl with pidfile /var/run/courier/imapd-ssl.pid
#   group mail
#   start program = "/etc/init.d/courier-imap-ssl start"
#   stop program = "/etc/init.d/courier-imap-ssl stop"
#   if failed host localhost port 993 type tcpssl sslauto protocol imap then restart
#   if 5 restarts within 5 cycles then timeout
#
#check process courier-pop3 with pidfile /var/run/courier/pop3d.pid
#   group mail
#   start program = "/etc/init.d/courier-pop start"
#   stop program = "/etc/init.d/courier-pop stop"
#   if failed host localhost port 110 type tcp protocol pop then restart
#   if 5 restarts within 5 cycles then timeout
#
#check process courier-pop3-ssl with pidfile /var/run/courier/pop3d-ssl.pid
#   group mail
#   start program = "/etc/init.d/courier-pop-ssl start"
#   stop program = "/etc/init.d/courier-pop-ssl stop"
#   if failed host localhost port 995 type tcpssl sslauto protocol pop then restart
#   if 5 restarts within 5 cycles then timeout
#
#check process dovecot with pidfile /var/run/dovecot/master.pid
#   group mail
#   start program = "/etc/init.d/dovecot start"
#   stop program = "/etc/init.d/dovecot stop"
#   if failed host localhost port 993 type tcpssl sslauto protocol imap then restart
#   if 5 restarts within 5 cycles then timeout

The configuration file is pretty self-explaining; if you are unsure about an option, take a look at the monit documentation: http://mmonit.com/monit/documentation/monit.html

In the apache part of the monit configuration you find this:

   if failed host www.example.com port 80 protocol http
      and request "/monit/token" then restart

which means that monit tries to connect to www.example.com on port 80 and tries to access the file /monit/token which is /var/www/www.example.com/web/monit/token because our web site’s document root is /var/www/www.example.com/web. If monit doesn’t succeed it means Apache isn’t running, and monit is going to restart it. Now we must create the file /var/www/www.example.com/web/monit/token and write some random string into it:

mkdir /var/www/www.example.com/web/monit
echo “hello” > /var/www/www.example.com/web/monit/token

Next we create the pem cert (/var/certs/monit.pem) we need for the SSL-encrypted monit web interface:

mkdir /var/certs
cd /var/certs

We need an OpenSSL configuration file to create our certificate. It can look like this:

vi /var/certs/monit.cnf

# create RSA certs - Server

RANDFILE = ./openssl.rnd

[ req ]
default_bits = 2048
encrypt_key = yes
distinguished_name = req_dn
x509_extensions = cert_type

[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default = MO

stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Monitoria

localityName                    = Locality Name (eg, city)
localityName_default            = Monittown

organizationName                = Organization Name (eg, company)
organizationName_default        = Monit Inc.

organizationalUnitName          = Organizational Unit Name (eg, section)
organizationalUnitName_default  = Dept. of Monitoring Technologies

commonName                      = Common Name (FQDN of your server)
commonName_default              = server.monit.mo

emailAddress                    = Email Address
emailAddress_default            = root@monit.mo

[ cert_type ]
nsCertType = server

Now we create the certificate like this:

openssl req -new -x509 -days 365 -nodes -config ./monit.cnf -out /var/certs/monit.pem -keyout /var/certs/monit.pem

openssl gendh 512 >> /var/certs/monit.pem

openssl x509 -subject -dates -fingerprint -noout -in /var/certs/monit.pem

chmod 700 /var/certs/monit.pem

Afterwards we edit /etc/default/monit to enable the monit daemon. Change startup to 1:

vi /etc/default/monit

# Defaults for monit initscript
# sourced by /etc/init.d/monit
# installed at /etc/default/monit by maintainer scripts
# Stefan Alfredsson <alfs@debian.org>

# You must set this variable to for monit to start
startup=1

# You can change the location of the state file here
# It can also be set in monitrc
# STATEFILE="/var/lib/monit/monit.state"

# To change the intervals which monit should run,
# edit the configuration file /etc/monit/monitrc
# It can no longer be configured here.

Finally, we can start monit:

/etc/init.d/monit start

Now point your browser to https://www.example.com:2812/ (make sure port 2812 isn’t blocked by your firewall), log in with admin and test, and you should see the monit web interface. It should look like this:

2

(Main Screen)

3

(Apache Status Page)

Depending on your configuration in /etc/monit/monitrc monit will restart your services if they fail and send notification emails if process IDs of services change, etc.

Have fun!

 

  • munin: http://munin-monitoring.org/
  • monit: http://mmonit.com/monit/

Comments

comments