Cheap VPS & Xen Server


Residential Proxy Network - Hourly & Monthly Packages

Server monitoring with Munin and Monit on CentOS 7.2


In this article, I will describe how you can monitor your CentOS 7 server with Munin and Monit. Munin produces nifty little graphics about nearly every aspect of your server (load average, memory usage, CPU usage, MySQL throughput, eth0 traffic, etc.) without much configuration, whereas Monit checks the availability of services like Apache, MySQL, Postfix and takes the appropriate action such as a restart if it finds a service is not behaving as expected. The combination of the two gives you full monitoring: graphics that lets you recognize current or upcoming problems (like “We need a bigger server soon, our load average is increasing rapidly.”), and a watchdog that ensures the availability of the monitored services.

Although Munin lets you monitor more than one server, we will only discuss the monitoring of the system where it is installed here.

This tutorial was written for CentOS 7.2, but the configuration should apply to other distributions like RHEL and Scientific Linux as well.

 

1 Preliminary Note

Our system’s hostname is server1.example.com, and we have a website www.example.com on it with the document root /var/www/html.

 

2 Enable the EPEL Repository

On CentOS 7, Munin and Monit are not available in the default CentOS repositories. Fortunately, we can install them from EPEL repository. To enable the EPEL repository, we run:

yum -y install epel-release

Import the EPEL GPG-key:

rpm –import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

and then run:

yum -y update

to ensure that the system is up to date and that the package list from EPEL is loaded before we start to install munin.

2 Install Apache web server

Munin requires a web server to serve and display its statistics files. I will install the apache httpd server here:

yum install httpd

Start apache and enable it to be started automatically at boot time.

systemctl enable httpd
systemctl start httpd

 

4 Install and Configure Munin

To install Munin on CentOS, we do this:

yum -y install munin munin-node

Then we create the system startup links for Munin and start it:

systemctl enable munin-node
systemctl start munin-node

Next, we must edit the Munin configuration file /etc/munin/munin.conf.

nano /etc/munin/munin.conf

We want munin to use the name server1.example.com instead of localhost in the HTML output, therefore we replace localhost with server1.example.com. Without the comments, the changed file looks like this:

[...]
# a simple host tree
[server1.example.com]
    address 127.0.0.1
    use_node_name yes
[...]

The munin statistics are protected with a username and password (Apache basic auth). In the next step we add a new user and password to the /etc/munin/munin-htpasswd file.

htpasswd /etc/munin/munin-htpasswd admin

The command will add a new user with the name “admin” and prompt twice for the new password.

munin_htpasswd

Now wait a few minutes so that munin can produce the first statistics output, then go to http://server1.example.com/munin/ in your browser, and you see the first statistics.

munin

(This is just a small excerpt of the many graphics that munin produces…)

5 Install and Configure Monit

Next, we will install Monit:

yum -y install monit

Then we create the system startup links for Monit:

systemctl enable monit
systemctl start monit

Monit’s default configuration file is /etc/monitrc where you can find some configuration examples (you can find more configuration examples on http://mmonit.com/wiki/Monit/ConfigurationExamples) that are all commented out, but it tells Monit to also look in the directory /etc/monit.d for configuration files.

In this case, I will monitor:

  • proftpd
  • sshd
  • MariaDB
  • apache
  • postfix

Furthermore, I will configure these settings for Monit:

  • Enable the Monit web interface on port 2812.
  • Use HTTPS for the web interface instead of HTTP.
  • Configure a password protected Login for the web interface.
  • Monit shall send email alerts to root@localhost.

First, I will configure the authentification settings. Open the file /etc/monitrc

nano /etc/monitrc

And scroll down until you find this section:

set httpd port 2812 and
   use address localhost # only accept connection from localhost
   allow localhost # allow localhost to connect to the server and
   allow admin:monit # require user 'admin' with password 'monit'
   allow @monit # allow users of group 'monit' to connect (rw)
   allow @users readonly # allow users of group 'users' to connect readonly

Replace it with the following settings:

set httpd port 2812 and
   use address 0.0.0.0
   SSL ENABLE
   PEMFILE /var/certs/monit.pem
   allow admin:test

The word “test” is the password, please replace that with a secure password and you might also want to change the username “admin” to a name that can not be guessed easily.

Now we add the configuration for the monitored services. Instead of modifying /etc/monitrc, we create a new configuration file /etc/monit.d/monitrc.

My file looks like this:

nano /etc/monit.d/monitrc

set logfile syslog facility log_daemon

# Send emails trough this mailserver
set mailserver localhost

# Set the From address of the alert emails
set mail-format { from: monit@server1.example.com }

# Send alerts to this address
set alert root@localhost

# Monitor the Proftpd service
check process proftpd with pidfile /var/run/proftpd/proftpd.pid
   start program = "/usr/bin/systemctl start proftpd"
   stop program  = "/usr/bin/systemctl stop proftpd"
   if failed port 21 protocol ftp then restart
   if 5 restarts within 5 cycles then timeout

# Monitor the SSH service
check process sshd with pidfile /var/run/sshd.pid
   start program  "/usr/bin/systemctl start sshd"
   stop program  "/usr/bin/systemctl stop sshd"
   if failed port 22 protocol ssh then restart
   if 5 restarts within 5 cycles then timeout

# Monitor MariaDB
check process mysql with pidfile /var/run/mariadb/mariadb.pid
   group database
   start program = "/usr/bin/systemctl start mariadb"
   stop program = "/usr/bin/systemctl stop mariadb"
   if failed host 127.0.0.1 port 3306 then restart
   if 5 restarts within 5 cycles then timeout

# Monitor the apache webserver
check process apache with pidfile /var/run/httpd/httpd.pid
   group www
   start program = "/usr/bin/systemctl start httpd"
   stop program  = "/usr/bin/systemctl stop httpd"
   if failed host localhost port 80 protocol http
      and request "/monit_token" then restart
   if cpu is greater than 60% for 2 cycles then alert
   if cpu > 80% for 5 cycles then restart
   if totalmem > 500 MB for 5 cycles then restart
   if children > 250 then restart
   if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout

# Monitor postfix mailserver
check process postfix with pidfile /var/spool/postfix/pid/master.pid
   group mail
   start program = "/usr/bin/systemctl start postfix"
   stop  program = "/usr/bin/systemctl stop postfix"
   if failed port 25 protocol smtp then restart
   if 5 restarts within 5 cycles then timeout

(Please make sure that you check processes only that really exist on your server – otherwise monit won’t start. I.e., if you tell monit to check Postfix, but Postfix isn’t installed on the system, monit won’t start.)

The configuration file is pretty self-explaining; if you are unsure about an option, take a look at the Monit documentation: http://mmonit.com/monit/documentation/monit.html

In the apache part of the Monit configuration you find this:

   if failed host localhost port 80 protocol http
      and request "/monit_token" then restart

which means that Monit tries to connect to localhost on port 80 and tries to access the file /monit_token which is /var/www/html/monit_token because our web site’s document root is /var/www/html. If Monit doesn’t succeed it means Apache isn’t running, and Monit is going to restart it. Now we must create the file /var/www/html/monit_token and write some random string into it:

touch /var/www/html/monit_token

Next, we create the SSL (pem) certificate (/var/certs/monit.pem) we need for the SSL-encrypted Monit web interface:

mkdir /var/certs
cd /var/certs

We need an OpenSSL configuration file to create our certificate. It can look like this:

nano /var/certs/monit.cnf

# create RSA certs - Server

RANDFILE = ./openssl.rnd

[ req ]
default_bits = 1024
encrypt_key = yes
distinguished_name = req_dn
x509_extensions = cert_type

[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default = MO

stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Monitoria

localityName                    = Locality Name (eg, city)
localityName_default            = Monittown

organizationName                = Organization Name (eg, company)
organizationName_default        = Monit Inc.

organizationalUnitName          = Organizational Unit Name (eg, section)
organizationalUnitName_default  = Dept. of Monitoring Technologies

commonName                      = Common Name (FQDN of your server)
commonName_default              = server.monit.mo

emailAddress                    = Email Address
emailAddress_default            = root@monit.mo

[ cert_type ]
nsCertType = server

Now we create the certificate like this:

openssl req -new -x509 -days 365 -nodes -config ./monit.cnf -out /var/certs/monit.pem -keyout /var/certs/monit.pemopenssl

openssl gendh 512 >> /var/certs/monit.pem

openssl x509 -subject -dates -fingerprint -noout -in /var/certs/monit.pem

chmod 700 /var/certs/monit.pem

Finally, we can start Monit:

systemctl restart monit

Now point your browser to https://www.example.com:2812/ (make sure port 2812 isn’t blocked by your firewall), log in with admin and test, and you should see the Monit web interface. It should look like this:

monit_dashboard

(Main Screen)

monit_apache

(Apache Status Page)

Depending on your configuration in /etc/monit.d/monitrc Monit will restart your services if they fail and send notification emails if process IDs of services change, etc.

 

  • munin: http://munin.projects.linpro.no
  • monit: http://mmonit.com/monit
  • CentOS: http://www.centos.org

 

Comments

comments