Cheap VPS & Xen Server

Residential Proxy Network - Hourly & Monthly Packages

Server Monitoring With Icinga On Ubuntu 11.10


Icinga is an enterprise grade open source monitoring system which keeps watch over networks and any conceivable network resource, notifies the user of errors and recoveries and generates performance data for reporting. It is a fork of Nagios. This tutorial explains how to install Icinga on an Ubuntu 11.10 server to monitor this server and another Ubuntu 11.10 server.

I do not issue any guarantee that this will work for you!

 

1 Preliminary Note

In this tutorial I use the following servers:

  • server1.example.com (IP: 192.168.0.100): Icinga server; this server has Apache, MySQL, Postfix, and Courier-IMAP/-POP3 installed that I want to monitor. I have a virtual host named www.example.com on this server.
  • server2.example.com (IP: 192.168.0.101): Icinga client; this server has Apache, MySQL, Postfix, and Dovecot-IMAP/-POP3 installed that I want to monitor.

Because we will run all the steps from this tutorial with root privileges, we can either prepend all commands in this tutorial with the string sudo, or we become root right now by typing

sudo su

 

2 Installing Icinga On The Icinga Host (server1)

server1.example.com:

Icinga can now be installed as follows:

apt-get install icinga icinga-doc icinga-phpapi

You will be asked the following questions:

Apache servers to configure for icinga: <– apache2

Please provide the password to be created with the “icingaadmin” user. Icinga web administration password: <– icingaadmin_password
Re-enter password to verify: <– icingaadmin_password

Configure database for icinga-idoutils with dbconfig-common? <– No

Afterwards you can access Icinga under http://www.example.com/icinga/. You will be asked for a username (icingaadmin) and the password you specified during installation:

1

This is how the Icinga web interface looks:

2

If you look around, you will notice that there are already a few service checks enabled for localhost (= server1.example.com):

2a

3

3 Configuring Icinga

server1.example.com:

The main Icinga configuration file is /etc/icinga/icinga.cfg, additional configurations are stored in /etc/icinga/commands.cfg and /etc/icinga/resource.cfg. Usually the default configuration is ok, so you don’t have to change these files.

The first thing you should change is the contact details in /etc/icinga/objects/contacts_icinga.cfg so that notifications are sent to the correct email address:

vi /etc/icinga/objects/contacts_icinga.cfg

[...]
define contact{
        contact_name                    root
        alias                           Falko Timme
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,r
        host_notification_options       d,r
        service_notification_commands   notify-service-by-email
        host_notification_commands      notify-host-by-email
        email                           me@myself.com
        }
[...]

The service checks for localhost (= server1.example.com) are defined in /etc/icinga/objects/localhost_icinga.cfg – take a look at that file:

cat /etc/icinga/objects/localhost_icinga.cfg

# A simple configuration file for monitoring the local host
# This can serve as an example for configuring other servers;
# Custom services specific to this host are added here, but services
# defined in icinga-common_services.cfg may also apply.
#

define host{
        use                     generic-host            ; Name of host template to use
        host_name               localhost
        alias                   localhost
        address                 127.0.0.1
        }

# Define a service to check the disk space of the root partition
# on the local machine.  Warning if < 20% free, critical if
# < 10% free space on partition.

define service{
        use                             generic-service         ; Name of service template to use
        host_name                       localhost
        service_description             Disk Space
        check_command                   check_all_disks!20%!10%
        }



# Define a service to check the number of currently logged in
# users on the local machine.  Warning if > 20 users, critical
# if > 50 users.

define service{
        use                             generic-service         ; Name of service template to use
        host_name                       localhost
        service_description             Current Users
        check_command                   check_users!20!50
        }


# Define a service to check the number of currently running procs
# on the local machine.  Warning if > 250 processes, critical if
# > 400 processes.

define service{
        use                             generic-service         ; Name of service template to use
        host_name                       localhost
        service_description             Total Processes
                check_command                   check_procs!250!400
        }



# Define a service to check the load on the local machine.

define service{
        use                             generic-service         ; Name of service template to use
        host_name                       localhost
        service_description             Current Load
                check_command                   check_load!5.0!4.0!3.0!10.0!6.0!4.0
        }

The check_command commands (like check_all_disks) are defined in the Nagios plugin configuration files in the /etc/nagios-plugins/config directory:

ls -l /etc/nagios-plugins/config

root@server1:~# ls -l /etc/nagios-plugins/config
total 144
-rw-r–r– 1 root root  277 2011-09-07 18:45 apt.cfg
-rw-r–r– 1 root root  182 2011-09-07 18:45 breeze.cfg
-rw-r–r– 1 root root  458 2011-09-07 18:45 dhcp.cfg
-rw-r–r– 1 root root  909 2011-09-07 18:45 disk.cfg
-rw-r–r– 1 root root 1722 2011-09-07 18:45 disk-smb.cfg
-rw-r–r– 1 root root  321 2011-09-07 18:45 dns.cfg
-rw-r–r– 1 root root  673 2011-09-07 18:45 dummy.cfg
-rw-r–r– 1 root root  146 2011-09-07 18:45 flexlm.cfg
-rw-r–r– 1 root root  159 2011-09-07 18:45 fping.cfg
-rw-r–r– 1 root root  414 2011-09-07 18:45 ftp.cfg
-rw-r–r– 1 root root  320 2011-09-07 18:45 games.cfg
-rw-r–r– 1 root root  157 2011-09-07 18:45 hppjd.cfg
-rw-r–r– 1 root root 3579 2011-09-07 18:45 http.cfg
-rw-r–r– 1 root root  818 2011-09-07 18:45 ifstatus.cfg
-rw-r–r– 1 root root  748 2011-09-07 18:45 ldap.cfg
-rw-r–r– 1 root root  195 2011-09-07 18:45 load.cfg
-rw-r–r– 1 root root 2062 2011-09-07 18:45 mail.cfg
-rw-r–r– 1 root root  708 2011-09-07 18:45 mailq.cfg
-rw-r–r– 1 root root  385 2011-09-07 18:45 mrtg.cfg
-rw-r–r– 1 root root  567 2011-09-07 18:45 mysql.cfg
-rw-r–r– 1 root root 2355 2011-09-07 18:45 netware.cfg
-rw-r–r– 1 root root  420 2011-09-07 18:45 news.cfg
-rw-r–r– 1 root root  497 2011-09-07 18:45 nt.cfg
-rw-r–r– 1 root root  466 2011-09-07 18:45 ntp.cfg
-rw-r–r– 1 root root  426 2011-09-07 18:45 pgsql.cfg
-rw-r–r– 1 root root 2026 2011-09-07 18:45 ping.cfg
-rw-r–r– 1 root root  511 2011-09-07 18:45 procs.cfg
-rw-r–r– 1 root root  240 2011-09-07 18:45 radius.cfg
-rw-r–r– 1 root root  397 2011-09-07 18:45 real.cfg
-rw-r–r– 1 root root  315 2011-09-07 18:45 rpc-nfs.cfg
-rw-r–r– 1 root root 5550 2011-09-07 18:45 snmp.cfg
-rw-r–r– 1 root root  753 2011-09-07 18:45 ssh.cfg
-rw-r–r– 1 root root  784 2011-09-07 18:45 tcp_udp.cfg
-rw-r–r– 1 root root  438 2011-09-07 18:45 telnet.cfg
-rw-r–r– 1 root root  155 2011-09-07 18:45 users.cfg
root@server1:~#

Let’s check out the /etc/nagios-plugins/config/disk.cfg file:

cat /etc/nagios-plugins/config/disk.cfg

# 'check_disk' command definition
define command{
        command_name    check_disk
        command_line    /usr/lib/nagios/plugins/check_disk -w '$ARG1$' -c '$ARG2$' -e -p '$ARG3$'
        }

# 'check_all_disks' command definition
define command{
        command_name    check_all_disks
        command_line    /usr/lib/nagios/plugins/check_disk -w '$ARG1$' -c '$ARG2$' -e
        }

# 'ssh_disk' command definition
define command{
        command_name    ssh_disk
        command_line    /usr/lib/nagios/plugins/check_by_ssh -H '$HOSTADDRESS$' -C "/usr/lib/nagios/plugins/check_disk -w '$ARG1$' -c '$ARG2$' -e -p '$ARG3$'"
        }

####
# use these checks, if you want to test IPv4 connectivity on IPv6 enabled systems
####

# 'ssh_disk_4' command definition
define command{
        command_name    ssh_disk_4
        command_line    /usr/lib/nagios/plugins/check_by_ssh -H '$HOSTADDRESS$' -C "/usr/lib/nagios/plugins/check_disk -w '$ARG1$' -c '$ARG2$' -e -p '$ARG3$'" -4
        }

As you see, the check_all_disks command is defined as /usr/lib/nagios/plugins/check_disk -w ‘$ARG1$’ -c ‘$ARG2$’ -e. If you take a look at the /etc/icinga/objects/localhost_icinga.cfg file again, you see that we have the line check_command check_all_disks!20%!10% in it. Icinga allows us to pass command line arguments to service checks by separating them with an exclamation mark (!), so check_command check_all_disks!20%!10% means we pass 20% as the first command line argument and 10% as the second command line argument to the /usr/lib/nagios/plugins/check_disk -w ‘$ARG1$’ -c ‘$ARG2$’ -e command so that it finally translates to /usr/lib/nagios/plugins/check_disk -w ‘20%’ -c ‘10%’ -e.

If you want to pass a command line argument that contains an exclamation mark, you must escape the exclamation mark with a backslash: \!

The Nagios plugins (i.e., the tools Icinga uses to run checks) are located in the /usr/lib/nagios/plugins directory:

ls -l /usr/lib/nagios/plugins

root@server1:~# ls -l /usr/lib/nagios/plugins
total 2456
-rwxr-xr-x 1 root root 108928 2011-09-07 18:46 check_apt
-rwxr-xr-x 1 root root   5369 2011-09-07 18:45 check_bgpstate
-rwxr-xr-x 1 root root   2242 2011-09-07 18:45 check_breeze
-rwxr-xr-x 1 root root  51976 2011-09-07 18:46 check_by_ssh
lrwxrwxrwx 1 root root      9 2011-09-07 18:46 check_clamd -> check_tcp
-rwxr-xr-x 1 root root  35136 2011-09-07 18:46 check_cluster
-rwxr-xr-x 1 root root  51552 2011-09-07 18:46 check_dhcp
-rwxr-xr-x 1 root root  47584 2011-09-07 18:46 check_dig
-rwxr-xr-x 1 root root 126016 2011-09-07 18:46 check_disk
-rwxr-xr-x 1 root root   8726 2011-09-07 18:45 check_disk_smb
-rwxr-xr-x 1 root root  47488 2011-09-07 18:46 check_dns
-rwxr-xr-x 1 root root  30704 2011-09-07 18:46 check_dummy
-rwxr-xr-x 1 root root   3053 2011-09-07 18:45 check_file_age
-rwxr-xr-x 1 root root   6315 2011-09-07 18:45 check_flexlm
lrwxrwxrwx 1 root root      9 2011-09-07 18:46 check_ftp -> check_tcp
lrwxrwxrwx 1 root root     10 2011-09-07 18:46 check_host -> check_icmp
-rwxr-xr-x 1 root root  47264 2011-09-07 18:46 check_hpjd
-rwxr-xr-x 1 root root 146656 2011-09-07 18:46 check_http
-rwxr-xr-x 1 root root  55328 2011-09-07 18:46 check_icmp
-rwxr-xr-x 1 root root  39360 2011-09-07 18:46 check_ide_smart
-rwxr-xr-x 1 root root  15134 2011-09-07 18:45 check_ifoperstatus
-rwxr-xr-x 1 root root  12598 2011-09-07 18:45 check_ifstatus
lrwxrwxrwx 1 root root      9 2011-09-07 18:46 check_imap -> check_tcp
-rwxr-xr-x 1 root root   6887 2011-09-07 18:45 check_ircd
lrwxrwxrwx 1 root root      9 2011-09-07 18:46 check_jabber -> check_tcp
-rwxr-xr-x 1 root root  43656 2011-09-07 18:46 check_ldap
lrwxrwxrwx 1 root root     10 2011-09-07 18:46 check_ldaps -> check_ldap
-rwxr-xr-x 1 root root   3407 2011-09-07 18:45 check_linux_raid
-rwxr-xr-x 1 root root  39104 2011-09-07 18:46 check_load
-rwxr-xr-x 1 root root   6026 2011-09-07 18:45 check_log
-rwxr-xr-x 1 root root  20284 2011-09-07 18:45 check_mailq
-rwxr-xr-x 1 root root  39296 2011-09-07 18:46 check_mrtg
-rwxr-xr-x 1 root root  39168 2011-09-07 18:46 check_mrtgtraf
-rwxr-xr-x 1 root root  47552 2011-09-07 18:46 check_mysql
-rwxr-xr-x 1 root root  47560 2011-09-07 18:46 check_mysql_query
-rwxr-xr-x 1 root root  39136 2011-09-07 18:46 check_nagios
lrwxrwxrwx 1 root root      9 2011-09-07 18:46 check_nntp -> check_tcp
lrwxrwxrwx 1 root root      9 2011-09-07 18:46 check_nntps -> check_tcp
-rwxr-xr-x 1 root root  51648 2011-09-07 18:46 check_nt
-rwxr-xr-x 1 root root  51616 2011-09-07 18:46 check_ntp
-rwxr-xr-x 1 root root  51880 2011-09-07 18:46 check_ntp_peer
-rwxr-xr-x 1 root root  47512 2011-09-07 18:46 check_ntp_time
-rwxr-xr-x 1 root root  63848 2011-09-07 18:46 check_nwstat
-rwxr-xr-x 1 root root   8326 2011-09-07 18:45 check_oracle
-rwxr-xr-x 1 root root  43336 2011-09-07 18:46 check_overcr
-rwxr-xr-x 1 root root  43552 2011-09-07 18:46 check_pgsql
-rwxr-xr-x 1 root root  51648 2011-09-07 18:46 check_ping
lrwxrwxrwx 1 root root      9 2011-09-07 18:46 check_pop -> check_tcp
-rwxr-xr-x 1 root root 117344 2011-09-07 18:46 check_procs
-rwxr-xr-x 1 root root  43496 2011-09-07 18:46 check_radius
-rwxr-xr-x 1 root root  43424 2011-09-07 18:46 check_real
-rwxr-xr-x 1 root root   9581 2011-09-07 18:45 check_rpc
lrwxrwxrwx 1 root root     10 2011-09-07 18:46 check_rta_multi -> check_icmp
-rwxr-xr-x 1 root root   1137 2011-09-07 18:45 check_sensors
lrwxrwxrwx 1 root root      9 2011-09-07 18:46 check_simap -> check_tcp
-rwxr-xr-x 1 root root 125632 2011-09-07 18:46 check_smtp
-rwxr-xr-x 1 root root 134272 2011-09-07 18:46 check_snmp
lrwxrwxrwx 1 root root      9 2011-09-07 18:46 check_spop -> check_tcp
-rwxr-xr-x 1 root root  39296 2011-09-07 18:46 check_ssh
lrwxrwxrwx 1 root root      9 2011-09-07 18:46 check_ssmtp -> check_tcp
-rwxr-xr-x 1 root root  43232 2011-09-07 18:46 check_swap
-rwxr-xr-x 1 root root  52064 2011-09-07 18:46 check_tcp
-rwxr-xr-x 1 root root  43392 2011-09-07 18:46 check_time
lrwxrwxrwx 1 root root      9 2011-09-07 18:46 check_udp -> check_tcp
-rwxr-xr-x 1 root root  47488 2011-09-07 18:46 check_ups
-rwxr-xr-x 1 root root  39072 2011-09-07 18:46 check_users
-rwxr-xr-x 1 root root   2936 2011-09-07 18:45 check_wave
-rwxr-xr-x 1 root root  43352 2011-09-07 18:46 negate
-rwxr-xr-x 1 root root  39032 2011-09-07 18:46 urlize
-rw-r–r– 1 root root   1938 2011-09-07 18:45 utils.pm
-rwxr-xr-x 1 root root    862 2011-09-07 18:45 utils.sh
root@server1:~#

To find out what command line arguments a plugin can take, call that plugin with the –help switch. For example, to find out how the check_disk plugin can be used, run

/usr/lib/nagios/plugins/check_disk –help

With this knowledge you can modify the service checks in /etc/icinga/objects/localhost_icinga.cfg to your likings, and you can add/modify plugin configurations in the /etc/nagios-plugins/config directory.

Now let’s assume we want to add a service check for MySQL, we first take a look at the appropriate plugin configuration:

cat /etc/nagios-plugins/config/mysql.cfg

# 'check_mysql' command definition
define command{
        command_name    check_mysql
        command_line    /usr/lib/nagios/plugins/check_mysql -H '$HOSTADDRESS$'
}

# 'check_mysql_cmdlinecred' command definition
define command{
        command_name    check_mysql_cmdlinecred
        command_line    /usr/lib/nagios/plugins/check_mysql -H '$HOSTADDRESS$' -u '$ARG1$' -p '$ARG2$'
}

# 'check_mysql_database' command definition
define command{
        command_name    check_mysql_database
        command_line    /usr/lib/nagios/plugins/check_mysql -d '$ARG3$' -H '$HOSTADDRESS$' -u '$ARG1$' -p '$ARG2$'
}

The command I want to use is check_mysql_cmdlinecred – this takes a MySQL username and a password as arguments (besides the host address which is taken from the host_name parameter of the service check definition. I want to use the MySQL user nagios with the password Kreationnext here, so I add the following section to /etc/icinga/objects/localhost_icinga.cfg:

vi /etc/icinga/objects/localhost_icinga.cfg

[...]
define service{
       use                             generic-service
       host_name                       localhost
       service_description             MySQL
       check_command                   check_mysql_cmdlinecred!nagios!Kreationnext
}

Before we restart Icinga, we must create the MySQL user nagios with the password Kreationnext:

mysql -u root -p

GRANT USAGE ON *.* TO nagios@localhost IDENTIFIED BY ‘Kreationnext‘;
GRANT USAGE ON *.* TO nagios@localhost.localdomain IDENTIFIED BY ‘Kreationnext‘;
FLUSH PRIVILEGES;

quit;

(The USAGE privilege is a synonym for ‘no privileges’, i.e., the nagios user can connect to MySQL, but not alter or read any data.)

Now we restart Icinga so that our changes take effect:

/etc/init.d/icinga restart

If you check localhost‘s services in the Icinga web interface now, you should see that a check for MySQL has been added:

4

Likewise, we can add checks for SMTP, POP3, and IMAP – these are just connection checks, so we don’t need any arguments:

vi /etc/icinga/objects/localhost_icinga.cfg

[...]
define service{
       use                             generic-service
       host_name                       localhost
       service_description             SMTP
       check_command                   check_smtp
}
define service{
       use                             generic-service
       host_name                       localhost
       service_description             POP3
       check_command                   check_pop
}
define service{
       use                             generic-service
       host_name                       localhost
       service_description             IMAP
       check_command                   check_imap
}

Restart Icinga…

/etc/init.d/icinga restart

… and a few moments later you should see the new checks in the Icinga web interface:

5

You might have noticed the SSH and HTTP checks for localhost which are not defined in /etc/icinga/objects/localhost_icinga.cfg. These are defined in hostgroups in the /etc/icinga/objects/hostgroups_icinga.cfg file. A hostgroup allows us to run a service check for multiple servers and define it only once. Take a look at that file:

cat /etc/icinga/objects/hostgroups_icinga.cfg

# Some generic hostgroup definitions

# A simple wildcard hostgroup
define hostgroup {
        hostgroup_name  all
                alias           All Servers
                members         *
        }

# A list of your Debian GNU/Linux servers
define hostgroup {
        hostgroup_name  debian-servers
                alias           Debian GNU/Linux Servers
                members         localhost
        }

# A list of your web servers
define hostgroup {
        hostgroup_name  http-servers
                alias           HTTP servers
                members         localhost
        }

# A list of your ssh-accessible servers
define hostgroup {
        hostgroup_name  ssh-servers
                alias           SSH servers
                members         localhost
        }

As you see, we have a hostgroup called http-servers and a hostgroup called ssh-servers, and localhost is a member of each of these groups. The service checks for the hostgroups are defined in /etc/icinga/objects/services_icinga.cfg. This file contains service checks and refers to the hostgroups to which these checks should be applied by using the hostgroup_name parameter:

cat /etc/icinga/objects/services_icinga.cfg

# check that web services are running
define service {
        hostgroup_name                  http-servers
        service_description             HTTP
        check_command                   check_http
        use                             generic-service
        notification_interval           0 ; set > 0 if you want to be renotified
}

# check that ssh services are running
define service {
        hostgroup_name                  ssh-servers
        service_description             SSH
        check_command                   check_ssh
        use                             generic-service
        notification_interval           0 ; set > 0 if you want to be renotified
}

As you see, the SSH and HTTP service checks are defined here.

4 Adding A Remote Server (server2) To Icinga

Monitoring localhost is nice, but of course, it would be even better if we could monitor all of our servers in one location. This is possible with Icinga, and this chapter describes how we can add our second Ubuntu 11.10 server (server2.example.com) to the setup.

To do this, we need to install the Nagios NRPE (Nagios Remote Plugin Executor) server on server2, and the Nagios NRPE plugin on server1. The NRPE server will listen on server2; server1 will connect to it using the NRPE plugin and pass commands to it that the NRPE server will execute on server2; it will pass back the results to server1.

First we install the nagios-nrpe-plugin package on server1:

server1:

apt-get install nagios-nrpe-plugin

Nagios web administration password: <– nagiosadmin_password
Password confirmation: <– nagiosadmin_password

Now we go to server2:

server2:

Install the nagios-nrpe-server package:

apt-get install nagios-nrpe-server

Now open /etc/nagios/nrpe.cfg:

vi /etc/nagios/nrpe.cfg

We must configure the NRPE server to allow server1 (IP: 192.168.0.100) to connect, therefore we add 192.168.0.100 to the allowed_hosts line:

[...]
# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address.  I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1,192.168.0.100
[...]

(If you don’t do this, you will get the following error when you run

/usr/lib/nagios/plugins/check_nrpe -H 192.168.0.101

on server1:

root@server1:/etc/nagios-plugins/config# /usr/lib/nagios/plugins/check_nrpe -H 192.168.0.101
CHECK_NRPE: Error – Could not complete SSL handshake.
root@server1:/etc/nagios-plugins/config#

)

Also, server1 needs to be allowed to pass command line arguments to the NRPE server, so still in the same file we set dont_blame_nrpe to 1:

[...]
# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed.  This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=1
[...]

(If you don’t do this, you will see the error

CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages.

for lots of remote service checks in the Icinga web interface, and in /var/log/syslog on server2 you will see these errors:

Aug 23 14:20:20 server2 nrpe[11496]: Error: Request contained command arguments, but argument option is not enabled!
Aug 23 14:20:20 server2 nrpe[11496]: Client request was invalid, bailing out…

)

Finally we must add command definitions for each service check we want to run on server2 and that is not already defined. I want to run the the check_procs, check_all_disks, and check_mysql_cmdlinecred checks on server2; these are not defined in /etc/nagios/nrpe.cfg, so I add them now (I also want to run the check_users and check_load checks, but these are already defined):

[...]
command[check_procs]=/usr/lib/nagios/plugins/check_procs -w 250 -c 400
command[check_all_disks]=/usr/lib/nagios/plugins/check_disk -w '20%' -c '10%' -e
command[check_mysql_cmdlinecred]=/usr/lib/nagios/plugins/check_mysql -H localhost -u 'nagios' -p 'Kreationnext'
[...]

(If you don’t do this, you will get errors like

NRPE: Command ‘check_all_disks’ not defined
NRPE: Command ‘check_mysql_cmdlinecred’ not defined
NRPE: Command ‘check_procs’ not defined

in the Icinga web interface.)

As you see I have hardcoded the command line arguments because using variables like command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ did not work for me. But still, when we configure the service checks for server2 on server1, we will have to pass command line arguments to these checks; server2 will ignore these because I have hardcoded the comand line arguments into /etc/nagios/nrpe.cfg, but if you leave them out, you will get errors like /usr/lib/nagios/plugins/check_nrpe: option requires an argument — ‘a’ in the Icinga web interface.

Now save the file and restart the NRPE server:

/etc/init.d/nagios-nrpe-server restart

Now check if the NRPE server is listening:

netstat -tap | grep nrpe

root@server2:~# netstat -tap | grep nrpe
tcp        0      0 *:nrpe                  *:*                     LISTEN       23668/nrpe
root@server2:~#

Now go back to server1

server1:

… and check if it can connect to the NRPE server on server2:

/usr/lib/nagios/plugins/check_nrpe -H 192.168.0.101

Output should be as follows in case of success:

root@server1:~# /usr/lib/nagios/plugins/check_nrpe -H 192.168.0.101
NRPE v2.12
root@server1:~#

Now go back to server2:

server2:

We want to check MySQL on server2; because we use the NRPE daemon, we can run the check locally on server2, i.e., we don’t have to open MySQL to the outside to allow server1 to run the check. Therefore I create the MySQL user nagios for localhost and localhost.localdomain instead of for 192.168.0.100 and server1.example.com:

mysql -u root -p

GRANT USAGE ON *.* TO nagios@localhost IDENTIFIED BY ‘Kreationnext‘;
GRANT USAGE ON *.* TO nagios@localhost.localdomain IDENTIFIED BY ‘Kreationnext‘;
FLUSH PRIVILEGES;

quit;

Now we go back to server1

server1:

… and create the Icinga configuration for server2:

vi /etc/icinga/objects/server2_icinga.cfg

define host{
       use generic-host
       host_name server2.example.com
       alias server2
       address 192.168.0.101
}
define service{
       use generic-service
       host_name server2.example.com
       service_description PING
       check_command check_ping!100.0,20%!500.0,60%
}
define service{
       use                             generic-service         ; Name of service template to use
       host_name                       server2.example.com
       service_description             Disk Space
       check_command                   check_nrpe!check_all_disks!20%!10%
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             Current Users
       check_command                   check_nrpe!check_users!20!50
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             Total Processes
       check_command                   check_nrpe!check_procs!250!400
}
define service{
        use                             generic-service         ; Name of service template to use
        host_name                       server2.example.com
        service_description             Current Load
        check_command                   check_nrpe!check_load!5.0!4.0!3.0!10.0!6.0!4.0
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             MySQL
       check_command                   check_nrpe!check_mysql_cmdlinecred!nagios!Kreationnext
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             SMTP
       check_command                   check_smtp
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             POP3
       check_command                   check_pop
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             IMAP
       check_command                   check_imap
}

(As I’ve mentioned before, although I have hardcoded the command line arguments for some commands into /etc/nagios/nrpe.cfg on server2, we still need to add command line arguments to certain these checks here.)

As you see, I use check_nrpe for some checks and pass the actual check (like check_all_disks) as a command line argument to check_nrpe. These are the checks that will be executed locally by the NRPE server on server2. check_nrpe is not needed for all checks. Checks that test a connection from the outside like check_ping or check_smtp can be run from server1.

To check the SSH and HTTP services on server2, we can EITHER add the following stanzas to /etc/icinga/objects/server2_icinga.cfg

[...]
define service {
        use                             generic-service
        host_name                       server2.example.com
        service_description             SSH
        check_command                   check_ssh
}
define service {
        use                             generic-service
        host_name                       server2.example.com
        service_description             HTTP
        check_command                   check_http
}

OR we add server2.example.com to the http-servers and ssh-servers hostgroups in /etc/icinga/objects/hostgroups_icinga.cfg:

vi /etc/icinga/objects/hostgroups_icinga.cfg

# Some generic hostgroup definitions

# A simple wildcard hostgroup
define hostgroup {
        hostgroup_name  all
                alias           All Servers
                members         *
        }

# A list of your Debian GNU/Linux servers
define hostgroup {
        hostgroup_name  debian-servers
                alias           Debian GNU/Linux Servers
                members         localhost,server2.example.com
        }

# A list of your web servers
define hostgroup {
        hostgroup_name  http-servers
                alias           HTTP servers
                members         localhost,server2.example.com
        }

# A list of your ssh-accessible servers
define hostgroup {
        hostgroup_name  ssh-servers
                alias           SSH servers
                members         localhost,server2.example.com
        }

Restart Icinga:

/etc/init.d/icinga restart

Afterwards you should find server2 in the Icinga web interface:

6

Here are the service checks for server2:

7

If you have added server2 to the hostgroups, it should be listed under Service Overview For All Host Groups as well:

8

  • Icinga: https://www.icinga.org/
  • Icinga Documentation: http://docs.icinga.org/latest/en/
  • Nagios: http://www.nagios.org/
  • Ubuntu: http://www.ubuntu.com/

 

Comments

comments