Cheap VPS & Xen Server

Residential Proxy Network - Hourly & Monthly Packages

Installing The Galera-Iworx Cluster

This document describes the installation of the Galera/Interworx cluster. This cluster provides the default Load Balancing that is available within Interworx, and additionally MySQL loadbalancing through MySQL Galera Clustering.



This guide assumes that you have installed your server with CentOS 6.4 (or later), and that you have double NIC’s with one External connection (192.168.120.x), and the other in a private VLAN (172.20.0.x).


Install the Atomic & EPEL repositories

Since we need some additional packages, we have to add some repositories to the server installation.

yum -y install wget wget rpm -Uvh epel-release-6-8.noarch.rpm wget -q -O – | sh

Add a line to the atomic repo (since we don’t want to use the atomic repo for mysql)

nano /etc/yum.repos.d/atomic.repo



Monitoring tools

In this part we will setup, the local monitoring tools

yum -y install nano git iftop ntop htop mytop lynx screen gcc mutt innotop iotop mtr man perl-DBD-MySQL


Other packages

Ok, now we are able to monitor the server we want to install the real software:

Webserver programs

yum -y install httpd clamav mysql mysql-server mysql-devel php-common php-dom php-pear php-soap php-pdo php-mysql php-devel php-gd php-ldap php-mbstring php-intl php-mcrypt phpmyadmin php-xmlrpc php-cli php-iconv php-ctype php-tokenizer aspell php-xcache xcache-admin

Common programs (especially needed if you use iscsi for Interworx later on)

yum -y install iscsi-initiator-utils lsscsi device-mapper-multipath dstat nfs-utils nfs-utils-lib


Set hostname

On all of the servers we will add the hostnames to the /etc/hosts file

echo master >> /etc/hosts
echo slave1 >> /etc/hosts
echo slave2 >> /etc/hosts
echo slave3 >> /etc/hosts
echo master >> /etc/hosts
echo slave1 >> /etc/hosts
echo slave2 >> /etc/hosts
echo slave3 >> /etc/hosts

If this is done, make sure that hostname and hostname -f returns the same value!!
and hostname -i   doesn’t give you or

hostname && hostname -f


SElinux & IPTables

Disable these services by running:

service iptables stop
setenforce 0

Edit the file /etc/sysconfig/selinux so it reads:



Configure ntpd

Since cluster services need the correct time we have to install the ntpd (timeserver deamon).

yum -y install ntp && chkconfig ntpd on

Let’s put in some timeserver as well:

nano /etc/ntp.conf



Create ssh-keys (on each server)

Because all the servers should be able to communicate with each other we have to create on each server a ssh-key.

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

Now add the created ssh public keys (/root/.ssh/ to all the servers in the file:

nano ~/.ssh/authorized_keys

If this step is completed try to logon to each server, and accept storage of the host key.

ssh master  Yes
ssh slave1 Yes
ssh slave2 Yes
ssh slave3  Yes

Now restart the server:

shutdown -r now

Configure MySQL

We are ready to go a step further and start with the configuration of MySQL.

First inititate MySQL for the first run:

mkdir /var/log/mysql
chown -R mysql:mysql /var/log/mysql

Now we can start mysql for the first time:

chkconfig mysqld on && service mysqld start

Create a MySQL restore set

To make it later on possible to install configure Interworx without messing up the Galera installation we have to copy some file to a secure location (on all nodes):

service mysqld stop
mkdir /tmp/interworx /tmp/interworx/etc/
cp /etc/my.cnf /tmp/interworx/etc/my.cnf
mkdir /tmp/interworx/etc/init.d
cp /etc/init.d/mysqld /tmp/interworx/etc/init.d/mysqld
mkdir /tmp/interworx/usr /tmp/interworx/usr/bin
cp /usr/bin/mysqld_safe /tmp/interworx/usr/bin/mysqld_safe
mkdir /tmp/interworx/usr/libexec
cp /usr/libexec/mysqld /tmp/interworx/usr/libexec/mysqld
mkdir /tmp/interworx/var/ /tmp/interworx/var/lib
cp -r /var/lib/mysql /tmp/interworx/var/lib



To configure the Galera Cluster we go to the website. We picked the following settings, but change according to your requirements:

Vendor:                      Codership (based on MySQL 5.5)
Infrastructure:              Other
Operating  System:           RHEL6 - Redhat 6.3/Fedora/Centos 6.3/OLN 6.3/Amazon AMI Platform:Linux  64-bit (x86_64)
Number of Galera  Servers:   3+1
MySQL PortNumber:            3333    (haproxy, interworx on 3306)
Galera PortNumber:           4567
Galera SST  PortNumber:      4444
SSH PortNumber:              22
OS User:                     root
MySQL Server  password:      pw4mydatabase
CMON DB password:            pw4myCMON
Firewall  (IPTables):        Disable
System Memory:               16 Gb
WAN:                         no
Skip DNS Resolve:            no
Database Size:               < 8GB
MySQL Usage:                 High write/High read
Number of cores:             16
Max connections per  server: 1500
Innodb_buffer_pool_size:     11319
Innodb_file_per_tabel: 	  yes
ClusterControl  Server:
Apache user:                 apache
WWWROOT:                     /var/www/html/
Config  directory:           /etc/
Server-ID 1:       
Datadir:                     /var/lib/mysql 
Server-ID 2:       
Server-ID  3:      
Email address:     

Click on Generate Deployment Scripts and retrieve them from your mailbox shortly after.

Install Galera Cluster

Now we are ready to install the Galera cluster software

cd /usr/local/src/
tar xvfz s9s-galera-2.2.0-rpm.tar.gz
cd s9s-galera-2.2.0-rpm/mysql/scripts/install

Before running the installer change the following:

echo “local_mysql_port=3333” >> ../../config/cmon.cnf.agent
echo “local_mysql_port=3333” >> ../../config/cmon.cnf.controller
echo “local_mysql_port=3333” >> ../../config/cmon_rrd.cnf

and add the bold line to the files ../../config/my.cnf and ../../config/my.cnf.cmon


Now run the installer:

./  2>&1  | tee cc.log

First the setup will try to ping all nodes. If ok, setup will start, since we disabled SELinux and Configured the SSH keys we can answer the following:

do you want to set SELinux to Permissive mode. n
Can you SSH from this host to all other hosts without password? y

After successfully configuring the Galera cluster, there should be a ClusterMonitor (cmon) page where you can connect to on the master node at:



Some pages require JPGraph, install this on the master:

cd /usr/local/src
mv download.php\?p\=5 jpgraph.tar.gz
tar xvfz jpgraph.tar.gz -C /var/www/html/cmon
cd /var/www/html/cmon
ln -s jpgraph-3.5.0b1 jpgraph
chown apache:apache -R jpgraph*


Change MySQL.cnf (configure UTF-8 etc.)

Since we want to use the UTF-8 character set in our database we add this option to the [MYSQLD] section of the file:

#Add some interworx  settings
# define default  character sets
   collation-server = utf8_unicode_ci
   init-connect='SET NAMES utf8'
   character-set-server  = utf8
   slow_query_log  = 1
   long_query_time  = 10
   myisam-sort-buffer-size  = 8M

Furthermore we change the environment variables to UTF8:

echo “LANG=en_US.utf-8” >> /etc/environment
echo “LC_ALL=en_US.utf-8” >> /etc/environment

Now we can restart MySQL to make this setting active for the databases that will be created in the future.

service mysql restart

Now we have to correct some installer files. Since mysql-libs is removed by the Galera setup. We have to use the –excludepath option to get around the following error:

file /usr/share/mysql/russian/errmsg.sys from install of MySQL-server-5.5.28_wsrep_23.7-1.rhel5.x86_64 conflicts with file from package mysql-libs-5.1.67-1.el6_3.x86_64

Download the mysql-libs rpm file and run it with the –excludepath option (which isn’t available when using yum ):

cd /usr/local/src
rpm -Uvh mysql-libs-5.1.67-1.el6_3.x86_64.rpm –excludepath=/usr/share/mysql/

Reinstall the packages that where removed because of the mysql-libs dependencies.

yum -y install nagios-plugins-all perl-DBD-MySQL innotop mytop


HAProxy setup

Next step is installing the HAProxy setup that uses cmon of the Galera cluster, to show its status information. At the master server we have to run the following commands:

# the installer needs to be extracted inside the galera installation folder!!!
cd /usr/local/src/s9s-galera-2.2.0-rpm/mysql/scripts/install
tar zxvf s9s-haproxy.tar.gz
cd haproxy

Before installing lets change some settings inside the following files:           HAPROXY_MYSQL_LISTEN_PORT="3306"                  "\tserver ${SPLIT[0]}  ${SPLIT[0]}:3333 check $STAT"           MYSQL_PORT="3333"     MYSQL_PORT="3333"

Now run the installer:

./ rhel galera

When the setup is completed without errors you have to grant the installer host rights to mysql instances on each server.

mysql -uroot -ppw4mydatabase
GRANT ALL ON *.* TO ‘root’@’’ IDENTIFIED BY ‘pw4mydatabase‘;

There is something what probably will go wrong tho, which is related to the defaults requiretty setting. It is possible that you get an error message like:

Pseudo-terminal will not be allocated because stdin is not a terminal.
sudo: sorry, you must have a tty to run sudo stdin is not a terminal.

If this happens edit the sudoers file (on all servers) by using:


and change the line

 Defaults    requirettys


#Defaults    requirettys

Now it should be possible to connect to the loadbalanced cluster on the following address:

mysql -h192.168.120.1 -P3306 -uroot -ppw4mydatabase

Check this from the slave nodes to make sure.

You can connect to the status page of the HAProxy,
(default admin/admin). You can change the username/password by editing the file:

nano /etc/haproxy/haproxy.cfg

group admin users admin
user admin insecure-password pw4adminHA
user stats insecure-password pw4userHA

Now we are ready to install Interworx.



Before installing stop haproxy & mysql & cmon on the nodes

service mysql stop && service cmon stop && service haproxy stop

Now we start moving around some file to restore the default MySQL environment (we have /var/lib/mysql on different disks, so if you dont, make a copy of the directories and restore these after installing interworx):

umount /var/lib/mysql
cp -r /tmp/interworx/var/lib/mysql /var/lib
chown -R mysql:mysql /var/lib/mysql
cp /tmp/interworx/etc/init.d/mysqld /etc/init.d/mysqld
chmod 755 /etc/init.d/mysqld
cp /tmp/interworx/usr/libexec/mysqld /usr/libexec/mysqld
chmod 755 /usr/libexec/mysqld
mkdir /var/run/mysqld
chown mysql:mysql /var/run/mysqld
mv /usr/bin/mysqld_safe /usr/bin/mysqld_safe.galera
cp /tmp/interworx/usr/bin/mysqld_safe /usr/bin/mysqld_safe
chmod 755 /usr/bin/mysqld_safe
mv /etc/my.cnf /etc/galera.cnf
cp /tmp/interworx/etc/my.cnf /etc/my.cnf
mv /etc/init.d/mysql /etc/init.d/

Edit the file

nano /etc/my.cnf


Since Interworx and Galera remove and install lots of packages, we have to take some precautions.  Default it isn’t possible to combine those to packages. The steps before (create a backup set of the default mysql) can resolve this problem.

We first have to download the installer file and modify it:

cd /usr/local/src && wget -q

Now we have to comment out line 321, to prevent Interworx to mess up the mysql install.


# mysqlinstall

Then we can save the file and start the installer.

sh ./

Press <enter> to begin the install…

After a while, you will get:

-=-=-=-=-= ALL DONE! THANK YOU! FOR USING InterWorx-CP =-=-=-=-=-

It’s good to check after installation the logs at  /home/interworx/var/log/error.log.
If ok, you’ll want to activate your InterWorx CP License.

Register now Interworx by using the website, since the command line doesn’t seem to work properly. Go to the website http://public IP Address:2080/nodeworx and enter the registration information:


After a while , you will probably get a blank page. To verify that the setup was completed you could test it with the command line method.


Scripted Activation Procedure

Run the goiworx.pex script with the following parameters (replace the bold text with the licensekeys below):

# Note: Make sure  the below is all one line. Replace LICENSE_KEY, EMAIL, and PASSWORD
 #~iworx/bin/goiworx.pex  --key=INTERWORX_YOURKEY --password=pw4yourIworx --ignorechecks

Now go to the website http://public IP Address/nodeworx and enter your login credentials. Accept the license agreement (some time you have to do this twice). If logged on you will be redirected to the page to configure your DNS servers:


Enter the requested primary/secondary DNS servers and click on update.  Click on System Services on the left side, followed by MySQL Server and Overview. Here we want to configure our Galera MySQL root password.




Click on the setup button underneath Interworx-CP Cluster Manager.


Choose the Quorum IP Address (HeartBeat LAN), and click setup. We now going to add the slave nodes.

Since we need a API-Key of each slave server we logon tot the slave servers, click on Nodeworx in the left menu, click on API-key and then on Generate on the right side panel.


After you click on generate you will see an API-key being generated.


Copy this key and go back to the master server.


Add the slave by using their IP-Address since the Heartbeat node isn’t separate available in the DNS.  Past the API-key, run test first, and then add the API-key. If all of this is working, and cmon also is green we are ready to go further. We start with rolling back the MySQL environment to the Galera environment. First stop on all servers the services!

service iworx stop && service mysqld stop

Now we rollback the Galera MySQL environment (on all servers). Again: we use separate disks for mysql, if you dont use the copies you made before.

mv /etc/my.cnf /etc/interworx.cnf
mv /etc/galera.cnf /etc/my.cnf
mv /usr/bin/mysqld_safe /usr/bin/mysqld_safe.interworx
mv /usr/bin/mysqld_safe.galera /usr/bin/mysqld_safe
rm -rf /var/run/mysqld
rm -rf /usr/libexec/mysqld
rm -rf /etc/init.d/mysqld
mv /var/lib/mysql /var/lib/mysql.iworx
mkdir /var/lib/mysql
chown -R mysql:mysql /var/lib/mysql/
mount -a
mv /etc/init.d/ /etc/init.d/mysql

Now start on all servers the services:

service mysql start && service cmon start && service iworx start && service haproxy start

Wait until cmon recovers again!

It is possible that MySQL want start the first time stop and start it until it works normal. There could be a lost pid file or subsys file, that you have to remove first. Anyway if it works normal we can remove some extra files

rm -rf /var/lib/mysql.iworx /usr/bin/mysqld_safe.interworx /tmp/interworx

If al servers are done, we have to insert the Iworx users into the Galera MSQL databases. On the master server and galera cluster we run the following mysql statements:
These values are available from Interworx if you install without the modified installer.

mysql --user=root  --password=pw4mydatabase
INSERT INTO  `mysql`.`user` VALUES ('localhost','iworx','[your iworx password in old password format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('','iworx','[your iworx master password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL);
GRANT ALL ON *.* TO  'iworx'@'' IDENTIFIED BY ‘pw4mydatabase’;

Wait until its up and running and in sync on the webpage http://master IP Address/cmon (use FireFox since this one updated the page better than IE).

Let’s start Iworx again:

service iworx start

Further we have to configure some status info inside the

nano /etc/httpd/conf/httpd.conf

ExtendedStatus On

<Location /server-status>
    SetHandler  server-status
    Order  deny,allow
    Deny from all
    Allow from
  <Location /server-info>
    SetHandler  server-info
    Order  deny,allow
    Deny from all
    Allow from

Configuring Interworx

Further we have to add the slave nodes to Interworx for loadbalancing:

Configure loadbalancer:









We have to reinstall packages that where uninstalled :

yum install -y phpMyAdmin xcache-admin mut


Reverse Proxy for HAProxy

We have to configure a (reverse)proxy inside the apache config so we can connect to the page like

echo “ProxyPass /haproxy http://localhost:9600” >> /etc/httpd/conf/httpd.conf
echo “ProxyPassReverse /haproxy http://localhost:9600” >> /etc/httpd/conf/httpd.conf

Restart the required services

service haproxy restart && service httpd restart


Upgrade to ClusterControl 1.2

Since ClusterControl 1.2 is released (7 march) , we should try to upgrade the current installation. Not only since there are some really nice features in it, but also to test if upgrading at all is possible!

When installing a new cluster from scratch you don’t have to do de upgrade steps since it will be 1.2 or higher right after install.


We start with the master node:

cd /usr/local/src
yum -y install php-curl
wget -O cmon-controller-1.2.0-1.x86_64.rpm
wget -O cmon-www-1.2.0-1.noarch.rpm
rpm -Uvh cmon-controller-1.2.0-1.x86_64.rpm
rpm -Uvh cmon-www-1.2.0-1.noarch.rpm
mysql -ucmon -ppw4myCMON < /usr/share/cmon/cmon_db_mods-1.1.36-1.2.0.sql
mysql -ucmon -ppw4myCMON < /usr/share/cmon/cmon_db.sql
mysql -ucmon -ppw4myCMON < /usr/share/cmon/cmon_data.sql
service cmon restart

Slave nodes

Then we repeat the following steps for all slave nodes

cd /usr/local/src
wget -O cmon-agent-1.2.0-1.x86_64.rpm
rpm -Uvh cmon-agent-1.2.0-1.x86_64.rpm
service cmon restart


ClusterControl 1.2 UI

To be sure you could check the master database, if the nodes are on release 1.2:

mysql -ucmon –ppw4myCMON
select hostname, cmon_version from cmon.hosts;

If ok, we go on with the GUI installer:

cd /usr/local/src
chmod +x

To finalize the installation open your web browser and go to this URL, to finalize the installation.

Now click on Install, and you will probably get an ClusterControl UI Installed message:


Click on “Click here to automatically register your cluster now! And you will be redirected to a Registerpage.


Now we are ready to login for the first time:


We copy and paste the URL, from the left top corner into the textbox on right bottom corner, and click register.

And there it is the new ClusterControl 1.2 UI.


Now you’re done! Enjoy your Interworx and MySQL Galera Cluster.