Cheap VPS & Xen Server


Residential Proxy Network - Hourly & Monthly Packages

Installing The Galera-Iworx Cluster


This document describes the installation of the Galera/Interworx cluster. This cluster provides the default Load Balancing that is available within Interworx, and additionally MySQL loadbalancing through MySQL Galera Clustering.

 

Pre-requisites

This guide assumes that you have installed your server with CentOS 6.4 (or later), and that you have double NIC’s with one External connection (192.168.120.x), and the other in a private VLAN (172.20.0.x).

 

Install the Atomic & EPEL repositories

Since we need some additional packages, we have to add some repositories to the server installation.

yum -y install wget wget http://ftp.nluug.nl/pub/os/Linux/distr/fedora-epel/6/i386/epel-release-6-8.noarch.rpm rpm -Uvh epel-release-6-8.noarch.rpm wget -q -O – http://www.atomicorp.com/installers/atomic | sh

Add a line to the atomic repo (since we don’t want to use the atomic repo for mysql)

nano /etc/yum.repos.d/atomic.repo

[atomic]
exclude=mysql*

 

Monitoring tools

In this part we will setup, the local monitoring tools

yum -y install nano git iftop ntop htop mytop lynx screen gcc mutt innotop iotop mtr man perl-DBD-MySQL

 

Other packages

Ok, now we are able to monitor the server we want to install the real software:

Webserver programs

yum -y install httpd clamav mysql mysql-server mysql-devel php-common php-dom php-pear php-soap php-pdo php-mysql php-devel php-gd php-ldap php-mbstring php-intl php-mcrypt phpmyadmin php-xmlrpc php-cli php-iconv php-ctype php-tokenizer aspell php-xcache xcache-admin

Common programs (especially needed if you use iscsi for Interworx later on)

yum -y install iscsi-initiator-utils lsscsi device-mapper-multipath dstat nfs-utils nfs-utils-lib

 

Set hostname

On all of the servers we will add the hostnames to the /etc/hosts file

echo 192.168.120.1 master.hosting.local master >> /etc/hosts
echo 192.168.120.2 slave1.hosting.local slave1 >> /etc/hosts
echo 192.168.120.3 slave2.hosting.local slave2 >> /etc/hosts
echo 192.168.120.4 slave3.hosting.local slave3 >> /etc/hosts
echo 172.20.0.1 master >> /etc/hosts
echo 172.20.0.2 slave1 >> /etc/hosts
echo 172.20.0.3 slave2 >> /etc/hosts
echo 172.20.0.4 slave3 >> /etc/hosts

If this is done, make sure that hostname and hostname -f returns the same value!!
and hostname -i   doesn’t give you 127.0.0.1 or 127.0.1.1.

hostname && hostname -f

 

SElinux & IPTables

Disable these services by running:

service iptables stop
setenforce 0

Edit the file /etc/sysconfig/selinux so it reads:

SELINUX=disabled

 

Configure ntpd

Since cluster services need the correct time we have to install the ntpd (timeserver deamon).

yum -y install ntp && chkconfig ntpd on

Let’s put in some timeserver as well:

nano /etc/ntp.conf

server pool.ntp.org

 

Create ssh-keys (on each server)

Because all the servers should be able to communicate with each other we have to create on each server a ssh-key.

ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

Now add the created ssh public keys (/root/.ssh/id_rsa.pub) to all the servers in the file:

nano ~/.ssh/authorized_keys

If this step is completed try to logon to each server, and accept storage of the host key.

ssh master  Yes
exit
ssh slave1 Yes
exit
ssh slave2 Yes
exit
ssh slave3  Yes
exit

Now restart the server:

shutdown -r now

Configure MySQL

We are ready to go a step further and start with the configuration of MySQL.

First inititate MySQL for the first run:

mkdir /var/log/mysql
chown -R mysql:mysql /var/log/mysql

Now we can start mysql for the first time:

chkconfig mysqld on && service mysqld start

Create a MySQL restore set

To make it later on possible to install configure Interworx without messing up the Galera installation we have to copy some file to a secure location (on all nodes):

service mysqld stop
mkdir /tmp/interworx /tmp/interworx/etc/
cp /etc/my.cnf /tmp/interworx/etc/my.cnf
mkdir /tmp/interworx/etc/init.d
cp /etc/init.d/mysqld /tmp/interworx/etc/init.d/mysqld
mkdir /tmp/interworx/usr /tmp/interworx/usr/bin
cp /usr/bin/mysqld_safe /tmp/interworx/usr/bin/mysqld_safe
mkdir /tmp/interworx/usr/libexec
cp /usr/libexec/mysqld /tmp/interworx/usr/libexec/mysqld
mkdir /tmp/interworx/var/ /tmp/interworx/var/lib
cp -r /var/lib/mysql /tmp/interworx/var/lib

 

Galera-Configuration

To configure the Galera Cluster we go to the http://www.severalnines.com/galera-configurator/ website. We picked the following settings, but change according to your requirements:

Vendor:                      Codership (based on MySQL 5.5)
Infrastructure:              Other
Operating  System:           RHEL6 - Redhat 6.3/Fedora/Centos 6.3/OLN 6.3/Amazon AMI Platform:Linux  64-bit (x86_64)
Number of Galera  Servers:   3+1
MySQL PortNumber:            3333    (haproxy, interworx on 3306)
Galera PortNumber:           4567
Galera SST  PortNumber:      4444
SSH PortNumber:              22
OS User:                     root
MySQL Server  password:      pw4mydatabase
CMON DB password:            pw4myCMON
Firewall  (IPTables):        Disable
System Memory:               16 Gb
WAN:                         no
Skip DNS Resolve:            no
Database Size:               < 8GB
MySQL Usage:                 High write/High read
Number of cores:             16
Max connections per  server: 1500
Innodb_buffer_pool_size:     11319
Innodb_file_per_tabel: 	  yes
ClusterControl  Server:      172.20.0.1
Apache user:                 apache
WWWROOT:                     /var/www/html/
Config  directory:           /etc/
Server-ID 1:                 172.20.0.2
Datadir:                     /var/lib/mysql 
Server-ID 2:                 172.20.0.3
Server-ID  3:                172.20.0.4
Email address:               your@email.com

Click on Generate Deployment Scripts and retrieve them from your mailbox shortly after.

Install Galera Cluster

Now we are ready to install the Galera cluster software

cd /usr/local/src/
wget http://www.severalnines.com/galera-configurator/tmp/bhhrgonbsft2e3o2vhlt0ip9v0/s9s-galera-2.2.0-rpm.tar.gz
tar xvfz s9s-galera-2.2.0-rpm.tar.gz
cd s9s-galera-2.2.0-rpm/mysql/scripts/install

Before running the installer change the following:

echo “local_mysql_port=3333” >> ../../config/cmon.cnf.agent
echo “local_mysql_port=3333” >> ../../config/cmon.cnf.controller
echo “local_mysql_port=3333” >> ../../config/cmon_rrd.cnf

and add the bold line to the files ../../config/my.cnf and ../../config/my.cnf.cmon

[MYSQLD]
old-passwords=1

Now run the installer:

./deploy.sh  2>&1  | tee cc.log

First the setup will try to ping all nodes. If ok, setup will start, since we disabled SELinux and Configured the SSH keys we can answer the following:

do you want to set SELinux to Permissive mode. n
Can you SSH from this host to all other hosts without password? y

After successfully configuring the Galera cluster, there should be a ClusterMonitor (cmon) page where you can connect to on the master node at:

http://192.168.120.1/cmon

 

JPGraph

Some pages require JPGraph, install this on the master:

cd /usr/local/src
wget http://jpgraph.net/download/download.php?p=5
mv download.php\?p\=5 jpgraph.tar.gz
tar xvfz jpgraph.tar.gz -C /var/www/html/cmon
cd /var/www/html/cmon
ln -s jpgraph-3.5.0b1 jpgraph
chown apache:apache -R jpgraph*

 

Change MySQL.cnf (configure UTF-8 etc.)

Since we want to use the UTF-8 character set in our database we add this option to the [MYSQLD] section of the file:

#Add some interworx  settings
 #symbolic-links=0
# define default  character sets
   collation-server = utf8_unicode_ci
   init-connect='SET NAMES utf8'
   character-set-server  = utf8
#log
   slow_query_log  = 1
   slow-query-log-file=slow-queries.log
   long_query_time  = 10
#OTHER THINGS,  BUFFERS ETC
   myisam-sort-buffer-size  = 8M
   skip-name-resolve
   memlock=0
   event_scheduler=1

Furthermore we change the environment variables to UTF8:

echo “LANG=en_US.utf-8” >> /etc/environment
echo “LC_ALL=en_US.utf-8” >> /etc/environment

Now we can restart MySQL to make this setting active for the databases that will be created in the future.

service mysql restart

Now we have to correct some installer files. Since mysql-libs is removed by the Galera setup. We have to use the –excludepath option to get around the following error:

file /usr/share/mysql/russian/errmsg.sys from install of MySQL-server-5.5.28_wsrep_23.7-1.rhel5.x86_64 conflicts with file from package mysql-libs-5.1.67-1.el6_3.x86_64

Download the mysql-libs rpm file and run it with the –excludepath option (which isn’t available when using yum ):

cd /usr/local/src
wget ftp://mirror.switch.ch/pool/1/mirror/scientificlinux/6rolling/x86_64/os/Packages/mysql-libs-5.1.67-1.el6_3.x86_64.rpm
rpm -Uvh mysql-libs-5.1.67-1.el6_3.x86_64.rpm –excludepath=/usr/share/mysql/

Reinstall the packages that where removed because of the mysql-libs dependencies.

yum -y install nagios-plugins-all perl-DBD-MySQL innotop mytop

 

HAProxy setup

Next step is installing the HAProxy setup that uses cmon of the Galera cluster, to show its status information. At the master server we have to run the following commands:

# the installer needs to be extracted inside the galera installation folder!!!
cd /usr/local/src/s9s-galera-2.2.0-rpm/mysql/scripts/install
wget http://severalnines.com/downloads/s9s-haproxy.tar.gz
tar zxvf s9s-haproxy.tar.gz
cd haproxy

Before installing lets change some settings inside the following files:

install-haproxy.sh:           HAPROXY_MYSQL_LISTEN_PORT="3306"
makecfg.sh:                  "\tserver ${SPLIT[0]}  ${SPLIT[0]}:3333 check $STAT"
mysqlchk.sh.galera           MYSQL_PORT="3333"
mysqlchk.sh.mysqlcluster     MYSQL_PORT="3333"

Now run the installer:

./install-haproxy.sh 172.20.0.1 rhel galera

When the setup is completed without errors you have to grant the installer host rights to mysql instances on each server.

mysql -uroot -ppw4mydatabase
GRANT ALL ON *.* TO ‘root’@’172.20.0.1’ IDENTIFIED BY ‘pw4mydatabase‘;
FLUSH PRIVILEGES;
exit

There is something what probably will go wrong tho, which is related to the defaults requiretty setting. It is possible that you get an error message like:

Pseudo-terminal will not be allocated because stdin is not a terminal.
sudo: sorry, you must have a tty to run sudo stdin is not a terminal.

If this happens edit the sudoers file (on all servers) by using:

visudo

and change the line

 Defaults    requirettys

into

#Defaults    requirettys

Now it should be possible to connect to the loadbalanced cluster on the following address:

mysql -h192.168.120.1 -P3306 -uroot -ppw4mydatabase

Check this from the slave nodes to make sure.

You can connect to the status page of the HAProxy, http://192.168.120.1:9600
(default admin/admin). You can change the username/password by editing the file:

nano /etc/haproxy/haproxy.cfg

userlist STATSUSERS
group admin users admin
user admin insecure-password pw4adminHA
user stats insecure-password pw4userHA

Now we are ready to install Interworx.

 

InterWorx-ControlPanel

Before installing stop haproxy & mysql & cmon on the nodes

service mysql stop && service cmon stop && service haproxy stop

Now we start moving around some file to restore the default MySQL environment (we have /var/lib/mysql on different disks, so if you dont, make a copy of the directories and restore these after installing interworx):

umount /var/lib/mysql
cp -r /tmp/interworx/var/lib/mysql /var/lib
chown -R mysql:mysql /var/lib/mysql
cp /tmp/interworx/etc/init.d/mysqld /etc/init.d/mysqld
chmod 755 /etc/init.d/mysqld
cp /tmp/interworx/usr/libexec/mysqld /usr/libexec/mysqld
chmod 755 /usr/libexec/mysqld
mkdir /var/run/mysqld
chown mysql:mysql /var/run/mysqld
mv /usr/bin/mysqld_safe /usr/bin/mysqld_safe.galera
cp /tmp/interworx/usr/bin/mysqld_safe /usr/bin/mysqld_safe
chmod 755 /usr/bin/mysqld_safe
mv /etc/my.cnf /etc/galera.cnf
cp /tmp/interworx/etc/my.cnf /etc/my.cnf
mv /etc/init.d/mysql /etc/init.d/mysql.org

Edit the file

nano /etc/my.cnf

[mysqld]
old-passwords=1

Since Interworx and Galera remove and install lots of packages, we have to take some precautions.  Default it isn’t possible to combine those to packages. The steps before (create a backup set of the default mysql) can resolve this problem.

We first have to download the installer file and modify it:

cd /usr/local/src && wget -q http://updates.interworx.com/iworx/scripts/iworx-cp-install.sh

Now we have to comment out line 321, to prevent Interworx to mess up the mysql install.

nano iworx-cp-install.sh

# mysqlinstall

Then we can save the file and start the installer.

sh ./iworx-cp-install.sh

Press <enter> to begin the install…

After a while, you will get:

-=-=-=-=-= ALL DONE! THANK YOU! FOR USING InterWorx-CP =-=-=-=-=-

It’s good to check after installation the logs at  /home/interworx/var/log/error.log.
If ok, you’ll want to activate your InterWorx CP License.

Register now Interworx by using the website, since the command line doesn’t seem to work properly. Go to the website http://public IP Address:2080/nodeworx and enter the registration information:

image1

After a while , you will probably get a blank page. To verify that the setup was completed you could test it with the command line method.

 

Scripted Activation Procedure

Run the goiworx.pex script with the following parameters (replace the bold text with the licensekeys below):

# Note: Make sure  the below is all one line. Replace LICENSE_KEY, EMAIL, and PASSWORD
 #~iworx/bin/goiworx.pex  --key=INTERWORX_YOURKEY --email=your@email.com --password=pw4yourIworx --ignorechecks

Now go to the website http://public IP Address/nodeworx and enter your login credentials. Accept the license agreement (some time you have to do this twice). If logged on you will be redirected to the page to configure your DNS servers:

image2

Enter the requested primary/secondary DNS servers and click on update.  Click on System Services on the left side, followed by MySQL Server and Overview. Here we want to configure our Galera MySQL root password.

image3

image4

 

Click on the setup button underneath Interworx-CP Cluster Manager.

image5

Choose the Quorum IP Address (HeartBeat LAN), and click setup. We now going to add the slave nodes.

Since we need a API-Key of each slave server we logon tot the slave servers, click on Nodeworx in the left menu, click on API-key and then on Generate on the right side panel.

image6

After you click on generate you will see an API-key being generated.

image7

Copy this key and go back to the master server.

image8

Add the slave by using their IP-Address since the Heartbeat node isn’t separate available in the DNS.  Past the API-key, run test first, and then add the API-key. If all of this is working, and cmon also is green we are ready to go further. We start with rolling back the MySQL environment to the Galera environment. First stop on all servers the services!

service iworx stop && service mysqld stop

Now we rollback the Galera MySQL environment (on all servers). Again: we use separate disks for mysql, if you dont use the copies you made before.

mv /etc/my.cnf /etc/interworx.cnf
mv /etc/galera.cnf /etc/my.cnf
mv /usr/bin/mysqld_safe /usr/bin/mysqld_safe.interworx
mv /usr/bin/mysqld_safe.galera /usr/bin/mysqld_safe
rm -rf /var/run/mysqld
rm -rf /usr/libexec/mysqld
rm -rf /etc/init.d/mysqld
mv /var/lib/mysql /var/lib/mysql.iworx
mkdir /var/lib/mysql
chown -R mysql:mysql /var/lib/mysql/
mount -a
mv /etc/init.d/mysql.org /etc/init.d/mysql

Now start on all servers the services:

service mysql start && service cmon start && service iworx start && service haproxy start

Wait until cmon recovers again!

It is possible that MySQL want start the first time stop and start it until it works normal. There could be a lost pid file or subsys file, that you have to remove first. Anyway if it works normal we can remove some extra files

rm -rf /var/lib/mysql.iworx /usr/bin/mysqld_safe.interworx /tmp/interworx

If al servers are done, we have to insert the Iworx users into the Galera MSQL databases. On the master server and galera cluster we run the following mysql statements:
These values are available from Interworx if you install without the modified installer.

mysql --user=root  --password=pw4mydatabase
INSERT INTO  `mysql`.`user` VALUES ('localhost','iworx','[your iworx password in old password format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('127.0.0.1','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('172.20.0.2','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('172.20.0.3','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('172.20.0.4','iworx','[your iworx password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL),
('172.20.0.1','iworx','[your iworx master password in old password  format]','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'',NULL);
GRANT ALL ON *.* TO  'iworx'@'172.20.0.1' IDENTIFIED BY ‘pw4mydatabase’;
FLUSH PRIVILEGES;
exit

Wait until its up and running and in sync on the webpage http://master IP Address/cmon (use FireFox since this one updated the page better than IE).

Let’s start Iworx again:

service iworx start

Further we have to configure some status info inside the

nano /etc/httpd/conf/httpd.conf

ExtendedStatus On

<Location /server-status>
    SetHandler  server-status
    Order  deny,allow
    Deny from all
    Allow from  127.0.0.1
  </Location>
  <Location /server-info>
    SetHandler  server-info
    Order  deny,allow
    Deny from all
    Allow from  127.0.0.1
  </Location>

Configuring Interworx

Further we have to add the slave nodes to Interworx for loadbalancing:

Configure loadbalancer:

image9

image10

image11

image12

image13

image14

image15

image16

We have to reinstall packages that where uninstalled :

yum install -y phpMyAdmin xcache-admin mut

 

Reverse Proxy for HAProxy

We have to configure a (reverse)proxy inside the apache config so we can connect to the page like http://192.168.120.1/haproxy

echo “ProxyPass /haproxy http://localhost:9600” >> /etc/httpd/conf/httpd.conf
echo “ProxyPassReverse /haproxy http://localhost:9600” >> /etc/httpd/conf/httpd.conf

Restart the required services

service haproxy restart && service httpd restart

 

Upgrade to ClusterControl 1.2

Since ClusterControl 1.2 is released (7 march) , we should try to upgrade the current installation. Not only since there are some really nice features in it, but also to test if upgrading at all is possible!

When installing a new cluster from scratch you don’t have to do de upgrade steps since it will be 1.2 or higher right after install.

Master

We start with the master node:

cd /usr/local/src
yum -y install php-curl
wget -O cmon-controller-1.2.0-1.x86_64.rpm http://www.severalnines.com/downloads/cmon/cmon-controller-1.2.0-1.x86_64.rpm
wget -O cmon-www-1.2.0-1.noarch.rpm http://www.severalnines.com/downloads/cmon/cmon-www-1.2.0-1.noarch.rpm
rpm -Uvh cmon-controller-1.2.0-1.x86_64.rpm
rpm -Uvh cmon-www-1.2.0-1.noarch.rpm
mysql -ucmon -ppw4myCMON < /usr/share/cmon/cmon_db_mods-1.1.36-1.2.0.sql
mysql -ucmon -ppw4myCMON < /usr/share/cmon/cmon_db.sql
mysql -ucmon -ppw4myCMON < /usr/share/cmon/cmon_data.sql
service cmon restart

Slave nodes

Then we repeat the following steps for all slave nodes

cd /usr/local/src
wget -O cmon-agent-1.2.0-1.x86_64.rpm http://www.severalnines.com/downloads/cmon/cmon-agent-1.2.0-1.x86_64.rpm
rpm -Uvh cmon-agent-1.2.0-1.x86_64.rpm
service cmon restart

 

ClusterControl 1.2 UI

To be sure you could check the master database, if the nodes are on release 1.2:

mysql -ucmon –ppw4myCMON
select hostname, cmon_version from cmon.hosts;
exit

If ok, we go on with the GUI installer:

cd /usr/local/src
wget http://severalnines.com/downloads/cmon/setup-cc-ui.sh
chmod +x setup-cc-ui.sh
./setup-cc-ui.sh

To finalize the installation open your web browser and go to this URL, http://192.168.120.1/install to finalize the installation.

Now click on Install, and you will probably get an ClusterControl UI Installed message:

image17

Click on “Click here to automatically register your cluster now! And you will be redirected to a Registerpage.

image18

Now we are ready to login for the first time:

image19

We copy and paste the URL, from the left top corner into the textbox on right bottom corner, and click register.

And there it is the new ClusterControl 1.2 UI.

image20

Now you’re done! Enjoy your Interworx and MySQL Galera Cluster.

 

 

Comments

comments