Cheap VPS & Xen Server


Residential Proxy Network - Hourly & Monthly Packages

Setting Up An Active/Active Samba CTDB Cluster Using GFS & DRBD (CentOS 5.5)


This article explains how to set up an Active/Active Samba CTDB Cluster, using GFS and DRBD. Prepared by Rafael Marangoni, from BRLink Servidor Linux Team.

 

Introduction

We use two nodes, both are active nodes, and the requests are loadbalanced. To replicate data between the nodes we use DRBD. To active/active cluster, we must use a ClusterFS (in this case GFS), to make the two nodes to write on the DRBD resource at same time.

 

1 Preliminary Note

Linux Distribution:

We are using the CentOS 5.5 (64bits) distribution, but will probably work on Fedora (and Red Hat, for sure). The installation of the CentOS is very simple and classical, select the base packages and other stuff that you like/need. One issue that must be remembered is that we use DRBD to replicate data between the nodes, then you’ll need to have a disk or partition exclusive to DRBD. Remember this before partitioning disks on CentOS installation.

Network Hardware/Topology:

We use two Gigabit NIC’s per node, one (eth0) connect to the network (LAN), and the other one (eth1) with a cross-over cable connecting both nodes.
The cross-over cable must be used to improve performance and confiability of the system, because DRBD won’t depends of network switchs or anything else to replicate data between the nodes.

In this tutorial we will use the physical nodes node1.clusterbr.int and node2.clusterbr.int:
node1.clustersmb.int: Uses IP 10.0.0.181 (LAN) and IP 172.16.0.1 (cross-over)
node2.clustersmb.int: Uses IP 10.0.0.182 (LAN) and IP 172.16.0.2 (cross-over)
vip1.clustersmb.int: It’s the Cluster IP, from node1 10.0.0.183.
vip2.clustersmb.int: It’s the Cluster IP, from node2 10.0.0.184.

Disks:

Both the nodes have two disks:
/dev/sda: to system OS;
/dev/sdb: to DRBD.
As I said before, you can use only one disk, if leaving one partition exclusive to DRBD.

CTDB:

It’s a cluster implementation of TDB, to use with Samba on cluster filesystems. We need it to have active/active cluster, then both the smb services can answers the network requests. More information on http://ctdb.samba.org.

 

2 Preparing The Nodes

Disabling SELINUX

We need to disable the SELINUX:

vi /etc/selinux/config

Change this line only (leaving everything else untouched):

SELINUX=disabled

 

Setting Hostname

We need to change the hostname and gateway of the nodes:

vi /etc/sysconfig/network

node1:

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node1.clustersmb.int
GATEWAY=10.0.0.9

node2:

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node2.clustersmb.int
GATEWAY=10.0.0.9

 

Configuring Network Interfaces

Next, we will configure the network interfaces:

node1:

The LAN interface:

vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=static
IPADDR=10.0.0.181
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=a6:1e:3d:67:66:78

The Cross-Over/DRBD interface:

vi /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
BOOTPROTO=static
IPADDR=172.16.0.1
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=ee:ef:ff:9a:9a:57

node2:

The LAN interface:

vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=static
IPADDR=10.0.0.182
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=52:52:a1:1a:62:32

The Cross-Over/DRBD interface:

vi /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
BOOTPROTO=static
IPADDR=172.16.0.2
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=1a:18:b2:50:96:1e

 

Setting DNS Configuration

Setting DNS configuration on both nodes (according to your network):

vi /etc/resolv.conf

search clustersmb.int
nameserver 10.0.0.9

 

Configuring Basic Hostname Resolution

Configuring /etc/hosts (same config on both nodes):

vi /etc/hosts

127.0.0.1               localhost.localdomain localhost
10.0.0.191              node1.clustersmb.int   node1
10.0.0.192              node2.clustersmb.int   node2

PS: You’ll probably want to set another lines on this file, to point to other addresses of your network.

 

Checking Network Connectivity

Let’s check if everything is fine:

node1:

Pinging node2 (thru LAN interface):

[root@node1 ~]# ping -c 2 node2

[root@node1 ~]# ping -c 2 node2
PING node2 (10.0.0.182) 56(84) bytes of data.
64 bytes from node2 (10.0.0.182): icmp_seq=1 ttl=64 time=0.089 ms
64 bytes from node2 (10.0.0.182): icmp_seq=2 ttl=64 time=0.082 ms
— node2 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.082/0.085/0.089/0.009 ms

Pinging node2 (thru cross-over interface):

[root@node1 ~]# ping -c 2 172.16.0.2

[root@node1 ~]# ping -c 2 172.16.0.2
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=0.083 ms
64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=0.083 ms
— 172.16.0.2 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.083/0.083/0.083/0.000 ms

node2:

Pinging node1 (thru LAN interface):

[root@node2 ~]# ping -c 2 node1

[root@node2 ~]# ping -c 2 node1
PING node1 (10.0.0.181) 56(84) bytes of data.
64 bytes from node1 (10.0.0.181): icmp_seq=1 ttl=64 time=0.068 ms
64 bytes from node1 (10.0.0.181): icmp_seq=2 ttl=64 time=0.063 ms
— node1 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.063/0.065/0.068/0.008 ms

Pinging node1 (thru cross-over interface):

[root@node2 ~]# ping -c 2 172.16.0.1

[root@node2 ~]# ping -c 2 172.16.0.1
PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
64 bytes from 172.16.0.1: icmp_seq=1 ttl=64 time=1.36 ms
64 bytes from 172.16.0.1: icmp_seq=2 ttl=64 time=0.075 ms
— 172.16.0.1 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.075/0.722/1.369/0.647 ms

 

Configuring Initialization Options

I like to set runlevel to 3.

vi /etc/inittab

Change this line only (leaving everything else untouched):

id:3:initdefault:

I like remove some services from automatic initialization, to maintain only services that really will be used.

These are the active services that we’ll need:

[root@node1 ~]# chkconfig –list | grep 3:sim

[root@node1 ~]# chkconfig –list | grep 3:sim
acpid           0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
anacron         0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
apmd            0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
atd             0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
cpuspeed        0:não  1:sim   2:sim   3:sim   4:sim   5:sim   6:não
crond           0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
irqbalance      0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
kudzu           0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
network         0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
rawdevices      0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
sshd            0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
syslog          0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não

At this point, we need to reboot both nodes to apply configuration.

reboot

This article explains how to set up an Active/Active Samba CTDB Cluster, using GFS and DRBD. Prepared by Rafael Marangoni, from BRLink Servidor Linux Team.

3. Installing Prerequisites And Cluster Packages

There are some packages that need to be installed before:

yum -y install drbd82 kmod-drbd82 samba

Let’s install Red Hat Cluster Suite:

yum -y groupinstall “Cluster Storage” “Clustering”

4. Configuring DRBD

First, we need to configure /etc/drbd.conf on both nodes:

vi /etc/drbd.conf

global {
    usage-count yes;
}

common {
  syncer {
                rate 100M;
                al-extents 257;
         }
}

resource r0 {

  protocol C;

  startup {
    become-primary-on both;              ### For Primary/Primary ###
    degr-wfc-timeout 60;
    wfc-timeout  30;
  }

  disk {
    on-io-error   detach;
  }

  net {
    allow-two-primaries;                 ### For Primary/Primary ###
    cram-hmac-alg sha1;
    shared-secret "mysecret";
    after-sb-0pri discard-zero-changes;
    after-sb-1pri violently-as0p;
    after-sb-2pri violently-as0p;
  }

  on node1.clustersmb.int {
    device     /dev/drbd0;
    disk       /dev/sdb;
    address    172.16.0.1:7788;
    meta-disk  internal;
  }

  on node2.clustersmb.int {
    device     /dev/drbd0;
    disk       /dev/sdb;
    address    172.16.0.1:7788;
    meta-disk  internal;
  }
}

The main points of configuration are:
resource: refers to the resource that will be manage by DRBD, note that we called “r0”
disk: refers to the device that DRBD will use (a disk or partition)
address: IP Address/port that DRBD will use (note that we points to cross-over interfaces)
syncer: the rate transfer between the nodes (we use 100M because we have Gigabit cards)
If you have doubts, please look at DRBD Users Guide: www.drbd.org/users-guide-emb/

Afterwards of that configuration, we can create the metadata on r0 resource. On both nodes do:

drbdadm create-md r0

Next, need to initiate the DRBD services, to create the initdb. On both nodes (at almost the same time), do:

/etc/init.d/drbd start

To put the two nodes as primary, on both nodes do:

drbdsetup /dev/drbd0 primary -o

To check the progress of the sync, and status of DRBD resource, look at /proc/drbd:

cat /proc/drbd

Now, we need to wait the end of the sync. This may take a long time, depends the size and performance of your disks. And of course, the speed of cluster network interfaces that is being used with the cross-over cable).

When the sync process ends, we can take a look at the status of the resource r0:

node1:

[root@node1 ~]# cat /proc/drbd

[root@node1 ~]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
0: cs:Connected st:Primary/Primary ds:UpToDate/UpToDate C r—
ns:2097052 nr:0 dw:0 dr:2097052 al:0 bm:128 lo:0 pe:0 ua:0 ap:0 oos:0

node2:

[root@node2 ~]# cat /proc/drbd

[root@node2 ~]# cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by buildsvn@c5-x8664-build, 2008-10-03 11:30:17
0: cs:Connected st:Primary/Primary ds:UpToDate/UpToDate C r—
ns:0 nr:2097052 dw:2097052 dr:0 al:0 bm:128 lo:0 pe:0 ua:0 ap:0 oos:0

It’s important to note that both servers are up to date (UpToDate/UpToDate) and primary (Primary/Primary).
To learn what means all the status information, take a look at: www.drbd.org/users-guide-emb/ch-admin.html#s-proc-drbd

We need to make the DRBD service start automaticaly at boot:

chkconfig –level 35 drbd on

 

5. Configuring GFS

Now, we must configure GFS (Red Hat Global File System), that it’s a cluster filesystem to use with DRBD:

First, we need to configure /etc/cluster/cluster.conf on both nodes:

vi /etc/cluster/cluster.conf

<?xml version="1.0\"?>
<cluster name="cluster1" config_version="3">

<cman two_node="1" expected_votes="1"/>

<clusternodes>
<clusternode name="node1.clustersmb.int" votes="1" nodeid="1">
        <fence>
                <method name="single">
                        <device name="manual" ipaddr="10.0.0.181"/>
                </method>
        </fence>
</clusternode>

<clusternode name="node2.clustersmb.int" votes="1" nodeid="2">
        <fence>
                <method name="single">
                        <device name="manual" ipaddr="10.0.0.182"/>
                </method>
        </fence>
</clusternode>
</clusternodes>

<fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="3"/>

<fencedevices>
        <fencedevice name="manual" agent="fence_manual"/>
</fencedevices>

</cluster>

Next, we need to start cman service, on both nodes (at same time):

/etc/init.d/cman start

Afterwards, we can start the other services, on both nodes:

/etc/init.d/clvmd start
/etc/init.d/gfs start
/etc/init.d/gfs2 start

We need to ensure that all the services are enabled at the initialization.
On both nodes, do:

chkconfig –level 35 cman on
chkconfig –level 35 clvmd on
chkconfig –level 35 gfs on
chkconfig –level 35 gfs2 on

Next, format the device, only on one node:

gfs_mkfs -p lock_dlm -t cluster1:gfs -j 2 /dev/drbd0

Now, we create the mountpoint and mount the drbd device, on both nodes:

mkdir /clusterdata
mount -t gfs /dev/drbd0 /clusterdata

Let’s insert the device on fstab, on both nodes:

vi /etc/fstab

Insert the following line:

/dev/drbd0          /clusterdata               gfs    defaults        0 0

Next, it’s good to check that the clusterfs is working:
Only on node1, do:

tar -zcvf /clusterdata/backup-test.tgz /etc/

Now, we check if the file exists on node2.
Only on node2, do:

ls -l /clusterdata/

[root@node2 ~]# ls -l /clusterdata
total 12576
-rw-r–r– 1 root root 12844520 Jul 23 16:01 backup-test.tgz

Now, let’s test if node2 can write.
Only on node2, do:

tar -zcvf /clusterdata/backup-test2.tgz /etc/

Now, we check if the two files exists on node1.
Only on node1, do:

ls -l /clusterdata/

[root@no2 ~]# ls -l /clusterdata/
total 25160
-rw-r–r– 1 root root 12850665 Jul 23 16:03 backup-test2.tgz
-rw-r–r– 1 root root 12844520 Jul 23 16:01 backup-test.tgz

If everything is fine, of course we can delete the test files.
Only one node, do:

rm -f /clusterdata/backup*

This article explains how to set up an Active/Active Samba CTDB Cluster, using GFS and DRBD. Prepared by Rafael Marangoni, from BRLink Servidor Linux Team.

6. Configuring Samba

The samba configuration is very common. In this article, we don’t cover how to configure samba. We only explain a very simple configuration.
Do it on both nodes:

First, we need to configure /etc/samba/smb.conf on both nodes:

vi /etc/samba/smb.conf

[global]

clustering = yes
idmap backend = tdb2
private dir=/clusterdata/ctdb
fileid:mapping = fsname
use mmap = no
nt acl support = yes
ea support = yes

[public]
comment = public share
path = /clusterdata/public
public = yes
writeable = yes
only guest = yes

We must to create the directories used by samba (only on one node):

mkdir /clusterdata/ctdb
mkdir /clusterdata/public
chmod 777 /clusterdata/public

Adding root on samba, only on one node:

smbpasswd -a root

 

7. Configuring CTDB

The process to install ctdb it’s very simple. Do it on both nodes:

First, we need to download it:

cd /usr/src
rsync -avz samba.org::ftp/unpacked/ctdb .
cd ctdb/

Then we can compile it:

cd /usr/src/ctdb/
./autogen.sh
./configure
make
make install

Creating the init scripts and config links to /etc:

cp config/ctdb.sysconfig /etc/sysconfig/ctdb
cp config/ctdb.init /etc/rc.d/init.d/ctdb
chmod +x /etc/init.d/ctdb

ln -s /usr/local/etc/ctdb/ /etc/ctdb
ln -s /usr/local/bin/ctdb /usr/bin/ctdb
ln -s /usr/local/sbin/ctdbd /usr/sbin/ctdbd

Next, we need to config /etc/sysconfig/ctdb on both nodes:

vi /etc/sysconfig/ctdb

CTDB_RECOVERY_LOCK="/dadoscluster/ctdb/storage"
CTDB_PUBLIC_INTERFACE=eth0
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_MANAGES_SAMBA=yes
ulimit -n 10000
CTDB_NODES=/etc/ctdb/nodes
CTDB_LOGFILE=/var/log/log.ctdb
CTDB_DEBUGLEVEL=2
CTDB_PUBLIC_NETWORK="10.0.0.0/24"
CTDB_PUBLIC_GATEWAY="10.0.0.9"

Now, config /etc/ctdb/public_addresses on both nodes:

vi /etc/ctdb/public_addresses

10.0.0.183/24
10.0.0.184/24

Then, config /etc/ctdb/nodes on both nodes:

vi /etc/ctdb/nodes

10.0.0.181
10.0.0.182

Then, config /etc/ctdb/events.d/11.route on both nodes:

vi /etc/ctdb/events.d/11.route

#!/bin/sh

. /etc/ctdb/functions
loadconfig ctdb

cmd="$1"
shift

case $cmd in
    takeip)
         # we ignore errors from this, as the route might be up already when we're grabbing
         # a 2nd IP on this interface
         /sbin/ip route add $CTDB_PUBLIC_NETWORK via $CTDB_PUBLIC_GATEWAY dev $1 2> /dev/null
         ;;
esac

exit 0

Set +x permission on script:

chmod +x /etc/ctdb/events.d/11.route

Finally, we need to bring services on:

/etc/init.d/ctdb start
/etc/init.d/smb start
chkconfig –level 35 ctdb on
chkconfig –level 35 smb on

 

References

CTDB: http://ctdb.samba.org/
CTDB: http://wiki.samba.org/index.php/CTDB_Setup#RedHat_GFS_filesystem
Howto: http://jun.homeunix.com/skills/server/GFS.DRBD8.CTDB.SAMBA.CentOS.txt

Comments

comments