Cheap VPS & Xen Server

Residential Proxy Network - Hourly & Monthly Packages

Disk Backup With Amanda On Debian Lenny


1. Introduction

Amanda is an open source client/server solution to back up filesystems. Backups are triggered by the backup server, backup definitions are located on the servers but exclusion lists are located on the client. Amanda is not yet to the level of backup solutions like IBM TSM/TDP or Legato Networker but it is the best open source solution I found so far.

2. Landscape

bckserver.mydomain.com (Debian lenny): The backup server that hosts amanda server.

bckclient.mydomain.com  (Debian lenny): The backup client is a web server with a MySQL version 5.0 database (no replication). The mysql backup is done with zrm version 2.2. I did not create a howto for zrm since the product is fairly easy to use and the zmanda Quick setup guide and user manual are really easy to follow. My biggest regret with zrm:  it does not backup to amanda tapes.

An ISCSI SAN as a library. backup to virtual tapes (disks)

 

3. AMANDA

3.1 Backup Server

3.1.1 Installation

As user root install the amanda-server package and … the amanda-client package or you will not be able to do restores from the bckclient.

# apt-get update

# apt-get install amanda-server amanda-client xinetd gawk gnuplot readline-common openssh-server

 Accept the additional packages required when prompted.

 

3.1.2  (If Applicable) Comment Amanda Entries In Inetd Configuration File

The packages configure both inetd and xinetd. In this howto we only use xinetd.

Edit /etc/inetd.conf and comment amanda related lines:

#:OTHER: Other services
#amandaidx stream tcp nowait backup /usr/sbin/tcpd /usr/lib/amanda/amindexd amindexd -auth=bsdtcp amdump amindexd amidxtaped
#amidxtape stream tcp nowait backup /usr/sbin/tcpd /usr/lib/amanda/amidxtaped amidxtaped -auth=bsdtcp amdump amindexd amidxtaped
#amanda dgram udp wait backup /usr/sbin/tcpd /usr/lib/amanda/amandad -auth=bsd amdump amindexd amidxtaped

 

3.1.3 Modify Amanda Daemons To Use auth bsdtcp

For details check the wiki.

Edit amanda, amidxtape and amandaidx files in /etc/xinetd.d/ to match the following:

# cat /etc/xinetd.d/amanda| grep -v “^#”

service amanda
{
        socket_type     = stream
        protocol        = tcp
        wait            = no
        user            = backup
        group           = backup
        groups          = yes
        server          = /usr/lib/amanda/amandad
        server_args     = -auth=bsdtcp amdump amindexd amidxtaped
        disable         = no
}

# cat /etc/xinetd.d/amidxtape | grep -v “^#”

service amidxtape
{
        socket_type     = stream
        protocol        = tcp
        wait            = no
        user            = backup
        group           = backup
        groups          = yes
        server          = /usr/lib/amanda/amidxtaped
        server_args     = -auth=bsdtcp amdump amindexd amidxtaped
        disable         = no
}

# cat /etc/xinetd.d/amandaidx | grep -v “^#”

service amandaidx
{
        socket_type     = stream
        protocol        = tcp
        wait            = no
        user            = backup
        group           = backup
        groups          = yes
        server          = /usr/lib/amanda/amindexd
        server_args     = -auth=bsdtcp amdump amindexd amidxtaped
        disable         = no
}

Enable the changes:

# /etc/init.d/xinetd restart

 

3.1.4 Backup Configuration

We will define daily backups and the configuration will be called DailySet1.

The backup user configured by default at the installation is:

user name: backup

user group: backup

home dir: /var/backups

#  su backup -c “mkdir /etc/amanda/DailySet1”

We create a minimum configuration file amanda.conf for DailySet1:

vi /etc/amanda/DailySet1/amanda.conf

org "DailySet1"         # your organization name for reports
mailto "backup_admin@mydomain.com"          # space separated list of operators at your site
dumpuser "backup"       # the user to run dumps under

logdir   "/etc/amanda/DailySet1/logs"           # log directory
infofile "/etc/amanda/DailySet1/curinfo"        # database filename
indexdir "/etc/amanda/DailySet1/index"          # index directory
tapelist "/etc/amanda/DailySet1/tapelist"       # list of used tapes

tapecycle 9 tapes
tapetype DISK
tpchanger "chg-disk"
changerfile "/etc/amanda/DailySet1/changer" # needed by amlabel
tapedev "file:/space/vtapes/DailySet1/slots"

define tapetype DISK {
  comment "Backup to HD"
  length 5 gbytes
}

# comment holding disk section if needed
holdingdisk hd1 {
  directory "/dumps/"
}

# to be used by any other dumptype
define dumptype global {
  comment "Global definitions"
  index yes
  record yes
  auth "bsdtcp"
}

define dumptype daily {
  global
  comment "daily dumptype"
  compress client fast
  program "GNUTAR"
  strategy standard
  priority high
  exclude list "/etc/amanda/DailySet1/global-debian-exclude.list"
}

Quick explanation:

– we plan to use 9 tapes of 5 GigaBytes. We recycle tapes after 9 backups.

– we plan to back up everything on the bckclient with some exceptions (ref exclude list section 3.2.6).

Check the amanda.conf man page for details on syntax.

Create the database repository for DailySet1 as specified in amanda.conf.

# su backup -c “mkdir /etc/amanda/DailySet1/logs”

# su backup -c “mkdir /etc/amanda/DailySet1/index”

# touch /etc/amanda/DailySet1/tapelist

# su backup -c “mkdir /etc/amanda/DailySet1/curinfo”

 

3.1.5 (Optional) Configure Holding Disks

The holding disk is used as a cache to store backup data from all Amanda clients (ref).

If you do not intend to use holding disks, review the amanda.conf file.

# comment holding disk section if needed
#holdingdisk hd1 {
#  directory "/dumps/"
#}

and  skip  this section.

If you plan to use holding disks:

# su backup -c “mkdir /dumps”

# chmod 750 /dumps

I use here a local disk. The details are not covered by this document.

# cat /etc/fstab | grep LABEL

LABEL=holdingdisk       /dumps           ext3    defaults        0      
LABEL=backup       /space  xfs     _netdev        0       0

3.1.6  Configure Virtual Tapes

# mkdir -p /space/

I use my ISCSI SAN disks here. Check this ISCSI howto for details.

# mkdir -p /space/vtapes/DailySet1/slots

# chown backup:backup -R  /space/

# chmod 750 /space/vtapes

# su – backup

$ cd /space/vtapes/DailySet1/slots

 Create the tapes:

$ for ((i=1; $i<=9; i++)); do mkdir  slot$i;done

Initialize tape changer:

$ ln -s slot1 data

 Test the virtual tapes:

$ ammt -f file:/space/vtapes/DailySet1/slots status

file:/space/vtapes/DailySet1/slots status: ONLINE

 Label the tapes:

$ for ((i=1; $i<=9;i++)); do amlabel DailySet1 DailySet1-0$i slot $i; done

Note: If you plan to use more than nine tapes (say 25 for example), do run a second amlabel command:

$ for ((i=10; $i<=25;i++)); do amlabel DailySet1 DailySet1-$i slot $i; done

 Reset the virtual tape changer back to the first slot.

$ amtape DailySet1 reset

amtape: changer is reset, slot 1 is loaded.

$ ls -l data

lrwxrwxrwx 1 backup backup  35 jun  7 02:27 data -> /space/vtapes/DailySet1/slots/slot1

 

3.1.7 Backup Definition: Disklist

In this section you specify for each backup client, what disks you intend to back up.

On the backup client:

As user root issue mount on the backup client to list mounted disks:

 # mount

/dev/mapper/debian-root on / type xfs
/dev/sda1 on /boot type ext2 (rw)
/dev/sdb1 on /var/www/www.mysite.com type ext3

On the backup server:

$ vi /etc/amanda/DailySet1/disklist

bckclient.mydomain.com /var/www/www.mysite.com/ daily
bckclient.mydomain.com / daily
bckclient.mydomain.com /boot/ daily

syntax: client.fqdn path dumptype (ref amanda.conf)

Note: Since we use fully qualified domain names, your name resolution (/etc/hosts or DNS and reverse DNS) must be correctly set up and tested. If you are unsure use IP addresses.

 

3.1.8 Authorization: amandahost

This file is used to authorize client or server connections:

$ vi /etc/amandahosts

bckserver.mydomain.com root amindexd amidxtaped
bckserver.mydomain.com backup amdump
bckclient.mydomain.com root amindexd amidxtaped

The line: “bckclient.mydomain.com root amindexd amidxtaped” allows user root from bckclient to perform restores.

The line “bckserver.mydomain.com backup amdump” allows user backup from bckserver to run backups on bckserver.

Details on file syntax and authentification  here.

Note: Since we use fully qualified domain names, your name resolution (/etc/hosts or DNS and reverse DNS) must be correctly set up and tested. If you are unsure use IP addresses.

# ln -s /var/backups/.amandahosts /etc/amandahosts

# chmod 400 etc/amandahosts

# ls -l /etc/amandahosts

-r——– 1 backup backup 157 mai 27 02:12 /etc/amandahosts

# ls -l /var/backups/.amandahosts

lrwxrwxrwx 1 root root 16 mai 17 00:41 /var/backups/.amandahosts -> /etc/amandahosts

3.2 Backup Client

3.2.1 Installation

As user root install the amanda-client package.

# apt-get update

# apt-get install amanda-client xinetd openssh-server

 Accept the additional packages required when prompted.

 

3.2.2 (If Applicable) Comment Amanda Entries In Inetd Configuration File

The packages configure both inetd and xinetd. In this howto we only use xinetd.

Edit /etc/inetd.conf and comment amanda related lines:

#:OTHER: Other services
#amanda dgram udp wait backup /usr/sbin/tcpd /usr/lib/amanda/amandad -auth=bsd amdump amindexd amidxtaped

 

3.2.3 Modify Amanda Daemons To Use auth bsdtcp

For details check the wiki.

Edit amanda, amidxtape and amandaidx files in /etc/xinetd.d/ to match the following:

# cat /etc/xinetd.d/amanda| grep -v “^#”

service amanda
{
        socket_type     = stream
        protocol        = tcp
        wait            = no
        user            = backup
        group           = backup
        groups          = yes
        server          = /usr/lib/amanda/amandad
        server_args     = -auth=bsdtcp amdump amindexd amidxtaped
        disable         = no
}

Enable changes:

# /etc/init.d/xinetd restart

 

3.2.4 Client Configuration: amanda-client.conf

# vi /etc/amanda/amanda-client.conf

conf "DailySet1" # your config name
index_server "bckserver.mydomain.com" # your amindexd server
tape_server "bckserver.mdomain.com" # your amidxtaped server


auth "bsdtcp"

 

3.2.5 Authorizations: amandahosts

Edit the amandahosts file to allow backups from bckserver:

# vi /etc/amanda/amandahosts

bckserver.mydomain.com backup amdump

The line “bckserver.mydomain.com backup amdump” allows user backup from bckserver to run backups on bckclient.

Details on file syntax and authentification  here.

Note: Since we use fully qualified domain names, your name resolution (/etc/hosts or DNS and reverse DNS) must be correctly set up and tested. If you are unsure use IP addresses.

# ls -la /var/backups/.amandahosts

lrwxrwxrwx 1 root root 23 mai 25 11:19 /var/backups/.amandahosts -> /etc/amanda/amandahosts

 

# ls -l /etc/amanda/amandahosts

-rw——- 1 backup backup 42 mai 26 23:21 /etc/amanda/amandahosts

 

3.2.6 Exclude List

As user backup create an exclude list. Bare in mind paths are relative. More details here.

$ vi /etc/amanda/DailySet1/global-debian-exclude.list

./proc
./media
./mnt
./dev

chmod 644 /etc/amanda/DailySet1/global-debian-exclude.list

Note: in my example the exclude list applies to the 3 filesystems. If you want to avoid this, define different exclude lists related to differents dumptypes and modify disklist file accordingly.

 

3.3 Tests

3.3.1 Config Test

Log on as user backup on bckserver and issue the following command:

$ amcheck DailySet1

Amanda Tape Server Host Check
—————————–
Holding disk /dumps/: 48800396 kB disk space available, using 48800396 kB
slot 2: read label `DailySet1-02′, date `20100607′
cannot overwrite active tape DailySet1-02
slot 3: read label `DailySet1-03′, date `X’
NOTE: skipping tape-writable test
Tape DailySet1-03 label ok
NOTE: host info dir /etc/amanda/DailySet1/curinfo/bckclient.mydomain.com does not exist
NOTE: it will be created on the next run.
NOTE: index dir /etc/amanda/DailySet1/index/bckclient.mydomainman .com does not exist
NOTE: it will be created on the next run.
Server check took 0.115 seconds

Amanda Backup Client Hosts Check

——————————–
Client check: 1 host checked in 0.132 seconds, 0 problems found

(brought to you by Amanda 2.5.2p1)

 

3.3.2 Backup Test

Log on as user backup on bckserver and issue the following command:

$ amdump DailySet1

Note: to force a full backup issue the follwing command before the amdump:

$ amadmin DailySet1 force bckclient.mydomain.com

On completion check backup_admin@mydomain.com mails for a backup notification.

 Hostname: bckserver
Org     : DailySet1
Config  : DailySet1
Date    : June 7, 2010

These dumps were to tape DailySet1-03.
The 
next tape Amanda expects to use is: a new tape.
The next new tape 
already labelled is: DailySet1-04.


STATISTICS:
            
                          Total       Full      Incr.
                        --------    --------   --------
Estimate Time (hrs:min)    0:00
Run Time      (hrs:min)    0:06
Dump Time (hrs:min)        0:05       0:05       0:00
Output Size (meg)         766.7      766.7        0.0
Original Size (meg)      1363.0     1363.0        0.0
Avg Compressed Size    56.3       56.3        -- 
Filesystems Dumped            1          1          0
Avg Dump Rate (k/s)      2730.3     2730.3        -- 
Tape Time (hrs:min)        0:01       0:01       0:00
Tape Size (meg)           766.8      766.8        0.0
Tape Used               15.0       15.0        0.0
Filesystems Taped             1          1          0
Chunks Taped                  0          0          0
Avg Tp Write Rate (k/s)  9490.5     9490.5        -- 

USAGE BY TAPE:
  Label              Time      Size      %    Nb    Nc
  DailySet1-03       0:01   785152k   15.0     1     0


NOTES:
  planner: tapecycle (9) <= runspercycle (10)
  planner: Adding new disk bckclient.mydomain.com:/.
  driver: WARNING: This is not the first amdump run today. Enable the usetimestamps option in the configuration file if you want to run amdump more than once per calendar
 day.
  taper: tape DailySet1-03 kb 785184 fm 1 [OK]
  small estimate: bckclient.mydomain.com / 0
                  est: 697920k    out 785152k


DUMP SUMMARY:
                      
                 DUMPER STATS               TAPER STATS 
HOSTNAME     DISK        L ORIG-kB  OUT-kB  COMP%  MMM:SS   KB/s MMM:SS   KB/s
-------------------------- ------------------------------------- -------------
bckclient    /           0 1395720  785152   56.3    4:48 2730.3   1:23 9490.4
bckclient    /boot       0   16320   15520   95.1    0:04 4324.4   0:00 98858.5
bckclient    -mysite.com 0   34750   11488   33.1    0:09 1228.3   0:01 16425.6

(brought to you by Amanda version 2.5.2p1)

$ amadmin DailySet1 find

date       host                           disk                    lv tape or file file part status
2010-06-07 bckclient.mydomain.com         /                        1 DailySet1-04    1   -- OK
2010-06-07 bckclient.mydomain.com         /                        1 DailySet1-05    1   -- OK
2010-06-07 bckclient.mydomain.com         /                        1 DailySet1-06    1   -- OK
2010-06-07 bckclient.mydomain.com         /                        0 DailySet1-01    0   -- FAILED (dumper) [port open: Connection timed out]
2010-06-07 bckclient.mydomain.com         /                        0 DailySet1-01    0   -- FAILED (dumper) [port open: Connection timed out]
2010-06-07 bckclient.mydomain.com         /                        0 DailySet1-02    0   -- FAILED (dumper) [port open: Connection timed out]
2010-06-07 bckclient.mydomain.com         /                        0 DailySet1-02    0   -- FAILED (dumper) [port open: Connection timed out]
2010-06-07 bckclient.mydomain.com         /                        0 DailySet1-03    1   -- OK
2010-06-07 bckclient.mydomain.com         /                        0 DailySet1-07    1   -- OK
2010-06-07 bckclient.mydomain.com         /                        0 DailySet1-08    3   -- OK
2010-06-07 bckclient.mydomain.com         /boot                    0 DailySet1-08    1   -- OK
2010-06-07 bckclient.mydomain.com         /var/www/www.mysite.com  0 DailySet1-08    2   -- OK

 

3.4 Backup Scheduling

Daily backup every day at 1:00AM:

# su  backup -c “crontab -e”

0 1 * * 1-7 /usr/sbin/amdump DailySet1

 

3.5 Restore

3.5.1 Backup Client Configuration

Log on as user root on the bckclient.

Create an amanda-client.conf file:

# su backup -c “mkdir /etc/amanda”

# vi /etc/amanda/amanda-client.conf

conf "DailySet1" # your config name (used for restore)
index_server "bckserver.mydomain.com" # your amindexd server
tape_server "bckserver.mydomain.com" # your amidxtaped server
auth "bsdtcp"

# ls -ltr /etc/amanda/DailySet1/amanda-client.conf

lrwxrwxrwx 1 root root 30 mai 25 17:50 /etc/amanda/DailySet1/amanda-client.conf -> /etc/amanda/amanda-client.conf

Note: If you mix weekly (long retention) and daily backups (short retention) for the same backup client you will have to specify the right conf (WeeklySet1 or DailySet1) for restore.

 

3.5.2 Recover

As user root on bckclient cd to a suitable place for restore (/tmp for example) and issue the following command:

# amrecover

AMRECOVER Version 2.5.2p1. Contacting server on bckserver.mydomain.com …
220 bckserver AMANDA index server (2.5.2p1) ready.
Setting restore date to today (2010-06-07)
200 Working date set to 2010-06-07.
200 Config set to DailySet1.
501 Host bckclient is not in your disklist.
Trying host bckclient.mydomain.com …
200 Dump host set to bckclient.mydomain.com.
Use the setdisk command to choose dump disk to recover

amrecover> listdisk

200- List of disk for host bckclient.mydomain.com
201- /

201- /var/www/www.mysite.com
201- /boot

amrecover> setdisk /var/www/www.mysite.com

200 Disk set to /var/www/www.mysite.com.

amrecover> ls

2010-06-07 xmlrpc/
2010-06-07 tmp/
2010-06-07 templates/
2010-06-07 robots.txt
2010-06-07 plugins/
2010-06-07 modules/
2010-06-07 logs/
2010-06-07 libraries/
2010-06-07 language/
2010-06-07 index2.php
2010-06-07 index.php
2010-06-07 includes/
2010-06-07 images/
2010-06-07 htaccess.txt
2010-06-07 configuration.php-dist
2010-06-07 configuration.php
2010-06-07 components/
2010-06-07 cache/
2010-06-07 aicontactsafe/
2010-06-07 administrator/
2010-06-07 LICENSES.php
2010-06-07 LICENSE.php
2010-06-07 INSTALL.php
2010-06-07 CREDITS.php
2010-06-07 COPYRIGHT.php
2010-06-07 CHANGELOG.php
2010-06-07 .htaccess
2010-06-07 .

amrecover> add *

Added dir /xmlrpc/ at date 2010-06-07
Added dir /tmp/ at date 2010-06-07
Added dir /templates/ at date 2010-06-07
Added file /robots.txt
Added dir /plugins/ at date 2010-06-07
Added dir /modules/ at date 2010-06-07
Added dir /logs/ at date 2010-06-07
Added dir /libraries/ at date 2010-06-07
Added dir /language/ at date 2010-06-07
Added file /index2.php
Added file /index.php
Added dir /includes/ at date 2010-06-07
Added dir /images/ at date 2010-06-07
Added file /htaccess.txt
Added file /configuration.php-dist
Added file /configuration.php
Added dir /components/ at date 2010-06-07
Added dir /cache/ at date 2010-06-07
Added dir /aicontactsafe/ at date 2010-06-07
Added dir /administrator/ at date 2010-06-07
Added file /LICENSES.php
Added file /LICENSE.php
Added file /INSTALL.php
Added file /CREDITS.php
Added file /COPYRIGHT.php
Added file /CHANGELOG.php
Added file /.htaccess

amrecover> extract

Extracting files using tape drive chg-disk on host bckserver.mydomain.com.
The following tapes are needed: DailySet1-08

Restoring files into directory /tmp

Continue [?/Y/n]? y

Extracting files using tape drive chg-disk on host bckserver.mydomain.com.
Load tape DailySet1-08 now

Continue [?/Y/n/s/t]? y

./administrator/
./administrator/backups/

Note:

tree navigation with cd

time navigation with setdate.

restore to another host with sethost

4. References Used To Build This Howto:

The 15-Minute Backup Solution

Quick start

Disk Based Backups With Amanda On Debian Etch

 

Comments

comments