Cheap VPS & Xen Server

Residential Proxy Network - Hourly & Monthly Packages

Creating Encrypted FTP Backups With duplicity And ftplicity On Debian Etch


When you rent a dedicated server nowadays, almost all providers give you FTP backup space for your server on one of the provider’s backup systems. This tutorial shows how you can use duplicity and ftplicity to create encrypted (so that nobody with access to the backup server can read sensitive data in your backups) backups on the provider’s remote backup server over FTP. ftplicity is a duplicity wrapper script (provided by the German computer magazine c’t) that allows us to use duplicity without interaction (i.e., you do not have to type in any passwords).

I do not issue any guarantee that this will work for you!

 

1 Preliminary Note

In this tutorial I call the backup server backup.example.com where I have an FTP account with the username backupuser and the password secret. On my own server, I want to backup the directory /home/exampleuser.

 

2 Installing duplicity

First we make sure our system is up-to-date:

apt-get update

apt-get upgrade

Then we install duplicity as follows:

apt-get install duplicity

 

3 Our First Backup

Now let’s do our first backup:

FTP_PASSWORD=secret duplicity /home/exampleuser ftp://backupuser@backup.example.com/

server2:/home/exampleuser# FTP_PASSWORD=secret duplicity /home/exampleuser ftp://backupuser@backup.example.com/
GnuPG passphrase:
Retype to confirm:
No signatures found, switching to full backup.
————–[ Backup Statistics ]————–
StartTime 1197306549.59 (Mon Dec 10 18:09:09 2007)
EndTime 1197306610.85 (Mon Dec 10 18:10:10 2007)
ElapsedTime 61.26 (1 minute 1.26 seconds)
SourceFiles 342
SourceFileSize 10545818 (10.1 MB)
NewFiles 342
NewFileSize 10545818 (10.1 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 342
RawDeltaSize 10513050 (10.0 MB)
TotalDestinationSizeChange 8968398 (8.55 MB)
Errors 0
————————————————-

server2:/home/exampleuser#

As you see you will be asked for a GnuPG passphrase. You can type in any password you like; this has to be done everytime you run duplicity. The backup will be encrypted with the help of GnuPG. Permissions and ownerships will be preserved in the backup.

To create the backup in a subdirectory on the backup server, you’d modify the command as follows:

FTP_PASSWORD=secret duplicity /home/exampleuser ftp://backupuser@backup.example.com/subdirectory

When you run duplicity for the first time, it will create a full backup; afterwards, it creates incremental backups. To force the creation of a full backup again, you can use the –full switch:

FTP_PASSWORD=secret duplicity –full /home/exampleuser ftp://backupuser@backup.example.com/

To exclude a directory from the backup, e.g. /home/exampleuser/tmp, you can use the –exclude switch:

FTP_PASSWORD=secret duplicity –exclude /home/exampleuser/tmp /home/exampleuser ftp://backupuser@backup.example.com/

If you are backing up the root directory /, remember to –exclude /proc, or else duplicity will probably crash.

To learn more about the available duplicity options, take a look at

man duplicity

 

4 Restore A Backup

Now let’s assume we have deleted everything in /home/exampleuser and want to restore it from our FTP backup. This is how it’s done:

FTP_PASSWORD=secret duplicity ftp://backupuser@backup.example.com/ /home/exampleuser

Please note that in this case the remote location comes before to local folder!

5 Automatic Backups With ftplicity

Because duplicity asks for a GnuPG password everytime we use it, it’s hard to use it for automatic backups (e.g. via cron). Fortunately there’s ftplicity, a duplicity wrapper script, which allows us to call duplicity without being asked for a password.

First we generate a GnuPG key that ftplicity will use (so that we don’t have to type in a password anymore):

gpg –-gen-key

server2:/home/exampleuser# gpg –gen-key
gpg (GnuPG) 1.4.6; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.

Please select what kind of key you want:
(1) DSA and Elgamal (default)
(2) DSA (sign only)
(5) RSA (sign only)
Your selection? <– ENTER
DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) <– ENTER
Requested keysize is 2048 bits
Please specify how long the key should be valid.
0 = key does not expire
<n>  = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0)
 <– ENTER
Key does not expire at all
Is this correct? (y/N)
 <– y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
“Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>”

Real name: <– your name, e.g. Falko Timme
Email address: <– your email address, e.g. ft@falkotimme.com
Comment:
You selected this USER-ID:
“Falko Timme <ft@falkotimme.com>”

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? <– O
You need a Passphrase to protect your secret key. <– Type in your desired password (twice to confirm it)

Now the key will be generated. It’s a good idea to open a second console and type some letters so that the random number generator can gain enough entropy:

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
+++++.++++++++++++++++++++.++++++++++..++++++++++.++++++++++++++++++++.
++++++++++.+++++++++++++++++++++++++++++++++++++++++++++++++++++++>++++++++++
…………………………………..+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
…..+++++…+++++.++++++++++.+++++.++++++++++..++++++++++.++++++++++++++
++++++++++++++++.++++++++++.++++++++++++++++++++++++++++++.+++++++++++++++.
+++++++++++++++>+++++…+++++>+++++>..+++++>+++++…………………….
<.+++++……………….>+++++…………………………………….
…….<+++++………………..+++++^^^^^
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 
7C6E958B marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   1024D/7C6E958B 2007-12-10
Key fingerprint = 1FDC 60FB 8A27 90D8 553C  3C3E 8E1F 66F7 7C6E 958B
uid                  Falko Timme <ft@falkotimme.com>
sub   2048g/F1BB98F4 2007-12-10

server2:/home/exampleuser#

I’ve highlighted the key ID (7C6e958B) because we’ll need it in a moment.

Next we download ftplicity and run it once so that it can create some needed files:

cd /tmp
wget ftp://ftp.heise.de/pub/ct/listings/0613-216.tar.gz
tar xvfz 0613-216.tar.gz
cp ftplicity-1.1.1/ftplicity /usr/local/bin
ftplicity

The output of the last command is in German; basically it’s saying the the ftplicity configuration is in the file /root/.ftplicity/conf:

server2:/tmp# ftplicity

Offenbar benutzen Sie ftplicity zum ersten Mal. Eine vorlaeufige
Konfigurationsdatei wurde unter /root/.ftplicity/conf erstellt.
Sie muessen dort die Daten des verwendeten GPG-Schluessels sowie
die Zugangsdaten fuer den FTP-Server eintragen, bevor Sie mit dem
Backup fortfahren koennen.

WICHTIG:
Sichern Sie das komplette Konfigurationsverzeichnis nach dem
ersten erfolgreichen Backup unbedingt auf einen vertrauenswuerdi-
gen externen Rechner und schuetzen Sie es vor unbefugtem Zugriff.

server2:/tmp#

Now we edit /root/.ftplicity/conf. It has the following variables:

  • GPG_KEY: the ID of our GnuPG key;
  • GPG_PW: the password we typed in when we created the GnuPG key;
  • ZIEL: the backup server (incl. the FTP username);
  • ZIEL_PW: the FTP password in the backup server;
  • QUELLE: the source directory (i.e., the directory to be backed up);
  • HOECHSTALTER: the age of the oldest backup; older backups will be deleted;
  • VERBOSITY: amount of information displayed on the screen by ftplicity;
  • TEMP_DIR: a directory for temporary files; when you restore a backup, this directory must at least have enough space for the biggest file in the backup.

vi /root/.ftplicity/conf

# Daten fuer GPG-Schluessel
GPG_KEY=7C6E958B
GPG_PW='gpg_key_password'
# Zugangsdaten fuer FTP-Server (URL-Format)
ZIEL='ftp://backupuser@backup.example.com/'
ZIEL_PW='secret'
# Basisverzeichnis fuers Backup
QUELLE='/home/exampleuser'
# aeltester Wiederherstellungszeitpunkt
HOECHSTALTER=1M
# Ausfuehrlichkeit der Bildschirmausgaben (9 fuer Fehlersuche)
VERBOSITY=4
# Verzeichnis fuer temporaere Dateien. Beim Restore muss dort
# mindestens Patz fuer die groesste Datei im Backup sein
TEMP_DIR=/tmp

Afterwards we change the permissions of the file so that only root has read and write permissions:

chmod 600 /root/.ftplicity/conf

We can now create three other files, although that is totally optional:

  • /root/.ftplicity/exclude: contains a list of directories to be excluded from the backup (one directory per line);
  • /root/.ftplicity/pre: contains command(s) to be executed prior to the backup (e.g. create a MySQL database dump);
  • /root/.ftplicity/post: contains command(s) to be executed after the backup.

Here’s a sample /root/.ftplicity/exclude and /root/.ftplicity/pre file (the syntax of /root/.ftplicity/post is the same as in /root/.ftplicity/pre):

vi /root/.ftplicity/exclude

/proc
/dev
/sys

chmod 600 /root/.ftplicity/exclude

vi /root/.ftplicity/pre

/usr/bin/mysqldump –-all-databases -u root -pyourrootsqlpassword > /home/exampleuser/db.sql

(It should be noted that the database gets locked during the creation of SQL dump; this might not be an issue for a small web site, but can be a problem for high-traffic web sites – your visitors won’t be able to access database-driven pages during mysqldump. Here’s a link to an interruption-free MySQL backup method: How To Back Up MySQL Databases Without Interrupting MySQL)

/root/.ftplicity/pre must be executable (as well as /root/.ftplicity/post if you choose to create one):

chmod 700 /root/.ftplicity/pre

As I said before, you only need /root/.ftplicity/conf; the other files are optional.

Now ftplicity is ready to be used; to create our backup, we simply run

ftplicity backup

If all goes well, you won’t be prompted for a password:

server2:~# ftplicity backup
Reading globbing filelist /root/.ftplicity/exclude
No signatures found, switching to full backup.
————–[ Backup Statistics ]————–
StartTime 1197309836.07 (Mon Dec 10 19:03:56 2007)
EndTime 1197309909.96 (Mon Dec 10 19:05:09 2007)
ElapsedTime 73.90 (1 minute 13.90 seconds)
SourceFiles 343
SourceFileSize 10859083 (10.4 MB)
NewFiles 343
NewFileSize 10859083 (10.4 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 343
RawDeltaSize 10826315 (10.3 MB)
TotalDestinationSizeChange 9052110 (8.63 MB)
Errors 0
————————————————-

server2:~#

To automate the backups, we can create a cron job (I create two cron jobs, one that runs daily and creates incremental backups, and one that runs once a month, creates a full backup and deletes old files):

crontab -e

# run the (incremental) backup each night at 03:23h
23 3 * * * /usr/local/bin/ftplicity backup
# do a full backup once per month & delete old backups
47 4 1 * * /usr/local/bin/ftplicity full && /usr/local/bin/ftplicity purge –-force

 

6 Restoring A Backup With ftplicity

Of course, you can use ftplicity also to restore a backup (again, you won’t be asked for a password). The syntax is very easy:

ftplicity restore /home/exampleuser

If you speak German, you can take a look at the ftplicity help tp learn what other options you have:

ftplicity –help

server2:~# ftplicity –help
Usage: ftplicity <Kommando> [<Optionen> …]

Kommandos:
backup:    inkrementelles Backup
full:      vollstaendiges Backup erzwingen
list:      Dateien im Backup auflisten
verify:    Backup pruefen und Aenderungen anzeigen
purge [–force]:
veraltete Backup-Archive anzeigen [und loeschen]
cleanup [–force]:
Backup-Verzeichnis aufraeumen (nach Programmabbruch)
restore <Zielpfad> [<Alter>]:
Backup nach <Zielpfad> restaurieren [vom Stand <Alter>]
fetch <Datei/Verzeichnis> <Ziel> [<Alter>]
Datei/Verzeichnis einzeln restaurieren (siehe Beispiel)

Beispiele:
Veraltete Backup-Archive anzeigen und loeschen:
ftplicity purge –force
Vollstaendiges Backup nach /mnt zurueckspielen:
ftplicity restore /mnt
Datei /etc/passwd nach /root/pw restaurieren, Stand vor 4 Tagen:
ftplicity fetch etc/passwd /root/pw 4D
(siehe “man duplicity”, Abschnitt TIME FORMATS)

Dateien in /root/.ftplicity
conf      Haupt-Konfigurationsdatei
pre       wird vor einem Backup ausgefuehrt
post      wird nach einem Backup ausgefuehrt
gpgkey    enthaelt exportierten GPG-Schluessel
exclude   Liste ausgeschlossener Dateien und Verzeichnisse
(siehe “man duplicity”, Abschnitt FILE SELECTION)

server2:~#

 

  • duplicity: http://duplicity.nongnu.org
  • ftplicity (Sourceforge Project Page): http://ftplicity.sourceforge.net/
  • ftplicity: http://www.heise.de/ct/06/13/links/216.shtml
  • Debian: http://www.debian.org

Comments

comments