Cheap VPS & Xen Server

Residential Proxy Network - Hourly & Monthly Packages

Creating Encrypted FTP Backups With duplicity And duply On Debian Squeeze


When you rent a dedicated server nowadays, almost all providers give you FTP backup space for your server on one of the provider’s backup systems. This tutorial shows how you can use duplicity and duply to create encrypted (so that nobody with access to the backup server can read sensitive data in your backups) backups on the provider’s remote backup server over FTP. duply is a duplicity wrapper script that allows us to use duplicity without interaction (i.e., you do not have to type in any passwords).

I do not issue any guarantee that this will work for you!

1 Preliminary Note

In this tutorial I call the backup server backup.example.com where I have an FTP account with the username backupuser and the password secret. On my own server, I want to backup the directory /home/exampleuser.

 

2 Installing duplicity

First we make sure our system is up-to-date:

apt-get update

apt-get upgrade

Then we install duplicity as follows:

apt-get install duplicity ncftp

 

3 Our First Backup

Now let’s do our first backup:

FTP_PASSWORD=secret duplicity /home/exampleuser ftp://backupuser@backup.example.com/

root@server1:/home/exampleuser# FTP_PASSWORD=secret duplicity /home/exampleuser ftp://backupuser@backup.example.com/
NcFTP version is 3.2.4
GnuPG passphrase:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
Retype passphrase to confirm:
————–[ Backup Statistics ]————–
StartTime 1341339005.14 (Tue Jul 3 20:10:05 2012)
EndTime 1341339006.46 (Tue Jul 3 20:10:06 2012)
ElapsedTime 1.32 (1.32 seconds)
SourceFiles 53
SourceFileSize 13494139 (12.9 MB)
NewFiles 53
NewFileSize 13494139 (12.9 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 53
RawDeltaSize 13461371 (12.8 MB)
TotalDestinationSizeChange 4687992 (4.47 MB)
Errors 0
————————————————-

root@server1:/home/exampleuser#

As you see you will be asked for a GnuPG passphrase. You can type in any password you like; this has to be done everytime you run duplicity. The backup will be encrypted with the help of GnuPG. Permissions and ownerships will be preserved in the backup.

To create the backup in a subdirectory on the backup server, you’d modify the command as follows:

FTP_PASSWORD=secret duplicity /home/exampleuser ftp://backupuser@backup.example.com/subdirectory

When you run duplicity for the first time, it will create a full backup; afterwards, it creates incremental backups. To force the creation of a full backup again, you can use the full switch:

FTP_PASSWORD=secret duplicity full /home/exampleuser ftp://backupuser@backup.example.com/

To exclude a directory from the backup, e.g. /home/exampleuser/tmp, you can use the –exclude switch:

FTP_PASSWORD=secret duplicity –exclude /home/exampleuser/tmp /home/exampleuser ftp://backupuser@backup.example.com/

If you are backing up the root directory /, remember to –exclude /proc, or else duplicity will probably crash.

To learn more about the available duplicity options, take a look at

man duplicity

 

4 Restore A Backup

Now let’s assume we have deleted everything in /home/exampleuser and want to restore it from our FTP backup. This is how it’s done:

FTP_PASSWORD=secret duplicity ftp://backupuser@backup.example.com/ /home/exampleuser

Please note that in this case the remote location comes before to local folder!

 

5 Automatic Backups With duply

Because duplicity asks for a GnuPG password everytime we use it, it’s hard to use it for automatic backups (e.g. via cron). Fortunately there’s duply, a duplicity wrapper script, which allows us to call duplicity without being asked for a password.

First we generate a GnuPG key that duply will use (so that we don’t have to type in a password anymore):

gpg –gen-key

server1:/home/exampleuser# gpg –gen-key
gpg (GnuPG) 1.4.10; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.

Please select what kind of key you want:
(1) DSA and Elgamal (default)
(2) DSA (sign only)
(5) RSA (sign only)
Your selection? <– ENTER
DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) <– ENTER
Requested keysize is 2048 bits
Please specify how long the key should be valid.
0 = key does not expire
<n>  = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0)
 <– ENTER
Key does not expire at all
Is this correct? (y/N)
 <– y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
“Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>”

Real name: <– your name, e.g. Falko Timme
Email address: <– your email address, e.g. ft@falkotimme.com
Comment:
You selected this USER-ID:
“Falko Timme <ft@falkotimme.com>”

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? <– O
You need a Passphrase to protect your secret key. <– Type in your desired password (twice to confirm it)

Now the key will be generated. It’s a good idea to open a second console and type some letters so that the random number generator can gain enough entropy:

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
….+++++
+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
….+++++
+++++
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 
7C6E958B marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   1024D/7C6E958B 2007-12-10
Key fingerprint = 1FDC 60FB 8A27 90D8 553C  3C3E 8E1F 66F7 7C6E 958B
uid                  Falko Timme <ft@falkotimme.com>
sub   2048g/F1BB98F4 2007-12-10

server1:/home/exampleuser#

I’ve highlighted the key ID (7C6e958B) because we’ll need it in a moment.

Next we download duply:

cd /tmp
wget http://downloads.sourceforge.net/project/ftplicity/duply%20%28simple%20duplicity%29/1.5.x/duply_1.5.7.tgz
tar xvfz duply_1.5.7.tgz
cp duply_1.5.7/duply /usr/local/bin

To use duply, we need to create at least one profile. Since I back up /home/exampleuser in this tutorial, I call the profile exampleuser. We can create the profile as follows:

duply exampleuser create

This creates the directory /root/.duply/exampleuser with the duply configuration file conf in it. Open the file:

vi /root/.duply/exampleuser/conf

The most important settings in this file are:

  • GPG_KEY: the ID of our GnuPG key;
  • GPG_PW: the password we typed in when we created the GnuPG key;
  • TARGET: the backup server (incl. the FTP username);
  • TARGET_PASS: the FTP password in the backup server;
  • SOURCE: the source directory (i.e., the directory to be backed up);
  • MAX_AGE: the age of the oldest backup; older backups will be deleted;
  • VERBOSITY: amount of information displayed on the screen by duply;
  • TEMP_DIR: a directory for temporary files; when you restore a backup, this directory must at least have enough space for the biggest file in the backup.

My file looks as follows

# gpg encryption settings, simple settings:
#  GPG_KEY='disabled' - disables encryption alltogether
#  GPG_KEY='<key1>[,<key2>]'; GPG_PW='pass' - encrypt with keys, sign
#    with key1 if secret key available and use GPG_PW for sign & decrypt
#  GPG_PW='passphrase' - symmetric encryption using passphrase only
GPG_KEY='7C6E958B'
GPG_PW='gpg_key_password'
# gpg encryption settings in detail (extended settings)
#  the above settings translate to the following more specific settings
#  GPG_KEYS_ENC='<keyid1>,[<keyid2>,...]' - list of pubkeys to encrypt to
#  GPG_KEY_SIGN='<keyid1>|disabled' - a secret key for signing
#  GPG_PW='<passphrase>' - needed for signing, decryption and symmetric
#   encryption. If you want to deliver different passphrases for e.g.
#   several keys or symmetric encryption plus key signing you can use
#   gpg-agent. Add '--use-agent' to the duplicity parameters below.
#   also see "A NOTE ON SYMMETRIC ENCRYPTION AND SIGNING" in duplicity manpage
# notes on en/decryption
#  private key and passphrase will only be needed for decryption or signing.
#  decryption happens on restore and incrementals (compare archdir contents).
#  for security reasons it makes sense to separate the signing key from the
#  encryption keys. https://answers.launchpad.net/duplicity/+question/107216
#GPG_KEYS_ENC='<pubkey1>,<pubkey2>,...'
#GPG_KEY_SIGN='<prvkey>'
# set if signing key passphrase differs from encryption (key) passphrase
# NOTE: available since duplicity 0.6.14, translates to SIGN_PASSPHRASE
#GPG_PW_SIGN='<signpass>'

# gpg options passed from duplicity to gpg process (default='')
# e.g. "--trust-model pgp|classic|direct|always"
#   or "--compress-algo=bzip2 --bzip2-compress-level=9"
#   or "--personal-cipher-preferences AES256,AES192,AES..."
#GPG_OPTS=''

# disable preliminary tests with the following setting
#GPG_TEST='disabled'

# credentials & server address of the backup target (URL-Format)
# syntax is
#   scheme://[user:password@]host[:port]/[/]path
# probably one out of
#   file://[/absolute_]path
#   ftp[s]://user[:password]@other.host[:port]/some_dir
#   hsi://user[:password]@other.host/some_dir
#   cf+http://container_name
#   imap[s]://user[:password]@host.com[/from_address_prefix]
#   rsync://user[:password]@other.host[:port]::/module/some_dir
#   # rsync over ssh (only keyauth)
#   rsync://user@other.host[:port]/relative_path
#   rsync://user@other.host[:port]//absolute_path
#   # for the s3 user/password are AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
#   s3://[user:password]@host/bucket_name[/prefix]
#   s3+http://[user:password]@bucket_name[/prefix]
#   # scp and sftp are aliases for the ssh backend
#   ssh://user[:password]@other.host[:port]/some_dir
#   tahoe://alias/directory
#   webdav[s]://user[:password]@other.host/some_dir
# ATTENTION: characters other than A-Za-z0-9.-_.~ in user,password,path have
#            to be replaced by their url encoded pendants, see
#            http://en.wikipedia.org/wiki/Url_encoding
#            if you define the credentials as TARGET_USER, TARGET_PASS below
#            duply will url_encode them for you
#TARGET='scheme://user[:password]@host[:port]/[/]path'
TARGET='ftp://backupuser@backup.example.com/'
# optionally the username/password can be defined as extra variables
# setting them here _and_ in TARGET results in an error
#TARGET_USER='_backend_username_'
TARGET_PASS='secret'

# base directory to backup
SOURCE='/home/exampleuser'

# exclude folders containing exclusion file (since duplicity 0.5.14)
# Uncomment the following two lines to enable this setting.
#FILENAME='.duplicity-ignore'
#DUPL_PARAMS="$DUPL_PARAMS --exclude-if-present '$FILENAME'"

# Time frame for old backups to keep, Used for the "purge" command.
# see duplicity man page, chapter TIME_FORMATS)
MAX_AGE=1M

# Number of full backups to keep. Used for the "purge-full" command.
# See duplicity man page, action "remove-all-but-n-full".
#MAX_FULL_BACKUPS=1

# activates duplicity --full-if-older-than option (since duplicity v0.4.4.RC3)
# forces a full backup if last full backup reaches a specified age, for the
# format of MAX_FULLBKP_AGE see duplicity man page, chapter TIME_FORMATS
# Uncomment the following two lines to enable this setting.
#MAX_FULLBKP_AGE=1M
#DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "

# sets duplicity --volsize option (available since v0.4.3.RC7)
# set the size of backup chunks to VOLSIZE MB instead of the default 25MB.
# VOLSIZE must be number of MB's to set the volume size to.
# Uncomment the following two lines to enable this setting.
#VOLSIZE=50
#DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "

# verbosity of output (error 0, warning 1-2, notice 3-4, info 5-8, debug 9)
# default is 4, if not set
VERBOSITY=4

# temporary file space. at least the size of the biggest file in backup
# for a successful restoration process. (default is '/tmp', if not set)
TEMP_DIR=/tmp

# Modifies archive-dir option (since 0.6.0) Defines a folder that holds
# unencrypted meta data of the backup, enabling new incrementals without the
# need to decrypt backend metadata first. If empty or deleted somehow, the
# private key and it's password are needed.
# NOTE: This is confidential data. Put it somewhere safe. It can grow quite
#       big over time so you might want to put it not in the home dir.
# default '~/.cache/duplicity/duply_<profile>/'
# if set  '${ARCH_DIR}/<profile>'
#ARCH_DIR=/some/space/safe/.duply-cache

# DEPRECATED setting
# sets duplicity --time-separator option (since v0.4.4.RC2) to allow users
# to change the time separator from ':' to another character that will work
# on their system.  HINT: For Windows SMB shares, use --time-separator='_'.
# NOTE: '-' is not valid as it conflicts with date separator.
# ATTENTION: only use this with duplicity < 0.5.10, since then default file
#            naming is compatible and this option is pending depreciation
#DUPL_PARAMS="$DUPL_PARAMS --time-separator _ "

# DEPRECATED setting
# activates duplicity --short-filenames option, when uploading to a file
# system that can't have filenames longer than 30 characters (e.g. Mac OS 8)
# or have problems with ':' as part of the filename (e.g. Microsoft Windows)
# ATTENTION: only use this with duplicity < 0.5.10, later versions default file
#            naming is compatible and this option is pending depreciation
#DUPL_PARAMS="$DUPL_PARAMS --short-filenames "

# more duplicity command line options can be added in the following way
# don't forget to leave a separating space char at the end
#DUPL_PARAMS="$DUPL_PARAMS --put_your_options_here "

We can now create three other files, although that is totally optional:

  • /root/.duply/exampleuser/exclude: contains a list of directories to be excluded from the backup (one directory per line);
  • /root/.duply/exampleuser/pre: contains command(s) to be executed prior to the backup (e.g. create a MySQL database dump);
  • /root/.duply/exampleuser/post: contains command(s) to be executed after the backup.

Here’s a sample /root/.duply/exampleuser/exclude and /root/.duply/exampleuser/pre file (the syntax of /root/.duply/exampleuser/post is the same as in /root/.duply/exampleuser/pre):

vi /root/.duply/exampleuser/exclude

/home/exampleuser/news
/home/exampleuser/messages
/home/exampleuser/tmp

chmod 600 /root/.duply/exampleuser/exclude

vi /root/.duply/exampleuser/pre

/usr/bin/mysqldump --all-databases -u root -pyourrootsqlpassword > /home/exampleuser/db.sql

(It should be noted that the database gets locked during the creation of SQL dump; this might not be an issue for a small web site, but can be a problem for high-traffic web sites – your visitors won’t be able to access database-driven pages during mysqldump. Here’s a link to an interruption-free MySQL backup method: How To Back Up MySQL Databases Without Interrupting MySQL)

/root/.duply/exampleuser/pre must be executable (as well as /root/.duply/exampleuser/post if you choose to create one):

chmod 700 /root/.duply/exampleuser/pre

As I said before, you only need /root/.duply/exampleuser/conf; the other files are optional.

Now duply is ready to be used; to create our backup, we simply run

duply exampleuser backup

If all goes well, you won’t be prompted for a password:

root@server1:~# duply exampleuser backup
Start duply v1.5.7, time is 2012-07-03 21:02:33.
Using profile ‘/root/.duply/exampleuser’.
Using installed duplicity version 0.6.08b, python 2.6.6, gpg 1.4.10 (Home: ~/.gnupg), awk ‘mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan’, bash ‘4.1.5(1)-release (x86_64-pc-linux-gnu)’.
Autoset found secret key of first GPG_KEY entry ‘7C6E958B’ for signing.
Test – Encrypt to 7C6E958B & Sign with 7C6E958B (OK)
Test – Decrypt (OK)
Test – Compare (OK)
Cleanup – Delete ‘/tmp/duply.4161.1341342154_*'(OK)

— Start running command PRE at 21:02:34.459 —
Running ‘/root/.duply/exampleuser/pre’ – OK
— Finished state OK at 21:02:34.535 – Runtime 00:00:00.075 —

— Start running command BKP at 21:02:34.599 —
NcFTP version is 3.2.4
Reading globbing filelist /root/.duply/exampleuser/exclude
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Jul 3 20:47:50 2012
————–[ Backup Statistics ]————–
StartTime 1341342156.07 (Tue Jul 3 21:02:36 2012)
EndTime 1341342156.13 (Tue Jul 3 21:02:36 2012)
ElapsedTime 0.06 (0.06 seconds)
SourceFiles 50
SourceFileSize 13490043 (12.9 MB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 0
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 716 (716 bytes)
Errors 0
————————————————-

— Finished state OK at 21:02:37.483 – Runtime 00:00:02.884 —

— Start running command POST at 21:02:37.556 —
Skipping n/a script ‘/root/.duply/exampleuser/post’.
— Finished state OK at 21:02:37.636 – Runtime 00:00:00.080 —
root@server1:~#

To automate the backups, we can create a cron job (I create two cron jobs, one that runs daily and creates incremental backups, and one that runs once a month, creates a full backup and deletes old files):

crontab -e

# run the (incremental) backup each night at 03:23h
23 3 * * * /usr/local/bin/duply exampleuser backup

# do a full backup once per month & delete old backups
47 4 1 * * /usr/local/bin/duply exampleuser full && /usr/local/bin/duply exampleuser purge --force

 

6 Restoring A Backup With duply

Of course, you can use duply also to restore a backup (again, you won’t be asked for a password). The syntax is very easy:

duply exampleuser restore /home/exampleuser

You can take a look at the duply help to learn what other options you have:

duply -h

root@server1:~# duply -h
VERSION:
duply version 1.5.7
(http://duply.net)

DESCRIPTION:
Duply deals as a wrapper for the mighty duplicity magic.
It simplifies running duplicity with cron or on command line by:

– keeping recurring settings in profiles per backup job
– enabling batch operations eg. backup_verify_purge
– executing pre/post scripts for every command
– precondition checking for flawless duplicity operation

For each backup job one configuration profile must be created.
The profile folder will be stored under ‘~/.duply/<profile>’
(where ~ is the current users home directory).
Hint:
If the folder ‘/etc/duply’ exists, the profiles for the super
user root will be searched & created there.

USAGE:
first time usage (profile creation):
duply <profile> create

general usage in single or batch mode (see EXAMPLES):
duply <profile> <command>[_<command>_…] [<options> …]

Non duply options are passed on to duplicity (see OPTIONS).
All conf parameters can also be defined in the environment instead.

PROFILE:
Indicated by a path or a profile name (<profile>), which is resolved
to ‘~/.duply/<profile>’ (~ expands to environment variable $HOME).

Superuser root can place profiles under ‘/etc/duply’. Simply create
the folder manually before running duply as superuser.
Note:
Already existing profiles in root’s profile folder will cease to work
unless there are moved to the new location manually.

example 1:   duply humbug backup

Alternatively a _path_ might be used e.g. useful for quick testing,
restoring or exotic locations. Shell expansion should work as usual.
Hint:
The path must contain at least one path separator ‘/’,
e.g. ‘./test’ instead of only ‘test’.

example 2:   duply ~/.duply/humbug backup

COMMANDS:
usage      get usage help text

create     creates a configuration profile
backup     backup with pre/post script execution (batch: pre_bkp_post),
full (if full_if_older matches or no earlier backup is found)
incremental (in all other cases)
pre/post   execute ‘<profile>/pre’, ‘<profile>/post’ scripts
bkp        as above but without executing pre/post scripts
full       force full backup
incr       force incremental backup
list [<age>]
list all files in backup (as it was at <age>, default: now)
status     prints backup sets and chains currently in repository
verify     list files changed since latest backup
restore <target_path> [<age>]
restore the complete backup to <target_path> [as it was at <age>]
fetch <src_path> <target_path> [<age>]
fetch single file/folder from backup [as it was at <age>]
purge [<max_age>] [–force]
list outdated backup files (older than $MAX_AGE)
[use –force to actually delete these files]
purge-full [<max_full_backups>] [–force]
list outdated backup files ($MAX_FULL_BACKUPS being the number of
full backups and associated incrementals to keep, counting in
reverse chronological order)
[use –force to actually delete these files]
cleanup [–force]
list broken backup chain files archives (e.g. after unfinished run)
[use –force to actually delete these files]

changelog  print changelog / todo list
txt2man    feature for package maintainers – create a manpage based on the
usage output. download txt2man from http://mvertes.free.fr/, put
it in the PATH and run ‘duply txt2man’ to create a man page.

OPTIONS:
–force    passed to duplicity (see commands: purge, purge-full, cleanup)
–preview  do nothing but print out generated duplicity command lines
–disable-encryption
disable encryption, overrides profile settings

PRE/POST SCRIPTS:
All internal duply variables will be readable in the scripts.
Some of interest might be

CONFDIR, SOURCE, TARGET_URL_<PROT|HOSTPATH|USER|PASS>,
GPG_<KEYS_ENC|KEY_SIGN|PW>, CMD_<PREV|NEXT>

The CMD_* variables were introduced to allow different actions according to
the command the scripts were attached to e.g. ‘pre_bkp_post_pre_verify_post’
will call the pre script two times, with CMD_NEXT variable set to ‘bkp’
on the first and to ‘verify’ on the second run.

EXAMPLES:
create profile ‘humbug’:
duply humbug create (now edit the resulting conf file)
backup ‘humbug’ now:
duply humbug backup
list available backup sets of profile ‘humbug’:
duply humbug status
list and delete obsolete backup archives of ‘humbug’:
duply humbug purge –force
restore latest backup of ‘humbug’ to /mnt/restore:
duply humbug restore /mnt/restore
restore /etc/passwd of ‘humbug’ from 4 days ago to /root/pw:
duply humbug fetch etc/passwd /root/pw 4D
(see “duplicity manpage”, section TIME FORMATS)
a one line batch job on ‘humbug’ for cron execution:
duply humbug backup_verify_purge –force

FILES:
in profile folder ‘~/.duply/<profile>’ or ‘/etc/duply’
conf             profile configuration file
pre,post         pre/post scripts (see above for details)
gpgkey.*.asc     exported GPG key files
exclude          a globbing list of included or excluded files/folders
(see “duplicity manpage”, section FILE SELECTION)

IMPORTANT:
Copy the _whole_ profile folder after the first backup to a safe place.
It contains everything needed to restore your backups. You will need
it if you have to restore the backup from another system (e.g. after a
system crash). Keep access to these files restricted as they contain
_all_ informations (gpg data, ftp data) to access and modify your backups.

Repeat this step after _all_ configuration changes. Some configuration
options are crucial for restoration.

SEE ALSO:
duplicity man page:
duplicity(1) or http://duplicity.nongnu.org/duplicity.1.html
root@server1:~#

  • duplicity: http://duplicity.nongnu.org/
  • duply: http://duply.net/
  • Debian: http://www.debian.org/

Comments

comments