Automated Encrypted Backups On Ubuntu With Duplicity, With and Without Déjà Dup
By way of a preface, I should note that while I'm working on Ubuntu 13.04 for the purposes of this post, what I outline below for Duplicity should work on any modern Linux, barring the Ubuntu-specific Debian-style package management commands. Any Linux will come with SSH and Duplicity can be installed if it isn't there already. The alternate approaches that use Déjà Dup are somewhat Ubuntu specific, however.
- Duplicity Setup and Scripts
- Setting Up a Cron Job
- An Alternative Approach #1: Déjà Dup, Local Archiving, Rsync
- An Alternative Approach #2: Déjà Dup Plus SFTP
- Déjà Dup and Root
- The Existence of Prying Eyes
Modern Ubuntu versions (12.04, 13.04, etc) come with Déjà Dup, a graphical frontend to the Duplicity command line tool for creating incremental encrypted backups and shipping those backups to remote locations. (And of course restoring from those backups when the inevitable happens). You launch Déjà Dup configuration from System Settings -> Backup, and each user maintains their own settings for what is to be backed up. Unfortunately Déjà Dup, while a very nice piece of work, needs a little massaging to make it happy with private key authentication for sending backup files to a remote location over SSH/SCP/SFTP.
So I put Déjà Dup to one side and spent a little time learning the wonderful world of command line Duplicity, followed by the wonderful world of bash scripting for encryption and private key authentication while avoiding prompts for user entry of passwords and passphrases. This is actually a somewhat frustrating arena if you don't happen to have a recipe to hand immediately. While I think there's a better solution than the scripts provided in this post just a little further down the path, one that involves creating and managing my own GPG keys for encryption with Duplicity, time constraints and the cost-benefit balance of annoyance versus utility versus learning versus getting it done eventually pushed me in the direction of putting the passphrase for encryption into an included file.
The particular use case here is an Ubuntu 13.04 machine with full disk encryption and screen-locking on idle that is only used by myself. Encrypted backups are pushed out to a cloud server. The passphrase for encryption is not shared by any of my other systems. So I feel relatively comfortable having this passphrase in the clear in its own isolated file off to one side of a script directory. It's no worse than any of the other sensitive information or important key files that might be present on a personal or work machine.
Duplicity Scripts and Setup
Create a folder, e.g. /home/username/backup-duplicity, and place the following files into it. These are set up to back up /home/username and /etc while skipping some of the unwanted directories under the /home/username, such as cache, trash, and downloads.
# A script to set values, included in other scripts. # Set these values to be appropriate to your use case. # # The destination for files is example.com, logging in as the ubuntu user. # Note that the path is under the user login directory, so it'll be # /home/ubuntu/backup in this case. ROOT_DESTINATION="ssh://firstname.lastname@example.org/backup" # The location of the private key used to authenticate with the server. PRIVATE_KEY="/home/username/backup-duplicity/example_id_rsa" # Build up ssh options to keep it obvious what we're doing - skipping check # on the host key to avoid a dialog that will cause the process to hang, and # setting the key. SSH_OPTIONS="-oStrictHostKeyChecking=no" SSH_OPTIONS="$SSH_OPTIONS -oIdentityFile=$PRIVATE_KEY" SSH_OPTIONS="$SSH_OPTIONS -oUserKnownHostsFile=/dev/null"
# The passphrase used to encrypt the files. Set this before using the scripts, # as backups will fail without it. # # Don't use core and important pass phrases here; any passphrase that is # in cleartext, even on a system with full disk encryption, should be # disposable. The purpose is just to ensure that backups can be pushed out to # the cloud with the expectation than people can't get into your data. export PASSPHRASE=""
+ /home/username - /home/username/Downloads - /home/username/.cache - /home/username/.local/share/Trash
#!/bin/bash # # Run a backup using the Duplicity tool. # # This creates an encrypted full or incremental backup and sends it to # example.com, assuming that this machine is properly set up with the # relevant keys. # # To set this up for use: # # 1) Set the PASSPHRASE variable in backup-passphrase.sh # 2) Set this script to run daily via cron. # # For more on Duplicity, see notes at: # # http://duplicity.nongnu.org/ # https://help.ubuntu.com/community/DuplicityBackupHowto # # Note the use of pexpect for the ssh-backend - this is so that all ssh options # work, as it just passes through to system ssh. The other option for the # ssh-backend only pays attention to oIdentityFile, which isn't so helpful. # Set needed variables by including a script in the same directory. DIR="$( cd "$( dirname "$0" )" && pwd)" source $DIR/backup-env.sh source $DIR/backup-passphrase.sh # Run a full backup every month, incremental otherwise. # Clean the remote backup space by deleting backups older than 6 months. # # Backup locations: echo "----------------------------" echo "Run backup for /home/username:" echo "----------------------------" DESTINATION="$ROOT_DESTINATION/home-username" duplicity --full-if-older-than 1M \ --ssh-backend=pexpect --ssh-options "$SSH_OPTIONS" \ --include-filelist $DIR/include-home-username \ /home/username $DESTINATION duplicity remove-older-than 6M --force \ --ssh-backend=pexpect --ssh-options "$SSH_OPTIONS" \ $DESTINATION echo "--------------------" echo "Run backup for /etc:" echo "--------------------" DESTINATION="$ROOT_DESTINATION/etc" duplicity --full-if-older-than 1M \ --ssh-backend=pexpect --ssh-options "$SSH_OPTIONS" \ /etc $DESTINATION duplicity remove-older-than 6M --force \ --ssh-backend=pexpect --ssh-options "$SSH_OPTIONS" \ $DESTINATION unset PASSPHRASE
#!/bin/bash # # Verify the latest backup made with the Duplicity tool. # # See run-backup.sh for notes on setup and options. # Set needed variables by loading from a script in the same directory DIR="$( cd "$( dirname "$0" )" && pwd)" source $DIR/backup-env.sh source $DIR/backup-passphrase.sh # Do the verification. echo "-----------------------------------------" echo "Verifying last backup for /home/username:" echo "-----------------------------------------" DESTINATION="$ROOT_DESTINATION/home-username" duplicity verify \ --ssh-backend=pexpect --ssh-options "$SSH_OPTIONS" \ --include-filelist $DIR/include-home-username \ $DESTINATION /home/username echo "-------------------------------" echo "Verifying last backup for /etc:" echo "-------------------------------" DESTINATION="$ROOT_DESTINATION/etc" duplicity verify \ --ssh-backend=pexpect --ssh-options "$SSH_OPTIONS" \ $DESTINATION /etc unset PASSPHRASE
The python-pexpect package is required and should be present on your system. If not, then it's easy enough to install:
sudo apt-get install python-pexpect
Given these scripts, you'll need to update the various settings in backup-env.sh to reflect your situation, alter the set of directories you want to include and exclude in the user directory in include-home-username, and set a passphrase in backup-passphrase.sh. You can then test to see if it works:
cd /home/username/backup-duplicity ./run-backup.sh
Setting Up a Cron Job
To get this backup script running on a daily basis, create a script file /etc/cron.daily/run-duplicity-backup:
#!/bin/bash # # Run the duplicity backup to push files out to the designated remote server. # # See run-backup.sh for notes, and see the general setup doc for what has to # happen in terms of prior setup. # /home/username/backup-duplicity/run-backup.sh &>> /var/log/duplicity/backup.log
You'll have to create the log folder:
sudo mkdir /var/log/duplicity
And there you have it, all done. If you need to restore, you'll be using the command line: take a look at the Duplicity documentation and man page for instructions on restoring all of a backed up folder or just a single file. Don't lose your passphrase! There's something to be said for putting a copy of backup-passphrase.sh into an encrypted archive and storing that somewhere else. Just in case.
An Alternative Approach #1: Déjà Dup, Local Archiving, Rsync
An alternative to the methodology above is to keep Déjà Dup on your Ubuntu system, the default GUI for Duplicity, and set up automated encrypted backups to that write to a local folder. Then create a cron job script that uses rsync to send those files to a remote server. This has some advantages and some disadvantages: you don't have to figure out Duplicity, but on the other hand you don't get to use all of its features and settings. It's arguably easier to get rsync to play nice with your private keys and SSH logins than it is to do the same with Duplicity. Your backup is now a two-stage process, so restoration requires some manual intervention to rsync your files back from where they are now. Also, you can't back up portions of the system such as /etc via the GUI while logged in as your own user - the backup runs under your user, so doesn't have suitable read permissions.
An Alternative Approach #2: Déjà Dup Plus SFTP
It is possible to use the Custom Location storage option in Déjà Dup in conjunction with an SSH config Host definition to ship your backups to a remote server with private key authentication.
Firstly, append this to /etc/ssh/ssh_config.
# Login to example.com for backups. # # e.g. ssh host-backup-example # # Turning off strict key checking prevents annoying hangups or # failures due to the key not being present (i.e. no login to that # server has been made, or the server is a cloud instance and has # been rebuilt, etc). # # If someone performs a MITM attack, then they get your encrypted # files. The only people likely to do this probably already have access # to the storage medium holding your backups, so it seems like a fair # tradeoff to remove the silent failure of backups due to host key # mismatch issues. # Host host-backup-example User ubuntu HostName example.com IdentityFile /home/username/.ssh/example_id_rsa StrictHostKeyChecking no UserKnownHostsFile /dev/null
Why the central ssh_config rather than the per-user /home/username/.ssh/config? It's a change in one place rather than several - your scenario might be different, but here I want to make the Host definition available to both my user and root, and possibly some other software or scripts that will run as root, some of which will choose to ignore anything declared in /root/.ssh/config. I'm not too concerned about the inability of other users or use cases to access the private key file.
- Logged in as username, start up the Déjà Dup preferences via System Settings -> Backup.
- Select the folders in /home/username to be backed up or excluded.
- Set the schedule: for example daily backups and keep backups for 6 months. Your options here are fairly limited.
- Under storage, select "Custom Location" and enter a URL of this form:
Here we are using the Host definition placed into /etc/ssh/ssh_config. Note that the path to the backup folders has to be absolute, and since we are logging in as the ubuntu user, it has to start with /home/ubuntu. These folders will be created in the backup process if they don't already exist.
Now head back to the main settings page for Déjà Dup and turn on backups. Then click the "Back Up Now" button. When the backup starts you will be prompted to choose a passphrase. Enter your passphrase: the same rules apply for this passphrase as those that have to sit in config files, which is to say don't use a passphrase shared with other systems. Since it's accessible in the system you should make it disposable and unique to this setup - so that it doesn't matter as much if it gets compromised, since it is only protecting the backups for this machine.
Déjà Dup and Root
The two alternative approaches above with the Déjà Dup GUI don't allow backup of things like /etc, where the permissions really require the root user to do the job. You can in fact easily configure the root user Déjà Dup preferences through the GUI while logged in as your own user by launching it as follows:
sudo su - deja-dup-preferences
But don't do this! It will cause you all sorts of issues going forward by associating Gnome runtime configuration and other items with the root user. Another problem here is actually running the backup via Déjà Dup as root. This requires a Gnome keyring to exist for root, or it will fail in an annoying fashion - popping up a message and request to create a default Gnome keyring by entering a passphrase, and not giving you anywhere near long enough to actually do that before failing silently.
Gnome is very much set up under the assumption that root will never log in or use Gnome directly, however. So some hacking might have to happen in order to get this to work, and I'm not completely comfortable with what seems to be involved there. So for backups that have to run as root, I'd probably fall back to the script and cron approach based on using Duplicity directly: that doesn't have to involve Gnome at all, and thus can avoid any of the workarounds or issues associated with Gnome and root. This is pretty much what the Gnome developers want - for you to entirely avoid using the desktop UI system with the root user.
The Existence of Prying Eyes
As should be fairly clear by this point, a large fraction of internet traffic and no doubt the contents of cloud servers are being copied and stored for posterity by various government organizations around the world. Assume that your offsite backups sent to a convenient cloud server are going to wind up in a government data warehouse, and pick your encryption methodologies and passphrase lengths accordingly. Longer is better.