2022-11-13 duplicity backups to Wasabi storage from unix

For multiple reasons, I picked Wasabi for my home backup cloud storage, after killing too many hard disks. Hunting around for decent backup software left me somewhat dispirited, but I settled on duplicity, which had the feature set I was after, and performed reasonably after getting the boto3 backend working.

The configuration for backing up to s3 compatible storage is a little daunting, and I've ended up with a "dup" wrapper that has all the options I'm after.

Note that I'm backing up from NetBSD, with software installed from pkgsrc, but this should work equally well from any similar Linux, FreeBSD, OpenBSD, etc, Unix-like system.

I've also not bothered to encrypt my backups - enabling that is left as an exercise to the reader.

I have also embedded the region name into the bucket name. I was playing with buckets in a few different regions, and this was the easiest way to disambigute them.

"dup" duplicity wrapper

Tweak for your use-case.

#!/bin/ksh

TS="date +%Y%m%d-%H%M%S"
RC=0
export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
FLAGS="--archive-dir /home/duplicity"
FLAGS="${FLAGS} --s3-use-new-style"
FLAGS="${FLAGS} --asynchronous-upload"
FLAGS="${FLAGS} --no-encryption"
FLAGS="${FLAGS} --exclude-other-filesystems"
FLAGS="${FLAGS} --exclude-device-files"
FLAGS="${FLAGS} --exclude-if-present .nobackup"
FLAGS="${FLAGS} --full-if-older-than 30D --no-encryption"

# https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-storage-regions-
# HOST=s3.us-east-1.wasabisys.com       # N. Virginia
# HOST=s3.us-west-1.wasabisys.com       # Oregon
# HOST=s3.ap-northeast-1.wasabisys.com  # Tokyo
# HOST=s3.ap-northeast-2.wasabisys.com  # Osaka
HOST=s3.ap-southeast-2.wasabisys.com    # Sydney
FLAGS="${FLAGS} --s3-endpoint-url https://${HOST}"
BUCKET=XXXXXXXX-duplicity-ap-southeast-2
ARC="boto3+s3://${BUCKET}"

function usage {
        echo "dup [--dry-run] {cmd} {args} ..." >&2
        echo "" >&2
        echo "Examples:" >&2
        echo "dup inc /home ARC/home" >&2
        echo "dup full /home ARC/home" >&2
        echo "dup full / ARC/root" >&2
        echo "dup list-current-files ARC/home" >&2
        echo "dup collection-status ARC/home" >&2
        echo "dup cleanup --force ARC/home" >&2
        echo "dup remove-all-but-n-full 2 --force ARC/home" >&2
        echo "dup --file-to-restore httpd/access_log restore ARC/home /tmp/restored_file" >&2
        echo "" >&2
        echo "ARC is substituted with \"${ARC}\"" >&2
        exit 1
}

case "$1" in
        help|-?|-h) usage;;
esac

CMD="/usr/pkg/bin/duplicity ${FLAGS}"

# Substitute any arg "ARC" by "${ARC}".
for i in "$@"; do
        if [ "${i}" == "ARC" ]; then
                echo "ARC given with no directory" >&2
                exit
        fi
        if [ "${i%%/*}" == "ARC" ]; then
                CMD="${CMD} \"${ARC}${i##ARC}\""
        else
                CMD="${CMD} \"${i}\""
        fi
done

echo "$($TS) Starting"
echo "$($TS) cmd: ${CMD}"
if [ $DRYRUN ]; then
        echo "$($TS) Not running, dryrun mode."
else
        echo "$($TS) Running..."
        eval $CMD
        RC=$?
fi
echo "$($TS) Complete."

exit $RC

Sample usage

Backing up the USB stick from my Prusa Mini 3d printer mounted under /mnt/sd0e. I'm passing --allow-source-mismatch since the mount point tends to change with what else I have mounted. In this case, nothing needed backing up, and the last full backup is recent enough.

ksh$ sudo dup incr --progress --allow-source-mismatch /mnt/sd0e ARC/prusa
20221113-143700 Starting
20221113-143700 cmd: /usr/pkg/bin/duplicity --archive-dir /home/duplicity --s3-use-new-style --asynchronous-upload --no-encryption --exclude-other-filesystems --exclude-device-files --exclude-if-present .nobackup --full-if-older-than 30D --no-encryption --s3-endpoint-url https://s3.ap-southeast-2.wasabisys.com "incr" "--progress" "--allow-source-mismatch" "/mnt/sd0e" "boto3+s3://XXXXXXXX-duplicity-ap-southeast-2/prusa"
20221113-143700 Running...
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Oct 30 14:41:43 2022
0.2KB 00:00:03 [0.0KB/s] [========================================>] 100% ETA 0sec
--------------[ Backup Statistics ]--------------
StartTime 1668310624.24 (Sun Nov 13 14:37:04 2022)
EndTime 1668310624.32 (Sun Nov 13 14:37:04 2022)
ElapsedTime 0.08 (0.08 seconds)
SourceFiles 170
SourceFileSize 954561788 (910 MB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 0
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 20 (20 bytes)
Errors 0
-------------------------------------------------

20221113-143708 Complete.

Look for and clean up old backup chains.

ksh$ sudo dup remove-all-but-n-full 2 --progress --force ARC/prusa
20221113-144159 Starting
20221113-144159 cmd: /usr/pkg/bin/duplicity --archive-dir /home/duplicity --s3-use-new-style --asynchronous-upload --no-encryption --exclude-other-filesystems --exclude-device-files --exclude-if-present .nobackup --full-if-older-than 30D --no-encryption --s3-endpoint-url https://s3.ap-southeast-2.wasabisys.com "remove-all-but-n-full" "2" "--progress" "--force" "boto3+s3://XXXXXXXX-duplicity-ap-southeast-2/prusa"
20221113-144159 Running...
Last full backup date: Sun Oct 30 14:41:43 2022
No old backup sets found, nothing deleted.
20221113-144201 Complete.