System administration

Creating backups using duplicati

Mikhail Kazhekin
· 3 min read

Duplicati is a very simple and straightforward backup tool, which is why I use it. In this example, duplicati will be running in a Docker container.

  1. Installing duplicati
  2. Backing up files
  3. Backing up the database and crontab

Installing duplicates

Let's create a duplicate folder

mkdir duplicati
cd duplicati

In the duplicati folder, create a "backups" folder. This folder will be used for database and crontab backups.

mkdir backups

We'll run duplicati in Docker. Here's an example docker-compose.yml

services:
  duplicati:
    image: lscr.io/linuxserver/duplicati:latest
    container_name: duplicati
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Etc/UTC
      - SETTINGS_ENCRYPTION_KEY=${ENCRYPTION_KEY}
      - CLI_ARGS= #optional
      - DUPLICATI__WEBSERVICE_PASSWORD= #optional
    volumes:
      - /path/to/duplicati/config:/config
      - /path/to/backups:/backups
      - /path/to/source:/source #forward the necessary folders
    ports:
      - 8200:8200
    restart: unless-stopped

docker-compose.yml

Launching the container

docker compose up -d

Creating a backup copy of files

Go to the control panel. Everything is intuitive there.

If you don't want to make backups of hidden files, you need to select hidden in the Exclude files by attribute drop-down list in the options section.

I recommend using smart backup retention because it saves you from having to set it up yourself.

Creating a backup copy of databases and crontab

To create database backups, we'll use a script and crontab. The idea is that we'll run a script in crontab on a schedule to create a database dump and a crontab backup. For example, at 2:00 AM, the db_backup.sh and crontab_backup.sh scripts run and save the dumps to a specific folder. At 2:05 AM, duplicati simply copies files from this specific folder, and at 3:00 AM, we delete this specific folder. And so on, repeating the process.

Create files in the backups folder

cd backups 
touch db_backup.sh && touch crontab_backup.sh && touch delete_backup.sh

We grant permission to execute these files.

sudo chmod +x db_backup.sh && sudo chmod +x crontab_backup.sh && sudo chmod +x delete_backup.sh

Sample code for db_backup.sh. In this example, we create a dump from Docker containers.

#!/bin/bash

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BACKUP_DIR="$SCRIPT_DIR/sql"
mkdir -p "$BACKUP_DIR"

CONTAINERS=("container1" "container2" "and so on")

DB_USER=
DB_PASS=

for CONTAINER in "${CONTAINERS[@]}"; do
    echo "Backup $CONTAINER"
    if [ "$CONTAINER" = "container1" ]; then
        databases=$(docker exec -i "$CONTAINER" mysql -u"$DB_USER" -p"$DB_PASS" -e "SHOW DATABASES;" 2>/dev/null | grep -Ev "^(Database|information_schema|performance_schema|sys|mysql)$" | tr -d '\r')
    else
        databases=$(docker exec -i "$CONTAINER" mariadb -u"$DB_USER" -p"$DB_PASS" -e "SHOW DATABASES;" 2>/dev/null | grep -Ev "^(Database|information_schema|performance_schema|sys|mysql)$" | tr -d '\r')
    fi 
    
    if [ -z "$databases" ]; then
        echo "Error"
        continue
    fi

    for db in $databases; do
        echo "Dump db $db"

        FILENAME="$BACKUP_DIR/${db}-$(date +%Y.%m.%d-%H:%M:%S).sql"
    
        if [ "$db" = "container1" ]; then
            docker exec -i $CONTAINER mysqldump -u "$DB_USER" -p"$DB_PASS" "$db" 2>/dev/null > "$FILENAME"
        else
            docker exec -i $CONTAINER mariadb-dump -u "$DB_USER" -p"$DB_PASS" "$db" 2>/dev/null > "$FILENAME"
        fi
    done
done

exit 0

db_backup.sh

This code fragment is responsible for obtaining a list of tables excluding service tables that are not needed in the backup.

if [ "$CONTAINER" = "container1" ]; then
        databases=$(docker exec -i "$CONTAINER" mysql -u"$DB_USER" -p"$DB_PASS" -e "SHOW DATABASES;" 2>/dev/null | grep -Ev "^(Database|information_schema|performance_schema|sys|mysql)$" | tr -d '\r')
    else
        databases=$(docker exec -i "$CONTAINER" mariadb -u"$DB_USER" -p"$DB_PASS" -e "SHOW DATABASES;" 2>/dev/null | grep -Ev "^(Database|information_schema|performance_schema|sys|mysql)$" | tr -d '\r')
    fi 
    
    if [ -z "$databases" ]; then
        echo "Error"
        continue
    fi

db_backup.sh

This code creates an exception for container1 because it runs on MySQL, so we need to retrieve the tables using the MySQL utility. If you don't need this, simply remove this condition.

This piece of code is directly responsible for creating the dump file.

for db in $databases; do
        echo "Dump db $db"

        FILENAME="$BACKUP_DIR/${db}-$(date +%Y.%m.%d-%H:%M:%S).sql"
    
        if [ "$db" = "container1" ]; then
            docker exec -i $CONTAINER mysqldump -u "$DB_USER" -p"$DB_PASS" "$db" 2>/dev/null > "$FILENAME"
        else
            docker exec -i $CONTAINER mariadb-dump -u "$DB_USER" -p"$DB_PASS" "$db" 2>/dev/null > "$FILENAME"
        fi
    done

db_backup.sh

If you have a database server running directly on the host, then you just need to remove docker exec -i $CONTAINER at the beginning of the command.

Setting up crontab and creating a crontab file dump

The final step is to create a crontab job.

Let's go to crontab

crontab -e

And at the very bottom of the file we insert these instructions.

00 2 * * * /home/user/duplicati/backups/make_dump.sh > /dev/null 2>&1
00 3 * * * /home/user/duplicati/backups/delete_dumps.sh > /dev/null 2>&1
00 2 * * * /home/user/duplicati/backups/crontab_backup.sh > /dev/null 2>&1

Now we paste this code into the crontab_backup.sh file

#!/bin/bash

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BACKUP_DIR="$SCRIPT_DIR/crontabs"
mkdir -p "$BACKUP_DIR"

cd /home/user/duplicati/backups/crontabs
crontab -l  > 'crontab-'$(date +"%Y.%m.%d-%H:%M:%S")'.txt'

crontab_backup.sh

Open the delete_backup.sh file and paste the code

#!/bin/bash
rm -rf /home/user/duplicati/backups/sql
rm -rf /home/user/duplicati/backups/crontabs

delete_backup.sh

I hope everything works out for you!🙃

Related Articles

Setting up your own CoreDNS DNS server in docker

Setting up your own CoreDNS DNS server in docker

· 1 min read