Comprehensive Backup Solution for Client-Server Environments

Version: 0.3 License: AGPL-3.0-or-later Author: Lars Duesing lars.duesing@camelotsweb.de Date: 2025-08-03

Table of Contents

1. Overview
2. Project Goals
3. General Backup Workflow
   • 3.1 Stage 1: Client-Side Snapshot Creation (local_backup.sh)
   • 3.2 Stage 2: Server-Side Fetching and Restic Storage (backup_server.sh)
   • 3.3 Stage 3: Server-Side Restic Repository Maintenance (restic_maintenance.sh)
4. Quick Start Guide
5. Build Requirements
6. Building Packages
7. Installation
8. License

IMPORTANT: A Backup is Worthless Without a Tested Recovery

This system is designed to create reliable backups, but a backup strategy is incomplete until the recovery process has been successfully tested. It is essential that you periodically perform recovery drills to ensure you can restore your data when needed.

Do not wait for a disaster to discover a flaw in your backup or recovery plan. See the docs/DISASTER_RECOVERY.md guide for detailed procedures and use the docs/TESTING_RECOVERY.md checklist to conduct regular drills.

1. Overview

This project provides a modular backup solution consisting of three main Debian packages built from this source tree:

1. backup-common: Contains shared configuration files (e.g., /etc/backup/common_config), shared library scripts (e.g., /opt/backup/lib/common_functions.sh), and the base plugin directory structure (/opt/backup/lib/plugins). It serves as a dependency for both the client and server packages.
2. backup-client: Installs the local_backup.sh orchestrator script (to /opt/backup/bin/), client-specific configuration (/etc/backup/client_config.yml), necessary plugins (to /opt/backup/lib/plugins/), and systemd units (local-backup.service, local-backup.timer) for running backups on client machines. This component is responsible for collecting data and creating local TAR archives.
3. backup-server: Installs the backup_server.sh script (to /opt/backup/bin/) for fetching client backups, the restic_maintenance.sh script (to /opt/backup/bin/), server-specific configuration (/etc/backup/server_config.yml), and systemd units (backup-server.service, backup-server.timer, restic-maintenance.service, restic-maintenance.timer) for running the backup server and Restic repository maintenance tasks.

This README.md provides a top-level overview. For specific details about each component, please refer to the README files within the backup-client/, backup-server/, configs/, plugins/, and systemd/ source directories (or their installed locations under /usr/share/doc/).

2. Project Goals

• Modularity: Separate components for client, server, and shared elements.
• Extensibility: A plugin-based architecture for the client to easily add support for backing up new types of services.
• Centralized Pull Model: A server pulls backups from clients.
• Secure Storage: Utilizes Restic on the server for encrypted, deduplicated backups.
• Automation: Systemd timers for scheduling both client backups and server operations.
• Robustness: Includes error handling, logging, file locking (both for script instances and shared directories), and configuration validation.
• Standardization: Aims to follow good shell scripting practices (e.g., Google Shell Style Guide) and Debian packaging conventions.

3. General Backup Workflow

The backup process is a multi-stage operation involving client-side snapshot creation and server-side fetching and storage:

Stage 1: Client-Side Snapshot Creation (local_backup.sh)

1. Initiation & Locking: The local_backup.sh script is typically run by a systemd timer on each client machine. It first acquires an instance lock (flock) to prevent multiple simultaneous runs of itself.
2. Configuration Loading: It loads common defaults (/etc/backup/common_config) and client-specific settings (/etc/backup/client_config.yml). Syntax checks are performed on these files.
3. Service Discovery: The script scans predefined directories (e.g., /etc/backup/docker/, /etc/backup/other/) for service.yaml files. Each service.yaml defines a specific service to be backed up.
4. Plugin Orchestration: For each service:
   • The core script identifies required backup tasks and calls relevant plugins from /opt/backup/lib/plugins/.
   • Plugins perform validation, preparation (e.g., stopping Docker services), execution (e.g., DB dumps, file rsync), and post-backup actions (e.g., restarting services).
5. Shared Directory Lock (TAR Creation): Before creating the final TAR archive, the client script attempts to acquire a directory-based lock (.backup_archive_in_progress.lock) in its BASE_BACKUP_DIR. This prevents the server from attempting to fetch an incomplete archive. Retries are performed if the lock is held.
6. Archive Creation & Verification: All collected data is consolidated into a compressed TAR archive (<CLIENT_HOSTNAME>-<TIMESTAMP>.tar.gz) and its integrity is verified.
7. Lock Release & Cleanup: The shared directory lock is released. Temporary files are removed, and old local TAR archives are pruned.

Stage 2: Server-Side Fetching and Restic Storage (backup_server.sh)

1. Initiation & Locking: The backup_server.sh script is typically run by a systemd timer on the central backup server and acquires its own instance lock (flock).
2. Configuration Loading: It loads common and server-specific configurations, performing syntax checks.
3. Client Iteration: For each configured client host:
   • Client Lock Check: The server connects via SSH and checks for the client-side .backup_archive_in_progress.lock in the client's remote_tar_dir. If present, it retries a few times before skipping the client for this run and sending a warning.
   • Fetch Archive: If no client lock, the server identifies and downloads the latest TAR archive using scp.
   • Unpack Archive: The archive is unpacked into a consistent, clean per-host temporary directory on the server.
   • Restic Newer Snapshot Check: Compares the TAR timestamp (via Restic tag) with existing snapshots to avoid redundant backups.
   • Restic Backup: restic backup --ignore-inode --host <client_hostname> is executed.
   • Remote & Local Cleanup: Processed TAR on client is managed; server-side temporary directories are removed.
4. Reporting: Per-host failures/warnings are logged and emailed. A final summary email details the status of all hosts.

Stage 3: Server-Side Restic Repository Maintenance (restic_maintenance.sh)

1. Initiation & Locking: Run by a separate timer, acquires its own instance lock.
2. Maintenance Tasks: For each Restic repository, it performs restic forget --prune (with per-host policy override) and restic check.
3. Reporting: Logs actions and sends a detailed email on failure, listing all repositories that failed maintenance.

This multi-stage process aims for reliable data collection, secure transfer, and efficient, encrypted storage.

4. Quick Start Guide

This guide provides the essential steps to set up a backup for a single client machine.

Step 1: Install Packages

Follow the build instructions in debian/README.md to create the .deb packages. Then, install them.

• On the Backup Server:

  sudo apt install ./backup-common*.deb ./backup-server*.deb

• On the Client Machine:

  sudo apt install ./backup-common*.deb ./backup-client*.deb

Step 2: Server-Side Setup

1. Create a User for Backups: Create a user on the client that the server will use to connect via SSH. This user needs read access to the data to be backed up and write access to the client's base_backup_dir.

2. Configure SSH Access:

   • On the server, generate an SSH key for the root user: sudo ssh-keygen -t ed25519.
   • Copy the public key (/root/.ssh/id_ed25519.pub) to the client's backup user's authorized keys file (e.g., /home/backupuser/.ssh/authorized_keys).
   • Test the connection from the server: sudo ssh backupuser@client-hostname.

3. Initialize Restic Repository:

   • On the server, create a directory for the client's repository: sudo mkdir -p /srv/restic_repos/client-hostname.

   • Edit /etc/backup/server_config.yml and set the restic.password_file path. Create this file and add a strong password to it (sudo chmod 600 /etc/backup/restic.pass).

   • Initialize the repo:

     sudo restic -r /srv/restic_repos/client-hostname --password-file /etc/backup/restic.pass init

4. Configure server_config.yml:

   • Edit /etc/backup/server_config.yml on the server.
   • Add the client to the hosts list, specifying its hostname, SSH user, and the remote directory where it stores TAR archives.

Step 3: Client-Side Setup

1. Configure client_config.yml:

   • On the client, edit /etc/backup/client_config.yml.
   • Set base_backup_dir (e.g., /var/tmp/backups), backup_user, and backup_group. The backup_user should be the same user the server connects as.

2. Create a Service Backup:

   • On the client, create a directory and a service.yaml file to define what to back up. For example, to back up /var/log:

     sudo mkdir -p /etc/backup/system/logs
     sudo nano /etc/backup/system/logs/service.yaml

   • Add the following content to service.yaml:

     service:
       name: "SystemLogs"
     files:
       paths:
         - "/var/log/"

Step 4: Perform a Manual Backup

1. On the client: sudo /opt/backup/bin/local_backup.sh -v
2. On the server: sudo /opt/backup/bin/backup_server.sh -v
3. On the server, verify: sudo restic -r /srv/restic_repos/client-hostname --password-file /etc/backup/restic.pass snapshots

Step 5: Automate

Once the manual run is successful, enable the systemd timers on both machines:

• On the client: sudo systemctl enable --now local-backup.timer
• On the server: sudo systemctl enable --now backup-server.timer restic-maintenance.timer

5. Build Requirements

To build the Debian packages from this source tree, you will need a Debian-based system with the following tools installed:

• dpkg-dev
• debhelper (version 12 or higher recommended)
• devscripts (provides dpkg-buildpackage)

Installation example:

sudo apt update
sudo apt install dpkg-dev debhelper devscripts

6. Building Packages

Detailed instructions for building the Debian packages (backup-common, backup-client, backup-server) are provided in the debian/README.md file within this source tree. This includes preparing the source, updating Debian control files, and running the dpkg-buildpackage command.

7. Installation

Install the generated .deb packages using dpkg -i or preferably apt install ./<package_name>*.deb (as apt handles dependencies).

1. On ALL machines (Client and Server): Install the common package first, as it's a dependency.

   sudo apt install ./backup-common*.deb

2. On CLIENT machines: Install the client package.

   sudo apt install ./backup-client*.deb

3. On the SERVER machine: Install the server package.

   sudo apt install ./backup-server*.deb

Post-installation configuration is required on both client and server machines. Refer to the specific README files installed with the packages (e.g., in /usr/share/doc/backup-client/) and the configuration files in /etc/backup/.

8. License

This project and its components are licensed under the GNU Affero General Public License v3.0 or later. See the LICENSE file for the full text.