diff --git a/deploy/README.md b/deploy/README.md index 7c8d3d7..5645ac4 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -103,6 +103,9 @@ also `root:root` and `0600`. The app runs as the dedicated `chesstrainer` system user. It must own `/opt/chesstrainer` so it can write `priv/static/backups`. +Database dumps live in `/var/backups/chesstrainer`, outside `/opt/chesstrainer` +and `priv/static`. See §Backups. + ## Copy the deploy artifacts Run these commands from a checkout of this repo on the VPS, or copy the @@ -441,6 +444,8 @@ The unit includes `ConditionPathExists=/opt/chesstrainer/current/bin/server`, so it is safe to enable before the first release. systemd skips it until a release is in place. +The backup unit and timer are installed separately; see §Backups. + ## Caddy Add the site block from `deploy/Caddyfile` by hand. It imports @@ -601,12 +606,13 @@ The script: 1. Extracts into `/opt/chesstrainer/releases/` 2. Fixes ownership -3. Flips `/opt/chesstrainer/current` -4. Restarts `chesstrainer.service` -5. Migrations run via `ExecStartPre` -6. Polls `/health` -7. Rolls back to the previous release on failure -8. Prunes old releases, keeping 5 by default +3. Takes a verified pre-deploy database dump, aborting if it fails (see §Backups) +4. Flips `/opt/chesstrainer/current` +5. Restarts `chesstrainer.service` +6. Migrations run via `ExecStartPre` +7. Polls `/health` +8. Rolls back to the previous release on failure +9. Prunes old releases, keeping 5 by default Verify locally before touching Caddy: @@ -641,6 +647,10 @@ KEEP_RELEASES=5 \ sudo -E /opt/chesstrainer/src/deploy/activate /tmp/chesstrainer-release.tar.gz ``` +Every activation takes a verified pre-deploy dump before migrations run. If +Postgres is intentionally unavailable and you accept the risk, add +`SKIP_PRE_DEPLOY_BACKUP=1` to the override list; see §Backups. + ## GitHub Actions CD Merging to `main` (or running the workflow manually) tests, builds, uploads, @@ -704,7 +714,8 @@ The workflow: 1. Runs `mix precommit` against Postgres. 2. Builds the release artifact on `ubuntu-24.04`. 3. On `main` or `workflow_dispatch`, uploads the tarball, installs the SSH key, - runs `sudo -n .../activate`, and performs public smoke checks. + runs `sudo -n .../activate`, and performs public smoke checks. Activation + takes a pre-deploy dump first and fails if it cannot. 4. A failed activation health check rolls back to the previous release. Manual dispatch is available from the Actions UI or: @@ -727,6 +738,10 @@ sudo /opt/chesstrainer/src/deploy/rollback 20261109120000 The script flips `current`, restarts the service, and checks `/health`. +Rollback reverts the release, not the database: migrations are not reversed. If +a migration changed data, restore the matching pre-deploy dump instead; see +§Backups. + ## Logs and health ```sh @@ -856,52 +871,338 @@ That shell speaks Erlang syntax, so quote Elixir module atoms, e.g. ## Backups -`docker exec` connects over the container's Unix socket, which the Postgres -image trusts locally, so `pg_dump` needs no password. Run it as the owner -(`chesstrainer`), not `chesstrainer_ro`: a full dump also needs sequence access -and ownership information. +Backups are produced on the VPS, encrypted there, and pulled to a workstation. +Nothing leaves the VPS unencrypted, and the VPS never holds the key that could +read the archives. -Take a compressed logical backup (custom format, restorable with `pg_restore`): +### What runs + +| Where | What | When | Retention | +|---|---|---|---| +| VPS | `chesstrainer-backup.timer` → `chesstrainer-backup.service` | daily 03:15 UTC, `Persistent=true`, 5 min jitter | 3 days plaintext, 14 days encrypted | +| VPS | the same script, called by `deploy/activate` | before every migration | last 5 pre-deploy sets | +| Workstation | `chesstrainer-backup-pull.timer` | daily 12:00 local, 10 min after boot | 7 daily, 4 weekly, 6 monthly | +| Workstation | `chesstrainer-backup-drill.timer` | monthly on the 1st at 09:00 | `drills.log` is never pruned | + +Each run writes one JSON line (`backup.completed`, `backup.failed`, +`backup.skipped`, `pull.completed`, `pull.failed`, `pull.skipped`, +`drill.completed`, `drill.failed`) so #140's log shipping can alert on it. A +`skipped` line means another run held the lock; it is not an alarm. + +### What a dump contains + +- `.dump` — `pg_dump -Fc` of `chesstrainer_prod` +- `.globals.sql` — `pg_dumpall --globals-only`, enough to recreate + `chesstrainer` and `chesstrainer_ro` on a fresh cluster +- `.counts` — row counts for `users`, `endgames`, `attempts`, used by the drill +- `.sha256` — checksums for the three files above + +Not backed up, by design: tablebases and the tablebase cache (both are +re-fetchable), anything under `priv/static`, and release tarballs. Dumps contain +user emails and bcrypt hashes: they are `0600 root` while plaintext, encrypted +before they leave the box, and pruned aggressively. + +### RPO and RTO + +RPO is 24 hours on the VPS and "the age of the newest pulled archive" off-box. +The pull runs daily, so a laptop that is off for a day leaves the newest +off-site copy around 48 hours old; the VPS keeps 14 days of encrypted archives, +which bounds how far behind the pull can fall. RTO is under an hour: create the +database, restore the newest dump, activate a release (see *Disaster recovery*). + +### One-time setup: VPS + +Install `age`. Encryption happens on the server; only the workstation can +decrypt. ```sh -sudo docker compose -f exec -T chesstrainer-db \ - pg_dump -U chesstrainer -d chesstrainer_prod -Fc \ - > /var/backups/chesstrainer-$(date -u +%Y%m%d%H%M%S).dump +sudo apt install age ``` -For a plain SQL file instead (human-readable, restorable with `psql`): +Create the configuration file. Generate `AGE_RECIPIENT` on the workstation +first (next section) and paste the public key. ```sh -sudo docker compose -f exec -T chesstrainer-db \ - pg_dump -U chesstrainer -d chesstrainer_prod \ - | gzip > /var/backups/chesstrainer-$(date -u +%Y%m%d%H%M%S).sql.gz +sudo install -m 600 /dev/null /etc/chesstrainer/backup.env +sudoedit /etc/chesstrainer/backup.env ``` -Check that a custom-format dump is readable before trusting it: +Start from `deploy/backup.env.example`. The required values are `AGE_RECIPIENT` +and `HEALTHCHECK_URL`. + +Create a healthchecks.io check with a period of 1 day and a grace time of 4 +hours, then paste its ping URL into `HEALTHCHECK_URL`. The script pings it after +a good run and `/fail` after a bad one, so silence means "no backup +finished", even when the VPS itself is down. + +Install the script and units: ```sh -sudo docker exec -i chesstrainer-db pg_restore -l \ - < /var/backups/.dump > /dev/null +sudo install -m 0755 /opt/chesstrainer/src/deploy/backup \ + /usr/local/sbin/chesstrainer-backup +sudo install -m 0644 \ + /opt/chesstrainer/src/deploy/chesstrainer-backup.service \ + /opt/chesstrainer/src/deploy/chesstrainer-backup.timer \ + /opt/chesstrainer/src/deploy/chesstrainer-backup-failure.service \ + /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now chesstrainer-backup.timer ``` -Restore into a scratch database (do not restore over the live database without -stopping `chesstrainer` first): +The timer runs the installed `/usr/local/sbin/chesstrainer-backup`, not the +copy under `/opt/chesstrainer/src`, because that tree is writable by the +application user. When `deploy/backup` changes, copy it to both places so the +timer and `deploy/activate` use the same script. + +Run it once and inspect the result: ```sh -sudo docker compose -f exec -T chesstrainer-db \ - createdb -U chesstrainer chesstrainer_restore -sudo docker exec -i chesstrainer-db pg_restore -U chesstrainer \ - -d chesstrainer_restore < /var/backups/.dump +sudo systemctl start chesstrainer-backup +sudo journalctl -u chesstrainer-backup -n 20 --no-pager -o cat +sudo ls -l /var/backups/chesstrainer /var/backups/chesstrainer/offsite +systemctl list-timers chesstrainer-backup.timer ``` -Example nightly entry in root's crontab (`sudo crontab -e`): +Expect a `.dump`, `.globals.sql`, `.counts` and `.sha256` at mode `0600`, and +one `.tar.age` in `offsite/` at mode `0644`. `offsite/` is the only thing the +workstation reads, which is why the pull needs no `sudo`. The vault directory is +`0711`, so the workstation user can traverse into `offsite/` but cannot list the +plaintext dumps. -```cron -15 3 * * * docker compose -f /home//docker-compose.yml exec -T chesstrainer-db pg_dump -U chesstrainer -d chesstrainer_prod -Fc > /var/backups/chesstrainer-$(date +\%Y\%m\%d).dump +### One-time setup: workstation + +```sh +sudo apt install age rsync +mkdir -p ~/.config/chesstrainer +chmod 700 ~/.config/chesstrainer +age-keygen -o ~/.config/chesstrainer/age.key +chmod 600 ~/.config/chesstrainer/age.key +grep 'public key' ~/.config/chesstrainer/age.key ``` -Keep `/var/backups` outside `/opt/chesstrainer/releases` and copy backups off -the VPS regularly. A backup that is only on the VPS is not a backup. +Copy that public key into `AGE_RECIPIENT` on the VPS. **Back up the private +key** (password manager, printed copy, offline disk): without it every archive +is unreadable, and the VPS deliberately cannot decrypt them. + +Create the pull configuration: + +```sh +install -m 600 /dev/null ~/.config/chesstrainer/backup-pull.env +$EDITOR ~/.config/chesstrainer/backup-pull.env +``` + +Start from `deploy/workstation/backup-pull.env.example`. `DEST` should point at +a real disk, for example `/mnt/Mint Data/chesstrainer-backups`. The script +refuses to run when `DEST`'s parent is missing, so an unmounted disk cannot +silently fill the root filesystem. + +Set `PULL_HEALTHCHECK_URL` to a second healthchecks.io check with a long period +(for example 72 hours) if you want to hear about it when the workstation stops +pulling. Leave it empty for journal-only reporting. + +Install the scripts and user units: + +```sh +REPO=~/Desktop/chess-trainer # wherever this repo is cloned + +install -d ~/.local/bin +install -m 0755 "$REPO/deploy/workstation/backup-pull" ~/.local/bin/chesstrainer-backup-pull +install -m 0755 "$REPO/deploy/workstation/backup-drill" ~/.local/bin/chesstrainer-backup-drill + +install -d ~/.config/systemd/user +install -m 0644 \ + "$REPO/deploy/workstation/chesstrainer-backup-pull.service" \ + "$REPO/deploy/workstation/chesstrainer-backup-pull.timer" \ + "$REPO/deploy/workstation/chesstrainer-backup-drill.service" \ + "$REPO/deploy/workstation/chesstrainer-backup-drill.timer" \ + "$REPO/deploy/workstation/chesstrainer-backup-alert@.service" \ + ~/.config/systemd/user/ + +systemctl --user daemon-reload +systemctl --user enable --now chesstrainer-backup-pull.timer +systemctl --user enable --now chesstrainer-backup-drill.timer +sudo loginctl enable-linger "$USER" +``` + +`enable-linger` lets the user timers run while the laptop is on but you are not +logged in. The pull also fires ten minutes after boot, and `Persistent=true` +catches up after the machine was off. Verify lingering stuck: + +```sh +loginctl show-user "$USER" -p Linger # Linger=yes +``` + +Run the first pull and drill by hand: + +```sh +~/.local/bin/chesstrainer-backup-pull +ls -l "/mnt/Mint Data/chesstrainer-backups" +~/.local/bin/chesstrainer-backup-drill +cat "/mnt/Mint Data/chesstrainer-backups/drills.log" +``` + +A passing drill line looks like: + +```json +{"timestamp":"2026-11-11T21:05:46Z","event":"drill.completed","archive":"20261111T120000Z.tar.age","result":"pass","counts":"users=3,endgames=42,attempts=900","duration_s":3} +``` + +The drill creates a throwaway `postgres:18.6` container, applies the globals +dump, asserts that `chesstrainer` and `chesstrainer_ro` exist, restores into +`chesstrainer_restore`, and compares the three row counts with the `.counts` +file from dump time. Restored counts may be higher (the counts are taken moments +before the dump); lower counts fail the drill. + +### Retention + +| Location | Plaintext | Encrypted | Notes | +|---|---|---|---| +| VPS `/var/backups/chesstrainer` | 3 days | 14 days | pre-deploy sets: last 5 | +| Workstation `DEST` | never | 7 daily, 4 weekly, 6 monthly | monthly copies outlive the VPS copies | + +The VPS is a staging area for the workstation; weekly and monthly generations +accumulate there. If the laptop is off for more than 14 days, the archives the +VPS dropped in the meantime are gone — that is the documented off-site RPO +trade-off. Moving to an always-on destination (a second host or object storage +with restic) is tracked in #151. + +### Restore a dump + +Always restore into a scratch database first. From an archive on the +workstation: + +```sh +WORK=$(mktemp -d) +age --decrypt -i ~/.config/chesstrainer/age.key \ + -o "$WORK/dump.tar" \ + "/mnt/Mint Data/chesstrainer-backups/.tar.age" +tar -xf "$WORK/dump.tar" -C "$WORK" +(cd "$WORK" && sha256sum --check --strict .sha256) +``` + +Then restore into a scratch container: + +```sh +docker run --rm -d --name restore-check \ + -e POSTGRES_PASSWORD=restore -e POSTGRES_HOST_AUTH_METHOD=trust postgres:18.6 +docker exec restore-check psql -U postgres -c 'create database chesstrainer_restore' +docker exec -i restore-check pg_restore -U postgres -d chesstrainer_restore \ + --single-transaction --exit-on-error < "$WORK/.dump" +docker exec restore-check psql -U postgres -d chesstrainer_restore \ + -c 'select count(*) from users; select count(*) from attempts;' +docker rm -f restore-check +rm -rf "$WORK" +``` + +### Disaster recovery + +1. Build a new VPS following this README through *First-time setup*, without + activating a release or switching Caddy. +2. Decrypt and unpack the newest archive on the workstation (above). +3. Copy the dump and globals to the new VPS: + `scp "$WORK/.dump" "$WORK/.globals.sql" @:/tmp/` +4. Recreate the cluster roles. Connect as `chesstrainer`: it is the container's + superuser (`POSTGRES_USER`), and the dump's final grants are issued + `GRANTED BY chesstrainer`, which another role cannot replay. The line that + recreates `chesstrainer` itself reports "already exists" and is harmless: + `sudo docker exec -i chesstrainer-db psql -U chesstrainer < /tmp/.globals.sql` +5. Create the database and restore it: + `sudo docker exec chesstrainer-db createdb -U chesstrainer chesstrainer_prod` + `sudo docker exec -i chesstrainer-db pg_restore -U chesstrainer -d chesstrainer_prod --single-transaction --exit-on-error < /tmp/.dump` +6. Verify the row counts, activate a release, then restore the Caddy block. +7. Take a fresh backup (`sudo systemctl start chesstrainer-backup`) and pull so + the new VPS is covered immediately. + +`pg_dump` and `pg_restore` must come from the container's Postgres 18, not the +host. + +### Pre-deploy dump + +`deploy/activate` calls `chesstrainer-backup` with `BACKUP_TAG=predeploy` after +extracting the release and before flipping `current`, so the state before +`bin/migrate` runs in `ExecStartPre` is always recoverable. A failed dump aborts +the activation. The escape hatch is deliberate and should be rare: + +```sh +sudo SKIP_PRE_DEPLOY_BACKUP=1 /opt/chesstrainer/src/deploy/activate /tmp/chesstrainer-release.tar.gz +``` + +If the installed script is missing, `activate` uses the `backup` next to it; if +neither exists it refuses to activate rather than migrate without a dump. The +GitHub Actions deploy job inherits the same behaviour. + +### Run it by hand + +Every timer can be triggered directly. On the VPS: + +```sh +# exactly what the timer runs +sudo systemctl start chesstrainer-backup +sudo journalctl -u chesstrainer-backup -n 20 -o cat + +# an ad-hoc dump with its own tag +sudo BACKUP_TAG=manual /usr/local/sbin/chesstrainer-backup + +# schedule, last result and last exit +systemctl list-timers chesstrainer-backup.timer +systemctl show chesstrainer-backup.service -p Result -p ExecMainExitTimestamp +``` + +On the workstation: + +```sh +# pull now, or through the unit (same PATH and OnFailure handling) +~/.local/bin/chesstrainer-backup-pull +systemctl --user start chesstrainer-backup-pull.service + +# drill the newest archive, or a named one +~/.local/bin/chesstrainer-backup-drill +~/.local/bin/chesstrainer-backup-drill 20261111T213734Z.tar.age + +# results and schedule +cat "/mnt/Mint Data/chesstrainer-backups/drills.log" +systemctl --user list-timers 'chesstrainer-backup-*' +``` + +A `backup.skipped` or `pull.skipped` line means another run held the lock; that +run reports the outcome. After fixing a failure, clear the unit with +`systemctl --user reset-failed chesstrainer-backup-pull.service` (or +`sudo systemctl reset-failed chesstrainer-backup.service`). + +Enabling the pull timer fires it once straight away: its `OnBootSec=10m` is +measured from boot, so enabling it hours after boot triggers immediately. That +is expected and harmless. + +### Monitoring + +```sh +systemctl list-timers chesstrainer-backup.timer +sudo journalctl -u chesstrainer-backup --since "2 days ago" -o cat +sudo systemctl --failed +systemctl --user list-timers chesstrainer-backup-pull.timer chesstrainer-backup-drill.timer +journalctl --user -u chesstrainer-backup-pull --since "2 days ago" -o cat +``` + +A failed run produces both `backup.failed` (from the script) and +`backup.unit-failed` (from the `OnFailure` unit). The healthchecks.io check is +the backstop: if the VPS dies, the ping stops and you are emailed. The pull +fails when the newest archive is more than 48 hours old, which catches a stopped +VPS timer the next time the workstation pulls. + +### Troubleshooting + +| Symptom | Likely cause | +|---|---| +| `backup.failed: missing command: age` | `sudo apt install age` | +| `backup.failed: only N MB free` | disk full on the VPS; prune old releases or grow the disk | +| `backup.failed: pg_dump failed` | `chesstrainer-db` is down or restarting — check `docker ps` | +| `backup.failed: globals dump is missing role …` | the role was renamed; update `EXPECTED_ROLES` | +| `pull.failed: rsync … failed` | SSH key, host key, or the VPS is unreachable | +| `pull.failed: newest archive … is Nh old` | the VPS backup timer stopped; check it before touching the workstation | +| `drill.failed: pg_restore failed` | archive is corrupt; keep it for diagnosis and restore an earlier one | + +Never put dumps anywhere under `priv/static` or the release directory, and never +restore over `chesstrainer_prod` while the application is running. ## New VPS checklist @@ -916,6 +1217,11 @@ the VPS regularly. A backup that is only on the VPS is not a backup. 8. Caddy block, `cloudflare-ranges.caddy`, and the CIDR refresh timer added; validated and reloaded. 9. Public `/`, `/privacy`, `/terms-conditions`, and `/health` verified. -10. Read-only role created, tunnel verified, backups scheduled and tested. -11. Remote shell attaches and exits cleanly: +10. Read-only role created and tunnel verified. +11. `age` installed, `/etc/chesstrainer/backup.env` written, backup script and + units installed, timer enabled, and one manual run produces a dump plus a + globals file. +12. Workstation age key and pull configuration installed, user timers and linger + enabled, first pull verified, first drill passes. +13. Remote shell attaches and exits cleanly: `sudo /opt/chesstrainer/current/bin/chesstrainer remote` (Ctrl-C twice). diff --git a/deploy/activate b/deploy/activate index ef71ff0..c68d033 100755 --- a/deploy/activate +++ b/deploy/activate @@ -7,7 +7,8 @@ set -euo pipefail # sudo deploy/activate /tmp/chesstrainer-20261109.tar.gz # # Environment overrides: -# APP_USER, APP_ROOT, SERVICE, HEALTH_URL, KEEP_RELEASES, RELEASE_ID +# APP_USER, APP_ROOT, SERVICE, HEALTH_URL, KEEP_RELEASES, RELEASE_ID, +# BACKUP_BIN, SKIP_PRE_DEPLOY_BACKUP=1 if [[ "${EUID:-$(id -u)}" -ne 0 ]]; then echo "activate: run with sudo" >&2 @@ -26,6 +27,8 @@ SERVICE="${SERVICE:-chesstrainer}" HEALTH_URL="${HEALTH_URL:-http://127.0.0.1:4100/health}" KEEP_RELEASES="${KEEP_RELEASES:-5}" RELEASE_ID="${RELEASE_ID:-$(date -u +%Y%m%d%H%M%S)}" +BACKUP_BIN="${BACKUP_BIN:-/usr/local/sbin/chesstrainer-backup}" +SKIP_PRE_DEPLOY_BACKUP="${SKIP_PRE_DEPLOY_BACKUP:-0}" if [[ ! "$RELEASE_ID" =~ ^[A-Za-z0-9._-]+$ ]]; then echo "activate: RELEASE_ID may contain only letters, digits, dot, underscore and dash" >&2 @@ -53,6 +56,29 @@ install -d -o "$APP_USER" -g "$APP_USER" "$RELEASE_DIR" tar -xzf "$TARBALL" -C "$RELEASE_DIR" chown -R "$APP_USER:$APP_USER" "$RELEASE_DIR" +if [[ "$SKIP_PRE_DEPLOY_BACKUP" == "1" ]]; then + echo "activate: SKIP_PRE_DEPLOY_BACKUP=1, not taking a pre-deploy dump" +else + backup_script="$BACKUP_BIN" + if [[ ! -x "$backup_script" ]]; then + backup_script="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/backup" + fi + + if [[ ! -x "$backup_script" ]]; then + echo "activate: no backup script at $BACKUP_BIN or next to activate" >&2 + echo "activate: install deploy/backup or set SKIP_PRE_DEPLOY_BACKUP=1" >&2 + exit 1 + fi + + # Migrations run in ExecStartPre after this, so the dump must exist first. + echo "activate: taking a pre-deploy dump" + if ! BACKUP_TAG=predeploy LOCK_WAIT=120 "$backup_script"; then + echo "activate: pre-deploy dump failed, aborting" >&2 + echo "activate: fix the backup or re-run with SKIP_PRE_DEPLOY_BACKUP=1" >&2 + exit 1 + fi +fi + echo "activate: pointing current at $RELEASE_DIR" ln -sfn "$RELEASE_DIR" "$APP_ROOT/current" diff --git a/deploy/backup b/deploy/backup new file mode 100755 index 0000000..3bed57a --- /dev/null +++ b/deploy/backup @@ -0,0 +1,266 @@ +#!/usr/bin/env bash +# +# Dump, verify, encrypt, prune and report on the production database. +# +# Installed as /usr/local/sbin/chesstrainer-backup (root-owned 0755) and run by +# chesstrainer-backup.timer. deploy/activate also calls it before migrations. +# Do not run the copy under /opt/chesstrainer/src as root: that tree belongs to +# the application user. +# +# sudo systemctl start chesstrainer-backup # daily run, as the timer does +# sudo BACKUP_TAG=manual chesstrainer-backup # ad-hoc, pruned by age +# +# Configuration lives in /etc/chesstrainer/backup.env (root:root 0600). See +# deploy/README.md §Backups. Every setting can also be passed as an environment +# variable. +# +# Each run prints exactly one JSON line (backup.completed or backup.failed) so +# journald, and #140's log shipping, can pick it up. + +set -euo pipefail +umask 077 + +ENV_FILE="${ENV_FILE:-/etc/chesstrainer/backup.env}" +if [[ -r "$ENV_FILE" ]]; then + # shellcheck disable=SC1090 + source "$ENV_FILE" +fi + +DB_CONTAINER="${DB_CONTAINER:-chesstrainer-db}" +PG_USER="${PG_USER:-chesstrainer}" +PG_DB="${PG_DB:-chesstrainer_prod}" +BACKUP_DIR="${BACKUP_DIR:-/var/backups/chesstrainer}" +EXPECTED_ROLES="${EXPECTED_ROLES:-chesstrainer chesstrainer_ro}" +AGE_RECIPIENT="${AGE_RECIPIENT:-}" +HEALTHCHECK_URL="${HEALTHCHECK_URL:-}" +SKIP_HEARTBEAT="${SKIP_HEARTBEAT:-0}" +MIN_FREE_MB="${MIN_FREE_MB:-2048}" +LOCAL_PLAINTEXT_DAYS="${LOCAL_PLAINTEXT_DAYS:-3}" +LOCAL_ENCRYPTED_DAYS="${LOCAL_ENCRYPTED_DAYS:-14}" +PREDEPLOY_KEEP="${PREDEPLOY_KEEP:-5}" +BACKUP_TAG="${BACKUP_TAG:-daily}" +LOCK_WAIT="${LOCK_WAIT:-0}" + +START_SECONDS=$SECONDS +FAILURE="" + +log_event() { + local event="$1" + local extra="${2:-}" + + printf '{"timestamp":"%s","event":"%s","tag":"%s"%s}\n' \ + "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$event" "$BACKUP_TAG" "$extra" +} + +heartbeat() { + local outcome="$1" + + [[ "$SKIP_HEARTBEAT" == "1" ]] && return 0 + [[ -n "$HEALTHCHECK_URL" ]] || return 0 + + local url="${HEALTHCHECK_URL%/}" + [[ "$outcome" == "fail" ]] && url="$url/fail" + + if ! curl --fail --silent --show-error \ + --max-time 10 --retry 2 --retry-delay 2 \ + --output /dev/null "$url"; then + echo "backup: heartbeat ping failed (run result is unaffected)" >&2 + fi +} + +fail() { + FAILURE="$*" + exit 1 +} + +on_exit() { + local status=$? + + if ((status != 0)); then + find "$BACKUP_DIR" -maxdepth 1 -name '*.partial' -delete 2>/dev/null || true + log_event backup.failed ",\"reason\":\"${FAILURE:-exit $status}\"" + heartbeat fail + fi + + exit "$status" +} + +trap on_exit EXIT +trap 'exit 143' TERM INT + +if [[ "${EUID:-$(id -u)}" -ne 0 ]]; then + fail "run as root" +fi + +[[ "$BACKUP_TAG" =~ ^[a-z0-9]+$ ]] || fail "BACKUP_TAG may contain only lowercase letters and digits" +[[ "$MIN_FREE_MB" =~ ^[0-9]+$ ]] || fail "MIN_FREE_MB must be an integer" +[[ "$LOCAL_PLAINTEXT_DAYS" =~ ^[0-9]+$ ]] || fail "LOCAL_PLAINTEXT_DAYS must be an integer" +[[ "$LOCAL_ENCRYPTED_DAYS" =~ ^[0-9]+$ ]] || fail "LOCAL_ENCRYPTED_DAYS must be an integer" +[[ "$PREDEPLOY_KEEP" =~ ^[0-9]+$ ]] || fail "PREDEPLOY_KEEP must be an integer" +[[ "$LOCK_WAIT" =~ ^[0-9]+$ ]] || fail "LOCK_WAIT must be an integer" + +[[ "$AGE_RECIPIENT" == age1* ]] || fail "AGE_RECIPIENT is not set to an age public key" + +if [[ "$SKIP_HEARTBEAT" != "1" && -z "$HEALTHCHECK_URL" ]]; then + fail "HEALTHCHECK_URL is not set (set SKIP_HEARTBEAT=1 to skip the ping)" +fi + +for cmd in docker flock curl age tar sha256sum; do + command -v "$cmd" >/dev/null 2>&1 || fail "missing command: $cmd" +done + +install -d -m 0711 -o root -g root "$BACKUP_DIR" +install -d -m 0755 -o root -g root "$BACKUP_DIR/offsite" +# 0711 on the vault lets the workstation user traverse to offsite/ without +# being able to list the plaintext dumps. +chmod 0711 "$BACKUP_DIR" +chmod 0755 "$BACKUP_DIR/offsite" + +exec 9>/run/lock/chesstrainer-backup.lock +if ! flock -w "$LOCK_WAIT" 9; then + if [[ "$LOCK_WAIT" == "0" ]]; then + # Overlapping timer/manual runs are not failures; the other run reports. + log_event backup.skipped ',"reason":"another backup is running"' + exit 0 + fi + + fail "another backup is still running" +fi + +free_mb="$(df -Pm "$BACKUP_DIR" | awk 'NR == 2 { print $4 }')" +[[ "$free_mb" =~ ^[0-9]+$ ]] || fail "cannot read free space for $BACKUP_DIR" +if ((free_mb < MIN_FREE_MB)); then + fail "only ${free_mb}MB free on $BACKUP_DIR, need ${MIN_FREE_MB}MB" +fi + +timestamp="$(date -u +%Y%m%dT%H%M%SZ)" +if [[ "$BACKUP_TAG" == "daily" ]]; then + prefix="$timestamp" +else + prefix="$timestamp-$BACKUP_TAG" +fi + +dump="$BACKUP_DIR/$prefix.dump" +globals="$BACKUP_DIR/$prefix.globals.sql" +counts="$BACKUP_DIR/$prefix.counts" +checksums="$BACKUP_DIR/$prefix.sha256" +archive="$BACKUP_DIR/offsite/$prefix.tar.age" + +# Row counts first: writes between this query and the dump can only push the +# restored counts up, so the drill treats this file as a lower bound. +read -r -d '' counts_sql <<'SQL' || true +select 'users=' || count(*) from users +union all select 'endgames=' || count(*) from endgames +union all select 'attempts=' || count(*) from attempts; +SQL + +docker exec "$DB_CONTAINER" psql -U "$PG_USER" -d "$PG_DB" \ + -At -v ON_ERROR_STOP=1 -c "$counts_sql" >"$counts.partial" || + fail "row count query failed" +[[ -s "$counts.partial" ]] || fail "row count query returned nothing" + +docker exec "$DB_CONTAINER" pg_dump -U "$PG_USER" -d "$PG_DB" -Fc \ + >"$dump.partial" || fail "pg_dump failed" +[[ -s "$dump.partial" ]] || fail "pg_dump produced an empty file" + +# pg_restore -l reads the archive header and table of contents. It needs the +# Postgres client matching the server, so run it inside the container. +docker exec -i "$DB_CONTAINER" pg_restore -l \ + <"$dump.partial" >/dev/null || fail "pg_restore -l rejected the dump" + +docker exec "$DB_CONTAINER" pg_dumpall -U "$PG_USER" --globals-only \ + >"$globals.partial" || fail "pg_dumpall --globals-only failed" +[[ -s "$globals.partial" ]] || fail "globals dump is empty" + +# shellcheck disable=SC2086 # EXPECTED_ROLES is a space-separated list +for role in $EXPECTED_ROLES; do + grep -Eq "^CREATE ROLE \"?${role}\"?( |;)" "$globals.partial" || + fail "globals dump is missing role $role" +done + +# Promote only after every check passes, so a broken dump never looks finished. +mv "$counts.partial" "$counts" +mv "$dump.partial" "$dump" +mv "$globals.partial" "$globals" + +( + cd "$BACKUP_DIR" + sha256sum "${dump##*/}" "${globals##*/}" "${counts##*/}" >"${checksums##*/}" +) + +tar -C "$BACKUP_DIR" -cf - \ + "${dump##*/}" "${globals##*/}" "${counts##*/}" "${checksums##*/}" | + age --recipient "$AGE_RECIPIENT" --output "$archive.partial" || + fail "encrypting the dump failed" + +chmod 0644 "$archive.partial" +mv "$archive.partial" "$archive" + +# --------------------------------------------------------------------------- +# Retention +# --------------------------------------------------------------------------- + +aged() { + [[ -n "$(find "$1" -maxdepth 0 -mtime +"$2" -print -quit 2>/dev/null)" ]] +} + +remove_plaintext_set() { + local prefix="$1" + + rm -f -- \ + "$BACKUP_DIR/$prefix.dump" \ + "$BACKUP_DIR/$prefix.globals.sql" \ + "$BACKUP_DIR/$prefix.counts" \ + "$BACKUP_DIR/$prefix.sha256" +} + +prune_plaintext_by_age() { + local dump base prefix + + while IFS= read -r dump; do + base="${dump##*/}" + prefix="${base%%.*}" + [[ "$base" == *-predeploy.dump ]] && continue + + if aged "$dump" "$LOCAL_PLAINTEXT_DAYS"; then + echo "backup: pruning $prefix (plaintext older than ${LOCAL_PLAINTEXT_DAYS}d)" >&2 + remove_plaintext_set "$prefix" + fi + done < <(find "$BACKUP_DIR" -maxdepth 1 -type f -name '*.dump') +} + +prune_predeploy_plaintext() { + local -a dumps + local i prefix + + mapfile -t dumps < <( + find "$BACKUP_DIR" -maxdepth 1 -type f -name '*-predeploy.dump' -printf '%f\n' | sort -r + ) + + for ((i = PREDEPLOY_KEEP; i < ${#dumps[@]}; i++)); do + prefix="${dumps[i]%%.*}" + echo "backup: pruning $prefix (keeping last ${PREDEPLOY_KEEP} pre-deploy dumps)" >&2 + remove_plaintext_set "$prefix" + done +} + +prune_encrypted() { + local archive + + while IFS= read -r archive; do + echo "backup: pruning ${archive##*/} (encrypted older than ${LOCAL_ENCRYPTED_DAYS}d)" >&2 + rm -f -- "$archive" + done < <( + find "$BACKUP_DIR/offsite" -maxdepth 1 -type f -name '*.tar.age' -mtime +"$LOCAL_ENCRYPTED_DAYS" + ) +} + +prune_plaintext_by_age +prune_predeploy_plaintext +prune_encrypted + +dump_bytes="$(stat -c %s "$dump")" +archive_bytes="$(stat -c %s "$archive")" + +log_event backup.completed ",\"dump\":\"$prefix\",\"dump_bytes\":$dump_bytes,\"offsite_bytes\":$archive_bytes,\"duration_s\":$((SECONDS - START_SECONDS))" +heartbeat ok diff --git a/deploy/backup.env.example b/deploy/backup.env.example new file mode 100644 index 0000000..4719877 --- /dev/null +++ b/deploy/backup.env.example @@ -0,0 +1,33 @@ +# /etc/chesstrainer/backup.env -- root:root, mode 0600. +# +# Sourced by /usr/local/sbin/chesstrainer-backup. Every value below shows the +# default; only AGE_RECIPIENT and HEALTHCHECK_URL must be set. Keep secrets out +# of the repo and out of world-readable files. + +# age public key, generated on the workstation with `age-keygen`. +# The private key never leaves the workstation. +AGE_RECIPIENT=age1... + +# healthchecks.io (or compatible) ping URL. The script pings it after a good +# run and pings /fail after a bad one. Set SKIP_HEARTBEAT=1 to run without +# monitoring (testing only). +HEALTHCHECK_URL=https://hc-ping.com/00000000-0000-0000-0000-000000000000 + +# Refuse to dump when the filesystem has less than this much free space. +MIN_FREE_MB=2048 + +# Retention. Dumps and their globals/counts/checksum siblings disappear as a +# set; only the encrypted archives in offsite/ live for the longer window. +LOCAL_PLAINTEXT_DAYS=3 +LOCAL_ENCRYPTED_DAYS=14 +PREDEPLOY_KEEP=5 + +# Postgres coordinates. The container trusts local Unix-socket connections, so +# no password belongs here. +DB_CONTAINER=chesstrainer-db +PG_USER=chesstrainer +PG_DB=chesstrainer_prod +BACKUP_DIR=/var/backups/chesstrainer + +# Roles the globals dump must be able to recreate. +EXPECTED_ROLES="chesstrainer chesstrainer_ro" diff --git a/deploy/chesstrainer-backup-failure.service b/deploy/chesstrainer-backup-failure.service new file mode 100644 index 0000000..ca481d6 --- /dev/null +++ b/deploy/chesstrainer-backup-failure.service @@ -0,0 +1,7 @@ +[Unit] +Description=Report a failed Inner Square Chess database backup +Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md + +[Service] +Type=oneshot +ExecStart=/usr/bin/logger --tag chesstrainer-backup --priority daemon.err --stderr backup.unit-failed: chesstrainer-backup.service failed, see journalctl -u chesstrainer-backup diff --git a/deploy/chesstrainer-backup.service b/deploy/chesstrainer-backup.service new file mode 100644 index 0000000..03bf2c8 --- /dev/null +++ b/deploy/chesstrainer-backup.service @@ -0,0 +1,16 @@ +[Unit] +Description=Inner Square Chess database backup +Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md +After=network-online.target docker.service +Wants=network-online.target +OnFailure=chesstrainer-backup-failure.service + +[Service] +Type=oneshot +User=root +Group=root +ExecStart=/usr/local/sbin/chesstrainer-backup +Nice=10 +IOSchedulingClass=idle +TimeoutStartSec=10min +SyslogIdentifier=chesstrainer-backup diff --git a/deploy/chesstrainer-backup.timer b/deploy/chesstrainer-backup.timer new file mode 100644 index 0000000..8c16233 --- /dev/null +++ b/deploy/chesstrainer-backup.timer @@ -0,0 +1,13 @@ +[Unit] +Description=Daily Inner Square Chess database backup +Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md + +[Timer] +OnCalendar=*-*-* 03:15:00 UTC +RandomizedDelaySec=5m +AccuracySec=1m +Persistent=true +Unit=chesstrainer-backup.service + +[Install] +WantedBy=timers.target diff --git a/deploy/workstation/backup-drill b/deploy/workstation/backup-drill new file mode 100755 index 0000000..d1e6c0d --- /dev/null +++ b/deploy/workstation/backup-drill @@ -0,0 +1,190 @@ +#!/usr/bin/env bash +# +# Restore the newest (or a named) archive into a throwaway Postgres container +# and prove that the dump, the roles and the row counts all came back. +# +# backup-drill # newest archive in DEST +# backup-drill 20261111T031500Z.tar.age +# +# Runs on the workstation. The production database is never contacted: the +# container is created and destroyed by this script. Results are appended to +# $DRILL_LOG (default $DEST/drills.log). +# +# Configuration: same file as backup-pull, +# ~/.config/chesstrainer/backup-pull.env. + +set -euo pipefail +umask 077 + +CONFIG="${PULL_ENV_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/chesstrainer/backup-pull.env}" +if [[ -r "$CONFIG" ]]; then + # shellcheck disable=SC1090 + source "$CONFIG" +fi + +DEST="${DEST:-$HOME/chesstrainer-backups}" +AGE_KEY="${AGE_KEY:-${XDG_CONFIG_HOME:-$HOME/.config}/chesstrainer/age.key}" +EXPECTED_ROLES="${EXPECTED_ROLES:-chesstrainer chesstrainer_ro}" +DRILL_IMAGE="${DRILL_IMAGE:-postgres:18.6}" +DRILL_DB="${DRILL_DB:-chesstrainer_restore}" +DRILL_LOG="${DRILL_LOG:-$DEST/drills.log}" +DRILL_CONTAINER="${DRILL_CONTAINER:-chesstrainer-drill-$$}" + +START_SECONDS=$SECONDS +FAILURE="" +TMPDIR_DRILL="" + +log_event() { + local event="$1" + local extra="${2:-}" + + printf '{"timestamp":"%s","event":"%s"%s}\n' \ + "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$event" "$extra" +} + +record() { + local line="$1" + + printf '%s\n' "$line" + printf '%s\n' "$line" >>"$DRILL_LOG" +} + +fail() { + FAILURE="$*" + exit 1 +} + +cleanup() { + local status=$? + + docker rm -f "$DRILL_CONTAINER" >/dev/null 2>&1 || true + + if [[ -n "$TMPDIR_DRILL" ]]; then + rm -rf "$TMPDIR_DRILL" + fi + + if ((status != 0)); then + if [[ -d "$DEST" ]]; then + record "$(log_event drill.failed ",\"reason\":\"${FAILURE:-exit $status}\"")" + else + log_event drill.failed ",\"reason\":\"${FAILURE:-exit $status}\"" + fi + fi + + exit "$status" +} + +trap cleanup EXIT +trap 'exit 143' TERM INT + +[[ -d "$DEST" ]] || fail "archive directory does not exist: $DEST" +[[ -r "$AGE_KEY" ]] || fail "age identity not readable: $AGE_KEY" + +for cmd in docker age tar sha256sum; do + command -v "$cmd" >/dev/null 2>&1 || fail "missing command: $cmd" +done + +if [[ $# -gt 0 ]]; then + case "$1" in + /*) archive="$1" ;; + *) archive="$DEST/$1" ;; + esac +else + archive="$(find "$DEST" -maxdepth 1 -type f -name '*.tar.age' -printf '%p\n' | sort | tail -n 1)" + [[ -n "$archive" ]] || fail "no archives in $DEST" +fi + +[[ -f "$archive" ]] || fail "archive not found: $archive" + +name="${archive##*/}" +[[ "$name" =~ ^[0-9]{8}T[0-9]{6}Z(-[a-z0-9]+)?\.tar\.age$ ]] || + fail "unexpected archive name: $name" + +prefix="${name%%.*}" +TMPDIR_DRILL="$(mktemp -d "${TMPDIR:-/tmp}/chesstrainer-drill.XXXXXX")" + +echo "backup-drill: decrypting $name" >&2 +age --decrypt --identity "$AGE_KEY" --output "$TMPDIR_DRILL/$prefix.tar" "$archive" || + fail "could not decrypt $name" + +tar -xf "$TMPDIR_DRILL/$prefix.tar" -C "$TMPDIR_DRILL" || + fail "archive does not contain a readable tar file" + +( + cd "$TMPDIR_DRILL" + sha256sum --check --strict --quiet "$prefix.sha256" +) || fail "checksum verification failed for $name" + +echo "backup-drill: starting $DRILL_IMAGE" >&2 +docker image inspect "$DRILL_IMAGE" >/dev/null 2>&1 || + docker pull "$DRILL_IMAGE" >/dev/null || + fail "could not pull $DRILL_IMAGE" + +docker run --rm --detach \ + --name "$DRILL_CONTAINER" \ + --env POSTGRES_USER=chesstrainer \ + --env POSTGRES_PASSWORD=drill \ + --env POSTGRES_HOST_AUTH_METHOD=trust \ + "$DRILL_IMAGE" >/dev/null || fail "could not start the drill container" + +ready=0 +for _ in $(seq 1 60); do + if docker exec "$DRILL_CONTAINER" pg_isready -U chesstrainer -d postgres >/dev/null 2>&1; then + ready=1 + break + fi + sleep 1 +done +((ready == 1)) || fail "postgres did not become ready" + +# Recreating the cluster roles is half of disaster recovery. The scratch +# cluster uses the same superuser name as production (`chesstrainer`), because +# the globals dump ends with grants issued "GRANTED BY chesstrainer". The dump +# always repeats the cluster's own superuser role, so "already exists" is +# expected; anything else is a real failure. +echo "backup-drill: applying globals" >&2 +docker exec -i "$DRILL_CONTAINER" psql -U chesstrainer -d postgres \ + -q -v ON_ERROR_STOP=0 -f - \ + <"$TMPDIR_DRILL/$prefix.globals.sql" >/dev/null 2>"$TMPDIR_DRILL/globals.err" || + fail "applying the globals dump failed" + +globals_errors="$(grep -v 'already exists' "$TMPDIR_DRILL/globals.err" | sed '/^$/d' || true)" +if [[ -n "$globals_errors" ]]; then + fail "globals restore reported errors: $(printf '%s' "$globals_errors" | head -n 1)" +fi + +for role in $EXPECTED_ROLES; do + found="$(docker exec "$DRILL_CONTAINER" psql -U chesstrainer -d postgres -Atc \ + "select 1 from pg_roles where rolname = '$role'")" + [[ "$found" == "1" ]] || fail "globals did not recreate role $role" +done + +owner="${EXPECTED_ROLES%% *}" + +docker exec "$DRILL_CONTAINER" psql -U chesstrainer -d postgres \ + -q -v ON_ERROR_STOP=1 -c "create database $DRILL_DB owner $owner" || + fail "could not create $DRILL_DB" + +echo "backup-drill: restoring into $DRILL_DB" >&2 +docker exec -i "$DRILL_CONTAINER" pg_restore -U chesstrainer -d "$DRILL_DB" \ + --single-transaction --exit-on-error <"$TMPDIR_DRILL/$prefix.dump" || + fail "pg_restore failed" + +restored_counts="" +for table in users endgames attempts; do + expected="$(awk -F= -v key="$table" '$1 == key { print $2 }' "$TMPDIR_DRILL/$prefix.counts")" + [[ "$expected" =~ ^[0-9]+$ ]] || fail "counts file has no value for $table" + + actual="$(docker exec "$DRILL_CONTAINER" psql -U chesstrainer -d "$DRILL_DB" -Atc \ + "select count(*) from $table")" + [[ "$actual" =~ ^[0-9]+$ ]] || fail "could not count $table" + + if ((actual < expected)); then + fail "restored $table=$actual is below the recorded $expected" + fi + + echo "backup-drill: $table restored=$actual recorded=$expected drift=$((actual - expected))" >&2 + restored_counts+="${restored_counts:+,}$table=$actual" +done + +record "$(log_event drill.completed ",\"archive\":\"$name\",\"result\":\"pass\",\"counts\":\"$restored_counts\",\"duration_s\":$((SECONDS - START_SECONDS))")" diff --git a/deploy/workstation/backup-pull b/deploy/workstation/backup-pull new file mode 100755 index 0000000..1582388 --- /dev/null +++ b/deploy/workstation/backup-pull @@ -0,0 +1,248 @@ +#!/usr/bin/env bash +# +# Pull the encrypted database archives from the VPS, verify them end to end, +# and apply the 7 daily / 4 weekly / 6 monthly retention policy. +# +# Runs on the workstation, never on the server. Install as +# ~/.local/bin/chesstrainer-backup-pull and enable +# chesstrainer-backup-pull.timer as a user unit. +# +# Configuration: ~/.config/chesstrainer/backup-pull.env (0600). See +# deploy/README.md §Backups. + +set -euo pipefail +umask 077 + +CONFIG="${PULL_ENV_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/chesstrainer/backup-pull.env}" +if [[ -r "$CONFIG" ]]; then + # shellcheck disable=SC1090 + source "$CONFIG" +fi + +VPS_HOST="${VPS_HOST:-}" +REMOTE_DIR="${REMOTE_DIR:-/var/backups/chesstrainer/offsite}" +DEST="${DEST:-$HOME/chesstrainer-backups}" +AGE_KEY="${AGE_KEY:-${XDG_CONFIG_HOME:-$HOME/.config}/chesstrainer/age.key}" +MAX_STALENESS_HOURS="${MAX_STALENESS_HOURS:-48}" +KEEP_DAILY_DAYS="${KEEP_DAILY_DAYS:-7}" +KEEP_WEEKLY_WEEKS="${KEEP_WEEKLY_WEEKS:-4}" +KEEP_MONTHLY_MONTHS="${KEEP_MONTHLY_MONTHS:-6}" +PULL_HEALTHCHECK_URL="${PULL_HEALTHCHECK_URL:-}" +SSH_OPTIONS="${SSH_OPTIONS:--o BatchMode=yes -o ConnectTimeout=10 -o ServerAliveInterval=15}" +RSYNC_ATTEMPTS="${RSYNC_ATTEMPTS:-3}" + +START_SECONDS=$SECONDS +FAILURE="" +VERIFIED=0 + +log_event() { + local event="$1" + local extra="${2:-}" + + printf '{"timestamp":"%s","event":"%s"%s}\n' \ + "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$event" "$extra" +} + +heartbeat() { + local outcome="$1" + + [[ -n "$PULL_HEALTHCHECK_URL" ]] || return 0 + + local url="${PULL_HEALTHCHECK_URL%/}" + [[ "$outcome" == "fail" ]] && url="$url/fail" + + if ! curl --fail --silent --show-error \ + --max-time 10 --retry 2 --retry-delay 2 \ + --output /dev/null "$url"; then + echo "backup-pull: heartbeat ping failed (run result is unaffected)" >&2 + fi +} + +fail() { + FAILURE="$*" + exit 1 +} + +cleanup() { + local status=$? + + if [[ -d "$DEST" ]]; then + find "$DEST" -maxdepth 1 -name '.verify.*' -type d -exec rm -rf {} + 2>/dev/null || true + fi + + if ((status != 0)); then + log_event pull.failed ",\"reason\":\"${FAILURE:-exit $status}\"" + heartbeat fail + fi + + exit "$status" +} + +trap cleanup EXIT +trap 'exit 143' TERM INT + +[[ -n "$VPS_HOST" ]] || fail "VPS_HOST is not set" +[[ -r "$AGE_KEY" ]] || fail "age identity not readable: $AGE_KEY" +[[ "$MAX_STALENESS_HOURS" =~ ^[0-9]+$ ]] || fail "MAX_STALENESS_HOURS must be an integer" +[[ "$KEEP_DAILY_DAYS" =~ ^[0-9]+$ ]] || fail "KEEP_DAILY_DAYS must be an integer" +[[ "$KEEP_WEEKLY_WEEKS" =~ ^[0-9]+$ ]] || fail "KEEP_WEEKLY_WEEKS must be an integer" +[[ "$KEEP_MONTHLY_MONTHS" =~ ^[0-9]+$ ]] || fail "KEEP_MONTHLY_MONTHS must be an integer" +[[ "$RSYNC_ATTEMPTS" =~ ^[0-9]+$ ]] || fail "RSYNC_ATTEMPTS must be an integer" + +for cmd in rsync ssh age tar sha256sum curl flock; do + command -v "$cmd" >/dev/null 2>&1 || fail "missing command: $cmd" +done + +# Never create DEST on the root filesystem when the backup disk is unmounted. +[[ -d "$(dirname "$DEST")" ]] || + fail "parent directory of $DEST does not exist (is the backup disk mounted?)" + +install -d -m 0700 "$DEST" +install -d -m 0700 "$DEST/.verified" + +exec 9>"$DEST/.pull.lock" +if ! flock -n 9; then + # A manual run overlapping the timer is not an error; the other run reports. + log_event pull.skipped ',"reason":"another pull is running"' + exit 0 +fi + +archive_epoch() { + local name="$1" + local ts="${name:0:16}" + + [[ "$ts" =~ ^[0-9]{8}T[0-9]{6}Z$ ]] || return 1 + + date -u -d "${ts:0:4}-${ts:4:2}-${ts:6:2} ${ts:9:2}:${ts:11:2}:${ts:13:2}Z" +%s +} + +verify_archive() { + local archive="$1" + local name="${archive##*/}" + local marker="$DEST/.verified/$name" + local sum tmp + + sum="$(sha256sum "$archive" | awk '{ print $1 }')" + + if [[ -f "$marker" && "$(cat "$marker")" == "$sum" ]]; then + return 0 + fi + + tmp="$(mktemp -d "$DEST/.verify.XXXXXX")" + + if ! age --decrypt --identity "$AGE_KEY" --output "$tmp/$name.tar" "$archive"; then + rm -rf "$tmp" + echo "backup-pull: $name failed to decrypt" >&2 + return 1 + fi + + if ! tar -xf "$tmp/$name.tar" -C "$tmp"; then + rm -rf "$tmp" + echo "backup-pull: $name did not contain a readable tar archive" >&2 + return 1 + fi + + if ! (cd "$tmp" && sha256sum --check --strict --quiet ./*.sha256); then + rm -rf "$tmp" + echo "backup-pull: $name failed its checksum verification" >&2 + return 1 + fi + + rm -rf "$tmp" + + printf '%s\n' "$sum" >"$marker.tmp" + mv "$marker.tmp" "$marker" + + VERIFIED=$((VERIFIED + 1)) +} + +# --------------------------------------------------------------------------- +# Copy +# --------------------------------------------------------------------------- + +attempt=1 +while true; do + if rsync --archive --partial --compress --timeout=120 \ + --include='*.tar.age' --exclude='*' \ + --rsh "ssh $SSH_OPTIONS" \ + "$VPS_HOST:$REMOTE_DIR/" "$DEST/"; then + break + fi + + if ((attempt >= RSYNC_ATTEMPTS)); then + fail "rsync from $VPS_HOST failed after $attempt attempts" + fi + + echo "backup-pull: rsync attempt $attempt failed, retrying" >&2 + attempt=$((attempt + 1)) + sleep 20 +done + +mapfile -t archives < <( + find "$DEST" -maxdepth 1 -type f -name '*.tar.age' -printf '%f\n' | sort +) + +((${#archives[@]} > 0)) || fail "no archives in $DEST after the pull" + +for name in "${archives[@]}"; do + [[ "$name" =~ ^[0-9]{8}T[0-9]{6}Z(-[a-z0-9]+)?\.tar\.age$ ]] || + fail "unexpected file in $DEST: $name" +done + +newest="${archives[${#archives[@]} - 1]}" +newest_epoch="$(archive_epoch "$newest")" || fail "cannot parse a timestamp from $newest" +age_seconds=$(($(date -u +%s) - newest_epoch)) + +if ((age_seconds > MAX_STALENESS_HOURS * 3600)); then + fail "newest archive $newest is $((age_seconds / 3600))h old (limit ${MAX_STALENESS_HOURS}h): VPS backups may have stopped" +fi + +# --------------------------------------------------------------------------- +# Verify every archive once; unchanged ciphertext is only re-hashed. +# --------------------------------------------------------------------------- + +for name in "${archives[@]}"; do + verify_archive "$DEST/$name" || fail "$name failed verification" +done + +# --------------------------------------------------------------------------- +# Retention: 7 daily, then one per ISO week for 4 weeks, then one per calendar +# month for 6 months. Archives outside all three windows are deleted. +# --------------------------------------------------------------------------- + +now="$(date -u +%s)" +pruned=0 +declare -A kept_week=() +declare -A kept_month=() + +mapfile -t newest_first < <( + find "$DEST" -maxdepth 1 -type f -name '*.tar.age' -printf '%f\n' | sort -r +) + +for name in "${newest_first[@]}"; do + epoch="$(archive_epoch "$name")" || continue + age_days=$(((now - epoch) / 86400)) + + if ((age_days < KEEP_DAILY_DAYS)); then + continue + fi + + week="$(date -u -d "@$epoch" +%G-W%V)" + if ((age_days < KEEP_WEEKLY_WEEKS * 7)) && [[ -z "${kept_week[$week]:-}" ]]; then + kept_week[$week]=1 + continue + fi + + month="$(date -u -d "@$epoch" +%Y-%m)" + if ((age_days < KEEP_MONTHLY_MONTHS * 31)) && [[ -z "${kept_month[$month]:-}" ]]; then + kept_month[$month]=1 + continue + fi + + echo "backup-pull: pruning $name" >&2 + rm -f -- "$DEST/$name" "$DEST/.verified/$name" + pruned=$((pruned + 1)) +done + +log_event pull.completed ",\"newest\":\"$newest\",\"archives\":$(( ${#archives[@]} - pruned )),\"verified\":$VERIFIED,\"pruned\":$pruned,\"duration_s\":$((SECONDS - START_SECONDS))" +heartbeat ok diff --git a/deploy/workstation/backup-pull.env.example b/deploy/workstation/backup-pull.env.example new file mode 100644 index 0000000..300b89c --- /dev/null +++ b/deploy/workstation/backup-pull.env.example @@ -0,0 +1,37 @@ +# ~/.config/chesstrainer/backup-pull.env -- mode 0600. +# +# Sourced by ~/.local/bin/chesstrainer-backup-pull. Values below are the +# defaults; VPS_HOST and AGE_KEY must be right. Quote anything containing +# spaces. + +VPS_HOST=owen@159.69.21.225 +REMOTE_DIR=/var/backups/chesstrainer/offsite +DEST="/mnt/Mint Data/chesstrainer-backups" + +# Private half of the age key. Back this file up somewhere safe: without it +# every archive is unreadable. +AGE_KEY="$HOME/.config/chesstrainer/age.key" + +# Fail the pull when the newest archive from the VPS is older than this, which +# means the VPS backup timer has stopped. +MAX_STALENESS_HOURS=48 + +# Keep every archive from the last 7 days, then one per ISO week for 4 weeks, +# then one per calendar month for 6 months. +KEEP_DAILY_DAYS=7 +KEEP_WEEKLY_WEEKS=4 +KEEP_MONTHLY_MONTHS=6 + +# Restore drill. It runs in a throwaway container matching the production +# Postgres major and appends results to DRILL_LOG. +# DRILL_IMAGE=postgres:18.6 +# DRILL_DB=chesstrainer_restore +# DRILL_LOG="$DEST/drills.log" +EXPECTED_ROLES="chesstrainer chesstrainer_ro" + +# Optional second heartbeat check with a long period (for example 72h), so you +# hear about it when the workstation stops pulling. Leave empty to disable. +# PULL_HEALTHCHECK_URL=https://hc-ping.com/00000000-0000-0000-0000-000000000000 + +# Extra ssh options, space separated. +# SSH_OPTIONS="-o BatchMode=yes -o ConnectTimeout=10 -o ServerAliveInterval=15" diff --git a/deploy/workstation/chesstrainer-backup-alert@.service b/deploy/workstation/chesstrainer-backup-alert@.service new file mode 100644 index 0000000..4633074 --- /dev/null +++ b/deploy/workstation/chesstrainer-backup-alert@.service @@ -0,0 +1,7 @@ +[Unit] +Description=Report a failed Inner Square Chess backup unit +Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md + +[Service] +Type=oneshot +ExecStart=/usr/bin/logger --tag chesstrainer-backup --priority user.err --stderr backup.unit-failed: %i failed, see journalctl --user -u %i diff --git a/deploy/workstation/chesstrainer-backup-drill.service b/deploy/workstation/chesstrainer-backup-drill.service new file mode 100644 index 0000000..6fc8345 --- /dev/null +++ b/deploy/workstation/chesstrainer-backup-drill.service @@ -0,0 +1,13 @@ +[Unit] +Description=Monthly restore drill for Inner Square Chess backups +Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md +OnFailure=chesstrainer-backup-alert@%N.service + +[Service] +Type=oneshot +ExecStart=%h/.local/bin/chesstrainer-backup-drill +WorkingDirectory=%h +Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin +Nice=10 +TimeoutStartSec=20min +SyslogIdentifier=chesstrainer-backup-drill diff --git a/deploy/workstation/chesstrainer-backup-drill.timer b/deploy/workstation/chesstrainer-backup-drill.timer new file mode 100644 index 0000000..a4e4340 --- /dev/null +++ b/deploy/workstation/chesstrainer-backup-drill.timer @@ -0,0 +1,13 @@ +[Unit] +Description=Monthly restore drill for Inner Square Chess backups +Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md + +[Timer] +OnCalendar=*-*-01 09:00:00 +RandomizedDelaySec=30m +AccuracySec=5m +Persistent=true +Unit=chesstrainer-backup-drill.service + +[Install] +WantedBy=timers.target diff --git a/deploy/workstation/chesstrainer-backup-pull.service b/deploy/workstation/chesstrainer-backup-pull.service new file mode 100644 index 0000000..09feba1 --- /dev/null +++ b/deploy/workstation/chesstrainer-backup-pull.service @@ -0,0 +1,15 @@ +[Unit] +Description=Pull encrypted Inner Square Chess backups from the VPS +Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md +After=network-online.target +Wants=network-online.target +OnFailure=chesstrainer-backup-alert@%N.service + +[Service] +Type=oneshot +ExecStart=%h/.local/bin/chesstrainer-backup-pull +WorkingDirectory=%h +Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin +Nice=10 +TimeoutStartSec=15min +SyslogIdentifier=chesstrainer-backup-pull diff --git a/deploy/workstation/chesstrainer-backup-pull.timer b/deploy/workstation/chesstrainer-backup-pull.timer new file mode 100644 index 0000000..8488700 --- /dev/null +++ b/deploy/workstation/chesstrainer-backup-pull.timer @@ -0,0 +1,14 @@ +[Unit] +Description=Daily pull of encrypted Inner Square Chess backups +Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md + +[Timer] +OnCalendar=*-*-* 12:00:00 +OnBootSec=10m +RandomizedDelaySec=10m +AccuracySec=1m +Persistent=true +Unit=chesstrainer-backup-pull.service + +[Install] +WantedBy=timers.target