mirror of
https://github.com/TheRealOwenRees/chess-endgame-trainer.git
synced 2026-11-18 03:32:06 +00:00
The VPS now takes a daily logical dump, verifies it, encrypts it, and hands it to a workstation; a monthly drill proves the archive restores. - deploy/backup: pg_dump -Fc plus pg_dumpall --globals-only, pg_restore -l and globals-role checks before anything is promoted, row counts for the drill, sha256, age encryption into offsite/, retention (3d plaintext, 14d encrypted, 5 pre-deploy sets), JSON logging and a healthchecks.io heartbeat - systemd service and timer (03:15 UTC, Persistent, jittered) plus an OnFailure unit that logs a distinct event - deploy/activate takes a pre-deploy dump after extracting and before flipping current, because migrations run in ExecStartPre; a failed dump aborts the activation, with SKIP_PRE_DEPLOY_BACKUP=1 as the escape hatch - deploy/workstation/backup-pull: rsync the encrypted archives over SSH, verify by decrypting and checking the inner sha256, retain 7d/4w/6m, and fail when the newest archive is older than 48h - deploy/workstation/backup-drill: restore the newest archive into a throwaway postgres:18.6, replay the globals dump as the same superuser, assert both roles, compare users/endgames/attempts, and append the result to drills.log - user units for the daily pull and monthly drill, with a shared OnFailure alert template - docs: install, key handling, retention, restore, disaster recovery, a run by hand section, monitoring and troubleshooting Verified against production: the first backup completed and pinged the heartbeat, the pull verified the archive, and the drill restored users=1 endgames=39 attempts=0, matching the live database.
This commit is contained in:
+341
-35
@@ -103,6 +103,9 @@ also `root:root` and `0600`.
|
||||
The app runs as the dedicated `chesstrainer` system user. It must own
|
||||
`/opt/chesstrainer` so it can write `priv/static/backups`.
|
||||
|
||||
Database dumps live in `/var/backups/chesstrainer`, outside `/opt/chesstrainer`
|
||||
and `priv/static`. See §Backups.
|
||||
|
||||
## Copy the deploy artifacts
|
||||
|
||||
Run these commands from a checkout of this repo on the VPS, or copy the
|
||||
@@ -441,6 +444,8 @@ The unit includes `ConditionPathExists=/opt/chesstrainer/current/bin/server`, so
|
||||
it is safe to enable before the first release. systemd skips it until a release
|
||||
is in place.
|
||||
|
||||
The backup unit and timer are installed separately; see §Backups.
|
||||
|
||||
## Caddy
|
||||
|
||||
Add the site block from `deploy/Caddyfile` by hand. It imports
|
||||
@@ -601,12 +606,13 @@ The script:
|
||||
|
||||
1. Extracts into `/opt/chesstrainer/releases/<timestamp>`
|
||||
2. Fixes ownership
|
||||
3. Flips `/opt/chesstrainer/current`
|
||||
4. Restarts `chesstrainer.service`
|
||||
5. Migrations run via `ExecStartPre`
|
||||
6. Polls `/health`
|
||||
7. Rolls back to the previous release on failure
|
||||
8. Prunes old releases, keeping 5 by default
|
||||
3. Takes a verified pre-deploy database dump, aborting if it fails (see §Backups)
|
||||
4. Flips `/opt/chesstrainer/current`
|
||||
5. Restarts `chesstrainer.service`
|
||||
6. Migrations run via `ExecStartPre`
|
||||
7. Polls `/health`
|
||||
8. Rolls back to the previous release on failure
|
||||
9. Prunes old releases, keeping 5 by default
|
||||
|
||||
Verify locally before touching Caddy:
|
||||
|
||||
@@ -641,6 +647,10 @@ KEEP_RELEASES=5 \
|
||||
sudo -E /opt/chesstrainer/src/deploy/activate /tmp/chesstrainer-release.tar.gz
|
||||
```
|
||||
|
||||
Every activation takes a verified pre-deploy dump before migrations run. If
|
||||
Postgres is intentionally unavailable and you accept the risk, add
|
||||
`SKIP_PRE_DEPLOY_BACKUP=1` to the override list; see §Backups.
|
||||
|
||||
## GitHub Actions CD
|
||||
|
||||
Merging to `main` (or running the workflow manually) tests, builds, uploads,
|
||||
@@ -704,7 +714,8 @@ The workflow:
|
||||
1. Runs `mix precommit` against Postgres.
|
||||
2. Builds the release artifact on `ubuntu-24.04`.
|
||||
3. On `main` or `workflow_dispatch`, uploads the tarball, installs the SSH key,
|
||||
runs `sudo -n .../activate`, and performs public smoke checks.
|
||||
runs `sudo -n .../activate`, and performs public smoke checks. Activation
|
||||
takes a pre-deploy dump first and fails if it cannot.
|
||||
4. A failed activation health check rolls back to the previous release.
|
||||
|
||||
Manual dispatch is available from the Actions UI or:
|
||||
@@ -727,6 +738,10 @@ sudo /opt/chesstrainer/src/deploy/rollback 20261109120000
|
||||
|
||||
The script flips `current`, restarts the service, and checks `/health`.
|
||||
|
||||
Rollback reverts the release, not the database: migrations are not reversed. If
|
||||
a migration changed data, restore the matching pre-deploy dump instead; see
|
||||
§Backups.
|
||||
|
||||
## Logs and health
|
||||
|
||||
```sh
|
||||
@@ -856,52 +871,338 @@ That shell speaks Erlang syntax, so quote Elixir module atoms, e.g.
|
||||
|
||||
## Backups
|
||||
|
||||
`docker exec` connects over the container's Unix socket, which the Postgres
|
||||
image trusts locally, so `pg_dump` needs no password. Run it as the owner
|
||||
(`chesstrainer`), not `chesstrainer_ro`: a full dump also needs sequence access
|
||||
and ownership information.
|
||||
Backups are produced on the VPS, encrypted there, and pulled to a workstation.
|
||||
Nothing leaves the VPS unencrypted, and the VPS never holds the key that could
|
||||
read the archives.
|
||||
|
||||
Take a compressed logical backup (custom format, restorable with `pg_restore`):
|
||||
### What runs
|
||||
|
||||
| Where | What | When | Retention |
|
||||
|---|---|---|---|
|
||||
| VPS | `chesstrainer-backup.timer` → `chesstrainer-backup.service` | daily 03:15 UTC, `Persistent=true`, 5 min jitter | 3 days plaintext, 14 days encrypted |
|
||||
| VPS | the same script, called by `deploy/activate` | before every migration | last 5 pre-deploy sets |
|
||||
| Workstation | `chesstrainer-backup-pull.timer` | daily 12:00 local, 10 min after boot | 7 daily, 4 weekly, 6 monthly |
|
||||
| Workstation | `chesstrainer-backup-drill.timer` | monthly on the 1st at 09:00 | `drills.log` is never pruned |
|
||||
|
||||
Each run writes one JSON line (`backup.completed`, `backup.failed`,
|
||||
`backup.skipped`, `pull.completed`, `pull.failed`, `pull.skipped`,
|
||||
`drill.completed`, `drill.failed`) so #140's log shipping can alert on it. A
|
||||
`skipped` line means another run held the lock; it is not an alarm.
|
||||
|
||||
### What a dump contains
|
||||
|
||||
- `<ts>.dump` — `pg_dump -Fc` of `chesstrainer_prod`
|
||||
- `<ts>.globals.sql` — `pg_dumpall --globals-only`, enough to recreate
|
||||
`chesstrainer` and `chesstrainer_ro` on a fresh cluster
|
||||
- `<ts>.counts` — row counts for `users`, `endgames`, `attempts`, used by the drill
|
||||
- `<ts>.sha256` — checksums for the three files above
|
||||
|
||||
Not backed up, by design: tablebases and the tablebase cache (both are
|
||||
re-fetchable), anything under `priv/static`, and release tarballs. Dumps contain
|
||||
user emails and bcrypt hashes: they are `0600 root` while plaintext, encrypted
|
||||
before they leave the box, and pruned aggressively.
|
||||
|
||||
### RPO and RTO
|
||||
|
||||
RPO is 24 hours on the VPS and "the age of the newest pulled archive" off-box.
|
||||
The pull runs daily, so a laptop that is off for a day leaves the newest
|
||||
off-site copy around 48 hours old; the VPS keeps 14 days of encrypted archives,
|
||||
which bounds how far behind the pull can fall. RTO is under an hour: create the
|
||||
database, restore the newest dump, activate a release (see *Disaster recovery*).
|
||||
|
||||
### One-time setup: VPS
|
||||
|
||||
Install `age`. Encryption happens on the server; only the workstation can
|
||||
decrypt.
|
||||
|
||||
```sh
|
||||
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
|
||||
pg_dump -U chesstrainer -d chesstrainer_prod -Fc \
|
||||
> /var/backups/chesstrainer-$(date -u +%Y%m%d%H%M%S).dump
|
||||
sudo apt install age
|
||||
```
|
||||
|
||||
For a plain SQL file instead (human-readable, restorable with `psql`):
|
||||
Create the configuration file. Generate `AGE_RECIPIENT` on the workstation
|
||||
first (next section) and paste the public key.
|
||||
|
||||
```sh
|
||||
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
|
||||
pg_dump -U chesstrainer -d chesstrainer_prod \
|
||||
| gzip > /var/backups/chesstrainer-$(date -u +%Y%m%d%H%M%S).sql.gz
|
||||
sudo install -m 600 /dev/null /etc/chesstrainer/backup.env
|
||||
sudoedit /etc/chesstrainer/backup.env
|
||||
```
|
||||
|
||||
Check that a custom-format dump is readable before trusting it:
|
||||
Start from `deploy/backup.env.example`. The required values are `AGE_RECIPIENT`
|
||||
and `HEALTHCHECK_URL`.
|
||||
|
||||
Create a healthchecks.io check with a period of 1 day and a grace time of 4
|
||||
hours, then paste its ping URL into `HEALTHCHECK_URL`. The script pings it after
|
||||
a good run and `<url>/fail` after a bad one, so silence means "no backup
|
||||
finished", even when the VPS itself is down.
|
||||
|
||||
Install the script and units:
|
||||
|
||||
```sh
|
||||
sudo docker exec -i chesstrainer-db pg_restore -l \
|
||||
< /var/backups/<timestamp>.dump > /dev/null
|
||||
sudo install -m 0755 /opt/chesstrainer/src/deploy/backup \
|
||||
/usr/local/sbin/chesstrainer-backup
|
||||
sudo install -m 0644 \
|
||||
/opt/chesstrainer/src/deploy/chesstrainer-backup.service \
|
||||
/opt/chesstrainer/src/deploy/chesstrainer-backup.timer \
|
||||
/opt/chesstrainer/src/deploy/chesstrainer-backup-failure.service \
|
||||
/etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now chesstrainer-backup.timer
|
||||
```
|
||||
|
||||
Restore into a scratch database (do not restore over the live database without
|
||||
stopping `chesstrainer` first):
|
||||
The timer runs the installed `/usr/local/sbin/chesstrainer-backup`, not the
|
||||
copy under `/opt/chesstrainer/src`, because that tree is writable by the
|
||||
application user. When `deploy/backup` changes, copy it to both places so the
|
||||
timer and `deploy/activate` use the same script.
|
||||
|
||||
Run it once and inspect the result:
|
||||
|
||||
```sh
|
||||
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
|
||||
createdb -U chesstrainer chesstrainer_restore
|
||||
sudo docker exec -i chesstrainer-db pg_restore -U chesstrainer \
|
||||
-d chesstrainer_restore < /var/backups/<timestamp>.dump
|
||||
sudo systemctl start chesstrainer-backup
|
||||
sudo journalctl -u chesstrainer-backup -n 20 --no-pager -o cat
|
||||
sudo ls -l /var/backups/chesstrainer /var/backups/chesstrainer/offsite
|
||||
systemctl list-timers chesstrainer-backup.timer
|
||||
```
|
||||
|
||||
Example nightly entry in root's crontab (`sudo crontab -e`):
|
||||
Expect a `.dump`, `.globals.sql`, `.counts` and `.sha256` at mode `0600`, and
|
||||
one `.tar.age` in `offsite/` at mode `0644`. `offsite/` is the only thing the
|
||||
workstation reads, which is why the pull needs no `sudo`. The vault directory is
|
||||
`0711`, so the workstation user can traverse into `offsite/` but cannot list the
|
||||
plaintext dumps.
|
||||
|
||||
```cron
|
||||
15 3 * * * docker compose -f /home/<username>/docker-compose.yml exec -T chesstrainer-db pg_dump -U chesstrainer -d chesstrainer_prod -Fc > /var/backups/chesstrainer-$(date +\%Y\%m\%d).dump
|
||||
### One-time setup: workstation
|
||||
|
||||
```sh
|
||||
sudo apt install age rsync
|
||||
mkdir -p ~/.config/chesstrainer
|
||||
chmod 700 ~/.config/chesstrainer
|
||||
age-keygen -o ~/.config/chesstrainer/age.key
|
||||
chmod 600 ~/.config/chesstrainer/age.key
|
||||
grep 'public key' ~/.config/chesstrainer/age.key
|
||||
```
|
||||
|
||||
Keep `/var/backups` outside `/opt/chesstrainer/releases` and copy backups off
|
||||
the VPS regularly. A backup that is only on the VPS is not a backup.
|
||||
Copy that public key into `AGE_RECIPIENT` on the VPS. **Back up the private
|
||||
key** (password manager, printed copy, offline disk): without it every archive
|
||||
is unreadable, and the VPS deliberately cannot decrypt them.
|
||||
|
||||
Create the pull configuration:
|
||||
|
||||
```sh
|
||||
install -m 600 /dev/null ~/.config/chesstrainer/backup-pull.env
|
||||
$EDITOR ~/.config/chesstrainer/backup-pull.env
|
||||
```
|
||||
|
||||
Start from `deploy/workstation/backup-pull.env.example`. `DEST` should point at
|
||||
a real disk, for example `/mnt/Mint Data/chesstrainer-backups`. The script
|
||||
refuses to run when `DEST`'s parent is missing, so an unmounted disk cannot
|
||||
silently fill the root filesystem.
|
||||
|
||||
Set `PULL_HEALTHCHECK_URL` to a second healthchecks.io check with a long period
|
||||
(for example 72 hours) if you want to hear about it when the workstation stops
|
||||
pulling. Leave it empty for journal-only reporting.
|
||||
|
||||
Install the scripts and user units:
|
||||
|
||||
```sh
|
||||
REPO=~/Desktop/chess-trainer # wherever this repo is cloned
|
||||
|
||||
install -d ~/.local/bin
|
||||
install -m 0755 "$REPO/deploy/workstation/backup-pull" ~/.local/bin/chesstrainer-backup-pull
|
||||
install -m 0755 "$REPO/deploy/workstation/backup-drill" ~/.local/bin/chesstrainer-backup-drill
|
||||
|
||||
install -d ~/.config/systemd/user
|
||||
install -m 0644 \
|
||||
"$REPO/deploy/workstation/chesstrainer-backup-pull.service" \
|
||||
"$REPO/deploy/workstation/chesstrainer-backup-pull.timer" \
|
||||
"$REPO/deploy/workstation/chesstrainer-backup-drill.service" \
|
||||
"$REPO/deploy/workstation/chesstrainer-backup-drill.timer" \
|
||||
"$REPO/deploy/workstation/chesstrainer-backup-alert@.service" \
|
||||
~/.config/systemd/user/
|
||||
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now chesstrainer-backup-pull.timer
|
||||
systemctl --user enable --now chesstrainer-backup-drill.timer
|
||||
sudo loginctl enable-linger "$USER"
|
||||
```
|
||||
|
||||
`enable-linger` lets the user timers run while the laptop is on but you are not
|
||||
logged in. The pull also fires ten minutes after boot, and `Persistent=true`
|
||||
catches up after the machine was off. Verify lingering stuck:
|
||||
|
||||
```sh
|
||||
loginctl show-user "$USER" -p Linger # Linger=yes
|
||||
```
|
||||
|
||||
Run the first pull and drill by hand:
|
||||
|
||||
```sh
|
||||
~/.local/bin/chesstrainer-backup-pull
|
||||
ls -l "/mnt/Mint Data/chesstrainer-backups"
|
||||
~/.local/bin/chesstrainer-backup-drill
|
||||
cat "/mnt/Mint Data/chesstrainer-backups/drills.log"
|
||||
```
|
||||
|
||||
A passing drill line looks like:
|
||||
|
||||
```json
|
||||
{"timestamp":"2026-11-11T21:05:46Z","event":"drill.completed","archive":"20261111T120000Z.tar.age","result":"pass","counts":"users=3,endgames=42,attempts=900","duration_s":3}
|
||||
```
|
||||
|
||||
The drill creates a throwaway `postgres:18.6` container, applies the globals
|
||||
dump, asserts that `chesstrainer` and `chesstrainer_ro` exist, restores into
|
||||
`chesstrainer_restore`, and compares the three row counts with the `.counts`
|
||||
file from dump time. Restored counts may be higher (the counts are taken moments
|
||||
before the dump); lower counts fail the drill.
|
||||
|
||||
### Retention
|
||||
|
||||
| Location | Plaintext | Encrypted | Notes |
|
||||
|---|---|---|---|
|
||||
| VPS `/var/backups/chesstrainer` | 3 days | 14 days | pre-deploy sets: last 5 |
|
||||
| Workstation `DEST` | never | 7 daily, 4 weekly, 6 monthly | monthly copies outlive the VPS copies |
|
||||
|
||||
The VPS is a staging area for the workstation; weekly and monthly generations
|
||||
accumulate there. If the laptop is off for more than 14 days, the archives the
|
||||
VPS dropped in the meantime are gone — that is the documented off-site RPO
|
||||
trade-off. Moving to an always-on destination (a second host or object storage
|
||||
with restic) is tracked in #151.
|
||||
|
||||
### Restore a dump
|
||||
|
||||
Always restore into a scratch database first. From an archive on the
|
||||
workstation:
|
||||
|
||||
```sh
|
||||
WORK=$(mktemp -d)
|
||||
age --decrypt -i ~/.config/chesstrainer/age.key \
|
||||
-o "$WORK/dump.tar" \
|
||||
"/mnt/Mint Data/chesstrainer-backups/<timestamp>.tar.age"
|
||||
tar -xf "$WORK/dump.tar" -C "$WORK"
|
||||
(cd "$WORK" && sha256sum --check --strict <timestamp>.sha256)
|
||||
```
|
||||
|
||||
Then restore into a scratch container:
|
||||
|
||||
```sh
|
||||
docker run --rm -d --name restore-check \
|
||||
-e POSTGRES_PASSWORD=restore -e POSTGRES_HOST_AUTH_METHOD=trust postgres:18.6
|
||||
docker exec restore-check psql -U postgres -c 'create database chesstrainer_restore'
|
||||
docker exec -i restore-check pg_restore -U postgres -d chesstrainer_restore \
|
||||
--single-transaction --exit-on-error < "$WORK/<timestamp>.dump"
|
||||
docker exec restore-check psql -U postgres -d chesstrainer_restore \
|
||||
-c 'select count(*) from users; select count(*) from attempts;'
|
||||
docker rm -f restore-check
|
||||
rm -rf "$WORK"
|
||||
```
|
||||
|
||||
### Disaster recovery
|
||||
|
||||
1. Build a new VPS following this README through *First-time setup*, without
|
||||
activating a release or switching Caddy.
|
||||
2. Decrypt and unpack the newest archive on the workstation (above).
|
||||
3. Copy the dump and globals to the new VPS:
|
||||
`scp "$WORK/<timestamp>.dump" "$WORK/<timestamp>.globals.sql" <VPS_USER>@<VPS_HOST>:/tmp/`
|
||||
4. Recreate the cluster roles. Connect as `chesstrainer`: it is the container's
|
||||
superuser (`POSTGRES_USER`), and the dump's final grants are issued
|
||||
`GRANTED BY chesstrainer`, which another role cannot replay. The line that
|
||||
recreates `chesstrainer` itself reports "already exists" and is harmless:
|
||||
`sudo docker exec -i chesstrainer-db psql -U chesstrainer < /tmp/<timestamp>.globals.sql`
|
||||
5. Create the database and restore it:
|
||||
`sudo docker exec chesstrainer-db createdb -U chesstrainer chesstrainer_prod`
|
||||
`sudo docker exec -i chesstrainer-db pg_restore -U chesstrainer -d chesstrainer_prod --single-transaction --exit-on-error < /tmp/<timestamp>.dump`
|
||||
6. Verify the row counts, activate a release, then restore the Caddy block.
|
||||
7. Take a fresh backup (`sudo systemctl start chesstrainer-backup`) and pull so
|
||||
the new VPS is covered immediately.
|
||||
|
||||
`pg_dump` and `pg_restore` must come from the container's Postgres 18, not the
|
||||
host.
|
||||
|
||||
### Pre-deploy dump
|
||||
|
||||
`deploy/activate` calls `chesstrainer-backup` with `BACKUP_TAG=predeploy` after
|
||||
extracting the release and before flipping `current`, so the state before
|
||||
`bin/migrate` runs in `ExecStartPre` is always recoverable. A failed dump aborts
|
||||
the activation. The escape hatch is deliberate and should be rare:
|
||||
|
||||
```sh
|
||||
sudo SKIP_PRE_DEPLOY_BACKUP=1 /opt/chesstrainer/src/deploy/activate /tmp/chesstrainer-release.tar.gz
|
||||
```
|
||||
|
||||
If the installed script is missing, `activate` uses the `backup` next to it; if
|
||||
neither exists it refuses to activate rather than migrate without a dump. The
|
||||
GitHub Actions deploy job inherits the same behaviour.
|
||||
|
||||
### Run it by hand
|
||||
|
||||
Every timer can be triggered directly. On the VPS:
|
||||
|
||||
```sh
|
||||
# exactly what the timer runs
|
||||
sudo systemctl start chesstrainer-backup
|
||||
sudo journalctl -u chesstrainer-backup -n 20 -o cat
|
||||
|
||||
# an ad-hoc dump with its own tag
|
||||
sudo BACKUP_TAG=manual /usr/local/sbin/chesstrainer-backup
|
||||
|
||||
# schedule, last result and last exit
|
||||
systemctl list-timers chesstrainer-backup.timer
|
||||
systemctl show chesstrainer-backup.service -p Result -p ExecMainExitTimestamp
|
||||
```
|
||||
|
||||
On the workstation:
|
||||
|
||||
```sh
|
||||
# pull now, or through the unit (same PATH and OnFailure handling)
|
||||
~/.local/bin/chesstrainer-backup-pull
|
||||
systemctl --user start chesstrainer-backup-pull.service
|
||||
|
||||
# drill the newest archive, or a named one
|
||||
~/.local/bin/chesstrainer-backup-drill
|
||||
~/.local/bin/chesstrainer-backup-drill 20261111T213734Z.tar.age
|
||||
|
||||
# results and schedule
|
||||
cat "/mnt/Mint Data/chesstrainer-backups/drills.log"
|
||||
systemctl --user list-timers 'chesstrainer-backup-*'
|
||||
```
|
||||
|
||||
A `backup.skipped` or `pull.skipped` line means another run held the lock; that
|
||||
run reports the outcome. After fixing a failure, clear the unit with
|
||||
`systemctl --user reset-failed chesstrainer-backup-pull.service` (or
|
||||
`sudo systemctl reset-failed chesstrainer-backup.service`).
|
||||
|
||||
Enabling the pull timer fires it once straight away: its `OnBootSec=10m` is
|
||||
measured from boot, so enabling it hours after boot triggers immediately. That
|
||||
is expected and harmless.
|
||||
|
||||
### Monitoring
|
||||
|
||||
```sh
|
||||
systemctl list-timers chesstrainer-backup.timer
|
||||
sudo journalctl -u chesstrainer-backup --since "2 days ago" -o cat
|
||||
sudo systemctl --failed
|
||||
systemctl --user list-timers chesstrainer-backup-pull.timer chesstrainer-backup-drill.timer
|
||||
journalctl --user -u chesstrainer-backup-pull --since "2 days ago" -o cat
|
||||
```
|
||||
|
||||
A failed run produces both `backup.failed` (from the script) and
|
||||
`backup.unit-failed` (from the `OnFailure` unit). The healthchecks.io check is
|
||||
the backstop: if the VPS dies, the ping stops and you are emailed. The pull
|
||||
fails when the newest archive is more than 48 hours old, which catches a stopped
|
||||
VPS timer the next time the workstation pulls.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
| Symptom | Likely cause |
|
||||
|---|---|
|
||||
| `backup.failed: missing command: age` | `sudo apt install age` |
|
||||
| `backup.failed: only N MB free` | disk full on the VPS; prune old releases or grow the disk |
|
||||
| `backup.failed: pg_dump failed` | `chesstrainer-db` is down or restarting — check `docker ps` |
|
||||
| `backup.failed: globals dump is missing role …` | the role was renamed; update `EXPECTED_ROLES` |
|
||||
| `pull.failed: rsync … failed` | SSH key, host key, or the VPS is unreachable |
|
||||
| `pull.failed: newest archive … is Nh old` | the VPS backup timer stopped; check it before touching the workstation |
|
||||
| `drill.failed: pg_restore failed` | archive is corrupt; keep it for diagnosis and restore an earlier one |
|
||||
|
||||
Never put dumps anywhere under `priv/static` or the release directory, and never
|
||||
restore over `chesstrainer_prod` while the application is running.
|
||||
|
||||
## New VPS checklist
|
||||
|
||||
@@ -916,6 +1217,11 @@ the VPS regularly. A backup that is only on the VPS is not a backup.
|
||||
8. Caddy block, `cloudflare-ranges.caddy`, and the CIDR refresh timer added;
|
||||
validated and reloaded.
|
||||
9. Public `/`, `/privacy`, `/terms-conditions`, and `/health` verified.
|
||||
10. Read-only role created, tunnel verified, backups scheduled and tested.
|
||||
11. Remote shell attaches and exits cleanly:
|
||||
10. Read-only role created and tunnel verified.
|
||||
11. `age` installed, `/etc/chesstrainer/backup.env` written, backup script and
|
||||
units installed, timer enabled, and one manual run produces a dump plus a
|
||||
globals file.
|
||||
12. Workstation age key and pull configuration installed, user timers and linger
|
||||
enabled, first pull verified, first drill passes.
|
||||
13. Remote shell attaches and exits cleanly:
|
||||
`sudo /opt/chesstrainer/current/bin/chesstrainer remote` (Ctrl-C twice).
|
||||
|
||||
+27
-1
@@ -7,7 +7,8 @@ set -euo pipefail
|
||||
# sudo deploy/activate /tmp/chesstrainer-20261109.tar.gz
|
||||
#
|
||||
# Environment overrides:
|
||||
# APP_USER, APP_ROOT, SERVICE, HEALTH_URL, KEEP_RELEASES, RELEASE_ID
|
||||
# APP_USER, APP_ROOT, SERVICE, HEALTH_URL, KEEP_RELEASES, RELEASE_ID,
|
||||
# BACKUP_BIN, SKIP_PRE_DEPLOY_BACKUP=1
|
||||
|
||||
if [[ "${EUID:-$(id -u)}" -ne 0 ]]; then
|
||||
echo "activate: run with sudo" >&2
|
||||
@@ -26,6 +27,8 @@ SERVICE="${SERVICE:-chesstrainer}"
|
||||
HEALTH_URL="${HEALTH_URL:-http://127.0.0.1:4100/health}"
|
||||
KEEP_RELEASES="${KEEP_RELEASES:-5}"
|
||||
RELEASE_ID="${RELEASE_ID:-$(date -u +%Y%m%d%H%M%S)}"
|
||||
BACKUP_BIN="${BACKUP_BIN:-/usr/local/sbin/chesstrainer-backup}"
|
||||
SKIP_PRE_DEPLOY_BACKUP="${SKIP_PRE_DEPLOY_BACKUP:-0}"
|
||||
|
||||
if [[ ! "$RELEASE_ID" =~ ^[A-Za-z0-9._-]+$ ]]; then
|
||||
echo "activate: RELEASE_ID may contain only letters, digits, dot, underscore and dash" >&2
|
||||
@@ -53,6 +56,29 @@ install -d -o "$APP_USER" -g "$APP_USER" "$RELEASE_DIR"
|
||||
tar -xzf "$TARBALL" -C "$RELEASE_DIR"
|
||||
chown -R "$APP_USER:$APP_USER" "$RELEASE_DIR"
|
||||
|
||||
if [[ "$SKIP_PRE_DEPLOY_BACKUP" == "1" ]]; then
|
||||
echo "activate: SKIP_PRE_DEPLOY_BACKUP=1, not taking a pre-deploy dump"
|
||||
else
|
||||
backup_script="$BACKUP_BIN"
|
||||
if [[ ! -x "$backup_script" ]]; then
|
||||
backup_script="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/backup"
|
||||
fi
|
||||
|
||||
if [[ ! -x "$backup_script" ]]; then
|
||||
echo "activate: no backup script at $BACKUP_BIN or next to activate" >&2
|
||||
echo "activate: install deploy/backup or set SKIP_PRE_DEPLOY_BACKUP=1" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Migrations run in ExecStartPre after this, so the dump must exist first.
|
||||
echo "activate: taking a pre-deploy dump"
|
||||
if ! BACKUP_TAG=predeploy LOCK_WAIT=120 "$backup_script"; then
|
||||
echo "activate: pre-deploy dump failed, aborting" >&2
|
||||
echo "activate: fix the backup or re-run with SKIP_PRE_DEPLOY_BACKUP=1" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "activate: pointing current at $RELEASE_DIR"
|
||||
ln -sfn "$RELEASE_DIR" "$APP_ROOT/current"
|
||||
|
||||
|
||||
Executable
+266
@@ -0,0 +1,266 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Dump, verify, encrypt, prune and report on the production database.
|
||||
#
|
||||
# Installed as /usr/local/sbin/chesstrainer-backup (root-owned 0755) and run by
|
||||
# chesstrainer-backup.timer. deploy/activate also calls it before migrations.
|
||||
# Do not run the copy under /opt/chesstrainer/src as root: that tree belongs to
|
||||
# the application user.
|
||||
#
|
||||
# sudo systemctl start chesstrainer-backup # daily run, as the timer does
|
||||
# sudo BACKUP_TAG=manual chesstrainer-backup # ad-hoc, pruned by age
|
||||
#
|
||||
# Configuration lives in /etc/chesstrainer/backup.env (root:root 0600). See
|
||||
# deploy/README.md §Backups. Every setting can also be passed as an environment
|
||||
# variable.
|
||||
#
|
||||
# Each run prints exactly one JSON line (backup.completed or backup.failed) so
|
||||
# journald, and #140's log shipping, can pick it up.
|
||||
|
||||
set -euo pipefail
|
||||
umask 077
|
||||
|
||||
ENV_FILE="${ENV_FILE:-/etc/chesstrainer/backup.env}"
|
||||
if [[ -r "$ENV_FILE" ]]; then
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
fi
|
||||
|
||||
DB_CONTAINER="${DB_CONTAINER:-chesstrainer-db}"
|
||||
PG_USER="${PG_USER:-chesstrainer}"
|
||||
PG_DB="${PG_DB:-chesstrainer_prod}"
|
||||
BACKUP_DIR="${BACKUP_DIR:-/var/backups/chesstrainer}"
|
||||
EXPECTED_ROLES="${EXPECTED_ROLES:-chesstrainer chesstrainer_ro}"
|
||||
AGE_RECIPIENT="${AGE_RECIPIENT:-}"
|
||||
HEALTHCHECK_URL="${HEALTHCHECK_URL:-}"
|
||||
SKIP_HEARTBEAT="${SKIP_HEARTBEAT:-0}"
|
||||
MIN_FREE_MB="${MIN_FREE_MB:-2048}"
|
||||
LOCAL_PLAINTEXT_DAYS="${LOCAL_PLAINTEXT_DAYS:-3}"
|
||||
LOCAL_ENCRYPTED_DAYS="${LOCAL_ENCRYPTED_DAYS:-14}"
|
||||
PREDEPLOY_KEEP="${PREDEPLOY_KEEP:-5}"
|
||||
BACKUP_TAG="${BACKUP_TAG:-daily}"
|
||||
LOCK_WAIT="${LOCK_WAIT:-0}"
|
||||
|
||||
START_SECONDS=$SECONDS
|
||||
FAILURE=""
|
||||
|
||||
log_event() {
|
||||
local event="$1"
|
||||
local extra="${2:-}"
|
||||
|
||||
printf '{"timestamp":"%s","event":"%s","tag":"%s"%s}\n' \
|
||||
"$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$event" "$BACKUP_TAG" "$extra"
|
||||
}
|
||||
|
||||
heartbeat() {
|
||||
local outcome="$1"
|
||||
|
||||
[[ "$SKIP_HEARTBEAT" == "1" ]] && return 0
|
||||
[[ -n "$HEALTHCHECK_URL" ]] || return 0
|
||||
|
||||
local url="${HEALTHCHECK_URL%/}"
|
||||
[[ "$outcome" == "fail" ]] && url="$url/fail"
|
||||
|
||||
if ! curl --fail --silent --show-error \
|
||||
--max-time 10 --retry 2 --retry-delay 2 \
|
||||
--output /dev/null "$url"; then
|
||||
echo "backup: heartbeat ping failed (run result is unaffected)" >&2
|
||||
fi
|
||||
}
|
||||
|
||||
fail() {
|
||||
FAILURE="$*"
|
||||
exit 1
|
||||
}
|
||||
|
||||
on_exit() {
|
||||
local status=$?
|
||||
|
||||
if ((status != 0)); then
|
||||
find "$BACKUP_DIR" -maxdepth 1 -name '*.partial' -delete 2>/dev/null || true
|
||||
log_event backup.failed ",\"reason\":\"${FAILURE:-exit $status}\""
|
||||
heartbeat fail
|
||||
fi
|
||||
|
||||
exit "$status"
|
||||
}
|
||||
|
||||
trap on_exit EXIT
|
||||
trap 'exit 143' TERM INT
|
||||
|
||||
if [[ "${EUID:-$(id -u)}" -ne 0 ]]; then
|
||||
fail "run as root"
|
||||
fi
|
||||
|
||||
[[ "$BACKUP_TAG" =~ ^[a-z0-9]+$ ]] || fail "BACKUP_TAG may contain only lowercase letters and digits"
|
||||
[[ "$MIN_FREE_MB" =~ ^[0-9]+$ ]] || fail "MIN_FREE_MB must be an integer"
|
||||
[[ "$LOCAL_PLAINTEXT_DAYS" =~ ^[0-9]+$ ]] || fail "LOCAL_PLAINTEXT_DAYS must be an integer"
|
||||
[[ "$LOCAL_ENCRYPTED_DAYS" =~ ^[0-9]+$ ]] || fail "LOCAL_ENCRYPTED_DAYS must be an integer"
|
||||
[[ "$PREDEPLOY_KEEP" =~ ^[0-9]+$ ]] || fail "PREDEPLOY_KEEP must be an integer"
|
||||
[[ "$LOCK_WAIT" =~ ^[0-9]+$ ]] || fail "LOCK_WAIT must be an integer"
|
||||
|
||||
[[ "$AGE_RECIPIENT" == age1* ]] || fail "AGE_RECIPIENT is not set to an age public key"
|
||||
|
||||
if [[ "$SKIP_HEARTBEAT" != "1" && -z "$HEALTHCHECK_URL" ]]; then
|
||||
fail "HEALTHCHECK_URL is not set (set SKIP_HEARTBEAT=1 to skip the ping)"
|
||||
fi
|
||||
|
||||
for cmd in docker flock curl age tar sha256sum; do
|
||||
command -v "$cmd" >/dev/null 2>&1 || fail "missing command: $cmd"
|
||||
done
|
||||
|
||||
install -d -m 0711 -o root -g root "$BACKUP_DIR"
|
||||
install -d -m 0755 -o root -g root "$BACKUP_DIR/offsite"
|
||||
# 0711 on the vault lets the workstation user traverse to offsite/ without
|
||||
# being able to list the plaintext dumps.
|
||||
chmod 0711 "$BACKUP_DIR"
|
||||
chmod 0755 "$BACKUP_DIR/offsite"
|
||||
|
||||
exec 9>/run/lock/chesstrainer-backup.lock
|
||||
if ! flock -w "$LOCK_WAIT" 9; then
|
||||
if [[ "$LOCK_WAIT" == "0" ]]; then
|
||||
# Overlapping timer/manual runs are not failures; the other run reports.
|
||||
log_event backup.skipped ',"reason":"another backup is running"'
|
||||
exit 0
|
||||
fi
|
||||
|
||||
fail "another backup is still running"
|
||||
fi
|
||||
|
||||
free_mb="$(df -Pm "$BACKUP_DIR" | awk 'NR == 2 { print $4 }')"
|
||||
[[ "$free_mb" =~ ^[0-9]+$ ]] || fail "cannot read free space for $BACKUP_DIR"
|
||||
if ((free_mb < MIN_FREE_MB)); then
|
||||
fail "only ${free_mb}MB free on $BACKUP_DIR, need ${MIN_FREE_MB}MB"
|
||||
fi
|
||||
|
||||
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
if [[ "$BACKUP_TAG" == "daily" ]]; then
|
||||
prefix="$timestamp"
|
||||
else
|
||||
prefix="$timestamp-$BACKUP_TAG"
|
||||
fi
|
||||
|
||||
dump="$BACKUP_DIR/$prefix.dump"
|
||||
globals="$BACKUP_DIR/$prefix.globals.sql"
|
||||
counts="$BACKUP_DIR/$prefix.counts"
|
||||
checksums="$BACKUP_DIR/$prefix.sha256"
|
||||
archive="$BACKUP_DIR/offsite/$prefix.tar.age"
|
||||
|
||||
# Row counts first: writes between this query and the dump can only push the
|
||||
# restored counts up, so the drill treats this file as a lower bound.
|
||||
read -r -d '' counts_sql <<'SQL' || true
|
||||
select 'users=' || count(*) from users
|
||||
union all select 'endgames=' || count(*) from endgames
|
||||
union all select 'attempts=' || count(*) from attempts;
|
||||
SQL
|
||||
|
||||
docker exec "$DB_CONTAINER" psql -U "$PG_USER" -d "$PG_DB" \
|
||||
-At -v ON_ERROR_STOP=1 -c "$counts_sql" >"$counts.partial" ||
|
||||
fail "row count query failed"
|
||||
[[ -s "$counts.partial" ]] || fail "row count query returned nothing"
|
||||
|
||||
docker exec "$DB_CONTAINER" pg_dump -U "$PG_USER" -d "$PG_DB" -Fc \
|
||||
>"$dump.partial" || fail "pg_dump failed"
|
||||
[[ -s "$dump.partial" ]] || fail "pg_dump produced an empty file"
|
||||
|
||||
# pg_restore -l reads the archive header and table of contents. It needs the
|
||||
# Postgres client matching the server, so run it inside the container.
|
||||
docker exec -i "$DB_CONTAINER" pg_restore -l \
|
||||
<"$dump.partial" >/dev/null || fail "pg_restore -l rejected the dump"
|
||||
|
||||
docker exec "$DB_CONTAINER" pg_dumpall -U "$PG_USER" --globals-only \
|
||||
>"$globals.partial" || fail "pg_dumpall --globals-only failed"
|
||||
[[ -s "$globals.partial" ]] || fail "globals dump is empty"
|
||||
|
||||
# shellcheck disable=SC2086 # EXPECTED_ROLES is a space-separated list
|
||||
for role in $EXPECTED_ROLES; do
|
||||
grep -Eq "^CREATE ROLE \"?${role}\"?( |;)" "$globals.partial" ||
|
||||
fail "globals dump is missing role $role"
|
||||
done
|
||||
|
||||
# Promote only after every check passes, so a broken dump never looks finished.
|
||||
mv "$counts.partial" "$counts"
|
||||
mv "$dump.partial" "$dump"
|
||||
mv "$globals.partial" "$globals"
|
||||
|
||||
(
|
||||
cd "$BACKUP_DIR"
|
||||
sha256sum "${dump##*/}" "${globals##*/}" "${counts##*/}" >"${checksums##*/}"
|
||||
)
|
||||
|
||||
tar -C "$BACKUP_DIR" -cf - \
|
||||
"${dump##*/}" "${globals##*/}" "${counts##*/}" "${checksums##*/}" |
|
||||
age --recipient "$AGE_RECIPIENT" --output "$archive.partial" ||
|
||||
fail "encrypting the dump failed"
|
||||
|
||||
chmod 0644 "$archive.partial"
|
||||
mv "$archive.partial" "$archive"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Retention
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
aged() {
|
||||
[[ -n "$(find "$1" -maxdepth 0 -mtime +"$2" -print -quit 2>/dev/null)" ]]
|
||||
}
|
||||
|
||||
remove_plaintext_set() {
|
||||
local prefix="$1"
|
||||
|
||||
rm -f -- \
|
||||
"$BACKUP_DIR/$prefix.dump" \
|
||||
"$BACKUP_DIR/$prefix.globals.sql" \
|
||||
"$BACKUP_DIR/$prefix.counts" \
|
||||
"$BACKUP_DIR/$prefix.sha256"
|
||||
}
|
||||
|
||||
prune_plaintext_by_age() {
|
||||
local dump base prefix
|
||||
|
||||
while IFS= read -r dump; do
|
||||
base="${dump##*/}"
|
||||
prefix="${base%%.*}"
|
||||
[[ "$base" == *-predeploy.dump ]] && continue
|
||||
|
||||
if aged "$dump" "$LOCAL_PLAINTEXT_DAYS"; then
|
||||
echo "backup: pruning $prefix (plaintext older than ${LOCAL_PLAINTEXT_DAYS}d)" >&2
|
||||
remove_plaintext_set "$prefix"
|
||||
fi
|
||||
done < <(find "$BACKUP_DIR" -maxdepth 1 -type f -name '*.dump')
|
||||
}
|
||||
|
||||
prune_predeploy_plaintext() {
|
||||
local -a dumps
|
||||
local i prefix
|
||||
|
||||
mapfile -t dumps < <(
|
||||
find "$BACKUP_DIR" -maxdepth 1 -type f -name '*-predeploy.dump' -printf '%f\n' | sort -r
|
||||
)
|
||||
|
||||
for ((i = PREDEPLOY_KEEP; i < ${#dumps[@]}; i++)); do
|
||||
prefix="${dumps[i]%%.*}"
|
||||
echo "backup: pruning $prefix (keeping last ${PREDEPLOY_KEEP} pre-deploy dumps)" >&2
|
||||
remove_plaintext_set "$prefix"
|
||||
done
|
||||
}
|
||||
|
||||
prune_encrypted() {
|
||||
local archive
|
||||
|
||||
while IFS= read -r archive; do
|
||||
echo "backup: pruning ${archive##*/} (encrypted older than ${LOCAL_ENCRYPTED_DAYS}d)" >&2
|
||||
rm -f -- "$archive"
|
||||
done < <(
|
||||
find "$BACKUP_DIR/offsite" -maxdepth 1 -type f -name '*.tar.age' -mtime +"$LOCAL_ENCRYPTED_DAYS"
|
||||
)
|
||||
}
|
||||
|
||||
prune_plaintext_by_age
|
||||
prune_predeploy_plaintext
|
||||
prune_encrypted
|
||||
|
||||
dump_bytes="$(stat -c %s "$dump")"
|
||||
archive_bytes="$(stat -c %s "$archive")"
|
||||
|
||||
log_event backup.completed ",\"dump\":\"$prefix\",\"dump_bytes\":$dump_bytes,\"offsite_bytes\":$archive_bytes,\"duration_s\":$((SECONDS - START_SECONDS))"
|
||||
heartbeat ok
|
||||
@@ -0,0 +1,33 @@
|
||||
# /etc/chesstrainer/backup.env -- root:root, mode 0600.
|
||||
#
|
||||
# Sourced by /usr/local/sbin/chesstrainer-backup. Every value below shows the
|
||||
# default; only AGE_RECIPIENT and HEALTHCHECK_URL must be set. Keep secrets out
|
||||
# of the repo and out of world-readable files.
|
||||
|
||||
# age public key, generated on the workstation with `age-keygen`.
|
||||
# The private key never leaves the workstation.
|
||||
AGE_RECIPIENT=age1...
|
||||
|
||||
# healthchecks.io (or compatible) ping URL. The script pings it after a good
|
||||
# run and pings <url>/fail after a bad one. Set SKIP_HEARTBEAT=1 to run without
|
||||
# monitoring (testing only).
|
||||
HEALTHCHECK_URL=https://hc-ping.com/00000000-0000-0000-0000-000000000000
|
||||
|
||||
# Refuse to dump when the filesystem has less than this much free space.
|
||||
MIN_FREE_MB=2048
|
||||
|
||||
# Retention. Dumps and their globals/counts/checksum siblings disappear as a
|
||||
# set; only the encrypted archives in offsite/ live for the longer window.
|
||||
LOCAL_PLAINTEXT_DAYS=3
|
||||
LOCAL_ENCRYPTED_DAYS=14
|
||||
PREDEPLOY_KEEP=5
|
||||
|
||||
# Postgres coordinates. The container trusts local Unix-socket connections, so
|
||||
# no password belongs here.
|
||||
DB_CONTAINER=chesstrainer-db
|
||||
PG_USER=chesstrainer
|
||||
PG_DB=chesstrainer_prod
|
||||
BACKUP_DIR=/var/backups/chesstrainer
|
||||
|
||||
# Roles the globals dump must be able to recreate.
|
||||
EXPECTED_ROLES="chesstrainer chesstrainer_ro"
|
||||
@@ -0,0 +1,7 @@
|
||||
[Unit]
|
||||
Description=Report a failed Inner Square Chess database backup
|
||||
Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/logger --tag chesstrainer-backup --priority daemon.err --stderr backup.unit-failed: chesstrainer-backup.service failed, see journalctl -u chesstrainer-backup
|
||||
@@ -0,0 +1,16 @@
|
||||
[Unit]
|
||||
Description=Inner Square Chess database backup
|
||||
Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
OnFailure=chesstrainer-backup-failure.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=root
|
||||
Group=root
|
||||
ExecStart=/usr/local/sbin/chesstrainer-backup
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
TimeoutStartSec=10min
|
||||
SyslogIdentifier=chesstrainer-backup
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Daily Inner Square Chess database backup
|
||||
Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 03:15:00 UTC
|
||||
RandomizedDelaySec=5m
|
||||
AccuracySec=1m
|
||||
Persistent=true
|
||||
Unit=chesstrainer-backup.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Executable
+190
@@ -0,0 +1,190 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Restore the newest (or a named) archive into a throwaway Postgres container
|
||||
# and prove that the dump, the roles and the row counts all came back.
|
||||
#
|
||||
# backup-drill # newest archive in DEST
|
||||
# backup-drill 20261111T031500Z.tar.age
|
||||
#
|
||||
# Runs on the workstation. The production database is never contacted: the
|
||||
# container is created and destroyed by this script. Results are appended to
|
||||
# $DRILL_LOG (default $DEST/drills.log).
|
||||
#
|
||||
# Configuration: same file as backup-pull,
|
||||
# ~/.config/chesstrainer/backup-pull.env.
|
||||
|
||||
set -euo pipefail
|
||||
umask 077
|
||||
|
||||
CONFIG="${PULL_ENV_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/chesstrainer/backup-pull.env}"
|
||||
if [[ -r "$CONFIG" ]]; then
|
||||
# shellcheck disable=SC1090
|
||||
source "$CONFIG"
|
||||
fi
|
||||
|
||||
DEST="${DEST:-$HOME/chesstrainer-backups}"
|
||||
AGE_KEY="${AGE_KEY:-${XDG_CONFIG_HOME:-$HOME/.config}/chesstrainer/age.key}"
|
||||
EXPECTED_ROLES="${EXPECTED_ROLES:-chesstrainer chesstrainer_ro}"
|
||||
DRILL_IMAGE="${DRILL_IMAGE:-postgres:18.6}"
|
||||
DRILL_DB="${DRILL_DB:-chesstrainer_restore}"
|
||||
DRILL_LOG="${DRILL_LOG:-$DEST/drills.log}"
|
||||
DRILL_CONTAINER="${DRILL_CONTAINER:-chesstrainer-drill-$$}"
|
||||
|
||||
START_SECONDS=$SECONDS
|
||||
FAILURE=""
|
||||
TMPDIR_DRILL=""
|
||||
|
||||
log_event() {
|
||||
local event="$1"
|
||||
local extra="${2:-}"
|
||||
|
||||
printf '{"timestamp":"%s","event":"%s"%s}\n' \
|
||||
"$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$event" "$extra"
|
||||
}
|
||||
|
||||
record() {
|
||||
local line="$1"
|
||||
|
||||
printf '%s\n' "$line"
|
||||
printf '%s\n' "$line" >>"$DRILL_LOG"
|
||||
}
|
||||
|
||||
fail() {
|
||||
FAILURE="$*"
|
||||
exit 1
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
local status=$?
|
||||
|
||||
docker rm -f "$DRILL_CONTAINER" >/dev/null 2>&1 || true
|
||||
|
||||
if [[ -n "$TMPDIR_DRILL" ]]; then
|
||||
rm -rf "$TMPDIR_DRILL"
|
||||
fi
|
||||
|
||||
if ((status != 0)); then
|
||||
if [[ -d "$DEST" ]]; then
|
||||
record "$(log_event drill.failed ",\"reason\":\"${FAILURE:-exit $status}\"")"
|
||||
else
|
||||
log_event drill.failed ",\"reason\":\"${FAILURE:-exit $status}\""
|
||||
fi
|
||||
fi
|
||||
|
||||
exit "$status"
|
||||
}
|
||||
|
||||
trap cleanup EXIT
|
||||
trap 'exit 143' TERM INT
|
||||
|
||||
[[ -d "$DEST" ]] || fail "archive directory does not exist: $DEST"
|
||||
[[ -r "$AGE_KEY" ]] || fail "age identity not readable: $AGE_KEY"
|
||||
|
||||
for cmd in docker age tar sha256sum; do
|
||||
command -v "$cmd" >/dev/null 2>&1 || fail "missing command: $cmd"
|
||||
done
|
||||
|
||||
if [[ $# -gt 0 ]]; then
|
||||
case "$1" in
|
||||
/*) archive="$1" ;;
|
||||
*) archive="$DEST/$1" ;;
|
||||
esac
|
||||
else
|
||||
archive="$(find "$DEST" -maxdepth 1 -type f -name '*.tar.age' -printf '%p\n' | sort | tail -n 1)"
|
||||
[[ -n "$archive" ]] || fail "no archives in $DEST"
|
||||
fi
|
||||
|
||||
[[ -f "$archive" ]] || fail "archive not found: $archive"
|
||||
|
||||
name="${archive##*/}"
|
||||
[[ "$name" =~ ^[0-9]{8}T[0-9]{6}Z(-[a-z0-9]+)?\.tar\.age$ ]] ||
|
||||
fail "unexpected archive name: $name"
|
||||
|
||||
prefix="${name%%.*}"
|
||||
TMPDIR_DRILL="$(mktemp -d "${TMPDIR:-/tmp}/chesstrainer-drill.XXXXXX")"
|
||||
|
||||
echo "backup-drill: decrypting $name" >&2
|
||||
age --decrypt --identity "$AGE_KEY" --output "$TMPDIR_DRILL/$prefix.tar" "$archive" ||
|
||||
fail "could not decrypt $name"
|
||||
|
||||
tar -xf "$TMPDIR_DRILL/$prefix.tar" -C "$TMPDIR_DRILL" ||
|
||||
fail "archive does not contain a readable tar file"
|
||||
|
||||
(
|
||||
cd "$TMPDIR_DRILL"
|
||||
sha256sum --check --strict --quiet "$prefix.sha256"
|
||||
) || fail "checksum verification failed for $name"
|
||||
|
||||
echo "backup-drill: starting $DRILL_IMAGE" >&2
|
||||
docker image inspect "$DRILL_IMAGE" >/dev/null 2>&1 ||
|
||||
docker pull "$DRILL_IMAGE" >/dev/null ||
|
||||
fail "could not pull $DRILL_IMAGE"
|
||||
|
||||
docker run --rm --detach \
|
||||
--name "$DRILL_CONTAINER" \
|
||||
--env POSTGRES_USER=chesstrainer \
|
||||
--env POSTGRES_PASSWORD=drill \
|
||||
--env POSTGRES_HOST_AUTH_METHOD=trust \
|
||||
"$DRILL_IMAGE" >/dev/null || fail "could not start the drill container"
|
||||
|
||||
ready=0
|
||||
for _ in $(seq 1 60); do
|
||||
if docker exec "$DRILL_CONTAINER" pg_isready -U chesstrainer -d postgres >/dev/null 2>&1; then
|
||||
ready=1
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
((ready == 1)) || fail "postgres did not become ready"
|
||||
|
||||
# Recreating the cluster roles is half of disaster recovery. The scratch
|
||||
# cluster uses the same superuser name as production (`chesstrainer`), because
|
||||
# the globals dump ends with grants issued "GRANTED BY chesstrainer". The dump
|
||||
# always repeats the cluster's own superuser role, so "already exists" is
|
||||
# expected; anything else is a real failure.
|
||||
echo "backup-drill: applying globals" >&2
|
||||
docker exec -i "$DRILL_CONTAINER" psql -U chesstrainer -d postgres \
|
||||
-q -v ON_ERROR_STOP=0 -f - \
|
||||
<"$TMPDIR_DRILL/$prefix.globals.sql" >/dev/null 2>"$TMPDIR_DRILL/globals.err" ||
|
||||
fail "applying the globals dump failed"
|
||||
|
||||
globals_errors="$(grep -v 'already exists' "$TMPDIR_DRILL/globals.err" | sed '/^$/d' || true)"
|
||||
if [[ -n "$globals_errors" ]]; then
|
||||
fail "globals restore reported errors: $(printf '%s' "$globals_errors" | head -n 1)"
|
||||
fi
|
||||
|
||||
for role in $EXPECTED_ROLES; do
|
||||
found="$(docker exec "$DRILL_CONTAINER" psql -U chesstrainer -d postgres -Atc \
|
||||
"select 1 from pg_roles where rolname = '$role'")"
|
||||
[[ "$found" == "1" ]] || fail "globals did not recreate role $role"
|
||||
done
|
||||
|
||||
owner="${EXPECTED_ROLES%% *}"
|
||||
|
||||
docker exec "$DRILL_CONTAINER" psql -U chesstrainer -d postgres \
|
||||
-q -v ON_ERROR_STOP=1 -c "create database $DRILL_DB owner $owner" ||
|
||||
fail "could not create $DRILL_DB"
|
||||
|
||||
echo "backup-drill: restoring into $DRILL_DB" >&2
|
||||
docker exec -i "$DRILL_CONTAINER" pg_restore -U chesstrainer -d "$DRILL_DB" \
|
||||
--single-transaction --exit-on-error <"$TMPDIR_DRILL/$prefix.dump" ||
|
||||
fail "pg_restore failed"
|
||||
|
||||
restored_counts=""
|
||||
for table in users endgames attempts; do
|
||||
expected="$(awk -F= -v key="$table" '$1 == key { print $2 }' "$TMPDIR_DRILL/$prefix.counts")"
|
||||
[[ "$expected" =~ ^[0-9]+$ ]] || fail "counts file has no value for $table"
|
||||
|
||||
actual="$(docker exec "$DRILL_CONTAINER" psql -U chesstrainer -d "$DRILL_DB" -Atc \
|
||||
"select count(*) from $table")"
|
||||
[[ "$actual" =~ ^[0-9]+$ ]] || fail "could not count $table"
|
||||
|
||||
if ((actual < expected)); then
|
||||
fail "restored $table=$actual is below the recorded $expected"
|
||||
fi
|
||||
|
||||
echo "backup-drill: $table restored=$actual recorded=$expected drift=$((actual - expected))" >&2
|
||||
restored_counts+="${restored_counts:+,}$table=$actual"
|
||||
done
|
||||
|
||||
record "$(log_event drill.completed ",\"archive\":\"$name\",\"result\":\"pass\",\"counts\":\"$restored_counts\",\"duration_s\":$((SECONDS - START_SECONDS))")"
|
||||
Executable
+248
@@ -0,0 +1,248 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Pull the encrypted database archives from the VPS, verify them end to end,
|
||||
# and apply the 7 daily / 4 weekly / 6 monthly retention policy.
|
||||
#
|
||||
# Runs on the workstation, never on the server. Install as
|
||||
# ~/.local/bin/chesstrainer-backup-pull and enable
|
||||
# chesstrainer-backup-pull.timer as a user unit.
|
||||
#
|
||||
# Configuration: ~/.config/chesstrainer/backup-pull.env (0600). See
|
||||
# deploy/README.md §Backups.
|
||||
|
||||
set -euo pipefail
|
||||
umask 077
|
||||
|
||||
CONFIG="${PULL_ENV_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/chesstrainer/backup-pull.env}"
|
||||
if [[ -r "$CONFIG" ]]; then
|
||||
# shellcheck disable=SC1090
|
||||
source "$CONFIG"
|
||||
fi
|
||||
|
||||
VPS_HOST="${VPS_HOST:-}"
|
||||
REMOTE_DIR="${REMOTE_DIR:-/var/backups/chesstrainer/offsite}"
|
||||
DEST="${DEST:-$HOME/chesstrainer-backups}"
|
||||
AGE_KEY="${AGE_KEY:-${XDG_CONFIG_HOME:-$HOME/.config}/chesstrainer/age.key}"
|
||||
MAX_STALENESS_HOURS="${MAX_STALENESS_HOURS:-48}"
|
||||
KEEP_DAILY_DAYS="${KEEP_DAILY_DAYS:-7}"
|
||||
KEEP_WEEKLY_WEEKS="${KEEP_WEEKLY_WEEKS:-4}"
|
||||
KEEP_MONTHLY_MONTHS="${KEEP_MONTHLY_MONTHS:-6}"
|
||||
PULL_HEALTHCHECK_URL="${PULL_HEALTHCHECK_URL:-}"
|
||||
SSH_OPTIONS="${SSH_OPTIONS:--o BatchMode=yes -o ConnectTimeout=10 -o ServerAliveInterval=15}"
|
||||
RSYNC_ATTEMPTS="${RSYNC_ATTEMPTS:-3}"
|
||||
|
||||
START_SECONDS=$SECONDS
|
||||
FAILURE=""
|
||||
VERIFIED=0
|
||||
|
||||
log_event() {
|
||||
local event="$1"
|
||||
local extra="${2:-}"
|
||||
|
||||
printf '{"timestamp":"%s","event":"%s"%s}\n' \
|
||||
"$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$event" "$extra"
|
||||
}
|
||||
|
||||
heartbeat() {
|
||||
local outcome="$1"
|
||||
|
||||
[[ -n "$PULL_HEALTHCHECK_URL" ]] || return 0
|
||||
|
||||
local url="${PULL_HEALTHCHECK_URL%/}"
|
||||
[[ "$outcome" == "fail" ]] && url="$url/fail"
|
||||
|
||||
if ! curl --fail --silent --show-error \
|
||||
--max-time 10 --retry 2 --retry-delay 2 \
|
||||
--output /dev/null "$url"; then
|
||||
echo "backup-pull: heartbeat ping failed (run result is unaffected)" >&2
|
||||
fi
|
||||
}
|
||||
|
||||
fail() {
|
||||
FAILURE="$*"
|
||||
exit 1
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
local status=$?
|
||||
|
||||
if [[ -d "$DEST" ]]; then
|
||||
find "$DEST" -maxdepth 1 -name '.verify.*' -type d -exec rm -rf {} + 2>/dev/null || true
|
||||
fi
|
||||
|
||||
if ((status != 0)); then
|
||||
log_event pull.failed ",\"reason\":\"${FAILURE:-exit $status}\""
|
||||
heartbeat fail
|
||||
fi
|
||||
|
||||
exit "$status"
|
||||
}
|
||||
|
||||
trap cleanup EXIT
|
||||
trap 'exit 143' TERM INT
|
||||
|
||||
[[ -n "$VPS_HOST" ]] || fail "VPS_HOST is not set"
|
||||
[[ -r "$AGE_KEY" ]] || fail "age identity not readable: $AGE_KEY"
|
||||
[[ "$MAX_STALENESS_HOURS" =~ ^[0-9]+$ ]] || fail "MAX_STALENESS_HOURS must be an integer"
|
||||
[[ "$KEEP_DAILY_DAYS" =~ ^[0-9]+$ ]] || fail "KEEP_DAILY_DAYS must be an integer"
|
||||
[[ "$KEEP_WEEKLY_WEEKS" =~ ^[0-9]+$ ]] || fail "KEEP_WEEKLY_WEEKS must be an integer"
|
||||
[[ "$KEEP_MONTHLY_MONTHS" =~ ^[0-9]+$ ]] || fail "KEEP_MONTHLY_MONTHS must be an integer"
|
||||
[[ "$RSYNC_ATTEMPTS" =~ ^[0-9]+$ ]] || fail "RSYNC_ATTEMPTS must be an integer"
|
||||
|
||||
for cmd in rsync ssh age tar sha256sum curl flock; do
|
||||
command -v "$cmd" >/dev/null 2>&1 || fail "missing command: $cmd"
|
||||
done
|
||||
|
||||
# Never create DEST on the root filesystem when the backup disk is unmounted.
|
||||
[[ -d "$(dirname "$DEST")" ]] ||
|
||||
fail "parent directory of $DEST does not exist (is the backup disk mounted?)"
|
||||
|
||||
install -d -m 0700 "$DEST"
|
||||
install -d -m 0700 "$DEST/.verified"
|
||||
|
||||
exec 9>"$DEST/.pull.lock"
|
||||
if ! flock -n 9; then
|
||||
# A manual run overlapping the timer is not an error; the other run reports.
|
||||
log_event pull.skipped ',"reason":"another pull is running"'
|
||||
exit 0
|
||||
fi
|
||||
|
||||
archive_epoch() {
|
||||
local name="$1"
|
||||
local ts="${name:0:16}"
|
||||
|
||||
[[ "$ts" =~ ^[0-9]{8}T[0-9]{6}Z$ ]] || return 1
|
||||
|
||||
date -u -d "${ts:0:4}-${ts:4:2}-${ts:6:2} ${ts:9:2}:${ts:11:2}:${ts:13:2}Z" +%s
|
||||
}
|
||||
|
||||
verify_archive() {
|
||||
local archive="$1"
|
||||
local name="${archive##*/}"
|
||||
local marker="$DEST/.verified/$name"
|
||||
local sum tmp
|
||||
|
||||
sum="$(sha256sum "$archive" | awk '{ print $1 }')"
|
||||
|
||||
if [[ -f "$marker" && "$(cat "$marker")" == "$sum" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
tmp="$(mktemp -d "$DEST/.verify.XXXXXX")"
|
||||
|
||||
if ! age --decrypt --identity "$AGE_KEY" --output "$tmp/$name.tar" "$archive"; then
|
||||
rm -rf "$tmp"
|
||||
echo "backup-pull: $name failed to decrypt" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
if ! tar -xf "$tmp/$name.tar" -C "$tmp"; then
|
||||
rm -rf "$tmp"
|
||||
echo "backup-pull: $name did not contain a readable tar archive" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
if ! (cd "$tmp" && sha256sum --check --strict --quiet ./*.sha256); then
|
||||
rm -rf "$tmp"
|
||||
echo "backup-pull: $name failed its checksum verification" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
rm -rf "$tmp"
|
||||
|
||||
printf '%s\n' "$sum" >"$marker.tmp"
|
||||
mv "$marker.tmp" "$marker"
|
||||
|
||||
VERIFIED=$((VERIFIED + 1))
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Copy
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
attempt=1
|
||||
while true; do
|
||||
if rsync --archive --partial --compress --timeout=120 \
|
||||
--include='*.tar.age' --exclude='*' \
|
||||
--rsh "ssh $SSH_OPTIONS" \
|
||||
"$VPS_HOST:$REMOTE_DIR/" "$DEST/"; then
|
||||
break
|
||||
fi
|
||||
|
||||
if ((attempt >= RSYNC_ATTEMPTS)); then
|
||||
fail "rsync from $VPS_HOST failed after $attempt attempts"
|
||||
fi
|
||||
|
||||
echo "backup-pull: rsync attempt $attempt failed, retrying" >&2
|
||||
attempt=$((attempt + 1))
|
||||
sleep 20
|
||||
done
|
||||
|
||||
mapfile -t archives < <(
|
||||
find "$DEST" -maxdepth 1 -type f -name '*.tar.age' -printf '%f\n' | sort
|
||||
)
|
||||
|
||||
((${#archives[@]} > 0)) || fail "no archives in $DEST after the pull"
|
||||
|
||||
for name in "${archives[@]}"; do
|
||||
[[ "$name" =~ ^[0-9]{8}T[0-9]{6}Z(-[a-z0-9]+)?\.tar\.age$ ]] ||
|
||||
fail "unexpected file in $DEST: $name"
|
||||
done
|
||||
|
||||
newest="${archives[${#archives[@]} - 1]}"
|
||||
newest_epoch="$(archive_epoch "$newest")" || fail "cannot parse a timestamp from $newest"
|
||||
age_seconds=$(($(date -u +%s) - newest_epoch))
|
||||
|
||||
if ((age_seconds > MAX_STALENESS_HOURS * 3600)); then
|
||||
fail "newest archive $newest is $((age_seconds / 3600))h old (limit ${MAX_STALENESS_HOURS}h): VPS backups may have stopped"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Verify every archive once; unchanged ciphertext is only re-hashed.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
for name in "${archives[@]}"; do
|
||||
verify_archive "$DEST/$name" || fail "$name failed verification"
|
||||
done
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Retention: 7 daily, then one per ISO week for 4 weeks, then one per calendar
|
||||
# month for 6 months. Archives outside all three windows are deleted.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
now="$(date -u +%s)"
|
||||
pruned=0
|
||||
declare -A kept_week=()
|
||||
declare -A kept_month=()
|
||||
|
||||
mapfile -t newest_first < <(
|
||||
find "$DEST" -maxdepth 1 -type f -name '*.tar.age' -printf '%f\n' | sort -r
|
||||
)
|
||||
|
||||
for name in "${newest_first[@]}"; do
|
||||
epoch="$(archive_epoch "$name")" || continue
|
||||
age_days=$(((now - epoch) / 86400))
|
||||
|
||||
if ((age_days < KEEP_DAILY_DAYS)); then
|
||||
continue
|
||||
fi
|
||||
|
||||
week="$(date -u -d "@$epoch" +%G-W%V)"
|
||||
if ((age_days < KEEP_WEEKLY_WEEKS * 7)) && [[ -z "${kept_week[$week]:-}" ]]; then
|
||||
kept_week[$week]=1
|
||||
continue
|
||||
fi
|
||||
|
||||
month="$(date -u -d "@$epoch" +%Y-%m)"
|
||||
if ((age_days < KEEP_MONTHLY_MONTHS * 31)) && [[ -z "${kept_month[$month]:-}" ]]; then
|
||||
kept_month[$month]=1
|
||||
continue
|
||||
fi
|
||||
|
||||
echo "backup-pull: pruning $name" >&2
|
||||
rm -f -- "$DEST/$name" "$DEST/.verified/$name"
|
||||
pruned=$((pruned + 1))
|
||||
done
|
||||
|
||||
log_event pull.completed ",\"newest\":\"$newest\",\"archives\":$(( ${#archives[@]} - pruned )),\"verified\":$VERIFIED,\"pruned\":$pruned,\"duration_s\":$((SECONDS - START_SECONDS))"
|
||||
heartbeat ok
|
||||
@@ -0,0 +1,37 @@
|
||||
# ~/.config/chesstrainer/backup-pull.env -- mode 0600.
|
||||
#
|
||||
# Sourced by ~/.local/bin/chesstrainer-backup-pull. Values below are the
|
||||
# defaults; VPS_HOST and AGE_KEY must be right. Quote anything containing
|
||||
# spaces.
|
||||
|
||||
VPS_HOST=owen@159.69.21.225
|
||||
REMOTE_DIR=/var/backups/chesstrainer/offsite
|
||||
DEST="/mnt/Mint Data/chesstrainer-backups"
|
||||
|
||||
# Private half of the age key. Back this file up somewhere safe: without it
|
||||
# every archive is unreadable.
|
||||
AGE_KEY="$HOME/.config/chesstrainer/age.key"
|
||||
|
||||
# Fail the pull when the newest archive from the VPS is older than this, which
|
||||
# means the VPS backup timer has stopped.
|
||||
MAX_STALENESS_HOURS=48
|
||||
|
||||
# Keep every archive from the last 7 days, then one per ISO week for 4 weeks,
|
||||
# then one per calendar month for 6 months.
|
||||
KEEP_DAILY_DAYS=7
|
||||
KEEP_WEEKLY_WEEKS=4
|
||||
KEEP_MONTHLY_MONTHS=6
|
||||
|
||||
# Restore drill. It runs in a throwaway container matching the production
|
||||
# Postgres major and appends results to DRILL_LOG.
|
||||
# DRILL_IMAGE=postgres:18.6
|
||||
# DRILL_DB=chesstrainer_restore
|
||||
# DRILL_LOG="$DEST/drills.log"
|
||||
EXPECTED_ROLES="chesstrainer chesstrainer_ro"
|
||||
|
||||
# Optional second heartbeat check with a long period (for example 72h), so you
|
||||
# hear about it when the workstation stops pulling. Leave empty to disable.
|
||||
# PULL_HEALTHCHECK_URL=https://hc-ping.com/00000000-0000-0000-0000-000000000000
|
||||
|
||||
# Extra ssh options, space separated.
|
||||
# SSH_OPTIONS="-o BatchMode=yes -o ConnectTimeout=10 -o ServerAliveInterval=15"
|
||||
@@ -0,0 +1,7 @@
|
||||
[Unit]
|
||||
Description=Report a failed Inner Square Chess backup unit
|
||||
Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/logger --tag chesstrainer-backup --priority user.err --stderr backup.unit-failed: %i failed, see journalctl --user -u %i
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Monthly restore drill for Inner Square Chess backups
|
||||
Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md
|
||||
OnFailure=chesstrainer-backup-alert@%N.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=%h/.local/bin/chesstrainer-backup-drill
|
||||
WorkingDirectory=%h
|
||||
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
|
||||
Nice=10
|
||||
TimeoutStartSec=20min
|
||||
SyslogIdentifier=chesstrainer-backup-drill
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Monthly restore drill for Inner Square Chess backups
|
||||
Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-01 09:00:00
|
||||
RandomizedDelaySec=30m
|
||||
AccuracySec=5m
|
||||
Persistent=true
|
||||
Unit=chesstrainer-backup-drill.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,15 @@
|
||||
[Unit]
|
||||
Description=Pull encrypted Inner Square Chess backups from the VPS
|
||||
Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
OnFailure=chesstrainer-backup-alert@%N.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=%h/.local/bin/chesstrainer-backup-pull
|
||||
WorkingDirectory=%h
|
||||
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
|
||||
Nice=10
|
||||
TimeoutStartSec=15min
|
||||
SyslogIdentifier=chesstrainer-backup-pull
|
||||
@@ -0,0 +1,14 @@
|
||||
[Unit]
|
||||
Description=Daily pull of encrypted Inner Square Chess backups
|
||||
Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 12:00:00
|
||||
OnBootSec=10m
|
||||
RandomizedDelaySec=10m
|
||||
AccuracySec=1m
|
||||
Persistent=true
|
||||
Unit=chesstrainer-backup-pull.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in New Issue
Block a user