Files
chess-endgame-trainer/deploy
owenrees b41c8bf1a6 Fix deploy docs: Bot Fight Mode status and probe reinstall (#167)
* Record that Bot Fight Mode is off

Both docs still claimed Bot Fight Mode was enabled. It is off; the Caddy
origin allowlist stays in force. Neither CI nor the on-box probe can get
past it on the free plan, so re-enabling needs the changes tracked in #153.

* Note that the probe must be installed to two places

The timer runs /usr/local/sbin/chesstrainer-probe, not the copy under
/opt/chesstrainer/src. Say so next to the install block, as the backup
section already does, so a changed deploy/probe does not silently drift
from what the timer runs.
2026-11-13 13:57:33 +01:00
..

Production deployment

Phoenix release on the shared Ubuntu VPS, behind Caddy, with Postgres in Docker.

Internet
  └─ Cloudflare (TLS, WAF, edge rate limiting)
       └─ Caddy :443 (cloudflare ranges only)
            └─ Phoenix release 127.0.0.1:4100
                 └─ Postgres 127.0.0.1:5432

Pinned versions

Thing Version
OS Ubuntu 24.04.5 LTS, x86_64
glibc 2.39
Erlang 28.5
Elixir 1.19.5-otp-28
Rust 1.98.1
Node.js 24 (asset build only)
Postgres postgres:18.6, pinned directly in the unified compose
Caddy 2.11.9
Docker 29.9.3
Docker Compose v5.7.0
GitHub runner ubuntu-24.04

The release ships its own ERTS. The host needs the runtime libraries for the Rust NIF (libstdc++6), OpenSSL, and CA certificates.

Host prerequisites

A new Ubuntu 24.04 VPS needs:

  • Docker Engine and the Compose plugin
  • Caddy
  • curl, ca-certificates, libstdc++6, openssl
  • Node.js 24 + npm if you build the release on this host instead of CI
  • DNS and TLS for the domain
  • A non-root deploy user with SSH access

Install the host libraries:

sudo apt update
sudo apt install -y curl ca-certificates libstdc++6 openssl

Install Docker and Caddy from their official repositories. The Caddy install commands already used for the static site are in deploy/static/README.md.

Firewall:

sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

Do not open 4100 or 5432. Phoenix binds to loopback, and Postgres is published only on 127.0.0.1.

DNS:

  • A/AAAA for innersquare.app and www point at the VPS.
  • Do not remove the existing mail records:
    • root MX records for support@innersquare.app
    • SES DKIM CNAMEs
    • the mail.innersquare.app custom MAIL FROM MX/TXT records
    • DMARC at _dmarc.innersquare.app

Shared-host choices

  • Port 4100 is used for Phoenix because another BEAM process already owns 4000 on this VPS.
  • Postgres runs as chesstrainer-db in the VPS's unified docker-compose.yml on 127.0.0.1:5432; that port is free. The image version is pinned directly in that file.
  • Caddy is shared and managed manually. This repo ships a reference block in deploy/Caddyfile; it is never installed automatically.
  • The Postgres volume is named chesstrainer_pgdata, so it cannot collide with other Docker projects.

Directory layout

/opt/chesstrainer/
├── releases/
│   ├── 20261109120000/
│   └── 20261109153000/
├── current -> releases/20261109153000
└── src/                 # optional checkout of this repo / deploy scripts

/etc/chesstrainer/env is the application environment file, owned by root:root with mode 0600.

/etc/chesstrainer/postgres.env holds the Postgres container credentials, also root:root and 0600.

/etc/chesstrainer/probe.env holds the public probe's URL and its healthchecks.io ping URL, also root:root and 0600. See §Monitoring and alerting.

The app runs as the dedicated chesstrainer system user. It must own /opt/chesstrainer so it can write priv/static/backups.

Database dumps live in /var/backups/chesstrainer, outside /opt/chesstrainer and priv/static. See §Backups.

Copy the deploy artifacts

Run these commands from a checkout of this repo on the VPS, or copy the deploy/ directory to /opt/chesstrainer/src/deploy.

sudo install -d -o root -g root /opt/chesstrainer/src/deploy
sudo cp -a deploy/. /opt/chesstrainer/src/deploy/

If you prefer to keep the checkout elsewhere, use that path in the commands below.

Application env file

/etc/chesstrainer/env:

SECRET_KEY_BASE=...
DATABASE_URL=ecto://chesstrainer:...@127.0.0.1:5432/chesstrainer_prod
PHX_HOST=innersquare.app
PORT=4100
POOL_SIZE=10
TRUST_FORWARDED_FOR=true

PRELAUNCH=true
MAILING_LIST_ENABLED=false

SES_REGION=eu-north-1
SES_ACCESS_KEY_ID=AKIA...
SES_SECRET_ACCESS_KEY=...

MATOMO_URL=https://analytics.owenrees.eu/
MATOMO_SITE_ID=6

TABLEBASE_PATH=/opt/chesstrainer/tablebases

# Optional:
# RELEASE_COOKIE=...

The production endpoint binds to 127.0.0.1 only. Caddy is the public entry point.

Matomo analytics

Set MATOMO_URL and MATOMO_SITE_ID to enable cookieless analytics on public pages and the guest trainer. Signed-in account and admin pages are excluded.

The privacy controls, Matomo server settings, retention, and opt-out behaviour are documented in docs/analytics.md.

Generate the two secrets:

openssl rand -base64 64   # SECRET_KEY_BASE
openssl rand -hex 32      # Postgres password

Secret key base

SECRET_KEY_BASE is required in production; the release raises at boot without it. It signs cookies, session tokens, LiveView tokens, and mailing-list confirmation links.

Generate it with mix phx.gen.secret or:

openssl rand -base64 64

Keep it stable. Rotating it signs every user out and invalidates outstanding confirmation and unsubscribe links. Never reuse the dev/test value or commit it.

Verify it is set without printing it:

sudo awk -F= '/^SECRET_KEY_BASE=/{print length($2)}' /etc/chesstrainer/env

Postgres env file

/etc/chesstrainer/postgres.env:

POSTGRES_USER=chesstrainer
POSTGRES_PASSWORD=...
POSTGRES_DB=chesstrainer_prod

Postgres 18 stores data under /var/lib/postgresql/18/docker, so the Compose volume mounts the parent:

volumes:
  - chesstrainer_pgdata:/var/lib/postgresql

Local tablebases (optional)

The production tablebase directory is /opt/chesstrainer/tablebases. Keep the .rtbw/.rtbz files flat in that directory, and make sure the chesstrainer user can read them:

sudo install -d -o chesstrainer -g chesstrainer /opt/chesstrainer/tablebases
sudo chown -R chesstrainer:chesstrainer /opt/chesstrainer/tablebases
sudo chmod -R u=rwX,g=rX,o= /opt/chesstrainer/tablebases

Set the path in /etc/chesstrainer/env:

TABLEBASE_PATH=/opt/chesstrainer/tablebases

Then restart the service, because TABLEBASE_PATH is read at boot:

sudo systemctl restart chesstrainer

Check /health. It should move from degraded / not_configured to ok / local:

curl -s http://127.0.0.1:4100/health
sudo journalctl -u chesstrainer -n 50 --no-pager | grep -i tablebase

If the path is missing, unreadable, or empty, the app falls back to Lichess and health stays degraded but still returns 200.

To refresh the files:

mix tablebase.fetch --path /opt/chesstrainer/tablebases
sudo chown -R chesstrainer:chesstrainer /opt/chesstrainer/tablebases
sudo systemctl restart chesstrainer

First-time setup

Run as root or with sudo:

useradd --system --user-group \
  --home-dir /opt/chesstrainer \
  --create-home \
  --shell /usr/sbin/nologin \
  chesstrainer

install -d -o chesstrainer -g chesstrainer /opt/chesstrainer/releases
install -d -o chesstrainer -g chesstrainer /opt/chesstrainer/src

install -d -m 755 /etc/chesstrainer
install -m 600 /dev/null /etc/chesstrainer/env
install -m 600 /dev/null /etc/chesstrainer/postgres.env
chown root:root /etc/chesstrainer/env /etc/chesstrainer/postgres.env

Edit both files with sudoedit and set the generated secrets and DATABASE_URL.

Unified Compose: add Postgres

Add the chesstrainer-db service and chesstrainer_pgdata volume from deploy/postgres.compose.yml to the VPS's existing unified docker-compose.yml. Do not run the reference file as a separate project.

The image is pinned directly in the unified compose:

image: postgres:18.6

For an immutable pin, replace it with the exact digest:

image: postgres:18.6@sha256:...

There is no POSTGRES_IMAGE environment variable involved.

Because postgres.env is root-owned 0600, run Compose with sudo:

sudo docker compose -f <UNIFIED_COMPOSE> config >/dev/null
sudo docker compose -f <UNIFIED_COMPOSE> up -d chesstrainer-db
sudo docker compose -f <UNIFIED_COMPOSE> ps chesstrainer-db
sudo docker compose -f <UNIFIED_COMPOSE> logs --tail=50 chesstrainer-db

Verify the database:

sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
  psql -U chesstrainer -d chesstrainer_prod -c 'select 1;'

To change the Postgres version:

# edit the image: line in the unified compose
sudo docker compose -f <UNIFIED_COMPOSE> pull chesstrainer-db
sudo docker compose -f <UNIFIED_COMPOSE> up -d chesstrainer-db

Read-only database access

The app connects as chesstrainer, which is a superuser. Use a separate read-only role for inspection, debugging, and dashboards.

Create it once per environment:

sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
  psql -U chesstrainer -d chesstrainer_prod <<'SQL'
CREATE ROLE chesstrainer_ro LOGIN PASSWORD '<generated-password>';
GRANT CONNECT ON DATABASE chesstrainer_prod TO chesstrainer_ro;
GRANT USAGE ON SCHEMA public TO chesstrainer_ro;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO chesstrainer_ro;

-- Cover tables created by future migrations. Migrations run as chesstrainer.
ALTER DEFAULT PRIVILEGES FOR ROLE chesstrainer IN SCHEMA public
  GRANT SELECT ON TABLES TO chesstrainer_ro;

-- Optional: read other sessions and server settings in pg_stat_* views.
GRANT pg_read_all_stats TO chesstrainer_ro;
SQL

Generate the password with openssl rand -hex 32 and keep it in a password manager. The only access it has is SELECT; there are no INSERT, UPDATE, DELETE, or object-creation grants. If the role already exists, skip the CREATE ROLE line and re-run the grants.

Optional defence in depth:

ALTER ROLE chesstrainer_ro SET default_transaction_read_only = on;

This is not a hard boundary (the role can override it per session), but it turns an accidental write into an error.

Connect from the VPS

docker exec connects over the container's Unix socket, which the Postgres image trusts locally, so no password is needed:

sudo docker exec -it chesstrainer-db \
  psql -U chesstrainer_ro -d chesstrainer_prod

Connect from a workstation over SSH

Postgres is published on 127.0.0.1:5432 only. OpenSSH also disables forwarding by default in /etc/ssh/sshd_config (AllowTcpForwarding no). Allow local forwarding for the deploy user only, restricted to Postgres, by appending this to the end of that file:

Match User <username>
    AllowTcpForwarding local
    PermitOpen 127.0.0.1:5432

Then validate and reload:

sudo sshd -t
sudo systemctl reload ssh
sudo sshd -T -C user=<username>,host=localhost,addr=127.0.0.1 | grep -iE 'allowtcpforwarding|permitopen'

Open the tunnel (local port 5433 because the dev database already binds 5432):

ssh -N -L 5433:127.0.0.1:5432 <VPS_USER>@<VPS_HOST>

Connect from a second terminal:

psql "postgres://chesstrainer_ro@127.0.0.1:5433/chesstrainer_prod"

Install postgresql-client if psql is not available on the workstation.

To avoid the password prompt, add this to ~/.pgpass (mode 0600):

127.0.0.1:5433:chesstrainer_prod:chesstrainer_ro:<password>

DBeaver uses the same tunnel. Create a PostgreSQL connection with host 127.0.0.1, port 5432, database chesstrainer_prod, user chesstrainer_ro, then on the SSH tab enable the tunnel with host <VPS_HOST>, port 22, user <VPS_USER>, and a public key. The AllowTcpForwarding local rule above is required for any tunnelled client.

Reset the role password:

sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
  psql -U chesstrainer -d chesstrainer_prod \
  -c "ALTER ROLE chesstrainer_ro PASSWORD '<new-password>';"

Verify the role is read-only:

psql "postgres://chesstrainer_ro@127.0.0.1:5433/chesstrainer_prod" \
  -c 'select current_user, current_database();'
psql "postgres://chesstrainer_ro@127.0.0.1:5433/chesstrainer_prod" \
  -c 'create table should_fail (i int);'

The second command must fail with permission denied for schema public.

systemd unit

From the deploy artifact copy:

sudo install -o root -g root -m 644 /opt/chesstrainer/src/deploy/chesstrainer.service \
  /etc/systemd/system/chesstrainer.service
sudo systemctl daemon-reload
sudo systemctl enable chesstrainer

The unit includes ConditionPathExists=/opt/chesstrainer/current/bin/server, so it is safe to enable before the first release. systemd skips it until a release is in place.

The backup unit and timer are installed separately; see §Backups.

Caddy

Add the site block from deploy/Caddyfile by hand. It imports cloudflare-ranges.caddy, which is generated by deploy/update-cloudflare-ranges.sh and refreshed by a systemd timer.

Caddy must only accept Cloudflare traffic and must replace X-Forwarded-For with Cloudflare's CF-Connecting-IP value:

innersquare.app {
    encode zstd gzip

    @cloudflare {
        import cloudflare-ranges.caddy
    }

    handle @cloudflare {
        reverse_proxy 127.0.0.1:4100 {
            header_up X-Forwarded-For {http.request.header.CF-Connecting-IP}
        }
    }

    handle {
        respond "Access Denied" 403
    }
}

www.innersquare.app {
    redir https://innersquare.app{uri} permanent
}

Install the range file and the hourly refresh timer:

sudo install -m 0755 /opt/chesstrainer/src/deploy/update-cloudflare-ranges.sh \
  /usr/local/sbin/chesstrainer-update-cloudflare-ranges
sudo install -m 0644 /opt/chesstrainer/src/deploy/cloudflare-ranges.caddy \
  /etc/caddy/cloudflare-ranges.caddy
sudo install -m 0644 /opt/chesstrainer/src/deploy/chesstrainer-cloudflare-ranges.service \
  /etc/systemd/system/chesstrainer-cloudflare-ranges.service
sudo install -m 0644 /opt/chesstrainer/src/deploy/chesstrainer-cloudflare-ranges.timer \
  /etc/systemd/system/chesstrainer-cloudflare-ranges.timer

The installed pieces are:

Path Mode Purpose
/usr/local/sbin/chesstrainer-update-cloudflare-ranges 0755 fetch/validate/reload script
/etc/caddy/cloudflare-ranges.caddy 0644 generated remote_ip list
/etc/systemd/system/chesstrainer-cloudflare-ranges.service 0644 oneshot service
/etc/systemd/system/chesstrainer-cloudflare-ranges.timer 0644 hourly timer

If /opt/chesstrainer/src/deploy is not present on the VPS, create the same four files directly with sudo tee from the canonical contents in deploy/. The timer only depends on the installed copies; it does not read the repo.

Run it once manually and enable the timer:

sudo systemctl daemon-reload
sudo systemctl enable --now chesstrainer-cloudflare-ranges.timer
sudo systemctl start chesstrainer-cloudflare-ranges.service
sudo journalctl -u chesstrainer-cloudflare-ranges.service -n 50 --no-pager

The script fetches https://www.cloudflare.com/ips-v4 and https://www.cloudflare.com/ips-v6, validates the result, and reloads Caddy only when the ranges actually changed. If the download, Caddy validation or reload fails, it keeps or restores the previous file.

Check the timer and last run:

systemctl list-timers chesstrainer-cloudflare-ranges.timer
systemctl status chesstrainer-cloudflare-ranges.service
journalctl -u chesstrainer-cloudflare-ranges.service --since "24 hours ago"

Do not edit /etc/caddy/cloudflare-ranges.caddy by hand; the next timer run overwrites it.

To update the refresher itself, replace the installed script or unit files from the deploy directory, then:

sudo systemctl daemon-reload
sudo systemctl restart chesstrainer-cloudflare-ranges.timer
sudo systemctl start chesstrainer-cloudflare-ranges.service

Without this, TRUST_FORWARDED_FOR=true records a Cloudflare edge address instead of the visitor, and the login throttle and guest puzzle limit bucket whole Cloudflare POPs together.

If a new VPS uses imports, put the site block in its own file instead; keep the import cloudflare-ranges.caddy path next to the Caddyfile.

Do not switch Caddy until a release is running and verified on 127.0.0.1:4100. Before that, keep the static coming-soon site live.

Validate and reload:

sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy

A direct request to the origin with a forged CF-Connecting-IP header should receive 403 Access Denied.

Do not let a deploy overwrite the shared Caddyfile.

See docs/security.md for the application throttle, real-IP setup and the Cloudflare edge controls. Production currently has Bot Fight Mode off (see #153) and the free-plan login flood rule: 10 POSTs per 10 seconds per IP, blocked for 10 seconds.

Building a release

Build on a machine matching Ubuntu 24.04 / glibc 2.39, or let CI do it. The release tarball is the contents of the release directory:

npm ci --prefix assets --no-audit --no-fund

export MIX_ENV=prod
mix deps.get --only prod
mix assets.deploy
mix release

tar -czf chesstrainer-release.tar.gz -C _build/prod/rel/chesstrainer .

npm ci is required before mix assets.deploy: Tailwind imports the Chessground CSS and esbuild bundles the package from assets/node_modules.

mix assets.deploy runs compile first so the generated LiveView colocated CSS exists.

First deploy and Caddy switch

Upload the tarball:

scp chesstrainer-release.tar.gz <VPS_USER>@<VPS_HOST>:/tmp/

On the VPS:

sudo /opt/chesstrainer/src/deploy/activate /tmp/chesstrainer-release.tar.gz

The script:

  1. Extracts into /opt/chesstrainer/releases/<timestamp>
  2. Fixes ownership
  3. Takes a verified pre-deploy database dump, aborting if it fails (see §Backups)
  4. Flips /opt/chesstrainer/current
  5. Restarts chesstrainer.service
  6. Migrations run via ExecStartPre
  7. Polls /health
  8. Rolls back to the previous release on failure
  9. Prunes old releases, keeping 5 by default

Verify locally before touching Caddy:

curl -i http://127.0.0.1:4100/health
curl -i http://127.0.0.1:4100/
sudo journalctl -u chesstrainer -n 100 --no-pager

Then add the Caddy block and reload. Verify publicly:

curl -I https://innersquare.app/
curl -I https://innersquare.app/privacy
curl -I https://innersquare.app/terms-conditions

Subsequent deploys

scp chesstrainer-release.tar.gz <VPS_USER>@<VPS_HOST>:/tmp/
ssh <VPS_USER>@<VPS_HOST>
sudo /opt/chesstrainer/src/deploy/activate /tmp/chesstrainer-release.tar.gz

Optional overrides:

RELEASE_ID=20261109120000 \
HEALTH_URL=http://127.0.0.1:4100/health \
KEEP_RELEASES=5 \
sudo -E /opt/chesstrainer/src/deploy/activate /tmp/chesstrainer-release.tar.gz

Every activation takes a verified pre-deploy dump before migrations run. If Postgres is intentionally unavailable and you accept the risk, add SKIP_PRE_DEPLOY_BACKUP=1 to the override list; see §Backups.

GitHub Actions CD

Merging to main (or running the workflow manually) tests, builds, uploads, and deploys the release. The deploy job uses SSH as <username>.

Deploy key

Generate a dedicated key, not your personal one:

ssh-keygen -t ed25519 -C "github-actions-chesstrainer" \
  -f ~/.ssh/chesstrainer_deploy -N ""
ssh-copy-id -i ~/.ssh/chesstrainer_deploy.pub <username>@<VPS_HOST>

VPS_HOST must be the public IP or DNS name reachable from GitHub. Do not use a local SSH alias such as personal-projects.

Passwordless sudo for the deploy scripts

sudo visudo -f /etc/sudoers.d/chesstrainer-deploy
<username> ALL=(root) NOPASSWD: /opt/chesstrainer/src/deploy/activate, /opt/chesstrainer/src/deploy/rollback

Validate:

sudo chown root:root /etc/sudoers.d/chesstrainer-deploy
sudo chmod 0440 /etc/sudoers.d/chesstrainer-deploy
sudo visudo -c
sudo -n -l /opt/chesstrainer/src/deploy/activate
sudo -n -l /opt/chesstrainer/src/deploy/rollback

GitHub environment

Create the production environment, optionally with required reviewers, and add environment secrets:

Secret Value
VPS_HOST public VPS IP or DNS name
VPS_USER <username>
VPS_SSH_KEY private deploy key
VPS_KNOWN_HOSTS ssh-keyscan -H <VPS_HOST> output

Generate known_hosts with:

ssh-keyscan -H <VPS_HOST>

Workflow

The workflow:

  1. Runs mix precommit against Postgres.
  2. Builds the release artifact on ubuntu-24.04.
  3. On main or workflow_dispatch, uploads the tarball, installs the SSH key, runs sudo -n .../activate, and performs public smoke checks. Activation takes a pre-deploy dump first and fails if it cannot.
  4. A failed activation health check rolls back to the previous release.

Manual dispatch is available from the Actions UI or:

gh workflow run ci.yml --ref main

Rollback

sudo /opt/chesstrainer/src/deploy/rollback

Or target a specific release:

sudo /opt/chesstrainer/src/deploy/rollback 20261109120000

The script flips current, restarts the service, and checks /health.

Rollback reverts the release, not the database: migrations are not reversed. If a migration changed data, restore the matching pre-deploy dump instead; see §Backups.

Logs and health

sudo systemctl status chesstrainer
sudo journalctl -u chesstrainer -n 100 --no-pager
curl -i http://127.0.0.1:4100/health

/health returns JSON with status, db, tablebase, and process. It returns 200 for ok/degraded and 503 for down. A missing local tablebase reports degraded but still 200 because Lichess fallback is available. See §Monitoring and alerting for the public probe that watches it, and for what a green check does and does not prove.

Reading production logs

Production emits one JSON object per line. Use -o cat so journald prints the raw line instead of its own field wrapper, then pipe to jq:

# Recent logs, pretty-printed
sudo journalctl -u chesstrainer -o cat -n 50 --no-pager | jq .

# Errors only
sudo journalctl -u chesstrainer -o cat --since "1 hour ago" | jq 'select(.level == "error")'

# Everything from one request
sudo journalctl -u chesstrainer -o cat | jq 'select(.request_id == "REQUEST_ID")'

# Crash reasons and stacktraces
sudo journalctl -u chesstrainer -o cat | jq -r 'select(.error) | .error.reason, (.error.stacktrace[]?)'

Each line has timestamp, level and message, plus request_id when one exists. Access lines add method, path, status and duration_ms; crashes add an error object with kind, reason and stacktrace. Query parameters and request bodies are never logged, token-shaped path segments are redacted, and email addresses or credentials in metadata render as [REDACTED].

The full field table, redaction rules and examples for adding background logs are in docs/logging.md.

Dev and test still use the readable [level] message formatter.

Keep the journal

By default some distributions keep journald logs in memory only. On the VPS, make sure they survive a reboot and have a retention budget:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
journalctl --disk-usage

Set Storage=persistent and a SystemMaxUse= value in /etc/systemd/journald.conf if the distribution does not already. journald rate-limits bursts; if a crash storm drops lines, raise RateLimitBurst or RateLimitIntervalSec there.

Monitoring and alerting

Three checks watch production at three different layers:

Where What Notices Alerts
VPS chesstrainer-probe.timer → chesstrainer-probe.service the user-facing HTTP path: DNS, Cloudflare, Caddy, Phoenix, Postgres healthchecks.io
VPS Beszel hub + agent host liveness, CPU, memory, disk, container restarts and OOM kills Beszel notifications
VPS chesstrainer-backup.timer heartbeat whether the nightly dump finished healthchecks.io (its own check)

The backup heartbeat is documented in §Backups. The probe and the Beszel gap are below.

On-box prober, off-box alerting

The probe runs on the VPS and curls the public hostname, so it crosses the same path a visitor does: Cloudflare's DNS and edge, the proxy back to Caddy, Caddy's Cloudflare allowlist, the Phoenix router, and one database query.

That is deliberately not 127.0.0.1:4100. A loopback probe stays green through every failure that makes the site unusable: a bad cloudflare-ranges.caddy reload, a stopped Caddy (521), an expired origin certificate (526), a deleted proxied DNS record, a WAF rule blocking /health, or a Cloudflare incident. Beszel is green through all of those too, because the host and the container are fine.

The trade-off is that the prober shares a host with the app. It cannot report that its own host is down, and it cannot prove the view from another network. Host death is caught by silence: the timer stops, no ping arrives, and healthchecks.io — which is off-box — alerts after the grace period. That is the whole reason the check-in URL points at a SaaS rather than at another box here. Do not describe this as external uptime monitoring.

The healthchecks.io check

Create the check at healthchecks.io:

  • Name innersquare.app /health, period 2 minutes, grace 2 minutes
  • Integrations: email and Discord (webhook). Use each integration's test notification once, and keep recovery notifications enabled so a resolved incident is visible.
  • Copy the ping URL. It is a secret: anyone who has it can mark the check green. It belongs only in the root-owned env file.

Every check-in carries evidence:

  • A healthy run posts the raw /health JSON.
  • A tablebase mismatch still posts a success, because a Lichess fallback is not an outage. The body carries the expected and actual values, so the mismatch is visible in the check history without alerting anyone.
  • A non-200, a malformed body, a db that is not ok, or a network/TLS failure posts to <ping-url>/fail with a JSON body containing the reason, the HTTP status, the curl exit code, the response body, and a loopback origin_status/origin_body for comparison. The origin fields are evidence only; the verdict comes from the public request.

A ping that cannot be delivered is a journal warning, not a second verdict: the next timer run reports the state again.

Install the probe

Write the env file, root-owned and 0600:

sudo install -m 600 /dev/null /etc/chesstrainer/probe.env
sudoedit /etc/chesstrainer/probe.env

Start from deploy/probe.env.example. PROBE_URL and HEALTHCHECK_URL are the only required values. Then install the script, the units, and the timer:

sudo install -m 0755 /opt/chesstrainer/src/deploy/probe \
  /usr/local/sbin/chesstrainer-probe
sudo install -m 0644 \
  /opt/chesstrainer/src/deploy/chesstrainer-probe.service \
  /opt/chesstrainer/src/deploy/chesstrainer-probe.timer \
  /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now chesstrainer-probe.timer

The timer runs the installed /usr/local/sbin/chesstrainer-probe, not the copy under /opt/chesstrainer/src, because that tree belongs to the application user. When deploy/probe changes, install it to both places so the timer and the repo agree.

Verify:

systemctl list-timers chesstrainer-probe.timer
sudo journalctl -u chesstrainer-probe -n 5 -o cat

Expect a probe.ok line every two minutes and a green check within one period. Each run prints exactly one JSON line (probe.ok, probe.degraded or probe.failed), like the backup script, so #140's log shipping can pick it up.

What /health does and does not mean

/health returns status, db, tablebase and process. It returns 200 for ok and degraded, and 503 for down.

  • db is one SELECT 1 through the connection pool. It proves the pool can reach Postgres. It does not prove migrations are current or that writes work.
  • tablebase reflects the local Syzygy directory. fallback means lookups go to Lichess. degraded is 200 because the fallback is usable.
  • process is always "ok". It is not independent evidence of liveness.

A green probe therefore means: DNS resolved, Cloudflare served the request, Caddy routed it under the Cloudflare allowlist, Phoenix answered, and Postgres answered one query. It does not mean background jobs are running, SES accepts mail, writes succeed, tablebase lookups are fast, or the site is fast under load. Beszel covers resource exhaustion and container restarts; the release health check and the logs cover deploy mistakes.

Beszel's blind spot

Beszel watches the host: agent connection, CPU, memory, disk, container restarts, OOM kills. Its hub runs on the same VPS as the agent, so a VPS outage or network loss silences it as well — that case is covered by the probe check going quiet. More importantly, Beszel never calls /health. A Phoenix release returning 500s on a healthy host, or Postgres refusing connections inside a running container, reads green in Beszel and red in the probe. The two are complementary, not redundant.

Drills

Nothing below touches production data, and the app keeps serving throughout.

--dry-run needs no network and no env file. It evaluates the success, degraded, fail, non-JSON and unreachable fixtures and sends no check-ins:

sudo /usr/local/sbin/chesstrainer-probe --dry-run

A degraded check-in, recorded but never alerting. A value passed on the command line wins over the env file:

sudo EXPECT_TABLEBASE=not_configured /usr/local/sbin/chesstrainer-probe

Expect exit 0, a green check, and a body showing tablebase: local against expected_tablebase: not_configured. No email and no Discord message.

The one fail ping, and the full alert path. Stop the timer first so the next scheduled run cannot mask the result:

sudo systemctl stop chesstrainer-probe.timer
sudo PROBE_URL=https://innersquare.app/health-does-not-exist \
  /usr/local/sbin/chesstrainer-probe

Expect exit 1, one probe.failed line with http_status: 404, one fail ping whose body shows origin_status: 200, one alert in Discord/email, and no second fail ping. Then send a real check-in and re-arm:

sudo /usr/local/sbin/chesstrainer-probe
sudo systemctl start chesstrainer-probe.timer

The real run posts a success, and healthchecks.io notifies that the check is back up. For the unreachable branch, point the probe at a closed port instead; the origin evidence should still show 200:

sudo PROBE_URL=https://127.0.0.1:9/health /usr/local/sbin/chesstrainer-probe

Planned maintenance: stopping the timer is reported as a down check after period plus grace, so about four minutes. Pause the check in the healthchecks.io UI for the duration, or accept the alert.

The probe reaches the edge as a plain curl. If a bot policy or WAF rule ever starts rejecting it, the fail body's origin_status shows the app itself is healthy; that is an edge-policy change, not an application outage. Re-enabling Bot Fight Mode is tracked in #153.

Interactive shell and user management

Attach an IEx shell to the running release. This does not restart the service and does not require direct database access:

sudo /opt/chesstrainer/current/bin/chesstrainer remote

You should get a prompt like iex(chesstrainer@<host>)1>. Exit with Ctrl-C twice. Never call System.halt/0, :init.stop/0, or Application.stop(:chesstrainer) from this shell — those stop production.

Use it for account and role management. Go through the application contexts (not SQL) so changesets, bcrypt hashing, timestamps, and token bookkeeping stay consistent:

# Look up by email or username
u = Chesstrainer.Accounts.get_user_by_login("owen")

# Promote or demote
{:ok, u} = Chesstrainer.Accounts.update_user_role(u, :super_admin)

# Confirm an account without sending email
{:ok, u} = Chesstrainer.Repo.update(Chesstrainer.Accounts.User.confirm_changeset(u))

# Create an account, generating its password server-side
pw = :crypto.strong_rand_bytes(18) |> Base.url_encode64(padding: false)

{:ok, u} =
  Chesstrainer.Accounts.register_user(%{
    email: "you@example.com",
    username: "owen",
    password: pw
  })

{:ok, u} = Chesstrainer.Repo.update(Chesstrainer.Accounts.User.confirm_changeset(u))
pw

Notes:

  • register_user/1 hashes the password with bcrypt; password is virtual and is never written to the database.
  • Do not type passwords into the shell if you can avoid it — IEx may keep a history file — and never paste a password from shell history. Generate one as above and change it later at /users/profile/settings.
  • mix accounts.promote still works when a compiled source checkout is present on the host, but the release shell is the supported production path.
  • There is no guard against demoting the last super_admin outside the future admin UI, so check the result before demoting.

If remote fails while booting its client VM (for example Kernel pid terminated (logger) with code_server/persistent_term in the trace), the deployed release is stale, mixed, or missing IEx: rebuild and redeploy it. To attach in the meantime, use the release's own ERTS:

sudo bash -c '
  COOKIE=$(cat /opt/chesstrainer/current/releases/COOKIE)
  exec /opt/chesstrainer/current/erts-*/bin/erl \
    -sname debug -setcookie "$COOKIE" -remsh chesstrainer
'

That shell speaks Erlang syntax, so quote Elixir module atoms, e.g. 'Elixir.Chesstrainer.Accounts':register_user(...).

Backups

Backups are produced on the VPS, encrypted there, and pulled to a workstation. Nothing leaves the VPS unencrypted, and the VPS never holds the key that could read the archives.

What runs

Where What When Retention
VPS chesstrainer-backup.timer → chesstrainer-backup.service daily 03:15 UTC, Persistent=true, 5 min jitter 3 days plaintext, 14 days encrypted
VPS the same script, called by deploy/activate before every migration last 5 pre-deploy sets
Workstation chesstrainer-backup-pull.timer daily 12:00 local, 10 min after boot 7 daily, 4 weekly, 6 monthly
Workstation chesstrainer-backup-drill.timer monthly on the 1st at 09:00 drills.log is never pruned

Each run writes one JSON line (backup.completed, backup.failed, backup.skipped, pull.completed, pull.failed, pull.skipped, drill.completed, drill.failed) so #140's log shipping can alert on it. A skipped line means another run held the lock; it is not an alarm.

What a dump contains

  • <ts>.dump — pg_dump -Fc of chesstrainer_prod
  • <ts>.globals.sql — pg_dumpall --globals-only, enough to recreate chesstrainer and chesstrainer_ro on a fresh cluster
  • <ts>.counts — row counts for users, endgames, attempts, used by the drill
  • <ts>.sha256 — checksums for the three files above

Not backed up, by design: tablebases and the tablebase cache (both are re-fetchable), anything under priv/static, and release tarballs. Dumps contain user emails and bcrypt hashes: they are 0600 root while plaintext, encrypted before they leave the box, and pruned aggressively.

RPO and RTO

RPO is 24 hours on the VPS and "the age of the newest pulled archive" off-box. The pull runs daily, so a laptop that is off for a day leaves the newest off-site copy around 48 hours old; the VPS keeps 14 days of encrypted archives, which bounds how far behind the pull can fall. RTO is under an hour: create the database, restore the newest dump, activate a release (see Disaster recovery).

One-time setup: VPS

Install age. Encryption happens on the server; only the workstation can decrypt.

sudo apt install age

Create the configuration file. Generate AGE_RECIPIENT on the workstation first (next section) and paste the public key.

sudo install -m 600 /dev/null /etc/chesstrainer/backup.env
sudoedit /etc/chesstrainer/backup.env

Start from deploy/backup.env.example. The required values are AGE_RECIPIENT and HEALTHCHECK_URL.

Create a healthchecks.io check with a period of 1 day and a grace time of 4 hours, then paste its ping URL into HEALTHCHECK_URL. The script pings it after a good run and <url>/fail after a bad one, so silence means "no backup finished", even when the VPS itself is down.

Install the script and units:

sudo install -m 0755 /opt/chesstrainer/src/deploy/backup \
  /usr/local/sbin/chesstrainer-backup
sudo install -m 0644 \
  /opt/chesstrainer/src/deploy/chesstrainer-backup.service \
  /opt/chesstrainer/src/deploy/chesstrainer-backup.timer \
  /opt/chesstrainer/src/deploy/chesstrainer-backup-failure.service \
  /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now chesstrainer-backup.timer

The timer runs the installed /usr/local/sbin/chesstrainer-backup, not the copy under /opt/chesstrainer/src, because that tree is writable by the application user. When deploy/backup changes, copy it to both places so the timer and deploy/activate use the same script.

Run it once and inspect the result:

sudo systemctl start chesstrainer-backup
sudo journalctl -u chesstrainer-backup -n 20 --no-pager -o cat
sudo ls -l /var/backups/chesstrainer /var/backups/chesstrainer/offsite
systemctl list-timers chesstrainer-backup.timer

Expect a .dump, .globals.sql, .counts and .sha256 at mode 0600, and one .tar.age in offsite/ at mode 0644. offsite/ is the only thing the workstation reads, which is why the pull needs no sudo. The vault directory is 0711, so the workstation user can traverse into offsite/ but cannot list the plaintext dumps.

One-time setup: workstation

sudo apt install age rsync
mkdir -p ~/.config/chesstrainer
chmod 700 ~/.config/chesstrainer
age-keygen -o ~/.config/chesstrainer/age.key
chmod 600 ~/.config/chesstrainer/age.key
grep 'public key' ~/.config/chesstrainer/age.key

Copy that public key into AGE_RECIPIENT on the VPS. Back up the private key (password manager, printed copy, offline disk): without it every archive is unreadable, and the VPS deliberately cannot decrypt them.

Create the pull configuration:

install -m 600 /dev/null ~/.config/chesstrainer/backup-pull.env
$EDITOR ~/.config/chesstrainer/backup-pull.env

Start from deploy/workstation/backup-pull.env.example. DEST should point at a real disk, for example /mnt/Mint Data/chesstrainer-backups. The script refuses to run when DEST's parent is missing, so an unmounted disk cannot silently fill the root filesystem.

Set PULL_HEALTHCHECK_URL to a second healthchecks.io check with a long period (for example 72 hours) if you want to hear about it when the workstation stops pulling. Leave it empty for journal-only reporting.

Install the scripts and user units:

REPO=~/Desktop/chess-trainer   # wherever this repo is cloned

install -d ~/.local/bin
install -m 0755 "$REPO/deploy/workstation/backup-pull" ~/.local/bin/chesstrainer-backup-pull
install -m 0755 "$REPO/deploy/workstation/backup-drill" ~/.local/bin/chesstrainer-backup-drill

install -d ~/.config/systemd/user
install -m 0644 \
  "$REPO/deploy/workstation/chesstrainer-backup-pull.service" \
  "$REPO/deploy/workstation/chesstrainer-backup-pull.timer" \
  "$REPO/deploy/workstation/chesstrainer-backup-drill.service" \
  "$REPO/deploy/workstation/chesstrainer-backup-drill.timer" \
  "$REPO/deploy/workstation/chesstrainer-backup-alert@.service" \
  ~/.config/systemd/user/

systemctl --user daemon-reload
systemctl --user enable --now chesstrainer-backup-pull.timer
systemctl --user enable --now chesstrainer-backup-drill.timer
sudo loginctl enable-linger "$USER"

enable-linger lets the user timers run while the laptop is on but you are not logged in. The pull also fires ten minutes after boot, and Persistent=true catches up after the machine was off. Verify lingering stuck:

loginctl show-user "$USER" -p Linger    # Linger=yes

Run the first pull and drill by hand:

~/.local/bin/chesstrainer-backup-pull
ls -l "/mnt/Mint Data/chesstrainer-backups"
~/.local/bin/chesstrainer-backup-drill
cat "/mnt/Mint Data/chesstrainer-backups/drills.log"

A passing drill line looks like:

{"timestamp":"2026-11-11T21:05:46Z","event":"drill.completed","archive":"20261111T120000Z.tar.age","result":"pass","counts":"users=3,endgames=42,attempts=900","duration_s":3}

The drill creates a throwaway postgres:18.6 container, applies the globals dump, asserts that chesstrainer and chesstrainer_ro exist, restores into chesstrainer_restore, and compares the three row counts with the .counts file from dump time. Restored counts may be higher (the counts are taken moments before the dump); lower counts fail the drill.

Retention

Location Plaintext Encrypted Notes
VPS /var/backups/chesstrainer 3 days 14 days pre-deploy sets: last 5
Workstation DEST never 7 daily, 4 weekly, 6 monthly monthly copies outlive the VPS copies

The VPS is a staging area for the workstation; weekly and monthly generations accumulate there. If the laptop is off for more than 14 days, the archives the VPS dropped in the meantime are gone — that is the documented off-site RPO trade-off. Moving to an always-on destination (a second host or object storage with restic) is tracked in #151.

Restore a dump

Always restore into a scratch database first. From an archive on the workstation:

WORK=$(mktemp -d)
age --decrypt -i ~/.config/chesstrainer/age.key \
  -o "$WORK/dump.tar" \
  "/mnt/Mint Data/chesstrainer-backups/<timestamp>.tar.age"
tar -xf "$WORK/dump.tar" -C "$WORK"
(cd "$WORK" && sha256sum --check --strict <timestamp>.sha256)

Then restore into a scratch container:

docker run --rm -d --name restore-check \
  -e POSTGRES_PASSWORD=restore -e POSTGRES_HOST_AUTH_METHOD=trust postgres:18.6
docker exec restore-check psql -U postgres -c 'create database chesstrainer_restore'
docker exec -i restore-check pg_restore -U postgres -d chesstrainer_restore \
  --single-transaction --exit-on-error < "$WORK/<timestamp>.dump"
docker exec restore-check psql -U postgres -d chesstrainer_restore \
  -c 'select count(*) from users; select count(*) from attempts;'
docker rm -f restore-check
rm -rf "$WORK"

Disaster recovery

  1. Build a new VPS following this README through First-time setup, without activating a release or switching Caddy.
  2. Decrypt and unpack the newest archive on the workstation (above).
  3. Copy the dump and globals to the new VPS: scp "$WORK/<timestamp>.dump" "$WORK/<timestamp>.globals.sql" <VPS_USER>@<VPS_HOST>:/tmp/
  4. Recreate the cluster roles. Connect as chesstrainer: it is the container's superuser (POSTGRES_USER), and the dump's final grants are issued GRANTED BY chesstrainer, which another role cannot replay. The line that recreates chesstrainer itself reports "already exists" and is harmless: sudo docker exec -i chesstrainer-db psql -U chesstrainer < /tmp/<timestamp>.globals.sql
  5. Create the database and restore it: sudo docker exec chesstrainer-db createdb -U chesstrainer chesstrainer_prod sudo docker exec -i chesstrainer-db pg_restore -U chesstrainer -d chesstrainer_prod --single-transaction --exit-on-error < /tmp/<timestamp>.dump
  6. Verify the row counts, activate a release, then restore the Caddy block.
  7. Take a fresh backup (sudo systemctl start chesstrainer-backup) and pull so the new VPS is covered immediately.

pg_dump and pg_restore must come from the container's Postgres 18, not the host.

Pre-deploy dump

deploy/activate calls chesstrainer-backup with BACKUP_TAG=predeploy after extracting the release and before flipping current, so the state before bin/migrate runs in ExecStartPre is always recoverable. A failed dump aborts the activation. The escape hatch is deliberate and should be rare:

sudo SKIP_PRE_DEPLOY_BACKUP=1 /opt/chesstrainer/src/deploy/activate /tmp/chesstrainer-release.tar.gz

If the installed script is missing, activate uses the backup next to it; if neither exists it refuses to activate rather than migrate without a dump. The GitHub Actions deploy job inherits the same behaviour.

Run it by hand

Every timer can be triggered directly. On the VPS:

# exactly what the timer runs
sudo systemctl start chesstrainer-backup
sudo journalctl -u chesstrainer-backup -n 20 -o cat

# an ad-hoc dump with its own tag
sudo BACKUP_TAG=manual /usr/local/sbin/chesstrainer-backup

# schedule, last result and last exit
systemctl list-timers chesstrainer-backup.timer
systemctl show chesstrainer-backup.service -p Result -p ExecMainExitTimestamp

On the workstation:

# pull now, or through the unit (same PATH and OnFailure handling)
~/.local/bin/chesstrainer-backup-pull
systemctl --user start chesstrainer-backup-pull.service

# drill the newest archive, or a named one
~/.local/bin/chesstrainer-backup-drill
~/.local/bin/chesstrainer-backup-drill 20261111T213734Z.tar.age

# results and schedule
cat "/mnt/Mint Data/chesstrainer-backups/drills.log"
systemctl --user list-timers 'chesstrainer-backup-*'

A backup.skipped or pull.skipped line means another run held the lock; that run reports the outcome. After fixing a failure, clear the unit with systemctl --user reset-failed chesstrainer-backup-pull.service (or sudo systemctl reset-failed chesstrainer-backup.service).

Enabling the pull timer fires it once straight away: its OnBootSec=10m is measured from boot, so enabling it hours after boot triggers immediately. That is expected and harmless.

Monitoring

systemctl list-timers chesstrainer-backup.timer
sudo journalctl -u chesstrainer-backup --since "2 days ago" -o cat
sudo systemctl --failed
systemctl --user list-timers chesstrainer-backup-pull.timer chesstrainer-backup-drill.timer
journalctl --user -u chesstrainer-backup-pull --since "2 days ago" -o cat

A failed run produces both backup.failed (from the script) and backup.unit-failed (from the OnFailure unit). The healthchecks.io check is the backstop: if the VPS dies, the ping stops and you are emailed. The pull fails when the newest archive is more than 48 hours old, which catches a stopped VPS timer the next time the workstation pulls.

Troubleshooting

Symptom Likely cause
backup.failed: missing command: age sudo apt install age
backup.failed: only N MB free disk full on the VPS; prune old releases or grow the disk
backup.failed: pg_dump failed chesstrainer-db is down or restarting — check docker ps
backup.failed: globals dump is missing role … the role was renamed; update EXPECTED_ROLES
pull.failed: rsync … failed SSH key, host key, or the VPS is unreachable
pull.failed: newest archive … is Nh old the VPS backup timer stopped; check it before touching the workstation
drill.failed: pg_restore failed archive is corrupt; keep it for diagnosis and restore an earlier one

Never put dumps anywhere under priv/static or the release directory, and never restore over chesstrainer_prod while the application is running.

New VPS checklist

  1. Ubuntu 24.04, DNS pointed at the VPS, firewall allows 22/80/443.
  2. Docker, Compose, Caddy, and runtime libraries installed.
  3. chesstrainer user, /opt/chesstrainer layout, root-owned env files.
  4. chesstrainer-db merged into the unified compose and healthy.
  5. systemd unit installed and enabled; ConditionPathExists keeps it skipped until a release exists.
  6. Release built on a matching Ubuntu runner and activated.
  7. /health and / verified on 127.0.0.1:4100.
  8. Caddy block, cloudflare-ranges.caddy, and the CIDR refresh timer added; validated and reloaded.
  9. Public /, /privacy, /terms-conditions, and /health verified.
  10. Read-only role created and tunnel verified.
  11. age installed, /etc/chesstrainer/backup.env written, backup script and units installed, timer enabled, and one manual run produces a dump plus a globals file.
  12. Workstation age key and pull configuration installed, user timers and linger enabled, first pull verified, first drill passes.
  13. /etc/chesstrainer/probe.env written, probe script and units installed, timer enabled, and the healthchecks.io check green within one period.
  14. Remote shell attaches and exits cleanly: sudo /opt/chesstrainer/current/bin/chesstrainer remote (Ctrl-C twice).