Add structured production logging (#105, #107) (#141)

* Update README.md

* Add structured JSON production logging (#105)

* Log requests and background failures (#107)

* Document production logging and the VPS runbook
This commit is contained in:
2026-11-10 22:00:05 +01:00
committed by GitHub
parent f8897fa5ae
commit 202b9b1f6a
20 changed files with 1380 additions and 15 deletions
+202 -11
View File
@@ -310,6 +310,121 @@ sudo docker compose -f <UNIFIED_COMPOSE> pull chesstrainer-db
sudo docker compose -f <UNIFIED_COMPOSE> up -d chesstrainer-db
```
## Read-only database access
The app connects as `chesstrainer`, which is a superuser. Use a separate
read-only role for inspection, debugging, and dashboards.
Create it once per environment:
```sh
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
psql -U chesstrainer -d chesstrainer_prod <<'SQL'
CREATE ROLE chesstrainer_ro LOGIN PASSWORD '<generated-password>';
GRANT CONNECT ON DATABASE chesstrainer_prod TO chesstrainer_ro;
GRANT USAGE ON SCHEMA public TO chesstrainer_ro;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO chesstrainer_ro;
-- Cover tables created by future migrations. Migrations run as chesstrainer.
ALTER DEFAULT PRIVILEGES FOR ROLE chesstrainer IN SCHEMA public
GRANT SELECT ON TABLES TO chesstrainer_ro;
-- Optional: read other sessions and server settings in pg_stat_* views.
GRANT pg_read_all_stats TO chesstrainer_ro;
SQL
```
Generate the password with `openssl rand -hex 32` and keep it in a password
manager. The only access it has is `SELECT`; there are no `INSERT`, `UPDATE`,
`DELETE`, or object-creation grants. If the role already exists, skip the
`CREATE ROLE` line and re-run the grants.
Optional defence in depth:
```sql
ALTER ROLE chesstrainer_ro SET default_transaction_read_only = on;
```
This is not a hard boundary (the role can override it per session), but it
turns an accidental write into an error.
### Connect from the VPS
`docker exec` connects over the container's Unix socket, which the Postgres
image trusts locally, so no password is needed:
```sh
sudo docker exec -it chesstrainer-db \
psql -U chesstrainer_ro -d chesstrainer_prod
```
### Connect from a workstation over SSH
Postgres is published on `127.0.0.1:5432` only. OpenSSH also disables
forwarding by default in `/etc/ssh/sshd_config` (`AllowTcpForwarding no`).
Allow local forwarding for the deploy user only, restricted to Postgres, by
appending this to the **end** of that file:
```text
Match User <username>
AllowTcpForwarding local
PermitOpen 127.0.0.1:5432
```
Then validate and reload:
```sh
sudo sshd -t
sudo systemctl reload ssh
sudo sshd -T -C user=<username>,host=localhost,addr=127.0.0.1 | grep -iE 'allowtcpforwarding|permitopen'
```
Open the tunnel (local port `5433` because the dev database already binds
`5432`):
```sh
ssh -N -L 5433:127.0.0.1:5432 <VPS_USER>@<VPS_HOST>
```
Connect from a second terminal:
```sh
psql "postgres://chesstrainer_ro@127.0.0.1:5433/chesstrainer_prod"
```
Install `postgresql-client` if `psql` is not available on the workstation.
To avoid the password prompt, add this to `~/.pgpass` (mode `0600`):
```text
127.0.0.1:5433:chesstrainer_prod:chesstrainer_ro:<password>
```
DBeaver uses the same tunnel. Create a PostgreSQL connection with host
`127.0.0.1`, port `5432`, database `chesstrainer_prod`, user
`chesstrainer_ro`, then on the **SSH** tab enable the tunnel with host
`<VPS_HOST>`, port `22`, user `<VPS_USER>`, and a public key. The
`AllowTcpForwarding local` rule above is required for any tunnelled client.
Reset the role password:
```sh
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
psql -U chesstrainer -d chesstrainer_prod \
-c "ALTER ROLE chesstrainer_ro PASSWORD '<new-password>';"
```
Verify the role is read-only:
```sh
psql "postgres://chesstrainer_ro@127.0.0.1:5433/chesstrainer_prod" \
-c 'select current_user, current_database();'
psql "postgres://chesstrainer_ro@127.0.0.1:5433/chesstrainer_prod" \
-c 'create table should_fail (i int);'
```
The second command must fail with `permission denied for schema public`.
## systemd unit
From the deploy artifact copy:
@@ -438,7 +553,7 @@ sudo -E /opt/chesstrainer/src/deploy/activate /tmp/chesstrainer-release.tar.gz
## GitHub Actions CD
Merging to `main` (or running the workflow manually) tests, builds, uploads,
and deploys the release. The deploy job uses SSH as `owen`.
and deploys the release. The deploy job uses SSH as `<username>`.
### Deploy key
@@ -447,7 +562,7 @@ Generate a dedicated key, not your personal one:
```sh
ssh-keygen -t ed25519 -C "github-actions-chesstrainer" \
-f ~/.ssh/chesstrainer_deploy -N ""
ssh-copy-id -i ~/.ssh/chesstrainer_deploy.pub owen@<VPS_HOST>
ssh-copy-id -i ~/.ssh/chesstrainer_deploy.pub <username>@<VPS_HOST>
```
`VPS_HOST` must be the public IP or DNS name reachable from GitHub. Do not use
@@ -460,7 +575,7 @@ sudo visudo -f /etc/sudoers.d/chesstrainer-deploy
```
```text
owen ALL=(root) NOPASSWD: /opt/chesstrainer/src/deploy/activate, /opt/chesstrainer/src/deploy/rollback
<username> ALL=(root) NOPASSWD: /opt/chesstrainer/src/deploy/activate, /opt/chesstrainer/src/deploy/rollback
```
Validate:
@@ -481,7 +596,7 @@ add environment secrets:
| Secret | Value |
|---|---|
| `VPS_HOST` | public VPS IP or DNS name |
| `VPS_USER` | `owen` |
| `VPS_USER` | `<username>` |
| `VPS_SSH_KEY` | private deploy key |
| `VPS_KNOWN_HOSTS` | `ssh-keyscan -H <VPS_HOST>` output |
@@ -533,24 +648,100 @@ curl -i http://127.0.0.1:4100/health
It returns 200 for `ok`/`degraded` and 503 for `down`. A missing local tablebase
reports `degraded` but still 200 because Lichess fallback is available.
## Backups
### Reading production logs
Take a logical backup of the production database:
Production emits one JSON object per line. Use `-o cat` so journald prints the
raw line instead of its own field wrapper, then pipe to `jq`:
```sh
sudo docker compose -f <UNIFIED_COMPOSE> \
exec -T chesstrainer-db pg_dump -U chesstrainer chesstrainer_prod \
# Recent logs, pretty-printed
sudo journalctl -u chesstrainer -o cat -n 50 --no-pager | jq .
# Errors only
sudo journalctl -u chesstrainer -o cat --since "1 hour ago" | jq 'select(.level == "error")'
# Everything from one request
sudo journalctl -u chesstrainer -o cat | jq 'select(.request_id == "REQUEST_ID")'
# Crash reasons and stacktraces
sudo journalctl -u chesstrainer -o cat | jq -r 'select(.error) | .error.reason, (.error.stacktrace[]?)'
```
Each line has `timestamp`, `level` and `message`, plus `request_id` when one
exists. Access lines add `method`, `path`, `status` and `duration_ms`; crashes
add an `error` object with `kind`, `reason` and `stacktrace`. Query parameters
and request bodies are never logged, token-shaped path segments are redacted,
and email addresses or credentials in metadata render as `[REDACTED]`.
The full field table, redaction rules and examples for adding background logs
are in [`docs/logging.md`](../docs/logging.md).
Dev and test still use the readable `[level] message` formatter.
### Keep the journal
By default some distributions keep journald logs in memory only. On the VPS,
make sure they survive a reboot and have a retention budget:
```sh
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
journalctl --disk-usage
```
Set `Storage=persistent` and a `SystemMaxUse=` value in
`/etc/systemd/journald.conf` if the distribution does not already. journald
rate-limits bursts; if a crash storm drops lines, raise `RateLimitBurst` or
`RateLimitIntervalSec` there.
## Backups
`docker exec` connects over the container's Unix socket, which the Postgres
image trusts locally, so `pg_dump` needs no password. Run it as the owner
(`chesstrainer`), not `chesstrainer_ro`: a full dump also needs sequence access
and ownership information.
Take a compressed logical backup (custom format, restorable with `pg_restore`):
```sh
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
pg_dump -U chesstrainer -d chesstrainer_prod -Fc \
> /var/backups/chesstrainer-$(date -u +%Y%m%d%H%M%S).dump
```
For a plain SQL file instead (human-readable, restorable with `psql`):
```sh
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
pg_dump -U chesstrainer -d chesstrainer_prod \
| gzip > /var/backups/chesstrainer-$(date -u +%Y%m%d%H%M%S).sql.gz
```
Check that a custom-format dump is readable before trusting it:
```sh
sudo docker exec -i chesstrainer-db pg_restore -l \
< /var/backups/<timestamp>.dump > /dev/null
```
Restore into a scratch database (do not restore over the live database without
stopping `chesstrainer` first):
```sh
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
createdb -U chesstrainer chesstrainer_restore
sudo docker exec -i chesstrainer-db pg_restore -U chesstrainer \
-d chesstrainer_restore < /var/backups/<timestamp>.dump
```
Example nightly entry in root's crontab (`sudo crontab -e`):
```cron
15 3 * * * docker compose -f /home/owen/docker-compose.yml exec -T chesstrainer-db pg_dump -U chesstrainer chesstrainer_prod | gzip > /var/backups/chesstrainer-$(date +\%Y\%m\%d).sql.gz
15 3 * * * docker compose -f /home/<username>/docker-compose.yml exec -T chesstrainer-db pg_dump -U chesstrainer -d chesstrainer_prod -Fc > /var/backups/chesstrainer-$(date +\%Y\%m\%d).dump
```
Keep `/var/backups` outside `/opt/chesstrainer/releases` and copy backups off
the VPS regularly.
the VPS regularly. A backup that is only on the VPS is not a backup.
## New VPS checklist
@@ -564,4 +755,4 @@ the VPS regularly.
7. `/health` and `/` verified on `127.0.0.1:4100`.
8. Caddy block added, validated, reloaded.
9. Public `/`, `/privacy`, `/terms-conditions`, and `/health` verified.
10. Backups scheduled and tested.
10. Read-only role created, tunnel verified, backups scheduled and tested.