Files
chess-endgame-trainer/docs/adr/0012-launch-observability-scope.md
owenrees 8934d191ad Record the launch observability decision and verify the error states (#172)
ADR 0012 states the launch bar, the deferred error tracking/metrics/log
shipping with a concrete revisit trigger for each, and the accepted risk of
journald plus off-box backups.

The verification page walks the #162 checklist against a local MIX_ENV=prod
build with forced failures: Postgres stopped, missing tablebase, empty pool,
expired tokens and exhausted guest budget. 14 rows pass, 6 fail; the eight
regressions are filed in #171. The audit's X1 row is corrected: an unknown
play code already returns the styled 404 page.
2026-11-13 16:10:42 +01:00

66 lines
3.3 KiB
Markdown

# Launch observability is the probe, honest error states, and the runbook
_Status: accepted._
At launch, observability means three things and no more: an external check that
the public app answers, error states that are honest and recoverable, and a
written runbook for when either fails. Everything else is deferred with a named
trigger, so the gaps are explicit rather than ambient.
## The launch bar
- **Probe.** A healthchecks.io check with a two-minute period and grace, fed by
an on-box prober with off-box alerting (#160, #161). It curls the public
`/health` through Cloudflare, so DNS, TLS, Caddy, the origin allowlist,
Phoenix and one Postgres query are covered. The loopback cross-check in a
failure body is evidence only. The prober shares a host with the app, so host
death is caught by silence, not by a ping.
- **Error states.** The audit (#162) and its fixes (#163), verified by #164.
Every failure mode has copy and a retry or a way home; exception details and
raw error terms do not reach the page.
- **Runbook.** `deploy/README.md` §Monitoring and alerting: the check, install,
what `/health` proves and does not, Beszel's blind spot, the drills, and the
dated drill log.
## Deferred, with triggers
- **Error tracking (#109, GlitchTip/Sentry).** Deferred because JSON logs
already carry the crash reason and stacktrace per request, and there is no
traffic to aggregate over. Revisit when the first production 500 cannot be
diagnosed from its journal line (missing or truncated stacktrace), or a user
reports an error without a request id, or unexplained errors exceed roughly
50 a day and per-line grep stops being practical.
- **Metrics and LiveDashboard (#111).** Deferred because Beszel covers host
resources and the probe covers the HTTP contract. Revisit when a capacity
question needs a number neither can answer (database pool saturation, p95
latency, queue depth), or daily active users pass 100, or a performance
incident takes more than an hour to localise.
- **Off-box log shipping and error-rate alerts (#140).** Deferred because
journald is persistent on the host and the probe alerts on downness. Revisit
when an alert must fire on a log pattern (error rate, authentication
failures), or the journal retention budget cannot cover a 30-day
investigation, or the host is rebuilt and local logs would be lost in the same
incident.
## Why journald and backups are enough at launch
Persistent journald (`Storage=persistent`, `SystemMaxUse`) holds one JSON object
per line: crashes carry a reason and stacktrace, access lines carry status and
duration, and token-shaped paths, emails and credentials are redacted. Off-box
encrypted backups cover the data.
The accepted risk: losing the host loses the logs with it, and there is no
aggregate error rate. At launch that is acceptable — one host, one operator,
low traffic, and an incident is diagnosed from the journal on the box or from
the probe's alert body. The triggers above are the point at which that risk
stops being acceptable.
## Consequences
- Investigation starts with `journalctl -u chesstrainer -o cat | jq`, not a
dashboard.
- Error tracking, metrics and log shipping stay explicit backlog items with
triggers rather than ambient gaps.
- The probe is an on-box prober with off-box alerting; it is not external
uptime monitoring and the runbook says so.