mirror of
https://github.com/TheRealOwenRees/chess-endgame-trainer.git
synced 2026-11-18 11:42:07 +00:00
ADR 0012 states the launch bar, the deferred error tracking/metrics/log shipping with a concrete revisit trigger for each, and the accepted risk of journald plus off-box backups. The verification page walks the #162 checklist against a local MIX_ENV=prod build with forced failures: Postgres stopped, missing tablebase, empty pool, expired tokens and exhausted guest budget. 14 rows pass, 6 fail; the eight regressions are filed in #171. The audit's X1 row is corrected: an unknown play code already returns the styled 404 page.
66 lines
3.3 KiB
Markdown
66 lines
3.3 KiB
Markdown
# Launch observability is the probe, honest error states, and the runbook
|
|
|
|
_Status: accepted._
|
|
|
|
At launch, observability means three things and no more: an external check that
|
|
the public app answers, error states that are honest and recoverable, and a
|
|
written runbook for when either fails. Everything else is deferred with a named
|
|
trigger, so the gaps are explicit rather than ambient.
|
|
|
|
## The launch bar
|
|
|
|
- **Probe.** A healthchecks.io check with a two-minute period and grace, fed by
|
|
an on-box prober with off-box alerting (#160, #161). It curls the public
|
|
`/health` through Cloudflare, so DNS, TLS, Caddy, the origin allowlist,
|
|
Phoenix and one Postgres query are covered. The loopback cross-check in a
|
|
failure body is evidence only. The prober shares a host with the app, so host
|
|
death is caught by silence, not by a ping.
|
|
- **Error states.** The audit (#162) and its fixes (#163), verified by #164.
|
|
Every failure mode has copy and a retry or a way home; exception details and
|
|
raw error terms do not reach the page.
|
|
- **Runbook.** `deploy/README.md` §Monitoring and alerting: the check, install,
|
|
what `/health` proves and does not, Beszel's blind spot, the drills, and the
|
|
dated drill log.
|
|
|
|
## Deferred, with triggers
|
|
|
|
- **Error tracking (#109, GlitchTip/Sentry).** Deferred because JSON logs
|
|
already carry the crash reason and stacktrace per request, and there is no
|
|
traffic to aggregate over. Revisit when the first production 500 cannot be
|
|
diagnosed from its journal line (missing or truncated stacktrace), or a user
|
|
reports an error without a request id, or unexplained errors exceed roughly
|
|
50 a day and per-line grep stops being practical.
|
|
- **Metrics and LiveDashboard (#111).** Deferred because Beszel covers host
|
|
resources and the probe covers the HTTP contract. Revisit when a capacity
|
|
question needs a number neither can answer (database pool saturation, p95
|
|
latency, queue depth), or daily active users pass 100, or a performance
|
|
incident takes more than an hour to localise.
|
|
- **Off-box log shipping and error-rate alerts (#140).** Deferred because
|
|
journald is persistent on the host and the probe alerts on downness. Revisit
|
|
when an alert must fire on a log pattern (error rate, authentication
|
|
failures), or the journal retention budget cannot cover a 30-day
|
|
investigation, or the host is rebuilt and local logs would be lost in the same
|
|
incident.
|
|
|
|
## Why journald and backups are enough at launch
|
|
|
|
Persistent journald (`Storage=persistent`, `SystemMaxUse`) holds one JSON object
|
|
per line: crashes carry a reason and stacktrace, access lines carry status and
|
|
duration, and token-shaped paths, emails and credentials are redacted. Off-box
|
|
encrypted backups cover the data.
|
|
|
|
The accepted risk: losing the host loses the logs with it, and there is no
|
|
aggregate error rate. At launch that is acceptable — one host, one operator,
|
|
low traffic, and an incident is diagnosed from the journal on the box or from
|
|
the probe's alert body. The triggers above are the point at which that risk
|
|
stops being acceptable.
|
|
|
|
## Consequences
|
|
|
|
- Investigation starts with `journalctl -u chesstrainer -o cat | jq`, not a
|
|
dashboard.
|
|
- Error tracking, metrics and log shipping stay explicit backlog items with
|
|
triggers rather than ambient gaps.
|
|
- The probe is an on-box prober with off-box alerting; it is not external
|
|
uptime monitoring and the runbook says so.
|