mirror of
https://github.com/TheRealOwenRees/chess-endgame-trainer.git
synced 2026-11-18 11:42:07 +00:00
Record the launch observability decision and verify the error states (#172)
ADR 0012 states the launch bar, the deferred error tracking/metrics/log shipping with a concrete revisit trigger for each, and the accepted risk of journald plus off-box backups. The verification page walks the #162 checklist against a local MIX_ENV=prod build with forced failures: Postgres stopped, missing tablebase, empty pool, expired tokens and exhausted guest budget. 14 rows pass, 6 fail; the eight regressions are filed in #171. The audit's X1 row is corrected: an unknown play code already returns the styled 404 page.
This commit is contained in:
@@ -0,0 +1,65 @@
|
||||
# Launch observability is the probe, honest error states, and the runbook
|
||||
|
||||
_Status: accepted._
|
||||
|
||||
At launch, observability means three things and no more: an external check that
|
||||
the public app answers, error states that are honest and recoverable, and a
|
||||
written runbook for when either fails. Everything else is deferred with a named
|
||||
trigger, so the gaps are explicit rather than ambient.
|
||||
|
||||
## The launch bar
|
||||
|
||||
- **Probe.** A healthchecks.io check with a two-minute period and grace, fed by
|
||||
an on-box prober with off-box alerting (#160, #161). It curls the public
|
||||
`/health` through Cloudflare, so DNS, TLS, Caddy, the origin allowlist,
|
||||
Phoenix and one Postgres query are covered. The loopback cross-check in a
|
||||
failure body is evidence only. The prober shares a host with the app, so host
|
||||
death is caught by silence, not by a ping.
|
||||
- **Error states.** The audit (#162) and its fixes (#163), verified by #164.
|
||||
Every failure mode has copy and a retry or a way home; exception details and
|
||||
raw error terms do not reach the page.
|
||||
- **Runbook.** `deploy/README.md` §Monitoring and alerting: the check, install,
|
||||
what `/health` proves and does not, Beszel's blind spot, the drills, and the
|
||||
dated drill log.
|
||||
|
||||
## Deferred, with triggers
|
||||
|
||||
- **Error tracking (#109, GlitchTip/Sentry).** Deferred because JSON logs
|
||||
already carry the crash reason and stacktrace per request, and there is no
|
||||
traffic to aggregate over. Revisit when the first production 500 cannot be
|
||||
diagnosed from its journal line (missing or truncated stacktrace), or a user
|
||||
reports an error without a request id, or unexplained errors exceed roughly
|
||||
50 a day and per-line grep stops being practical.
|
||||
- **Metrics and LiveDashboard (#111).** Deferred because Beszel covers host
|
||||
resources and the probe covers the HTTP contract. Revisit when a capacity
|
||||
question needs a number neither can answer (database pool saturation, p95
|
||||
latency, queue depth), or daily active users pass 100, or a performance
|
||||
incident takes more than an hour to localise.
|
||||
- **Off-box log shipping and error-rate alerts (#140).** Deferred because
|
||||
journald is persistent on the host and the probe alerts on downness. Revisit
|
||||
when an alert must fire on a log pattern (error rate, authentication
|
||||
failures), or the journal retention budget cannot cover a 30-day
|
||||
investigation, or the host is rebuilt and local logs would be lost in the same
|
||||
incident.
|
||||
|
||||
## Why journald and backups are enough at launch
|
||||
|
||||
Persistent journald (`Storage=persistent`, `SystemMaxUse`) holds one JSON object
|
||||
per line: crashes carry a reason and stacktrace, access lines carry status and
|
||||
duration, and token-shaped paths, emails and credentials are redacted. Off-box
|
||||
encrypted backups cover the data.
|
||||
|
||||
The accepted risk: losing the host loses the logs with it, and there is no
|
||||
aggregate error rate. At launch that is acceptable — one host, one operator,
|
||||
low traffic, and an incident is diagnosed from the journal on the box or from
|
||||
the probe's alert body. The triggers above are the point at which that risk
|
||||
stops being acceptable.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Investigation starts with `journalctl -u chesstrainer -o cat | jq`, not a
|
||||
dashboard.
|
||||
- Error tracking, metrics and log shipping stay explicit backlog items with
|
||||
triggers rather than ambient gaps.
|
||||
- The probe is an on-box prober with off-box alerting; it is not external
|
||||
uptime monitoring and the runbook says so.
|
||||
Reference in New Issue
Block a user