Record the launch observability decision and verify the error states (#172)

ADR 0012 states the launch bar, the deferred error tracking/metrics/log
shipping with a concrete revisit trigger for each, and the accepted risk of
journald plus off-box backups.

The verification page walks the #162 checklist against a local MIX_ENV=prod
build with forced failures: Postgres stopped, missing tablebase, empty pool,
expired tokens and exhausted guest budget. 14 rows pass, 6 fail; the eight
regressions are filed in #171. The audit's X1 row is corrected: an unknown
play code already returns the styled 404 page.
This commit is contained in:
2026-11-13 16:10:42 +01:00
committed by GitHub
parent 021e412167
commit 8934d191ad
3 changed files with 127 additions and 3 deletions
@@ -0,0 +1,65 @@
# Launch observability is the probe, honest error states, and the runbook
_Status: accepted._
At launch, observability means three things and no more: an external check that
the public app answers, error states that are honest and recoverable, and a
written runbook for when either fails. Everything else is deferred with a named
trigger, so the gaps are explicit rather than ambient.
## The launch bar
- **Probe.** A healthchecks.io check with a two-minute period and grace, fed by
an on-box prober with off-box alerting (#160, #161). It curls the public
`/health` through Cloudflare, so DNS, TLS, Caddy, the origin allowlist,
Phoenix and one Postgres query are covered. The loopback cross-check in a
failure body is evidence only. The prober shares a host with the app, so host
death is caught by silence, not by a ping.
- **Error states.** The audit (#162) and its fixes (#163), verified by #164.
Every failure mode has copy and a retry or a way home; exception details and
raw error terms do not reach the page.
- **Runbook.** `deploy/README.md` §Monitoring and alerting: the check, install,
what `/health` proves and does not, Beszel's blind spot, the drills, and the
dated drill log.
## Deferred, with triggers
- **Error tracking (#109, GlitchTip/Sentry).** Deferred because JSON logs
already carry the crash reason and stacktrace per request, and there is no
traffic to aggregate over. Revisit when the first production 500 cannot be
diagnosed from its journal line (missing or truncated stacktrace), or a user
reports an error without a request id, or unexplained errors exceed roughly
50 a day and per-line grep stops being practical.
- **Metrics and LiveDashboard (#111).** Deferred because Beszel covers host
resources and the probe covers the HTTP contract. Revisit when a capacity
question needs a number neither can answer (database pool saturation, p95
latency, queue depth), or daily active users pass 100, or a performance
incident takes more than an hour to localise.
- **Off-box log shipping and error-rate alerts (#140).** Deferred because
journald is persistent on the host and the probe alerts on downness. Revisit
when an alert must fire on a log pattern (error rate, authentication
failures), or the journal retention budget cannot cover a 30-day
investigation, or the host is rebuilt and local logs would be lost in the same
incident.
## Why journald and backups are enough at launch
Persistent journald (`Storage=persistent`, `SystemMaxUse`) holds one JSON object
per line: crashes carry a reason and stacktrace, access lines carry status and
duration, and token-shaped paths, emails and credentials are redacted. Off-box
encrypted backups cover the data.
The accepted risk: losing the host loses the logs with it, and there is no
aggregate error rate. At launch that is acceptable — one host, one operator,
low traffic, and an incident is diagnosed from the journal on the box or from
the probe's alert body. The triggers above are the point at which that risk
stops being acceptable.
## Consequences
- Investigation starts with `journalctl -u chesstrainer -o cat | jq`, not a
dashboard.
- Error tracking, metrics and log shipping stay explicit backlog items with
triggers rather than ambient gaps.
- The probe is an on-box prober with off-box alerting; it is not external
uptime monitoring and the runbook says so.
+59
View File
@@ -0,0 +1,59 @@
# Error-state verification
The #162 checklist, walked on 2026-11-13 against a local production-like build.
Failures are filed as #171; this page is linked from the audit.
## Method
- **Build:** `MIX_ENV=prod` `mix phx.server` on `127.0.0.1:4010`, commit
`021e412` plus this branch's documentation.
- **Database:** a scratch `postgres:18.6` on port 55432. Empty at first, then
one confirmed user and one 8+ piece endgame (inserted directly, because the
admin changeset rejects positions beyond seven pieces).
- **Configuration:** `PRELAUNCH=false`, `MAILING_LIST_ENABLED=true`,
`TABLEBASE_PATH` pointing at a missing directory, dummy SES credentials.
- **Forced failures:** Postgres stopped, missing tablebase directory, empty
puzzle pool, expired reset and confirmation tokens (backdated
`users_tokens.inserted_at`), stale and invalid mailing-list forms, guest
budget exhausted, unknown public play code.
- **Observation:** `curl` for controller-rendered pages, headless Chrome over
the DevTools Protocol for LiveView-connected pages, and `mix test` for the
interactive states that need a click. Each row names its evidence.
- **Not forced:** a real Lichess 429 (the cooldown and classifier paths were
exercised by the tests), a crash after connect, and the two admin-only import
leaks (code inspection and tests). The first run rejected the socket origin
because `PHX_HOST` defaulted to `example.com`; setting `PHX_HOST=127.0.0.1`
fixed it. That was local configuration, not a product finding.
## Results
| Checklist row | Result | Evidence |
|---|---|---|
| T1 cooldown | **pass** | LiveView test: mount with a cooldown shows "Tablebase cooling down"; Retry re-fetches and returns to play. |
| T2 429 | **pass** | LiveView test: classifier maps 429 to "Tablebase rate-limited"; HTML carries no `429`, `too_many_requests` or upstream body. |
| T3 not covered | **pass** | Prod run: guest page with a 12-piece endgame shows "Position not covered… can't be graded", no `not_covered` in the DOM, Next puzzle or back available. |
| T4 generic | **pass** | LiveView test: classifier returns "Tablebase unavailable"; the reason is logged and never rendered. |
| T5/T6 load and reply failure | **pass** | T5 covered by the cooldown-at-mount test and Retry. T6 shares the blocked panel and `reset/1` path; not separately forced. |
| T8 invalid FEN | **fail** | Existing test: the flash still contains the client message and the raw FEN. #171. |
| S1/S2/S3/S5 empty pools | **pass** | S1 prod: Medium picker shows the in-page empty state and Retry. S5 prod: guest shows "No endgames are available yet" and Retry. S2/S3 tests: exhausted state, Retry and Change difficulty. |
| S4 guest limit | **pass** | Prod run: five visits exhaust the budget; the wall shows the reset time, Sign up free and Log in. |
| A1 reset token | **pass** | Prod run: an expired token flashes "Reset password link is invalid or it has expired." on the log-in page, with Forgot your password available. |
| A2 confirmation token | **fail** | Prod run: an expired token flashes "User not found or token is invalid." and lands on registration, not log in. #171. |
| A3/A4 login | **pass** | Tests: invalid credentials use one message; the throttle reports the remaining minutes. |
| M1–M5 mailing list | **pass** | Prod run: invalid confirmation link, removed address, stale form, invalid email and an SES send failure each show their specific copy with the form reachable. |
| W1/W2 404 and 500 | **pass** | Prod run: styled 404 for an unknown path; with Postgres stopped, `/` returns the styled 500 with no exception or stacktrace. |
| W3 crash | **fail** | The server-error banner has no reload action; a repeating crash has no manual escape. Not separately forced; same banner as W4. #171. |
| W4 disconnect | **pass** | Observed "Something went wrong! Attempting to reconnect" while the socket was rejected, with automatic retries; the banner hides on reconnect. |
| W5 database down | **pass** | Prod run: `/health` returns 503 `db:down`; `/` returns the 500 page. Note the loop: the home link also needs the database, so it 500s too. |
| D1/D2 export | **fail** | Prod run: an export with the database stopped returns the 500 page (no export-specific copy); `/users/profile/export/xml` returns plain text "Unknown export format". #171. |
| H1 empty-database home | **fail** | Prod run: the fallback board renders, but "Try today's puzzle" points at the auth-gated `/endgames`. #171. |
| X1 unknown play code | **pass** | Prod run: `/endgames/nope/play` returns the styled 404 page. This corrects the audit's earlier claim that it 500ed. |
| X2/X3 admin leaks | **fail** | Code inspection and tests: both importer messages still interpolate `inspect(reason)`. Not forced. #171. |
## Tally
- 14 checklist rows pass, 6 fail (8 individual regressions, because D1/D2 and
X2/X3 share a row).
- Every failure row is a state that was already listed as "improve" in the
audit and was outside #163; no new regression came from the #163 changes.
- The regressions are filed together in #171 rather than fixed silently.
+3 -3
View File
@@ -16,7 +16,8 @@ Verdicts:
status body, a stacktrace, or a raw FEN appearing in rendered HTML. status body, a stacktrace, or a raw FEN appearing in rendered HTML.
**Status:** #163 implements T1–T6, S1–S3 and S5. The rows below describe the **Status:** #163 implements T1–T6, S1–S3 and S5. The rows below describe the
state before that change; #164 records what was verified afterwards. state before that change. #164 walked this checklist on a local prod build:
see [error-states-verification.md](error-states-verification.md).
## Play and tablebase ## Play and tablebase
@@ -84,7 +85,7 @@ state before that change; #164 records what was verified afterwards.
| ID | State | Trigger | What the user sees today | Verdict | Action | | ID | State | Trigger | What the user sees today | Verdict | Action |
|---|---|---|---|---|---| |---|---|---|---|---|---|
| X1 | Unknown public play code | `/endgames/:code/play` with a code that does not exist (`play.ex:79`) | `get_endgame_by_public_code!/1` raises → the 500 page | improve | Fetch without the bang and render the 404 page (or redirect to `/endgames` with a flash). A stale link is not a server error. | | X1 | Unknown public play code | `/endgames/:code/play` with a code that does not exist (`play.ex:79`) | The styled 404 page: `Ecto.NoResultsError` maps to 404 through `Plug.Exception` | keep | Verified by #164; no change. |
| X2 | Bulk import lookup crash | `handle_async` exits (`import.ex:271`) | Flash **"Could not look up the FENs: #{inspect(reason)}"** | improve | Generic copy plus a logged reason. Admin-only, but still an internal leak. | | X2 | Bulk import lookup crash | `handle_async` exits (`import.ex:271`) | Flash **"Could not look up the FENs: #{inspect(reason)}"** | improve | Generic copy plus a logged reason. Admin-only, but still an internal leak. |
| X3 | Bulk import tablebase load failure | `Local.check/0` returns an unexpected error (`import.ex:287`) | **"Local tablebase failed to load: #{inspect(reason)}"** | improve | Generic copy plus a logged reason. Admin-only, but still an internal leak. | | X3 | Bulk import tablebase load failure | `Local.check/0` returns an unexpected error (`import.ex:287`) | **"Local tablebase failed to load: #{inspect(reason)}"** | improve | Generic copy plus a logged reason. Admin-only, but still an internal leak. |
@@ -146,7 +147,6 @@ Separate changes, each independently testable:
- **D1** — rescue export errors into a flash on the data tab. - **D1** — rescue export errors into a flash on the data tab.
- **D2** — styled 404 for unknown export formats. - **D2** — styled 404 for unknown export formats.
- **H1** — empty-database call to action. - **H1** — empty-database call to action.
- **X1** — 404 instead of 500 for an unknown public code.
- **X2/X3** — generic admin import copy; log the details. - **X2/X3** — generic admin import copy; log the details.
No code changes are part of this audit. No code changes are part of this audit.