diff --git a/docs/adr/0012-launch-observability-scope.md b/docs/adr/0012-launch-observability-scope.md new file mode 100644 index 0000000..6281fe0 --- /dev/null +++ b/docs/adr/0012-launch-observability-scope.md @@ -0,0 +1,65 @@ +# Launch observability is the probe, honest error states, and the runbook + +_Status: accepted._ + +At launch, observability means three things and no more: an external check that +the public app answers, error states that are honest and recoverable, and a +written runbook for when either fails. Everything else is deferred with a named +trigger, so the gaps are explicit rather than ambient. + +## The launch bar + +- **Probe.** A healthchecks.io check with a two-minute period and grace, fed by + an on-box prober with off-box alerting (#160, #161). It curls the public + `/health` through Cloudflare, so DNS, TLS, Caddy, the origin allowlist, + Phoenix and one Postgres query are covered. The loopback cross-check in a + failure body is evidence only. The prober shares a host with the app, so host + death is caught by silence, not by a ping. +- **Error states.** The audit (#162) and its fixes (#163), verified by #164. + Every failure mode has copy and a retry or a way home; exception details and + raw error terms do not reach the page. +- **Runbook.** `deploy/README.md` §Monitoring and alerting: the check, install, + what `/health` proves and does not, Beszel's blind spot, the drills, and the + dated drill log. + +## Deferred, with triggers + +- **Error tracking (#109, GlitchTip/Sentry).** Deferred because JSON logs + already carry the crash reason and stacktrace per request, and there is no + traffic to aggregate over. Revisit when the first production 500 cannot be + diagnosed from its journal line (missing or truncated stacktrace), or a user + reports an error without a request id, or unexplained errors exceed roughly + 50 a day and per-line grep stops being practical. +- **Metrics and LiveDashboard (#111).** Deferred because Beszel covers host + resources and the probe covers the HTTP contract. Revisit when a capacity + question needs a number neither can answer (database pool saturation, p95 + latency, queue depth), or daily active users pass 100, or a performance + incident takes more than an hour to localise. +- **Off-box log shipping and error-rate alerts (#140).** Deferred because + journald is persistent on the host and the probe alerts on downness. Revisit + when an alert must fire on a log pattern (error rate, authentication + failures), or the journal retention budget cannot cover a 30-day + investigation, or the host is rebuilt and local logs would be lost in the same + incident. + +## Why journald and backups are enough at launch + +Persistent journald (`Storage=persistent`, `SystemMaxUse`) holds one JSON object +per line: crashes carry a reason and stacktrace, access lines carry status and +duration, and token-shaped paths, emails and credentials are redacted. Off-box +encrypted backups cover the data. + +The accepted risk: losing the host loses the logs with it, and there is no +aggregate error rate. At launch that is acceptable — one host, one operator, +low traffic, and an incident is diagnosed from the journal on the box or from +the probe's alert body. The triggers above are the point at which that risk +stops being acceptable. + +## Consequences + +- Investigation starts with `journalctl -u chesstrainer -o cat | jq`, not a + dashboard. +- Error tracking, metrics and log shipping stay explicit backlog items with + triggers rather than ambient gaps. +- The probe is an on-box prober with off-box alerting; it is not external + uptime monitoring and the runbook says so. diff --git a/docs/error-states-verification.md b/docs/error-states-verification.md new file mode 100644 index 0000000..b496ddd --- /dev/null +++ b/docs/error-states-verification.md @@ -0,0 +1,59 @@ +# Error-state verification + +The #162 checklist, walked on 2026-11-13 against a local production-like build. +Failures are filed as #171; this page is linked from the audit. + +## Method + +- **Build:** `MIX_ENV=prod` `mix phx.server` on `127.0.0.1:4010`, commit + `021e412` plus this branch's documentation. +- **Database:** a scratch `postgres:18.6` on port 55432. Empty at first, then + one confirmed user and one 8+ piece endgame (inserted directly, because the + admin changeset rejects positions beyond seven pieces). +- **Configuration:** `PRELAUNCH=false`, `MAILING_LIST_ENABLED=true`, + `TABLEBASE_PATH` pointing at a missing directory, dummy SES credentials. +- **Forced failures:** Postgres stopped, missing tablebase directory, empty + puzzle pool, expired reset and confirmation tokens (backdated + `users_tokens.inserted_at`), stale and invalid mailing-list forms, guest + budget exhausted, unknown public play code. +- **Observation:** `curl` for controller-rendered pages, headless Chrome over + the DevTools Protocol for LiveView-connected pages, and `mix test` for the + interactive states that need a click. Each row names its evidence. +- **Not forced:** a real Lichess 429 (the cooldown and classifier paths were + exercised by the tests), a crash after connect, and the two admin-only import + leaks (code inspection and tests). The first run rejected the socket origin + because `PHX_HOST` defaulted to `example.com`; setting `PHX_HOST=127.0.0.1` + fixed it. That was local configuration, not a product finding. + +## Results + +| Checklist row | Result | Evidence | +|---|---|---| +| T1 cooldown | **pass** | LiveView test: mount with a cooldown shows "Tablebase cooling down"; Retry re-fetches and returns to play. | +| T2 429 | **pass** | LiveView test: classifier maps 429 to "Tablebase rate-limited"; HTML carries no `429`, `too_many_requests` or upstream body. | +| T3 not covered | **pass** | Prod run: guest page with a 12-piece endgame shows "Position not covered… can't be graded", no `not_covered` in the DOM, Next puzzle or back available. | +| T4 generic | **pass** | LiveView test: classifier returns "Tablebase unavailable"; the reason is logged and never rendered. | +| T5/T6 load and reply failure | **pass** | T5 covered by the cooldown-at-mount test and Retry. T6 shares the blocked panel and `reset/1` path; not separately forced. | +| T8 invalid FEN | **fail** | Existing test: the flash still contains the client message and the raw FEN. #171. | +| S1/S2/S3/S5 empty pools | **pass** | S1 prod: Medium picker shows the in-page empty state and Retry. S5 prod: guest shows "No endgames are available yet" and Retry. S2/S3 tests: exhausted state, Retry and Change difficulty. | +| S4 guest limit | **pass** | Prod run: five visits exhaust the budget; the wall shows the reset time, Sign up free and Log in. | +| A1 reset token | **pass** | Prod run: an expired token flashes "Reset password link is invalid or it has expired." on the log-in page, with Forgot your password available. | +| A2 confirmation token | **fail** | Prod run: an expired token flashes "User not found or token is invalid." and lands on registration, not log in. #171. | +| A3/A4 login | **pass** | Tests: invalid credentials use one message; the throttle reports the remaining minutes. | +| M1–M5 mailing list | **pass** | Prod run: invalid confirmation link, removed address, stale form, invalid email and an SES send failure each show their specific copy with the form reachable. | +| W1/W2 404 and 500 | **pass** | Prod run: styled 404 for an unknown path; with Postgres stopped, `/` returns the styled 500 with no exception or stacktrace. | +| W3 crash | **fail** | The server-error banner has no reload action; a repeating crash has no manual escape. Not separately forced; same banner as W4. #171. | +| W4 disconnect | **pass** | Observed "Something went wrong! Attempting to reconnect" while the socket was rejected, with automatic retries; the banner hides on reconnect. | +| W5 database down | **pass** | Prod run: `/health` returns 503 `db:down`; `/` returns the 500 page. Note the loop: the home link also needs the database, so it 500s too. | +| D1/D2 export | **fail** | Prod run: an export with the database stopped returns the 500 page (no export-specific copy); `/users/profile/export/xml` returns plain text "Unknown export format". #171. | +| H1 empty-database home | **fail** | Prod run: the fallback board renders, but "Try today's puzzle" points at the auth-gated `/endgames`. #171. | +| X1 unknown play code | **pass** | Prod run: `/endgames/nope/play` returns the styled 404 page. This corrects the audit's earlier claim that it 500ed. | +| X2/X3 admin leaks | **fail** | Code inspection and tests: both importer messages still interpolate `inspect(reason)`. Not forced. #171. | + +## Tally + +- 14 checklist rows pass, 6 fail (8 individual regressions, because D1/D2 and + X2/X3 share a row). +- Every failure row is a state that was already listed as "improve" in the + audit and was outside #163; no new regression came from the #163 changes. +- The regressions are filed together in #171 rather than fixed silently. diff --git a/docs/error-states.md b/docs/error-states.md index 69d13ee..b37e98e 100644 --- a/docs/error-states.md +++ b/docs/error-states.md @@ -16,7 +16,8 @@ Verdicts: status body, a stacktrace, or a raw FEN appearing in rendered HTML. **Status:** #163 implements T1–T6, S1–S3 and S5. The rows below describe the -state before that change; #164 records what was verified afterwards. +state before that change. #164 walked this checklist on a local prod build: +see [error-states-verification.md](error-states-verification.md). ## Play and tablebase @@ -84,7 +85,7 @@ state before that change; #164 records what was verified afterwards. | ID | State | Trigger | What the user sees today | Verdict | Action | |---|---|---|---|---|---| -| X1 | Unknown public play code | `/endgames/:code/play` with a code that does not exist (`play.ex:79`) | `get_endgame_by_public_code!/1` raises → the 500 page | improve | Fetch without the bang and render the 404 page (or redirect to `/endgames` with a flash). A stale link is not a server error. | +| X1 | Unknown public play code | `/endgames/:code/play` with a code that does not exist (`play.ex:79`) | The styled 404 page: `Ecto.NoResultsError` maps to 404 through `Plug.Exception` | keep | Verified by #164; no change. | | X2 | Bulk import lookup crash | `handle_async` exits (`import.ex:271`) | Flash **"Could not look up the FENs: #{inspect(reason)}"** | improve | Generic copy plus a logged reason. Admin-only, but still an internal leak. | | X3 | Bulk import tablebase load failure | `Local.check/0` returns an unexpected error (`import.ex:287`) | **"Local tablebase failed to load: #{inspect(reason)}"** | improve | Generic copy plus a logged reason. Admin-only, but still an internal leak. | @@ -146,7 +147,6 @@ Separate changes, each independently testable: - **D1** — rescue export errors into a flash on the data tab. - **D2** — styled 404 for unknown export formats. - **H1** — empty-database call to action. -- **X1** — 404 instead of 500 for an unknown public code. - **X2/X3** — generic admin import copy; log the details. No code changes are part of this audit.