mirror of
https://github.com/TheRealOwenRees/chess-endgame-trainer.git
synced 2026-11-18 11:42:07 +00:00
Record the launch observability decision and verify the error states (#172)
ADR 0012 states the launch bar, the deferred error tracking/metrics/log shipping with a concrete revisit trigger for each, and the accepted risk of journald plus off-box backups. The verification page walks the #162 checklist against a local MIX_ENV=prod build with forced failures: Postgres stopped, missing tablebase, empty pool, expired tokens and exhausted guest budget. 14 rows pass, 6 fail; the eight regressions are filed in #171. The audit's X1 row is corrected: an unknown play code already returns the styled 404 page.
This commit is contained in:
@@ -0,0 +1,65 @@
|
||||
# Launch observability is the probe, honest error states, and the runbook
|
||||
|
||||
_Status: accepted._
|
||||
|
||||
At launch, observability means three things and no more: an external check that
|
||||
the public app answers, error states that are honest and recoverable, and a
|
||||
written runbook for when either fails. Everything else is deferred with a named
|
||||
trigger, so the gaps are explicit rather than ambient.
|
||||
|
||||
## The launch bar
|
||||
|
||||
- **Probe.** A healthchecks.io check with a two-minute period and grace, fed by
|
||||
an on-box prober with off-box alerting (#160, #161). It curls the public
|
||||
`/health` through Cloudflare, so DNS, TLS, Caddy, the origin allowlist,
|
||||
Phoenix and one Postgres query are covered. The loopback cross-check in a
|
||||
failure body is evidence only. The prober shares a host with the app, so host
|
||||
death is caught by silence, not by a ping.
|
||||
- **Error states.** The audit (#162) and its fixes (#163), verified by #164.
|
||||
Every failure mode has copy and a retry or a way home; exception details and
|
||||
raw error terms do not reach the page.
|
||||
- **Runbook.** `deploy/README.md` §Monitoring and alerting: the check, install,
|
||||
what `/health` proves and does not, Beszel's blind spot, the drills, and the
|
||||
dated drill log.
|
||||
|
||||
## Deferred, with triggers
|
||||
|
||||
- **Error tracking (#109, GlitchTip/Sentry).** Deferred because JSON logs
|
||||
already carry the crash reason and stacktrace per request, and there is no
|
||||
traffic to aggregate over. Revisit when the first production 500 cannot be
|
||||
diagnosed from its journal line (missing or truncated stacktrace), or a user
|
||||
reports an error without a request id, or unexplained errors exceed roughly
|
||||
50 a day and per-line grep stops being practical.
|
||||
- **Metrics and LiveDashboard (#111).** Deferred because Beszel covers host
|
||||
resources and the probe covers the HTTP contract. Revisit when a capacity
|
||||
question needs a number neither can answer (database pool saturation, p95
|
||||
latency, queue depth), or daily active users pass 100, or a performance
|
||||
incident takes more than an hour to localise.
|
||||
- **Off-box log shipping and error-rate alerts (#140).** Deferred because
|
||||
journald is persistent on the host and the probe alerts on downness. Revisit
|
||||
when an alert must fire on a log pattern (error rate, authentication
|
||||
failures), or the journal retention budget cannot cover a 30-day
|
||||
investigation, or the host is rebuilt and local logs would be lost in the same
|
||||
incident.
|
||||
|
||||
## Why journald and backups are enough at launch
|
||||
|
||||
Persistent journald (`Storage=persistent`, `SystemMaxUse`) holds one JSON object
|
||||
per line: crashes carry a reason and stacktrace, access lines carry status and
|
||||
duration, and token-shaped paths, emails and credentials are redacted. Off-box
|
||||
encrypted backups cover the data.
|
||||
|
||||
The accepted risk: losing the host loses the logs with it, and there is no
|
||||
aggregate error rate. At launch that is acceptable — one host, one operator,
|
||||
low traffic, and an incident is diagnosed from the journal on the box or from
|
||||
the probe's alert body. The triggers above are the point at which that risk
|
||||
stops being acceptable.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Investigation starts with `journalctl -u chesstrainer -o cat | jq`, not a
|
||||
dashboard.
|
||||
- Error tracking, metrics and log shipping stay explicit backlog items with
|
||||
triggers rather than ambient gaps.
|
||||
- The probe is an on-box prober with off-box alerting; it is not external
|
||||
uptime monitoring and the runbook says so.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Error-state verification
|
||||
|
||||
The #162 checklist, walked on 2026-11-13 against a local production-like build.
|
||||
Failures are filed as #171; this page is linked from the audit.
|
||||
|
||||
## Method
|
||||
|
||||
- **Build:** `MIX_ENV=prod` `mix phx.server` on `127.0.0.1:4010`, commit
|
||||
`021e412` plus this branch's documentation.
|
||||
- **Database:** a scratch `postgres:18.6` on port 55432. Empty at first, then
|
||||
one confirmed user and one 8+ piece endgame (inserted directly, because the
|
||||
admin changeset rejects positions beyond seven pieces).
|
||||
- **Configuration:** `PRELAUNCH=false`, `MAILING_LIST_ENABLED=true`,
|
||||
`TABLEBASE_PATH` pointing at a missing directory, dummy SES credentials.
|
||||
- **Forced failures:** Postgres stopped, missing tablebase directory, empty
|
||||
puzzle pool, expired reset and confirmation tokens (backdated
|
||||
`users_tokens.inserted_at`), stale and invalid mailing-list forms, guest
|
||||
budget exhausted, unknown public play code.
|
||||
- **Observation:** `curl` for controller-rendered pages, headless Chrome over
|
||||
the DevTools Protocol for LiveView-connected pages, and `mix test` for the
|
||||
interactive states that need a click. Each row names its evidence.
|
||||
- **Not forced:** a real Lichess 429 (the cooldown and classifier paths were
|
||||
exercised by the tests), a crash after connect, and the two admin-only import
|
||||
leaks (code inspection and tests). The first run rejected the socket origin
|
||||
because `PHX_HOST` defaulted to `example.com`; setting `PHX_HOST=127.0.0.1`
|
||||
fixed it. That was local configuration, not a product finding.
|
||||
|
||||
## Results
|
||||
|
||||
| Checklist row | Result | Evidence |
|
||||
|---|---|---|
|
||||
| T1 cooldown | **pass** | LiveView test: mount with a cooldown shows "Tablebase cooling down"; Retry re-fetches and returns to play. |
|
||||
| T2 429 | **pass** | LiveView test: classifier maps 429 to "Tablebase rate-limited"; HTML carries no `429`, `too_many_requests` or upstream body. |
|
||||
| T3 not covered | **pass** | Prod run: guest page with a 12-piece endgame shows "Position not covered… can't be graded", no `not_covered` in the DOM, Next puzzle or back available. |
|
||||
| T4 generic | **pass** | LiveView test: classifier returns "Tablebase unavailable"; the reason is logged and never rendered. |
|
||||
| T5/T6 load and reply failure | **pass** | T5 covered by the cooldown-at-mount test and Retry. T6 shares the blocked panel and `reset/1` path; not separately forced. |
|
||||
| T8 invalid FEN | **fail** | Existing test: the flash still contains the client message and the raw FEN. #171. |
|
||||
| S1/S2/S3/S5 empty pools | **pass** | S1 prod: Medium picker shows the in-page empty state and Retry. S5 prod: guest shows "No endgames are available yet" and Retry. S2/S3 tests: exhausted state, Retry and Change difficulty. |
|
||||
| S4 guest limit | **pass** | Prod run: five visits exhaust the budget; the wall shows the reset time, Sign up free and Log in. |
|
||||
| A1 reset token | **pass** | Prod run: an expired token flashes "Reset password link is invalid or it has expired." on the log-in page, with Forgot your password available. |
|
||||
| A2 confirmation token | **fail** | Prod run: an expired token flashes "User not found or token is invalid." and lands on registration, not log in. #171. |
|
||||
| A3/A4 login | **pass** | Tests: invalid credentials use one message; the throttle reports the remaining minutes. |
|
||||
| M1–M5 mailing list | **pass** | Prod run: invalid confirmation link, removed address, stale form, invalid email and an SES send failure each show their specific copy with the form reachable. |
|
||||
| W1/W2 404 and 500 | **pass** | Prod run: styled 404 for an unknown path; with Postgres stopped, `/` returns the styled 500 with no exception or stacktrace. |
|
||||
| W3 crash | **fail** | The server-error banner has no reload action; a repeating crash has no manual escape. Not separately forced; same banner as W4. #171. |
|
||||
| W4 disconnect | **pass** | Observed "Something went wrong! Attempting to reconnect" while the socket was rejected, with automatic retries; the banner hides on reconnect. |
|
||||
| W5 database down | **pass** | Prod run: `/health` returns 503 `db:down`; `/` returns the 500 page. Note the loop: the home link also needs the database, so it 500s too. |
|
||||
| D1/D2 export | **fail** | Prod run: an export with the database stopped returns the 500 page (no export-specific copy); `/users/profile/export/xml` returns plain text "Unknown export format". #171. |
|
||||
| H1 empty-database home | **fail** | Prod run: the fallback board renders, but "Try today's puzzle" points at the auth-gated `/endgames`. #171. |
|
||||
| X1 unknown play code | **pass** | Prod run: `/endgames/nope/play` returns the styled 404 page. This corrects the audit's earlier claim that it 500ed. |
|
||||
| X2/X3 admin leaks | **fail** | Code inspection and tests: both importer messages still interpolate `inspect(reason)`. Not forced. #171. |
|
||||
|
||||
## Tally
|
||||
|
||||
- 14 checklist rows pass, 6 fail (8 individual regressions, because D1/D2 and
|
||||
X2/X3 share a row).
|
||||
- Every failure row is a state that was already listed as "improve" in the
|
||||
audit and was outside #163; no new regression came from the #163 changes.
|
||||
- The regressions are filed together in #171 rather than fixed silently.
|
||||
@@ -16,7 +16,8 @@ Verdicts:
|
||||
status body, a stacktrace, or a raw FEN appearing in rendered HTML.
|
||||
|
||||
**Status:** #163 implements T1–T6, S1–S3 and S5. The rows below describe the
|
||||
state before that change; #164 records what was verified afterwards.
|
||||
state before that change. #164 walked this checklist on a local prod build:
|
||||
see [error-states-verification.md](error-states-verification.md).
|
||||
|
||||
## Play and tablebase
|
||||
|
||||
@@ -84,7 +85,7 @@ state before that change; #164 records what was verified afterwards.
|
||||
|
||||
| ID | State | Trigger | What the user sees today | Verdict | Action |
|
||||
|---|---|---|---|---|---|
|
||||
| X1 | Unknown public play code | `/endgames/:code/play` with a code that does not exist (`play.ex:79`) | `get_endgame_by_public_code!/1` raises → the 500 page | improve | Fetch without the bang and render the 404 page (or redirect to `/endgames` with a flash). A stale link is not a server error. |
|
||||
| X1 | Unknown public play code | `/endgames/:code/play` with a code that does not exist (`play.ex:79`) | The styled 404 page: `Ecto.NoResultsError` maps to 404 through `Plug.Exception` | keep | Verified by #164; no change. |
|
||||
| X2 | Bulk import lookup crash | `handle_async` exits (`import.ex:271`) | Flash **"Could not look up the FENs: #{inspect(reason)}"** | improve | Generic copy plus a logged reason. Admin-only, but still an internal leak. |
|
||||
| X3 | Bulk import tablebase load failure | `Local.check/0` returns an unexpected error (`import.ex:287`) | **"Local tablebase failed to load: #{inspect(reason)}"** | improve | Generic copy plus a logged reason. Admin-only, but still an internal leak. |
|
||||
|
||||
@@ -146,7 +147,6 @@ Separate changes, each independently testable:
|
||||
- **D1** — rescue export errors into a flash on the data tab.
|
||||
- **D2** — styled 404 for unknown export formats.
|
||||
- **H1** — empty-database call to action.
|
||||
- **X1** — 404 instead of 500 for an unknown public code.
|
||||
- **X2/X3** — generic admin import copy; log the details.
|
||||
|
||||
No code changes are part of this audit.
|
||||
|
||||
Reference in New Issue
Block a user