Files
chess-endgame-trainer/docs/error-states-verification.md
T
owenrees aebb2aecfb Fix the error-state regressions from the #164 verification (#171) (#173)
T8: the invalid-FEN flash is generic; the client message and FEN go to the
log. A2: an expired confirmation link says so and lands on log-in, where an
unconfirmed account resends the email. W3: the connection banner gains Reload
and Home actions. D1: a failed export logs and redirects to the data tab with
a support address instead of a bare 500. D2: an unknown export format renders
the styled 404. H1: the empty-database home CTA points at the guest trainer
instead of auth-gated /endgames. X2/X3: the importer hides inspect(reason)
behind generic copy and logs it.

Adds coverage for the T8 copy, A2 destination, D2 404 page, H1 CTA and the W3
actions; the H1 tests now restore the curated-code override so async page
tests cannot poison each other.
2026-11-13 16:46:29 +01:00

62 lines
5.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Error-state verification
The #162 checklist, walked on 2026-11-13 against a local production-like build.
Failures were filed as #171 and fixed in the follow-up for that issue; this page
records the verification run, not the current state. It is linked from the
audit.
## Method
- **Build:** `MIX_ENV=prod` `mix phx.server` on `127.0.0.1:4010`, commit
`021e412` plus this branch's documentation.
- **Database:** a scratch `postgres:18.6` on port 55432. Empty at first, then
one confirmed user and one 8+ piece endgame (inserted directly, because the
admin changeset rejects positions beyond seven pieces).
- **Configuration:** `PRELAUNCH=false`, `MAILING_LIST_ENABLED=true`,
`TABLEBASE_PATH` pointing at a missing directory, dummy SES credentials.
- **Forced failures:** Postgres stopped, missing tablebase directory, empty
puzzle pool, expired reset and confirmation tokens (backdated
`users_tokens.inserted_at`), stale and invalid mailing-list forms, guest
budget exhausted, unknown public play code.
- **Observation:** `curl` for controller-rendered pages, headless Chrome over
the DevTools Protocol for LiveView-connected pages, and `mix test` for the
interactive states that need a click. Each row names its evidence.
- **Not forced:** a real Lichess 429 (the cooldown and classifier paths were
exercised by the tests), a crash after connect, and the two admin-only import
leaks (code inspection and tests). The first run rejected the socket origin
because `PHX_HOST` defaulted to `example.com`; setting `PHX_HOST=127.0.0.1`
fixed it. That was local configuration, not a product finding.
## Results
| Checklist row | Result | Evidence |
|---|---|---|
| T1 cooldown | **pass** | LiveView test: mount with a cooldown shows "Tablebase cooling down"; Retry re-fetches and returns to play. |
| T2 429 | **pass** | LiveView test: classifier maps 429 to "Tablebase rate-limited"; HTML carries no `429`, `too_many_requests` or upstream body. |
| T3 not covered | **pass** | Prod run: guest page with a 12-piece endgame shows "Position not covered… can't be graded", no `not_covered` in the DOM, Next puzzle or back available. |
| T4 generic | **pass** | LiveView test: classifier returns "Tablebase unavailable"; the reason is logged and never rendered. |
| T5/T6 load and reply failure | **pass** | T5 covered by the cooldown-at-mount test and Retry. T6 shares the blocked panel and `reset/1` path; not separately forced. |
| T8 invalid FEN | **fail** | Existing test: the flash still contains the client message and the raw FEN. #171. |
| S1/S2/S3/S5 empty pools | **pass** | S1 prod: Medium picker shows the in-page empty state and Retry. S5 prod: guest shows "No endgames are available yet" and Retry. S2/S3 tests: exhausted state, Retry and Change difficulty. |
| S4 guest limit | **pass** | Prod run: five visits exhaust the budget; the wall shows the reset time, Sign up free and Log in. |
| A1 reset token | **pass** | Prod run: an expired token flashes "Reset password link is invalid or it has expired." on the log-in page, with Forgot your password available. |
| A2 confirmation token | **fail** | Prod run: an expired token flashes "User not found or token is invalid." and lands on registration, not log in. #171. |
| A3/A4 login | **pass** | Tests: invalid credentials use one message; the throttle reports the remaining minutes. |
| M1–M5 mailing list | **pass** | Prod run: invalid confirmation link, removed address, stale form, invalid email and an SES send failure each show their specific copy with the form reachable. |
| W1/W2 404 and 500 | **pass** | Prod run: styled 404 for an unknown path; with Postgres stopped, `/` returns the styled 500 with no exception or stacktrace. |
| W3 crash | **fail** | The server-error banner has no reload action; a repeating crash has no manual escape. Not separately forced; same banner as W4. #171. |
| W4 disconnect | **pass** | Observed "Something went wrong! Attempting to reconnect" while the socket was rejected, with automatic retries; the banner hides on reconnect. |
| W5 database down | **pass** | Prod run: `/health` returns 503 `db:down`; `/` returns the 500 page. Note the loop: the home link also needs the database, so it 500s too. |
| D1/D2 export | **fail** | Prod run: an export with the database stopped returns the 500 page (no export-specific copy); `/users/profile/export/xml` returns plain text "Unknown export format". #171. |
| H1 empty-database home | **fail** | Prod run: the fallback board renders, but "Try today's puzzle" points at the auth-gated `/endgames`. #171. |
| X1 unknown play code | **pass** | Prod run: `/endgames/nope/play` returns the styled 404 page. This corrects the audit's earlier claim that it 500ed. |
| X2/X3 admin leaks | **fail** | Code inspection and tests: both importer messages still interpolate `inspect(reason)`. Not forced. #171. |
## Tally
- 14 checklist rows pass, 6 fail (8 individual regressions, because D1/D2 and
X2/X3 share a row).
- Every failure row is a state that was already listed as "improve" in the
audit and was outside #163; no new regression came from the #163 changes.
- The regressions are filed together in #171 rather than fixed silently.