Record the probe outage drill and its outcome (#168)

Adds the stop-Phoenix drill to the runbook, using forced probe runs so the
outage stays under a minute, and a dated drill log with the first run's
results. The log notes that the manual probe was started twice during the
outage, which is why two notifications arrived for one down/up transition.
This commit is contained in:
2026-11-13 14:48:01 +01:00
committed by GitHub
parent b41c8bf1a6
commit 0ae4c5b6fd
+33
View File
@@ -979,6 +979,25 @@ the origin evidence should still show `200`:
sudo PROBE_URL=https://127.0.0.1:9/health /usr/local/sbin/chesstrainer-probe
```
The outage drill: stop Phoenix, prove the alert, bring it back. Forcing each
probe run instead of waiting for the timer keeps the outage under a minute.
Start the probe **once** per step: every run sends its own check-in, so a
second start during the outage is a second alert.
```sh
sudo systemctl start chesstrainer-probe.service # baseline: probe.ok
sudo systemctl stop chesstrainer
sudo systemctl start chesstrainer-probe.service || true # probe.failed, http 502
# confirm one Discord message and one email, then:
sudo systemctl start chesstrainer
curl -sS -o /dev/null -w '%{http_code}\n' https://innersquare.app/health # 200
sudo systemctl start chesstrainer-probe.service # probe.ok, recovery
```
Expect `http_status: 502` from Caddy and `origin_status: 000` with
`origin_curl_exit: 7` in the fail body: Phoenix is down, while the host and the
edge are fine.
Planned maintenance: stopping the timer is reported as a down check after
period plus grace, so about four minutes. Pause the check in the healthchecks.io
UI for the duration, or accept the alert.
@@ -988,6 +1007,20 @@ starts rejecting it, the fail body's `origin_status` shows the app itself is
healthy; that is an edge-policy change, not an application outage. Re-enabling
Bot Fight Mode is tracked in #153.
### Drill log
- **2026-11-13 — install and first drills.** Script and units installed 12:34
UTC, timer enabled, first scheduled `probe.ok` at 12:35. Forced-failure drill
(`PROBE_URL` at a missing path): one fail ping with `http_status: 404` and
`origin_status: 200`, one Discord message and one email, then one recovery
notification. Degraded drill (`EXPECT_TABLEBASE=not_configured`): green
check-in with the mismatch in the body, no notification. Outage drill: the
app was stopped 13:32:55 and started 13:33:42 UTC; the manual probe was
started twice while it was down, so `probe.failed` fired at 13:33:06 and
13:33:14 (`http 502`) and two notifications arrived for one down → up
transition. `probe.ok` at 13:34:03 and one recovery notification. No
production data was touched. Start the probe once per drill step.
## Interactive shell and user management
Attach an IEx shell to the **running** release. This does not restart the