mirror of
https://github.com/TheRealOwenRees/chess-endgame-trainer.git
synced 2026-11-18 03:32:06 +00:00
Add a public health probe and monitoring runbook (#166)
* move theme picker position in add endgames form * Add branded favicons and web manifest - Replace the placeholder favicon with the knight mark: .ico (16/32/48), SVG, and 16/32 PNGs. - Add opaque apple-touch and Android launcher icons plus a site webmanifest naming the app and using the brand palette. - Link the icon set, manifest, and theme-color from the root layout and the 404/500 pages. - Whitelist the new static files in static_paths/0 and the prelaunch allowlist so they are served in production and behind the gate. - Sync the coming-soon site's favicon.ico and drop the unused Phoenix logo. * Add a public health probe and monitoring runbook Probe https://<PHX_HOST>/health through Cloudflare every two minutes and report to a healthchecks.io check. A healthy run posts the raw health JSON; a tablebase mismatch is a green check-in with the mismatch recorded; a non-200, non-JSON, down database or network failure posts one fail ping with the response body and a loopback cross-check as evidence. --dry-run evaluates the success, degraded, fail, non-JSON and unreachable fixtures without reading config or touching the network. The runbook covers the healthchecks.io check, install, what /health proves, Beszel's blind spot and the drills. * Fix theme picker search after moving it into the endgame form The picker moved inside the endgame form, which nested the component's own <form>. Browsers, and the LiveView test DOM, drop a nested form along with its phx-change and phx-submit bindings, so the search box did nothing and three form tests lost their #theme-form selector. The component now renders no form: phx-change sits on the search input, which works both inside the endgame form and standalone on the bulk importer. The unreachable create_theme_from_search handlers go away; theme creation uses the existing button. Tests search through the input and click that button.
This commit is contained in:
+187
-2
@@ -100,6 +100,10 @@ DNS:
|
||||
`/etc/chesstrainer/postgres.env` holds the Postgres container credentials,
|
||||
also `root:root` and `0600`.
|
||||
|
||||
`/etc/chesstrainer/probe.env` holds the public probe's URL and its
|
||||
healthchecks.io ping URL, also `root:root` and `0600`. See §Monitoring and
|
||||
alerting.
|
||||
|
||||
The app runs as the dedicated `chesstrainer` system user. It must own
|
||||
`/opt/chesstrainer` so it can write `priv/static/backups`.
|
||||
|
||||
@@ -752,7 +756,9 @@ curl -i http://127.0.0.1:4100/health
|
||||
|
||||
`/health` returns JSON with `status`, `db`, `tablebase`, and `process`.
|
||||
It returns 200 for `ok`/`degraded` and 503 for `down`. A missing local tablebase
|
||||
reports `degraded` but still 200 because Lichess fallback is available.
|
||||
reports `degraded` but still 200 because Lichess fallback is available. See
|
||||
§Monitoring and alerting for the public probe that watches it, and for what a
|
||||
green check does and does not prove.
|
||||
|
||||
### Reading production logs
|
||||
|
||||
@@ -800,6 +806,183 @@ Set `Storage=persistent` and a `SystemMaxUse=` value in
|
||||
rate-limits bursts; if a crash storm drops lines, raise `RateLimitBurst` or
|
||||
`RateLimitIntervalSec` there.
|
||||
|
||||
## Monitoring and alerting
|
||||
|
||||
Three checks watch production at three different layers:
|
||||
|
||||
| Where | What | Notices | Alerts |
|
||||
|---|---|---|---|
|
||||
| VPS | `chesstrainer-probe.timer` → `chesstrainer-probe.service` | the user-facing HTTP path: DNS, Cloudflare, Caddy, Phoenix, Postgres | healthchecks.io |
|
||||
| VPS | Beszel hub + agent | host liveness, CPU, memory, disk, container restarts and OOM kills | Beszel notifications |
|
||||
| VPS | `chesstrainer-backup.timer` heartbeat | whether the nightly dump finished | healthchecks.io (its own check) |
|
||||
|
||||
The backup heartbeat is documented in §Backups. The probe and the Beszel gap
|
||||
are below.
|
||||
|
||||
### On-box prober, off-box alerting
|
||||
|
||||
The probe runs on the VPS and curls the **public** hostname, so it crosses the
|
||||
same path a visitor does: Cloudflare's DNS and edge, the proxy back to Caddy,
|
||||
Caddy's Cloudflare allowlist, the Phoenix router, and one database query.
|
||||
|
||||
That is deliberately not `127.0.0.1:4100`. A loopback probe stays green through
|
||||
every failure that makes the site unusable: a bad `cloudflare-ranges.caddy`
|
||||
reload, a stopped Caddy (`521`), an expired origin certificate (`526`), a
|
||||
deleted proxied DNS record, a WAF rule blocking `/health`, or a Cloudflare
|
||||
incident. Beszel is green through all of those too, because the host and the
|
||||
container are fine.
|
||||
|
||||
The trade-off is that the prober shares a host with the app. It cannot report
|
||||
that its own host is down, and it cannot prove the view from another network.
|
||||
Host death is caught by **silence**: the timer stops, no ping arrives, and
|
||||
healthchecks.io — which is off-box — alerts after the grace period. That is the
|
||||
whole reason the check-in URL points at a SaaS rather than at another box here.
|
||||
Do not describe this as external uptime monitoring.
|
||||
|
||||
### The healthchecks.io check
|
||||
|
||||
Create the check at healthchecks.io:
|
||||
|
||||
- Name `innersquare.app /health`, period **2 minutes**, grace **2 minutes**
|
||||
- Integrations: email and Discord (webhook). Use each integration's test
|
||||
notification once, and keep recovery notifications enabled so a resolved
|
||||
incident is visible.
|
||||
- Copy the ping URL. It is a secret: anyone who has it can mark the check
|
||||
green. It belongs only in the root-owned env file.
|
||||
|
||||
Every check-in carries evidence:
|
||||
|
||||
- A healthy run posts the raw `/health` JSON.
|
||||
- A tablebase mismatch still posts a **success**, because a Lichess fallback is
|
||||
not an outage. The body carries the expected and actual values, so the
|
||||
mismatch is visible in the check history without alerting anyone.
|
||||
- A non-200, a malformed body, a `db` that is not `ok`, or a network/TLS
|
||||
failure posts to `<ping-url>/fail` with a JSON body containing the reason,
|
||||
the HTTP status, the curl exit code, the response body, and a loopback
|
||||
`origin_status`/`origin_body` for comparison. The origin fields are evidence
|
||||
only; the verdict comes from the public request.
|
||||
|
||||
A ping that cannot be delivered is a journal warning, not a second verdict:
|
||||
the next timer run reports the state again.
|
||||
|
||||
### Install the probe
|
||||
|
||||
Write the env file, root-owned and `0600`:
|
||||
|
||||
```sh
|
||||
sudo install -m 600 /dev/null /etc/chesstrainer/probe.env
|
||||
sudoedit /etc/chesstrainer/probe.env
|
||||
```
|
||||
|
||||
Start from `deploy/probe.env.example`. `PROBE_URL` and `HEALTHCHECK_URL` are the
|
||||
only required values. Then install the script, the units, and the timer:
|
||||
|
||||
```sh
|
||||
sudo install -m 0755 /opt/chesstrainer/src/deploy/probe \
|
||||
/usr/local/sbin/chesstrainer-probe
|
||||
sudo install -m 0644 \
|
||||
/opt/chesstrainer/src/deploy/chesstrainer-probe.service \
|
||||
/opt/chesstrainer/src/deploy/chesstrainer-probe.timer \
|
||||
/etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now chesstrainer-probe.timer
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```sh
|
||||
systemctl list-timers chesstrainer-probe.timer
|
||||
sudo journalctl -u chesstrainer-probe -n 5 -o cat
|
||||
```
|
||||
|
||||
Expect a `probe.ok` line every two minutes and a green check within one period.
|
||||
Each run prints exactly one JSON line (`probe.ok`, `probe.degraded` or
|
||||
`probe.failed`), like the backup script, so #140's log shipping can pick it up.
|
||||
|
||||
### What `/health` does and does not mean
|
||||
|
||||
`/health` returns `status`, `db`, `tablebase` and `process`. It returns 200 for
|
||||
`ok` and `degraded`, and 503 for `down`.
|
||||
|
||||
- `db` is one `SELECT 1` through the connection pool. It proves the pool can
|
||||
reach Postgres. It does not prove migrations are current or that writes work.
|
||||
- `tablebase` reflects the local Syzygy directory. `fallback` means lookups go
|
||||
to Lichess. `degraded` is 200 because the fallback is usable.
|
||||
- `process` is always `"ok"`. It is not independent evidence of liveness.
|
||||
|
||||
A green probe therefore means: DNS resolved, Cloudflare served the request,
|
||||
Caddy routed it under the Cloudflare allowlist, Phoenix answered, and Postgres
|
||||
answered one query. It does not mean background jobs are running, SES accepts
|
||||
mail, writes succeed, tablebase lookups are fast, or the site is fast under
|
||||
load. Beszel covers resource exhaustion and container restarts; the release
|
||||
health check and the logs cover deploy mistakes.
|
||||
|
||||
### Beszel's blind spot
|
||||
|
||||
Beszel watches the host: agent connection, CPU, memory, disk, container
|
||||
restarts, OOM kills. Its hub runs on the same VPS as the agent, so a VPS
|
||||
outage or network loss silences it as well — that case is covered by the probe
|
||||
check going quiet. More importantly, Beszel never calls `/health`. A Phoenix
|
||||
release returning 500s on a healthy host, or Postgres refusing connections
|
||||
inside a running container, reads green in Beszel and red in the probe. The two
|
||||
are complementary, not redundant.
|
||||
|
||||
### Drills
|
||||
|
||||
Nothing below touches production data, and the app keeps serving throughout.
|
||||
|
||||
`--dry-run` needs no network and no env file. It evaluates the success,
|
||||
degraded, fail, non-JSON and unreachable fixtures and sends no check-ins:
|
||||
|
||||
```sh
|
||||
sudo /usr/local/sbin/chesstrainer-probe --dry-run
|
||||
```
|
||||
|
||||
A degraded check-in, recorded but never alerting. A value passed on the command
|
||||
line wins over the env file:
|
||||
|
||||
```sh
|
||||
sudo EXPECT_TABLEBASE=not_configured /usr/local/sbin/chesstrainer-probe
|
||||
```
|
||||
|
||||
Expect exit 0, a green check, and a body showing `tablebase: local` against
|
||||
`expected_tablebase: not_configured`. No email and no Discord message.
|
||||
|
||||
The one fail ping, and the full alert path. Stop the timer first so the next
|
||||
scheduled run cannot mask the result:
|
||||
|
||||
```sh
|
||||
sudo systemctl stop chesstrainer-probe.timer
|
||||
sudo PROBE_URL=https://innersquare.app/health-does-not-exist \
|
||||
/usr/local/sbin/chesstrainer-probe
|
||||
```
|
||||
|
||||
Expect exit 1, one `probe.failed` line with `http_status: 404`, one fail ping
|
||||
whose body shows `origin_status: 200`, one alert in Discord/email, and no
|
||||
second fail ping. Then send a real check-in and re-arm:
|
||||
|
||||
```sh
|
||||
sudo /usr/local/sbin/chesstrainer-probe
|
||||
sudo systemctl start chesstrainer-probe.timer
|
||||
```
|
||||
|
||||
The real run posts a success, and healthchecks.io notifies that the check is
|
||||
back up. For the unreachable branch, point the probe at a closed port instead;
|
||||
the origin evidence should still show `200`:
|
||||
|
||||
```sh
|
||||
sudo PROBE_URL=https://127.0.0.1:9/health /usr/local/sbin/chesstrainer-probe
|
||||
```
|
||||
|
||||
Planned maintenance: stopping the timer is reported as a down check after
|
||||
period plus grace, so about four minutes. Pause the check in the healthchecks.io
|
||||
UI for the duration, or accept the alert.
|
||||
|
||||
The probe reaches the edge as a plain `curl`. If a bot policy or WAF rule ever
|
||||
starts rejecting it, the fail body's `origin_status` shows the app itself is
|
||||
healthy; that is an edge-policy change, not an application outage. Re-enabling
|
||||
Bot Fight Mode is tracked in #153.
|
||||
|
||||
## Interactive shell and user management
|
||||
|
||||
Attach an IEx shell to the **running** release. This does not restart the
|
||||
@@ -1223,5 +1406,7 @@ restore over `chesstrainer_prod` while the application is running.
|
||||
globals file.
|
||||
12. Workstation age key and pull configuration installed, user timers and linger
|
||||
enabled, first pull verified, first drill passes.
|
||||
13. Remote shell attaches and exits cleanly:
|
||||
13. `/etc/chesstrainer/probe.env` written, probe script and units installed,
|
||||
timer enabled, and the healthchecks.io check green within one period.
|
||||
14. Remote shell attaches and exits cleanly:
|
||||
`sudo /opt/chesstrainer/current/bin/chesstrainer remote` (Ctrl-C twice).
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Probe the public Inner Square Chess app and report to healthchecks.io
|
||||
Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=root
|
||||
Group=root
|
||||
ExecStart=/usr/local/sbin/chesstrainer-probe
|
||||
TimeoutStartSec=90s
|
||||
SyslogIdentifier=chesstrainer-probe
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Probe the public Inner Square Chess app every two minutes
|
||||
Documentation=https://github.com/TheRealOwenRees/chess-endgame-trainer/blob/main/deploy/README.md
|
||||
|
||||
[Timer]
|
||||
OnBootSec=30s
|
||||
OnUnitActiveSec=2min
|
||||
AccuracySec=5s
|
||||
Unit=chesstrainer-probe.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Executable
+359
@@ -0,0 +1,359 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Probe the public Inner Square Chess app and report the verdict to
|
||||
# healthchecks.io.
|
||||
#
|
||||
# Installed as /usr/local/sbin/chesstrainer-probe (root-owned 0755) and run by
|
||||
# chesstrainer-probe.timer every two minutes. The probe deliberately goes
|
||||
# through the public hostname, not loopback: DNS, Cloudflare, Caddy and the
|
||||
# origin allowlist are part of what a visitor depends on. A loopback health
|
||||
# URL is still consulted on failure, as evidence only.
|
||||
#
|
||||
# sudo /usr/local/sbin/chesstrainer-probe
|
||||
# sudo /usr/local/sbin/chesstrainer-probe --dry-run # no network, no config
|
||||
#
|
||||
# Configuration lives in /etc/chesstrainer/probe.env (root:root 0600). See
|
||||
# deploy/README.md §Monitoring and alerting. Individual settings can be passed
|
||||
# as environment variables to a manual run, which is how the drills vary the
|
||||
# URL and the expected tablebase.
|
||||
#
|
||||
# Each run prints exactly one JSON line (probe.ok, probe.degraded or
|
||||
# probe.failed) so journald, and #140's log shipping, can pick it up.
|
||||
|
||||
set -euo pipefail
|
||||
umask 077
|
||||
|
||||
ENV_FILE="${ENV_FILE:-/etc/chesstrainer/probe.env}"
|
||||
|
||||
START_SECONDS=$SECONDS
|
||||
DRY_RUN=0
|
||||
TMPDIR_PROBE=""
|
||||
HTTP_BODY_FILE=""
|
||||
HTTP_HEADER_FILE=""
|
||||
HTTP_ERROR_FILE=""
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage: chesstrainer-probe [--dry-run]
|
||||
|
||||
--dry-run print the verdict for the success, degraded, fail, non-JSON and
|
||||
unreachable fixtures without reading the env file or making a
|
||||
network request
|
||||
EOF
|
||||
}
|
||||
|
||||
while (($# > 0)); do
|
||||
case "$1" in
|
||||
--dry-run) DRY_RUN=1 ;;
|
||||
-h | --help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
usage >&2
|
||||
echo "probe: unknown argument: $1" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
log_event() {
|
||||
local event="$1"
|
||||
local extra="${2:-}"
|
||||
|
||||
printf '{"timestamp":"%s","event":"%s"%s}\n' \
|
||||
"$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$event" "$extra"
|
||||
}
|
||||
|
||||
fail() {
|
||||
echo "probe: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
json_escape() {
|
||||
local text="$1"
|
||||
|
||||
text="${text//\\/\\\\}"
|
||||
text="${text//\"/\\\"}"
|
||||
text="${text//$'\n'/\\n}"
|
||||
text="${text//$'\r'/\\r}"
|
||||
text="${text//$'\t'/\\t}"
|
||||
text="$(printf '%s' "$text" | LC_ALL=C tr -d '[:cntrl:]')"
|
||||
|
||||
printf '%s' "$text"
|
||||
}
|
||||
|
||||
body_snippet() {
|
||||
local file="$1"
|
||||
|
||||
if [[ -s "$file" ]]; then
|
||||
head -c "$MAX_BODY_BYTES" "$file"
|
||||
fi
|
||||
}
|
||||
|
||||
json_field() {
|
||||
local key="$1"
|
||||
local json="$2"
|
||||
|
||||
if [[ "$json" =~ \"$key\"[[:space:]]*:[[:space:]]*\"([^\"]*)\" ]]; then
|
||||
printf '%s' "${BASH_REMATCH[1]}"
|
||||
fi
|
||||
}
|
||||
|
||||
http_get() {
|
||||
local url="$1"
|
||||
local prefix="$2"
|
||||
local proto="$3"
|
||||
local status=""
|
||||
local exit_code=0
|
||||
|
||||
HTTP_BODY_FILE="$prefix.body"
|
||||
HTTP_HEADER_FILE="$prefix.headers"
|
||||
HTTP_ERROR_FILE="$prefix.error"
|
||||
: >"$HTTP_BODY_FILE"
|
||||
: >"$HTTP_HEADER_FILE"
|
||||
: >"$HTTP_ERROR_FILE"
|
||||
|
||||
if status="$(curl --silent --show-error --location \
|
||||
--proto "$proto" \
|
||||
--connect-timeout 5 \
|
||||
--max-time "$PROBE_TIMEOUT_SECONDS" \
|
||||
--output "$HTTP_BODY_FILE" \
|
||||
--dump-header "$HTTP_HEADER_FILE" \
|
||||
--write-out '%{http_code}' \
|
||||
"$url")"; then
|
||||
exit_code=0
|
||||
else
|
||||
exit_code=$?
|
||||
fi
|
||||
|
||||
HTTP_STATUS="${status:-000}"
|
||||
CURL_EXIT="$exit_code"
|
||||
}
|
||||
|
||||
# Sets VERDICT (ok, degraded, failed), REASON, DB and TABLEBASE. The database is
|
||||
# the only hard dependency; a missing local tablebase still serves users from
|
||||
# Lichess, so it is reported as a green but degraded check-in.
|
||||
evaluate() {
|
||||
local status="$1"
|
||||
local exit_code="$2"
|
||||
local body="$3"
|
||||
local trimmed=""
|
||||
|
||||
VERDICT="failed"
|
||||
REASON=""
|
||||
DB=""
|
||||
TABLEBASE=""
|
||||
|
||||
if ((exit_code != 0)); then
|
||||
REASON="unreachable: curl exit $exit_code"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$status" != "200" ]]; then
|
||||
REASON="http $status"
|
||||
return
|
||||
fi
|
||||
|
||||
trimmed="${body#"${body%%[![:space:]]*}"}"
|
||||
trimmed="${trimmed%"${trimmed##*[![:space:]]}"}"
|
||||
if [[ "${trimmed:0:1}" != "{" || "${trimmed: -1}" != "}" ]]; then
|
||||
REASON="non-JSON response"
|
||||
return
|
||||
fi
|
||||
|
||||
DB="$(json_field db "$body")"
|
||||
TABLEBASE="$(json_field tablebase "$body")"
|
||||
|
||||
if [[ -z "$DB" ]]; then
|
||||
REASON="response has no db field"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$DB" != "ok" ]]; then
|
||||
REASON="db is $DB"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$EXPECT_TABLEBASE" != "any" && "$TABLEBASE" != "$EXPECT_TABLEBASE" ]]; then
|
||||
VERDICT="degraded"
|
||||
REASON="tablebase is ${TABLEBASE:-unknown}, expected $EXPECT_TABLEBASE"
|
||||
return
|
||||
fi
|
||||
|
||||
VERDICT="ok"
|
||||
}
|
||||
|
||||
# Exactly one call site per run, and one call site per outcome, so a forced
|
||||
# failure can only ever produce one fail ping.
|
||||
ping_check() {
|
||||
local outcome="$1"
|
||||
local body="$2"
|
||||
local url=""
|
||||
local body_file=""
|
||||
|
||||
if [[ -z "$HEALTHCHECK_URL" ]]; then
|
||||
echo "probe: HEALTHCHECK_URL is not set, not reporting" >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
url="${HEALTHCHECK_URL%/}"
|
||||
if [[ "$outcome" == "fail" ]]; then
|
||||
url="$url/fail"
|
||||
fi
|
||||
|
||||
body_file="$(mktemp)"
|
||||
printf '%s' "$body" >"$body_file"
|
||||
|
||||
if ! curl --silent --show-error --fail \
|
||||
--max-time 10 --retry 2 --retry-delay 2 \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data-binary "@$body_file" \
|
||||
--output /dev/null "$url"; then
|
||||
echo "probe: check-in ping failed (run result is unaffected)" >&2
|
||||
fi
|
||||
|
||||
rm -f "$body_file"
|
||||
}
|
||||
|
||||
origin_evidence() {
|
||||
local prefix="$1"
|
||||
|
||||
http_get "$ORIGIN_URL" "$prefix" '=https,http'
|
||||
ORIGIN_STATUS="$HTTP_STATUS"
|
||||
ORIGIN_EXIT="$CURL_EXIT"
|
||||
ORIGIN_BODY="$(body_snippet "$HTTP_BODY_FILE")"
|
||||
ORIGIN_ERROR="$(body_snippet "$HTTP_ERROR_FILE")"
|
||||
}
|
||||
|
||||
fail_evidence() {
|
||||
local evidence=""
|
||||
|
||||
evidence="$(printf '{"probe":"failed","reason":"%s","url":"%s","http_status":"%s","curl_exit":%s' \
|
||||
"$(json_escape "$REASON")" "$(json_escape "$PROBE_URL")" "$PUBLIC_STATUS" "$PUBLIC_EXIT")"
|
||||
|
||||
if [[ -n "$ERROR_TEXT" ]]; then
|
||||
evidence+="$(printf ',"error":"%s"' "$(json_escape "$ERROR_TEXT")")"
|
||||
fi
|
||||
|
||||
if [[ -n "$BODY" ]]; then
|
||||
evidence+="$(printf ',"body":"%s"' "$(json_escape "$BODY")")"
|
||||
fi
|
||||
|
||||
evidence+="$(printf ',"origin_url":"%s","origin_status":"%s","origin_curl_exit":%s' \
|
||||
"$(json_escape "$ORIGIN_URL")" "$ORIGIN_STATUS" "$ORIGIN_EXIT")"
|
||||
|
||||
if [[ -n "$ORIGIN_BODY" ]]; then
|
||||
evidence+="$(printf ',"origin_body":"%s"' "$(json_escape "$ORIGIN_BODY")")"
|
||||
fi
|
||||
|
||||
if [[ -n "$ORIGIN_ERROR" ]]; then
|
||||
evidence+="$(printf ',"origin_error":"%s"' "$(json_escape "$ORIGIN_ERROR")")"
|
||||
fi
|
||||
|
||||
printf '%s}' "$evidence"
|
||||
}
|
||||
|
||||
dry_run() {
|
||||
local -a scenarios=(
|
||||
'HTTP 200, healthy|200|0|{"process":"ok","status":"ok","db":"ok","tablebase":"local"}'
|
||||
'HTTP 200, tablebase fallback|200|0|{"process":"ok","status":"degraded","db":"ok","tablebase":"fallback"}'
|
||||
'HTTP 503, db down|503|0|{"process":"ok","status":"down","db":"down","tablebase":"local"}'
|
||||
'HTTP 200, non-JSON|200|0|<html><body>502 Bad Gateway</body></html>'
|
||||
'timeout, no response|000|28|'
|
||||
)
|
||||
local row name status exit_code body action
|
||||
|
||||
printf 'probe: dry run, no network calls and no configuration read\n'
|
||||
printf 'probe: every fixture is evaluated with EXPECT_TABLEBASE=local\n\n'
|
||||
printf '%-28s %-9s %-46s %s\n' "scenario" "verdict" "reason" "would ping"
|
||||
|
||||
for row in "${scenarios[@]}"; do
|
||||
IFS='|' read -r name status exit_code body <<<"$row"
|
||||
EXPECT_TABLEBASE="local"
|
||||
evaluate "$status" "$exit_code" "$body"
|
||||
|
||||
case "$VERDICT" in
|
||||
ok) action="success, raw /health JSON" ;;
|
||||
degraded) action="success, mismatch evidence" ;;
|
||||
*) action="fail, evidence JSON" ;;
|
||||
esac
|
||||
|
||||
printf '%-28s %-9s %-46s %s\n' "$name" "$VERDICT" "${REASON:--}" "$action"
|
||||
done
|
||||
|
||||
printf '\nprobe: dry run complete, no check-ins sent\n'
|
||||
}
|
||||
|
||||
if ((DRY_RUN)); then
|
||||
dry_run
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Overrides passed on the command line survive the env file, so the drills can
|
||||
# vary the URL or the expected tablebase without editing a root-only file.
|
||||
_OVERRIDE_PROBE_URL="${PROBE_URL:-}"
|
||||
_OVERRIDE_EXPECT_TABLEBASE="${EXPECT_TABLEBASE:-}"
|
||||
_OVERRIDE_PROBE_TIMEOUT_SECONDS="${PROBE_TIMEOUT_SECONDS:-}"
|
||||
|
||||
if [[ -r "$ENV_FILE" ]]; then
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
else
|
||||
fail "cannot read $ENV_FILE (set ENV_FILE to override)"
|
||||
fi
|
||||
|
||||
PROBE_URL="${_OVERRIDE_PROBE_URL:-${PROBE_URL:-}}"
|
||||
EXPECT_TABLEBASE="${_OVERRIDE_EXPECT_TABLEBASE:-${EXPECT_TABLEBASE:-local}}"
|
||||
PROBE_TIMEOUT_SECONDS="${_OVERRIDE_PROBE_TIMEOUT_SECONDS:-${PROBE_TIMEOUT_SECONDS:-10}}"
|
||||
|
||||
HEALTHCHECK_URL="${HEALTHCHECK_URL:-}"
|
||||
ORIGIN_URL="${ORIGIN_URL:-http://127.0.0.1:4100/health}"
|
||||
MAX_BODY_BYTES="${MAX_BODY_BYTES:-2000}"
|
||||
|
||||
[[ "$PROBE_URL" == https://* ]] || fail "PROBE_URL must be an https URL"
|
||||
[[ -n "$HEALTHCHECK_URL" ]] || fail "HEALTHCHECK_URL is not set"
|
||||
[[ "$EXPECT_TABLEBASE" =~ ^(local|fallback|not_configured|any)$ ]] ||
|
||||
fail "EXPECT_TABLEBASE must be local, fallback, not_configured or any"
|
||||
[[ "$PROBE_TIMEOUT_SECONDS" =~ ^[0-9]+$ ]] || fail "PROBE_TIMEOUT_SECONDS must be an integer"
|
||||
command -v curl >/dev/null 2>&1 || fail "missing command: curl"
|
||||
|
||||
TMPDIR_PROBE="$(mktemp -d)"
|
||||
|
||||
cleanup() {
|
||||
if [[ -n "$TMPDIR_PROBE" && -d "$TMPDIR_PROBE" ]]; then
|
||||
rm -rf "$TMPDIR_PROBE"
|
||||
fi
|
||||
}
|
||||
|
||||
trap cleanup EXIT
|
||||
trap 'exit 143' TERM INT
|
||||
|
||||
http_get "$PROBE_URL" "$TMPDIR_PROBE/public" '=https'
|
||||
PUBLIC_STATUS="$HTTP_STATUS"
|
||||
PUBLIC_EXIT="$CURL_EXIT"
|
||||
BODY="$(body_snippet "$HTTP_BODY_FILE")"
|
||||
ERROR_TEXT="$(body_snippet "$HTTP_ERROR_FILE")"
|
||||
|
||||
evaluate "$PUBLIC_STATUS" "$PUBLIC_EXIT" "$BODY"
|
||||
|
||||
case "$VERDICT" in
|
||||
ok)
|
||||
log_event probe.ok ",\"status\":\"$PUBLIC_STATUS\",\"tablebase\":\"$(json_escape "$TABLEBASE")\",\"duration_s\":$((SECONDS - START_SECONDS))"
|
||||
ping_check ok "$BODY"
|
||||
;;
|
||||
degraded)
|
||||
evidence="$(printf '{"expected_tablebase":"%s","health":%s}' \
|
||||
"$(json_escape "$EXPECT_TABLEBASE")" "$BODY")"
|
||||
log_event probe.degraded ",\"reason\":\"$(json_escape "$REASON")\",\"tablebase\":\"$(json_escape "$TABLEBASE")\",\"duration_s\":$((SECONDS - START_SECONDS))"
|
||||
ping_check ok "$evidence"
|
||||
;;
|
||||
*)
|
||||
origin_evidence "$TMPDIR_PROBE/origin"
|
||||
evidence="$(fail_evidence)"
|
||||
log_event probe.failed ",\"reason\":\"$(json_escape "$REASON")\",\"http_status\":\"$PUBLIC_STATUS\",\"curl_exit\":$PUBLIC_EXIT,\"duration_s\":$((SECONDS - START_SECONDS))"
|
||||
ping_check fail "$evidence"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@@ -0,0 +1,28 @@
|
||||
# /etc/chesstrainer/probe.env -- root:root, mode 0600.
|
||||
#
|
||||
# Sourced by /usr/local/sbin/chesstrainer-probe. The check-in URL is a secret:
|
||||
# anyone who has it can mark the check green, so keep it out of the repo and
|
||||
# out of world-readable files.
|
||||
|
||||
# The public URL a visitor actually requests. Probing the edge rather than
|
||||
# loopback is deliberate: DNS, TLS, Cloudflare, Caddy and the origin allowlist
|
||||
# are part of the contract. Must be https.
|
||||
PROBE_URL=https://innersquare.app/health
|
||||
|
||||
# healthchecks.io ping URL for the two-minute probe check. A healthy run posts
|
||||
# to it; a failed run posts to <url>/fail.
|
||||
HEALTHCHECK_URL=https://hc-ping.com/00000000-0000-0000-0000-000000000000
|
||||
|
||||
# The tablebase value /health should report. "local" means Syzygy files are
|
||||
# loaded, "fallback" means Lichess is used, "not_configured" means
|
||||
# TABLEBASE_PATH is unset. A mismatch is reported as degraded, never as down.
|
||||
# Set to "any" to accept whatever the app reports.
|
||||
EXPECT_TABLEBASE=local
|
||||
|
||||
# Whole-request timeout. The timer runs every two minutes, so keep this well
|
||||
# under that.
|
||||
PROBE_TIMEOUT_SECONDS=10
|
||||
|
||||
# Loopback health URL. It is consulted only when the public probe fails, and
|
||||
# only as evidence: it does not change the verdict. Defaults to the value shown.
|
||||
# ORIGIN_URL=http://127.0.0.1:4100/health
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 152 B After Width: | Height: | Size: 15 KiB |
Reference in New Issue
Block a user