Files
chess-endgame-trainer/docs/logging.md
T
owenrees 624ddab644 Admin panel, login throttling, and Cloudflare edge hardening (#150)
* Document the remote shell for production user management

* add sitemap to footer links

* remove border under theme selection

* Add admin theme and user management (#144)

Add /admin/themes CRUD and /admin/users list/detail with role and plan editing, guarded by a shared Policy module. Include last-super-admin and self-role protections plus structured id-only admin logs.

Also add admin invitations, confirmation resends and password-reset emails, which work in development through the Swoosh mailbox before SES is configured.

* Rate-limit password logins and emit auth events (#142)

Add a two-bucket login throttle (5 failures per login, 10 per IP in a 15-minute window) and a shared client-address helper for controllers and LiveViews. Blocked attempts skip the database and bcrypt.

Log auth.login.failed and auth.login.throttled warnings with boot-scoped HMAC addresses only, and stop repeated confirmation emails on unconfirmed login attempts.

* Harden the production login edge for Cloudflare

Document the layered login defence: the app throttle is primary, Caddy must pass Cloudflare's CF-Connecting-IP to the app, and Cloudflare edge rate limiting covers request floods.

Update the reference Caddyfile to accept only Cloudflare ranges and reject direct-to-origin traffic, and document the exact Cloudflare dashboard rules.

* Document 10-second Cloudflare rate-limit window

Cloudflare may only offer a 10-second period. Keep the edge rule at 10 login POSTs per 10 seconds per IP and rely on the app's 15-minute login/address throttle for credential guessing.

* Document Cloudflare limits for LiveView registration

Registration submits over the shared LiveView socket, so a Cloudflare rule on POST /users/register will not count account creations. Note the app-level throttle/Turnstile options instead.

* Document current Cloudflare free-plan edge controls

Record the production setup: Cloudflare-only Caddy origins, Bot Fight Mode, and the 10-second login flood rule (10 POSTs per 10 seconds, 10-second block). Clarify that the application's 15-minute throttle remains the real credential lockout.

* Auto-refresh Cloudflare CIDRs for Caddy

Move the origin allowlist into a generated cloudflare-ranges.caddy imported by Caddy. Add an hourly systemd timer and script that fetch Cloudflare's v4/v6 lists, reject malformed responses, validate the candidate Caddyfile, and reload Caddy only when the ranges change. Restore the previous file on validation or reload failure.

* Document Cloudflare CIDR refresher setup and operations

Record the installed paths, permissions, timer checks, manual run, update procedure, and the direct-on-VPS setup fallback when the deploy directory is absent.
2026-11-11 21:27:59 +01:00

224 lines
8.0 KiB
Markdown

# Logging
Production writes one JSON object per line to standard output, which systemd
captures in journald. Development and test keep the readable
`[level] message` formatter.
## Configuration
`config/prod.exs` swaps the default handler's formatter for
`Chesstrainer.JSONLogFormatter`; nothing else changes per environment. The
formatter adds no runtime dependency beyond `Jason`, which the app already uses
for Phoenix JSON.
## Line shape
```json
{
"timestamp": "2026-11-10T21:36:28.489000Z",
"level": "error",
"message": "GET /endgames 500 in 12.5ms",
"request_id": "F7x...",
"method": "GET",
"path": "/endgames",
"status": 500,
"duration_ms": 12.5,
"error": {
"kind": "RuntimeError",
"reason": "bad thing",
"stacktrace": ["lib/foo.ex:10: Foo.bar/1"]
}
}
```
`timestamp`, `level` and `message` are always present. Metadata keys are
flattened into the object; reserved internal keys (`time`, `gl`, `report_cb`,
`error_logger`, `exception`, `stacktrace`, `crash_reason`, `reason`, `kind`)
are either unpacked into the line or omitted. Fixed fields win over metadata
with the same name.
| Field | When | Meaning |
|---|---|---|
| `timestamp` | always | ISO-8601 UTC, microsecond precision |
| `level` | always | `debug`, `info`, `notice`, `warning`, `error`, … |
| `message` | always | Rendered log message, truncated to 8 KiB |
| `request_id` | request-scoped logs | `Plug.RequestId` value, including LiveView processes |
| `method`, `path`, `status`, `duration_ms` | access lines | One HTTP request; no query string or body |
| `error.kind` | exceptions/crashes | Exception module, or the crash reason's kind |
| `error.reason` | exceptions/crashes | `Exception.message/1` or an inspected reason |
| `error.stacktrace` | exceptions/crashes | Array of formatted frames, capped at 50 |
| everything else | as supplied | Flattened atom metadata, sanitised and redacted |
Exceptions (`Logger.error(..., exception: ..., stacktrace: ...)`) and OTP crash
reports (`meta.crash_reason`, including Bandit request crashes and GenServer
terminations) become the `error` object. Stacktrace frames are an array of
strings so `jq -r '.error.stacktrace[]'` prints them readably.
## Redaction
The formatter redacts:
- metadata keys named `email`, `*_email`, `email_address`, `contact_email` or
`recipient`;
- any email-shaped string anywhere in metadata, nested `params`, or the rendered
message;
- credentials in `password`, `secret`, `token`, `authorization`, `cookie`,
`api_key`, `private_key` and `credential` keys;
- `token=...`/`password=...` assignments in messages.
Path segments of 20 or more URL-safe characters (reset/confirmation tokens,
UUIDs) are replaced with `[REDACTED]` in access lines. Query parameters and
request bodies are never logged.
## Access lines
`ChesstrainerWeb.RequestLogger` listens to `[:phoenix, :endpoint, :stop]` and
emits one `:info` line per HTTP request that reaches the endpoint, including a
LiveView's initial request. Phoenix's own endpoint start/stop pair is disabled
with `log: false` in `ChesstrainerWeb.Endpoint` so there is exactly one line.
`ChesstrainerWeb.Plugs.PutRequestId` copies `Plug.RequestId`'s id into the
session, and `UserAuth.mount_current_scope/2` puts it into `Logger.metadata/1`,
so logs from the LiveView process keep the id after the HTTP request ends.
## Writing logs
Normal `Logger` calls need no changes. Atom metadata keys are flattened into the
JSON object:
```elixir
require Logger
Logger.info("tablebase cache miss", table: :tablebase, key: key)
Logger.warning("tablebase fetch failed", status: 503, reason: reason)
```
For a caught exception, pass the exception and stacktrace so the formatter can
build the `error` object:
```elixir
Logger.error("cache backup failed", exception: exception, stacktrace: __STACKTRACE__)
```
For background work, wrap the body so `{:error, reason}` returns, throws, exits
and crashes all produce a log line:
```elixir
Task.Supervisor.start_child(Chesstrainer.Cache.BackupSupervisor, fn ->
Chesstrainer.Background.run("send digest", [user_id: user.id], fn ->
send_digest(user)
end)
end)
```
Prefer identifiers such as `user_id` or `table` over emails, and never log
params or request bodies. The formatter is a safety net, not a licence to log
PII.
## Authentication events
`ChesstrainerWeb.UserSessionController` emits warning lines consumed by
host-level tooling such as fail2ban:
| Message | Metadata | Meaning |
|---|---|---|
| `auth.login.failed` | `reason` (`invalid_credentials` / `unconfirmed`), `ip` | A password login failed. |
| `auth.login.throttled` | `reason` (`ip` / `login`), `ip` | The address or login bucket blocked the attempt before bcrypt. |
`ip` is a boot-scoped HMAC of the client address, not the address itself. The
raw login identifier and password are never included. The login throttle is
configured in `config/config.exs` and defaults to five failures per login and
ten per address in a fifteen-minute window.
## Querying logs
`journalctl` prints the raw line only with `-o cat`; the default output wraps it
in journald fields. Install `jq` once on the VPS with
`sudo apt-get install -y jq`, then:
```sh
# Pretty-print recent lines
sudo journalctl -u chesstrainer -o cat -n 50 --no-pager | jq .
# Errors only
sudo journalctl -u chesstrainer -o cat --since "1 hour ago" \
| jq 'select(.level == "error")'
# Everything from one request id
sudo journalctl -u chesstrainer -o cat \
| jq 'select(.request_id == "REQUEST_ID")'
# Crash reason and stacktrace
sudo journalctl -u chesstrainer -o cat \
| jq -r 'select(.error) | "\(.timestamp) \(.error.kind): \(.error.reason)\n\((.error.stacktrace // []) | join("\n"))"'
# Slow requests
sudo journalctl -u chesstrainer -o cat \
| jq 'select(.duration_ms > 1000) | {timestamp, method, path, status, duration_ms}'
# Most common error kinds in the current boot
sudo journalctl -u chesstrainer -o cat -b \
| jq -r 'select(.error.kind) | .error.kind' | sort | uniq -c | sort -rn
```
`Plug.RequestId` also returns the id in the `x-request-id` response header, so a
user can report the id from a failed request and you can filter straight to it.
## Local checks
The formatter tests run without a production release:
```sh
mix test test/chesstrainer/json_log_formatter_test.exs \
test/chesstrainer/background_test.exs \
test/chesstrainer_web/request_logger_test.exs
```
To render one event without touching the Logger configuration:
```sh
mix run --no-start -e '
event = %{
level: :error,
msg: {:string, "boom"},
meta: %{
time: System.system_time(:microsecond),
request_id: "req-1",
email: "user@example.com",
exception: %RuntimeError{message: "bad"},
stacktrace: [{Foo, :bar, 1, [file: ~c"lib/foo.ex", line: 10]}]
}
}
IO.puts(Chesstrainer.JSONLogFormatter.format(event, %{}))
'
```
That prints one JSON line with the email redacted and the exception unpacked
into `error`.
## Background work audit (#107)
Checked:
- `Chesstrainer.Cache` scheduled backup: `Task.Supervisor.start_child` ignored
the task's return value and `Persist.backup/1` returns `{:error, reason}`
without logging. Now wrapped in `Chesstrainer.Background.run/3`, which logs
`{:error, reason}` returns and crashes (with exception, reason and
stacktrace). A failure to start the task is logged too.
- `Chesstrainer.Cache.Persist.restore/1`: an unreadable backup was silently
skipped. It now logs a warning with the reason and starts with an empty cache.
- `Chesstrainer.Endgames.Tablebase.Parser`: Lichess 5xx and network failures
were returned to the caller (which shows a flash) but never logged. They now
log a warning with the status or reason. 429 cooldowns stay quiet because they
are expected.
- `Mix.Tasks.Tablebase.Fetch`: already aborts loudly with `Mix.raise` on HTTP or
network failure; no change.
- Email delivery: sends are synchronous and their error is returned to the
controller, so nothing is silent today. Future background sends should use
`Chesstrainer.Background.run/3`.
## Reading logs on the VPS
See the "Reading production logs" and "Keep the journal" sections in
[`deploy/README.md`](../deploy/README.md).