mirror of
https://github.com/TheRealOwenRees/chess-endgame-trainer.git
synced 2026-11-18 11:42:07 +00:00
* Update README.md * Add structured JSON production logging (#105) * Log requests and background failures (#107) * Document production logging and the VPS runbook
This commit is contained in:
+208
@@ -0,0 +1,208 @@
|
||||
# Logging
|
||||
|
||||
Production writes one JSON object per line to standard output, which systemd
|
||||
captures in journald. Development and test keep the readable
|
||||
`[level] message` formatter.
|
||||
|
||||
## Configuration
|
||||
|
||||
`config/prod.exs` swaps the default handler's formatter for
|
||||
`Chesstrainer.JSONLogFormatter`; nothing else changes per environment. The
|
||||
formatter adds no runtime dependency beyond `Jason`, which the app already uses
|
||||
for Phoenix JSON.
|
||||
|
||||
## Line shape
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": "2026-11-10T21:36:28.489000Z",
|
||||
"level": "error",
|
||||
"message": "GET /endgames 500 in 12.5ms",
|
||||
"request_id": "F7x...",
|
||||
"method": "GET",
|
||||
"path": "/endgames",
|
||||
"status": 500,
|
||||
"duration_ms": 12.5,
|
||||
"error": {
|
||||
"kind": "RuntimeError",
|
||||
"reason": "bad thing",
|
||||
"stacktrace": ["lib/foo.ex:10: Foo.bar/1"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`timestamp`, `level` and `message` are always present. Metadata keys are
|
||||
flattened into the object; reserved internal keys (`time`, `gl`, `report_cb`,
|
||||
`error_logger`, `exception`, `stacktrace`, `crash_reason`, `reason`, `kind`)
|
||||
are either unpacked into the line or omitted. Fixed fields win over metadata
|
||||
with the same name.
|
||||
|
||||
| Field | When | Meaning |
|
||||
|---|---|---|
|
||||
| `timestamp` | always | ISO-8601 UTC, microsecond precision |
|
||||
| `level` | always | `debug`, `info`, `notice`, `warning`, `error`, … |
|
||||
| `message` | always | Rendered log message, truncated to 8 KiB |
|
||||
| `request_id` | request-scoped logs | `Plug.RequestId` value, including LiveView processes |
|
||||
| `method`, `path`, `status`, `duration_ms` | access lines | One HTTP request; no query string or body |
|
||||
| `error.kind` | exceptions/crashes | Exception module, or the crash reason's kind |
|
||||
| `error.reason` | exceptions/crashes | `Exception.message/1` or an inspected reason |
|
||||
| `error.stacktrace` | exceptions/crashes | Array of formatted frames, capped at 50 |
|
||||
| everything else | as supplied | Flattened atom metadata, sanitised and redacted |
|
||||
|
||||
Exceptions (`Logger.error(..., exception: ..., stacktrace: ...)`) and OTP crash
|
||||
reports (`meta.crash_reason`, including Bandit request crashes and GenServer
|
||||
terminations) become the `error` object. Stacktrace frames are an array of
|
||||
strings so `jq -r '.error.stacktrace[]'` prints them readably.
|
||||
|
||||
## Redaction
|
||||
|
||||
The formatter redacts:
|
||||
|
||||
- metadata keys named `email`, `*_email`, `email_address`, `contact_email` or
|
||||
`recipient`;
|
||||
- any email-shaped string anywhere in metadata, nested `params`, or the rendered
|
||||
message;
|
||||
- credentials in `password`, `secret`, `token`, `authorization`, `cookie`,
|
||||
`api_key`, `private_key` and `credential` keys;
|
||||
- `token=...`/`password=...` assignments in messages.
|
||||
|
||||
Path segments of 20 or more URL-safe characters (reset/confirmation tokens,
|
||||
UUIDs) are replaced with `[REDACTED]` in access lines. Query parameters and
|
||||
request bodies are never logged.
|
||||
|
||||
## Access lines
|
||||
|
||||
`ChesstrainerWeb.RequestLogger` listens to `[:phoenix, :endpoint, :stop]` and
|
||||
emits one `:info` line per HTTP request that reaches the endpoint, including a
|
||||
LiveView's initial request. Phoenix's own endpoint start/stop pair is disabled
|
||||
with `log: false` in `ChesstrainerWeb.Endpoint` so there is exactly one line.
|
||||
|
||||
`ChesstrainerWeb.Plugs.PutRequestId` copies `Plug.RequestId`'s id into the
|
||||
session, and `UserAuth.mount_current_scope/2` puts it into `Logger.metadata/1`,
|
||||
so logs from the LiveView process keep the id after the HTTP request ends.
|
||||
|
||||
## Writing logs
|
||||
|
||||
Normal `Logger` calls need no changes. Atom metadata keys are flattened into the
|
||||
JSON object:
|
||||
|
||||
```elixir
|
||||
require Logger
|
||||
|
||||
Logger.info("tablebase cache miss", table: :tablebase, key: key)
|
||||
Logger.warning("tablebase fetch failed", status: 503, reason: reason)
|
||||
```
|
||||
|
||||
For a caught exception, pass the exception and stacktrace so the formatter can
|
||||
build the `error` object:
|
||||
|
||||
```elixir
|
||||
Logger.error("cache backup failed", exception: exception, stacktrace: __STACKTRACE__)
|
||||
```
|
||||
|
||||
For background work, wrap the body so `{:error, reason}` returns, throws, exits
|
||||
and crashes all produce a log line:
|
||||
|
||||
```elixir
|
||||
Task.Supervisor.start_child(Chesstrainer.Cache.BackupSupervisor, fn ->
|
||||
Chesstrainer.Background.run("send digest", [user_id: user.id], fn ->
|
||||
send_digest(user)
|
||||
end)
|
||||
end)
|
||||
```
|
||||
|
||||
Prefer identifiers such as `user_id` or `table` over emails, and never log
|
||||
params or request bodies. The formatter is a safety net, not a licence to log
|
||||
PII.
|
||||
|
||||
## Querying logs
|
||||
|
||||
`journalctl` prints the raw line only with `-o cat`; the default output wraps it
|
||||
in journald fields. Install `jq` once on the VPS with
|
||||
`sudo apt-get install -y jq`, then:
|
||||
|
||||
```sh
|
||||
# Pretty-print recent lines
|
||||
sudo journalctl -u chesstrainer -o cat -n 50 --no-pager | jq .
|
||||
|
||||
# Errors only
|
||||
sudo journalctl -u chesstrainer -o cat --since "1 hour ago" \
|
||||
| jq 'select(.level == "error")'
|
||||
|
||||
# Everything from one request id
|
||||
sudo journalctl -u chesstrainer -o cat \
|
||||
| jq 'select(.request_id == "REQUEST_ID")'
|
||||
|
||||
# Crash reason and stacktrace
|
||||
sudo journalctl -u chesstrainer -o cat \
|
||||
| jq -r 'select(.error) | "\(.timestamp) \(.error.kind): \(.error.reason)\n\((.error.stacktrace // []) | join("\n"))"'
|
||||
|
||||
# Slow requests
|
||||
sudo journalctl -u chesstrainer -o cat \
|
||||
| jq 'select(.duration_ms > 1000) | {timestamp, method, path, status, duration_ms}'
|
||||
|
||||
# Most common error kinds in the current boot
|
||||
sudo journalctl -u chesstrainer -o cat -b \
|
||||
| jq -r 'select(.error.kind) | .error.kind' | sort | uniq -c | sort -rn
|
||||
```
|
||||
|
||||
`Plug.RequestId` also returns the id in the `x-request-id` response header, so a
|
||||
user can report the id from a failed request and you can filter straight to it.
|
||||
|
||||
## Local checks
|
||||
|
||||
The formatter tests run without a production release:
|
||||
|
||||
```sh
|
||||
mix test test/chesstrainer/json_log_formatter_test.exs \
|
||||
test/chesstrainer/background_test.exs \
|
||||
test/chesstrainer_web/request_logger_test.exs
|
||||
```
|
||||
|
||||
To render one event without touching the Logger configuration:
|
||||
|
||||
```sh
|
||||
mix run --no-start -e '
|
||||
event = %{
|
||||
level: :error,
|
||||
msg: {:string, "boom"},
|
||||
meta: %{
|
||||
time: System.system_time(:microsecond),
|
||||
request_id: "req-1",
|
||||
email: "user@example.com",
|
||||
exception: %RuntimeError{message: "bad"},
|
||||
stacktrace: [{Foo, :bar, 1, [file: ~c"lib/foo.ex", line: 10]}]
|
||||
}
|
||||
}
|
||||
IO.puts(Chesstrainer.JSONLogFormatter.format(event, %{}))
|
||||
'
|
||||
```
|
||||
|
||||
That prints one JSON line with the email redacted and the exception unpacked
|
||||
into `error`.
|
||||
|
||||
## Background work audit (#107)
|
||||
|
||||
Checked:
|
||||
|
||||
- `Chesstrainer.Cache` scheduled backup: `Task.Supervisor.start_child` ignored
|
||||
the task's return value and `Persist.backup/1` returns `{:error, reason}`
|
||||
without logging. Now wrapped in `Chesstrainer.Background.run/3`, which logs
|
||||
`{:error, reason}` returns and crashes (with exception, reason and
|
||||
stacktrace). A failure to start the task is logged too.
|
||||
- `Chesstrainer.Cache.Persist.restore/1`: an unreadable backup was silently
|
||||
skipped. It now logs a warning with the reason and starts with an empty cache.
|
||||
- `Chesstrainer.Endgames.Tablebase.Parser`: Lichess 5xx and network failures
|
||||
were returned to the caller (which shows a flash) but never logged. They now
|
||||
log a warning with the status or reason. 429 cooldowns stay quiet because they
|
||||
are expected.
|
||||
- `Mix.Tasks.Tablebase.Fetch`: already aborts loudly with `Mix.raise` on HTTP or
|
||||
network failure; no change.
|
||||
- Email delivery: sends are synchronous and their error is returned to the
|
||||
controller, so nothing is silent today. Future background sends should use
|
||||
`Chesstrainer.Background.run/3`.
|
||||
|
||||
## Reading logs on the VPS
|
||||
|
||||
See the "Reading production logs" and "Keep the journal" sections in
|
||||
[`deploy/README.md`](../deploy/README.md).
|
||||
Reference in New Issue
Block a user