* Update README.md * Add structured JSON production logging (#105) * Log requests and background failures (#107) * Document production logging and the VPS runbook
7.3 KiB
Logging
Production writes one JSON object per line to standard output, which systemd
captures in journald. Development and test keep the readable
[level] message formatter.
Configuration
config/prod.exs swaps the default handler's formatter for
Chesstrainer.JSONLogFormatter; nothing else changes per environment. The
formatter adds no runtime dependency beyond Jason, which the app already uses
for Phoenix JSON.
Line shape
{
"timestamp": "2026-11-10T21:36:28.489000Z",
"level": "error",
"message": "GET /endgames 500 in 12.5ms",
"request_id": "F7x...",
"method": "GET",
"path": "/endgames",
"status": 500,
"duration_ms": 12.5,
"error": {
"kind": "RuntimeError",
"reason": "bad thing",
"stacktrace": ["lib/foo.ex:10: Foo.bar/1"]
}
}
timestamp, level and message are always present. Metadata keys are
flattened into the object; reserved internal keys (time, gl, report_cb,
error_logger, exception, stacktrace, crash_reason, reason, kind)
are either unpacked into the line or omitted. Fixed fields win over metadata
with the same name.
| Field | When | Meaning |
|---|---|---|
timestamp |
always | ISO-8601 UTC, microsecond precision |
level |
always | debug, info, notice, warning, error, … |
message |
always | Rendered log message, truncated to 8 KiB |
request_id |
request-scoped logs | Plug.RequestId value, including LiveView processes |
method, path, status, duration_ms |
access lines | One HTTP request; no query string or body |
error.kind |
exceptions/crashes | Exception module, or the crash reason's kind |
error.reason |
exceptions/crashes | Exception.message/1 or an inspected reason |
error.stacktrace |
exceptions/crashes | Array of formatted frames, capped at 50 |
| everything else | as supplied | Flattened atom metadata, sanitised and redacted |
Exceptions (Logger.error(..., exception: ..., stacktrace: ...)) and OTP crash
reports (meta.crash_reason, including Bandit request crashes and GenServer
terminations) become the error object. Stacktrace frames are an array of
strings so jq -r '.error.stacktrace[]' prints them readably.
Redaction
The formatter redacts:
- metadata keys named
email,*_email,email_address,contact_emailorrecipient; - any email-shaped string anywhere in metadata, nested
params, or the rendered message; - credentials in
password,secret,token,authorization,cookie,api_key,private_keyandcredentialkeys; token=.../password=...assignments in messages.
Path segments of 20 or more URL-safe characters (reset/confirmation tokens,
UUIDs) are replaced with [REDACTED] in access lines. Query parameters and
request bodies are never logged.
Access lines
ChesstrainerWeb.RequestLogger listens to [:phoenix, :endpoint, :stop] and
emits one :info line per HTTP request that reaches the endpoint, including a
LiveView's initial request. Phoenix's own endpoint start/stop pair is disabled
with log: false in ChesstrainerWeb.Endpoint so there is exactly one line.
ChesstrainerWeb.Plugs.PutRequestId copies Plug.RequestId's id into the
session, and UserAuth.mount_current_scope/2 puts it into Logger.metadata/1,
so logs from the LiveView process keep the id after the HTTP request ends.
Writing logs
Normal Logger calls need no changes. Atom metadata keys are flattened into the
JSON object:
require Logger
Logger.info("tablebase cache miss", table: :tablebase, key: key)
Logger.warning("tablebase fetch failed", status: 503, reason: reason)
For a caught exception, pass the exception and stacktrace so the formatter can
build the error object:
Logger.error("cache backup failed", exception: exception, stacktrace: __STACKTRACE__)
For background work, wrap the body so {:error, reason} returns, throws, exits
and crashes all produce a log line:
Task.Supervisor.start_child(Chesstrainer.Cache.BackupSupervisor, fn ->
Chesstrainer.Background.run("send digest", [user_id: user.id], fn ->
send_digest(user)
end)
end)
Prefer identifiers such as user_id or table over emails, and never log
params or request bodies. The formatter is a safety net, not a licence to log
PII.
Querying logs
journalctl prints the raw line only with -o cat; the default output wraps it
in journald fields. Install jq once on the VPS with
sudo apt-get install -y jq, then:
# Pretty-print recent lines
sudo journalctl -u chesstrainer -o cat -n 50 --no-pager | jq .
# Errors only
sudo journalctl -u chesstrainer -o cat --since "1 hour ago" \
| jq 'select(.level == "error")'
# Everything from one request id
sudo journalctl -u chesstrainer -o cat \
| jq 'select(.request_id == "REQUEST_ID")'
# Crash reason and stacktrace
sudo journalctl -u chesstrainer -o cat \
| jq -r 'select(.error) | "\(.timestamp) \(.error.kind): \(.error.reason)\n\((.error.stacktrace // []) | join("\n"))"'
# Slow requests
sudo journalctl -u chesstrainer -o cat \
| jq 'select(.duration_ms > 1000) | {timestamp, method, path, status, duration_ms}'
# Most common error kinds in the current boot
sudo journalctl -u chesstrainer -o cat -b \
| jq -r 'select(.error.kind) | .error.kind' | sort | uniq -c | sort -rn
Plug.RequestId also returns the id in the x-request-id response header, so a
user can report the id from a failed request and you can filter straight to it.
Local checks
The formatter tests run without a production release:
mix test test/chesstrainer/json_log_formatter_test.exs \
test/chesstrainer/background_test.exs \
test/chesstrainer_web/request_logger_test.exs
To render one event without touching the Logger configuration:
mix run --no-start -e '
event = %{
level: :error,
msg: {:string, "boom"},
meta: %{
time: System.system_time(:microsecond),
request_id: "req-1",
email: "user@example.com",
exception: %RuntimeError{message: "bad"},
stacktrace: [{Foo, :bar, 1, [file: ~c"lib/foo.ex", line: 10]}]
}
}
IO.puts(Chesstrainer.JSONLogFormatter.format(event, %{}))
'
That prints one JSON line with the email redacted and the exception unpacked
into error.
Background work audit (#107)
Checked:
Chesstrainer.Cachescheduled backup:Task.Supervisor.start_childignored the task's return value andPersist.backup/1returns{:error, reason}without logging. Now wrapped inChesstrainer.Background.run/3, which logs{:error, reason}returns and crashes (with exception, reason and stacktrace). A failure to start the task is logged too.Chesstrainer.Cache.Persist.restore/1: an unreadable backup was silently skipped. It now logs a warning with the reason and starts with an empty cache.Chesstrainer.Endgames.Tablebase.Parser: Lichess 5xx and network failures were returned to the caller (which shows a flash) but never logged. They now log a warning with the status or reason. 429 cooldowns stay quiet because they are expected.Mix.Tasks.Tablebase.Fetch: already aborts loudly withMix.raiseon HTTP or network failure; no change.- Email delivery: sends are synchronous and their error is returned to the
controller, so nothing is silent today. Future background sends should use
Chesstrainer.Background.run/3.
Reading logs on the VPS
See the "Reading production logs" and "Keep the journal" sections in
deploy/README.md.