Files
chess-endgame-trainer/docs/logging.md
T
owenrees 202b9b1f6a Add structured production logging (#105, #107) (#141)
* Update README.md

* Add structured JSON production logging (#105)

* Log requests and background failures (#107)

* Document production logging and the VPS runbook
2026-11-10 22:00:05 +01:00

7.3 KiB

Logging

Production writes one JSON object per line to standard output, which systemd captures in journald. Development and test keep the readable [level] message formatter.

Configuration

config/prod.exs swaps the default handler's formatter for Chesstrainer.JSONLogFormatter; nothing else changes per environment. The formatter adds no runtime dependency beyond Jason, which the app already uses for Phoenix JSON.

Line shape

{
  "timestamp": "2026-11-10T21:36:28.489000Z",
  "level": "error",
  "message": "GET /endgames 500 in 12.5ms",
  "request_id": "F7x...",
  "method": "GET",
  "path": "/endgames",
  "status": 500,
  "duration_ms": 12.5,
  "error": {
    "kind": "RuntimeError",
    "reason": "bad thing",
    "stacktrace": ["lib/foo.ex:10: Foo.bar/1"]
  }
}

timestamp, level and message are always present. Metadata keys are flattened into the object; reserved internal keys (time, gl, report_cb, error_logger, exception, stacktrace, crash_reason, reason, kind) are either unpacked into the line or omitted. Fixed fields win over metadata with the same name.

Field When Meaning
timestamp always ISO-8601 UTC, microsecond precision
level always debug, info, notice, warning, error, …
message always Rendered log message, truncated to 8 KiB
request_id request-scoped logs Plug.RequestId value, including LiveView processes
method, path, status, duration_ms access lines One HTTP request; no query string or body
error.kind exceptions/crashes Exception module, or the crash reason's kind
error.reason exceptions/crashes Exception.message/1 or an inspected reason
error.stacktrace exceptions/crashes Array of formatted frames, capped at 50
everything else as supplied Flattened atom metadata, sanitised and redacted

Exceptions (Logger.error(..., exception: ..., stacktrace: ...)) and OTP crash reports (meta.crash_reason, including Bandit request crashes and GenServer terminations) become the error object. Stacktrace frames are an array of strings so jq -r '.error.stacktrace[]' prints them readably.

Redaction

The formatter redacts:

  • metadata keys named email, *_email, email_address, contact_email or recipient;
  • any email-shaped string anywhere in metadata, nested params, or the rendered message;
  • credentials in password, secret, token, authorization, cookie, api_key, private_key and credential keys;
  • token=.../password=... assignments in messages.

Path segments of 20 or more URL-safe characters (reset/confirmation tokens, UUIDs) are replaced with [REDACTED] in access lines. Query parameters and request bodies are never logged.

Access lines

ChesstrainerWeb.RequestLogger listens to [:phoenix, :endpoint, :stop] and emits one :info line per HTTP request that reaches the endpoint, including a LiveView's initial request. Phoenix's own endpoint start/stop pair is disabled with log: false in ChesstrainerWeb.Endpoint so there is exactly one line.

ChesstrainerWeb.Plugs.PutRequestId copies Plug.RequestId's id into the session, and UserAuth.mount_current_scope/2 puts it into Logger.metadata/1, so logs from the LiveView process keep the id after the HTTP request ends.

Writing logs

Normal Logger calls need no changes. Atom metadata keys are flattened into the JSON object:

require Logger

Logger.info("tablebase cache miss", table: :tablebase, key: key)
Logger.warning("tablebase fetch failed", status: 503, reason: reason)

For a caught exception, pass the exception and stacktrace so the formatter can build the error object:

Logger.error("cache backup failed", exception: exception, stacktrace: __STACKTRACE__)

For background work, wrap the body so {:error, reason} returns, throws, exits and crashes all produce a log line:

Task.Supervisor.start_child(Chesstrainer.Cache.BackupSupervisor, fn ->
  Chesstrainer.Background.run("send digest", [user_id: user.id], fn ->
    send_digest(user)
  end)
end)

Prefer identifiers such as user_id or table over emails, and never log params or request bodies. The formatter is a safety net, not a licence to log PII.

Querying logs

journalctl prints the raw line only with -o cat; the default output wraps it in journald fields. Install jq once on the VPS with sudo apt-get install -y jq, then:

# Pretty-print recent lines
sudo journalctl -u chesstrainer -o cat -n 50 --no-pager | jq .

# Errors only
sudo journalctl -u chesstrainer -o cat --since "1 hour ago" \
  | jq 'select(.level == "error")'

# Everything from one request id
sudo journalctl -u chesstrainer -o cat \
  | jq 'select(.request_id == "REQUEST_ID")'

# Crash reason and stacktrace
sudo journalctl -u chesstrainer -o cat \
  | jq -r 'select(.error) | "\(.timestamp) \(.error.kind): \(.error.reason)\n\((.error.stacktrace // []) | join("\n"))"'

# Slow requests
sudo journalctl -u chesstrainer -o cat \
  | jq 'select(.duration_ms > 1000) | {timestamp, method, path, status, duration_ms}'

# Most common error kinds in the current boot
sudo journalctl -u chesstrainer -o cat -b \
  | jq -r 'select(.error.kind) | .error.kind' | sort | uniq -c | sort -rn

Plug.RequestId also returns the id in the x-request-id response header, so a user can report the id from a failed request and you can filter straight to it.

Local checks

The formatter tests run without a production release:

mix test test/chesstrainer/json_log_formatter_test.exs \
  test/chesstrainer/background_test.exs \
  test/chesstrainer_web/request_logger_test.exs

To render one event without touching the Logger configuration:

mix run --no-start -e '
event = %{
  level: :error,
  msg: {:string, "boom"},
  meta: %{
    time: System.system_time(:microsecond),
    request_id: "req-1",
    email: "user@example.com",
    exception: %RuntimeError{message: "bad"},
    stacktrace: [{Foo, :bar, 1, [file: ~c"lib/foo.ex", line: 10]}]
  }
}
IO.puts(Chesstrainer.JSONLogFormatter.format(event, %{}))
'

That prints one JSON line with the email redacted and the exception unpacked into error.

Background work audit (#107)

Checked:

  • Chesstrainer.Cache scheduled backup: Task.Supervisor.start_child ignored the task's return value and Persist.backup/1 returns {:error, reason} without logging. Now wrapped in Chesstrainer.Background.run/3, which logs {:error, reason} returns and crashes (with exception, reason and stacktrace). A failure to start the task is logged too.
  • Chesstrainer.Cache.Persist.restore/1: an unreadable backup was silently skipped. It now logs a warning with the reason and starts with an empty cache.
  • Chesstrainer.Endgames.Tablebase.Parser: Lichess 5xx and network failures were returned to the caller (which shows a flash) but never logged. They now log a warning with the status or reason. 429 cooldowns stay quiet because they are expected.
  • Mix.Tasks.Tablebase.Fetch: already aborts loudly with Mix.raise on HTTP or network failure; no change.
  • Email delivery: sends are synchronous and their error is returned to the controller, so nothing is silent today. Future background sends should use Chesstrainer.Background.run/3.

Reading logs on the VPS

See the "Reading production logs" and "Keep the journal" sections in deploy/README.md.