* Document the remote shell for production user management * add sitemap to footer links * remove border under theme selection * Add admin theme and user management (#144) Add /admin/themes CRUD and /admin/users list/detail with role and plan editing, guarded by a shared Policy module. Include last-super-admin and self-role protections plus structured id-only admin logs. Also add admin invitations, confirmation resends and password-reset emails, which work in development through the Swoosh mailbox before SES is configured. * Rate-limit password logins and emit auth events (#142) Add a two-bucket login throttle (5 failures per login, 10 per IP in a 15-minute window) and a shared client-address helper for controllers and LiveViews. Blocked attempts skip the database and bcrypt. Log auth.login.failed and auth.login.throttled warnings with boot-scoped HMAC addresses only, and stop repeated confirmation emails on unconfirmed login attempts. * Harden the production login edge for Cloudflare Document the layered login defence: the app throttle is primary, Caddy must pass Cloudflare's CF-Connecting-IP to the app, and Cloudflare edge rate limiting covers request floods. Update the reference Caddyfile to accept only Cloudflare ranges and reject direct-to-origin traffic, and document the exact Cloudflare dashboard rules. * Document 10-second Cloudflare rate-limit window Cloudflare may only offer a 10-second period. Keep the edge rule at 10 login POSTs per 10 seconds per IP and rely on the app's 15-minute login/address throttle for credential guessing. * Document Cloudflare limits for LiveView registration Registration submits over the shared LiveView socket, so a Cloudflare rule on POST /users/register will not count account creations. Note the app-level throttle/Turnstile options instead. * Document current Cloudflare free-plan edge controls Record the production setup: Cloudflare-only Caddy origins, Bot Fight Mode, and the 10-second login flood rule (10 POSTs per 10 seconds, 10-second block). Clarify that the application's 15-minute throttle remains the real credential lockout. * Auto-refresh Cloudflare CIDRs for Caddy Move the origin allowlist into a generated cloudflare-ranges.caddy imported by Caddy. Add an hourly systemd timer and script that fetch Cloudflare's v4/v6 lists, reject malformed responses, validate the candidate Caddyfile, and reload Caddy only when the ranges change. Restore the previous file on validation or reload failure. * Document Cloudflare CIDR refresher setup and operations Record the installed paths, permissions, timer checks, manual run, update procedure, and the direct-on-VPS setup fallback when the deploy directory is absent.
8.0 KiB
Logging
Production writes one JSON object per line to standard output, which systemd
captures in journald. Development and test keep the readable
[level] message formatter.
Configuration
config/prod.exs swaps the default handler's formatter for
Chesstrainer.JSONLogFormatter; nothing else changes per environment. The
formatter adds no runtime dependency beyond Jason, which the app already uses
for Phoenix JSON.
Line shape
{
"timestamp": "2026-11-10T21:36:28.489000Z",
"level": "error",
"message": "GET /endgames 500 in 12.5ms",
"request_id": "F7x...",
"method": "GET",
"path": "/endgames",
"status": 500,
"duration_ms": 12.5,
"error": {
"kind": "RuntimeError",
"reason": "bad thing",
"stacktrace": ["lib/foo.ex:10: Foo.bar/1"]
}
}
timestamp, level and message are always present. Metadata keys are
flattened into the object; reserved internal keys (time, gl, report_cb,
error_logger, exception, stacktrace, crash_reason, reason, kind)
are either unpacked into the line or omitted. Fixed fields win over metadata
with the same name.
| Field | When | Meaning |
|---|---|---|
timestamp |
always | ISO-8601 UTC, microsecond precision |
level |
always | debug, info, notice, warning, error, … |
message |
always | Rendered log message, truncated to 8 KiB |
request_id |
request-scoped logs | Plug.RequestId value, including LiveView processes |
method, path, status, duration_ms |
access lines | One HTTP request; no query string or body |
error.kind |
exceptions/crashes | Exception module, or the crash reason's kind |
error.reason |
exceptions/crashes | Exception.message/1 or an inspected reason |
error.stacktrace |
exceptions/crashes | Array of formatted frames, capped at 50 |
| everything else | as supplied | Flattened atom metadata, sanitised and redacted |
Exceptions (Logger.error(..., exception: ..., stacktrace: ...)) and OTP crash
reports (meta.crash_reason, including Bandit request crashes and GenServer
terminations) become the error object. Stacktrace frames are an array of
strings so jq -r '.error.stacktrace[]' prints them readably.
Redaction
The formatter redacts:
- metadata keys named
email,*_email,email_address,contact_emailorrecipient; - any email-shaped string anywhere in metadata, nested
params, or the rendered message; - credentials in
password,secret,token,authorization,cookie,api_key,private_keyandcredentialkeys; token=.../password=...assignments in messages.
Path segments of 20 or more URL-safe characters (reset/confirmation tokens,
UUIDs) are replaced with [REDACTED] in access lines. Query parameters and
request bodies are never logged.
Access lines
ChesstrainerWeb.RequestLogger listens to [:phoenix, :endpoint, :stop] and
emits one :info line per HTTP request that reaches the endpoint, including a
LiveView's initial request. Phoenix's own endpoint start/stop pair is disabled
with log: false in ChesstrainerWeb.Endpoint so there is exactly one line.
ChesstrainerWeb.Plugs.PutRequestId copies Plug.RequestId's id into the
session, and UserAuth.mount_current_scope/2 puts it into Logger.metadata/1,
so logs from the LiveView process keep the id after the HTTP request ends.
Writing logs
Normal Logger calls need no changes. Atom metadata keys are flattened into the
JSON object:
require Logger
Logger.info("tablebase cache miss", table: :tablebase, key: key)
Logger.warning("tablebase fetch failed", status: 503, reason: reason)
For a caught exception, pass the exception and stacktrace so the formatter can
build the error object:
Logger.error("cache backup failed", exception: exception, stacktrace: __STACKTRACE__)
For background work, wrap the body so {:error, reason} returns, throws, exits
and crashes all produce a log line:
Task.Supervisor.start_child(Chesstrainer.Cache.BackupSupervisor, fn ->
Chesstrainer.Background.run("send digest", [user_id: user.id], fn ->
send_digest(user)
end)
end)
Prefer identifiers such as user_id or table over emails, and never log
params or request bodies. The formatter is a safety net, not a licence to log
PII.
Authentication events
ChesstrainerWeb.UserSessionController emits warning lines consumed by
host-level tooling such as fail2ban:
| Message | Metadata | Meaning |
|---|---|---|
auth.login.failed |
reason (invalid_credentials / unconfirmed), ip |
A password login failed. |
auth.login.throttled |
reason (ip / login), ip |
The address or login bucket blocked the attempt before bcrypt. |
ip is a boot-scoped HMAC of the client address, not the address itself. The
raw login identifier and password are never included. The login throttle is
configured in config/config.exs and defaults to five failures per login and
ten per address in a fifteen-minute window.
Querying logs
journalctl prints the raw line only with -o cat; the default output wraps it
in journald fields. Install jq once on the VPS with
sudo apt-get install -y jq, then:
# Pretty-print recent lines
sudo journalctl -u chesstrainer -o cat -n 50 --no-pager | jq .
# Errors only
sudo journalctl -u chesstrainer -o cat --since "1 hour ago" \
| jq 'select(.level == "error")'
# Everything from one request id
sudo journalctl -u chesstrainer -o cat \
| jq 'select(.request_id == "REQUEST_ID")'
# Crash reason and stacktrace
sudo journalctl -u chesstrainer -o cat \
| jq -r 'select(.error) | "\(.timestamp) \(.error.kind): \(.error.reason)\n\((.error.stacktrace // []) | join("\n"))"'
# Slow requests
sudo journalctl -u chesstrainer -o cat \
| jq 'select(.duration_ms > 1000) | {timestamp, method, path, status, duration_ms}'
# Most common error kinds in the current boot
sudo journalctl -u chesstrainer -o cat -b \
| jq -r 'select(.error.kind) | .error.kind' | sort | uniq -c | sort -rn
Plug.RequestId also returns the id in the x-request-id response header, so a
user can report the id from a failed request and you can filter straight to it.
Local checks
The formatter tests run without a production release:
mix test test/chesstrainer/json_log_formatter_test.exs \
test/chesstrainer/background_test.exs \
test/chesstrainer_web/request_logger_test.exs
To render one event without touching the Logger configuration:
mix run --no-start -e '
event = %{
level: :error,
msg: {:string, "boom"},
meta: %{
time: System.system_time(:microsecond),
request_id: "req-1",
email: "user@example.com",
exception: %RuntimeError{message: "bad"},
stacktrace: [{Foo, :bar, 1, [file: ~c"lib/foo.ex", line: 10]}]
}
}
IO.puts(Chesstrainer.JSONLogFormatter.format(event, %{}))
'
That prints one JSON line with the email redacted and the exception unpacked
into error.
Background work audit (#107)
Checked:
Chesstrainer.Cachescheduled backup:Task.Supervisor.start_childignored the task's return value andPersist.backup/1returns{:error, reason}without logging. Now wrapped inChesstrainer.Background.run/3, which logs{:error, reason}returns and crashes (with exception, reason and stacktrace). A failure to start the task is logged too.Chesstrainer.Cache.Persist.restore/1: an unreadable backup was silently skipped. It now logs a warning with the reason and starts with an empty cache.Chesstrainer.Endgames.Tablebase.Parser: Lichess 5xx and network failures were returned to the caller (which shows a flash) but never logged. They now log a warning with the status or reason. 429 cooldowns stay quiet because they are expected.Mix.Tasks.Tablebase.Fetch: already aborts loudly withMix.raiseon HTTP or network failure; no change.- Email delivery: sends are synchronous and their error is returned to the
controller, so nothing is silent today. Future background sends should use
Chesstrainer.Background.run/3.
Reading logs on the VPS
See the "Reading production logs" and "Keep the journal" sections in
deploy/README.md.