Files
owenrees 624ddab644 Admin panel, login throttling, and Cloudflare edge hardening (#150)
* Document the remote shell for production user management

* add sitemap to footer links

* remove border under theme selection

* Add admin theme and user management (#144)

Add /admin/themes CRUD and /admin/users list/detail with role and plan editing, guarded by a shared Policy module. Include last-super-admin and self-role protections plus structured id-only admin logs.

Also add admin invitations, confirmation resends and password-reset emails, which work in development through the Swoosh mailbox before SES is configured.

* Rate-limit password logins and emit auth events (#142)

Add a two-bucket login throttle (5 failures per login, 10 per IP in a 15-minute window) and a shared client-address helper for controllers and LiveViews. Blocked attempts skip the database and bcrypt.

Log auth.login.failed and auth.login.throttled warnings with boot-scoped HMAC addresses only, and stop repeated confirmation emails on unconfirmed login attempts.

* Harden the production login edge for Cloudflare

Document the layered login defence: the app throttle is primary, Caddy must pass Cloudflare's CF-Connecting-IP to the app, and Cloudflare edge rate limiting covers request floods.

Update the reference Caddyfile to accept only Cloudflare ranges and reject direct-to-origin traffic, and document the exact Cloudflare dashboard rules.

* Document 10-second Cloudflare rate-limit window

Cloudflare may only offer a 10-second period. Keep the edge rule at 10 login POSTs per 10 seconds per IP and rely on the app's 15-minute login/address throttle for credential guessing.

* Document Cloudflare limits for LiveView registration

Registration submits over the shared LiveView socket, so a Cloudflare rule on POST /users/register will not count account creations. Note the app-level throttle/Turnstile options instead.

* Document current Cloudflare free-plan edge controls

Record the production setup: Cloudflare-only Caddy origins, Bot Fight Mode, and the 10-second login flood rule (10 POSTs per 10 seconds, 10-second block). Clarify that the application's 15-minute throttle remains the real credential lockout.

* Auto-refresh Cloudflare CIDRs for Caddy

Move the origin allowlist into a generated cloudflare-ranges.caddy imported by Caddy. Add an hourly systemd timer and script that fetch Cloudflare's v4/v6 lists, reject malformed responses, validate the candidate Caddyfile, and reload Caddy only when the ranges change. Restore the previous file on validation or reload failure.

* Document Cloudflare CIDR refresher setup and operations

Record the installed paths, permissions, timer checks, manual run, update procedure, and the direct-on-VPS setup fallback when the deploy directory is absent.
2026-11-11 21:27:59 +01:00

8.0 KiB

Logging

Production writes one JSON object per line to standard output, which systemd captures in journald. Development and test keep the readable [level] message formatter.

Configuration

config/prod.exs swaps the default handler's formatter for Chesstrainer.JSONLogFormatter; nothing else changes per environment. The formatter adds no runtime dependency beyond Jason, which the app already uses for Phoenix JSON.

Line shape

{
  "timestamp": "2026-11-10T21:36:28.489000Z",
  "level": "error",
  "message": "GET /endgames 500 in 12.5ms",
  "request_id": "F7x...",
  "method": "GET",
  "path": "/endgames",
  "status": 500,
  "duration_ms": 12.5,
  "error": {
    "kind": "RuntimeError",
    "reason": "bad thing",
    "stacktrace": ["lib/foo.ex:10: Foo.bar/1"]
  }
}

timestamp, level and message are always present. Metadata keys are flattened into the object; reserved internal keys (time, gl, report_cb, error_logger, exception, stacktrace, crash_reason, reason, kind) are either unpacked into the line or omitted. Fixed fields win over metadata with the same name.

Field When Meaning
timestamp always ISO-8601 UTC, microsecond precision
level always debug, info, notice, warning, error, …
message always Rendered log message, truncated to 8 KiB
request_id request-scoped logs Plug.RequestId value, including LiveView processes
method, path, status, duration_ms access lines One HTTP request; no query string or body
error.kind exceptions/crashes Exception module, or the crash reason's kind
error.reason exceptions/crashes Exception.message/1 or an inspected reason
error.stacktrace exceptions/crashes Array of formatted frames, capped at 50
everything else as supplied Flattened atom metadata, sanitised and redacted

Exceptions (Logger.error(..., exception: ..., stacktrace: ...)) and OTP crash reports (meta.crash_reason, including Bandit request crashes and GenServer terminations) become the error object. Stacktrace frames are an array of strings so jq -r '.error.stacktrace[]' prints them readably.

Redaction

The formatter redacts:

  • metadata keys named email, *_email, email_address, contact_email or recipient;
  • any email-shaped string anywhere in metadata, nested params, or the rendered message;
  • credentials in password, secret, token, authorization, cookie, api_key, private_key and credential keys;
  • token=.../password=... assignments in messages.

Path segments of 20 or more URL-safe characters (reset/confirmation tokens, UUIDs) are replaced with [REDACTED] in access lines. Query parameters and request bodies are never logged.

Access lines

ChesstrainerWeb.RequestLogger listens to [:phoenix, :endpoint, :stop] and emits one :info line per HTTP request that reaches the endpoint, including a LiveView's initial request. Phoenix's own endpoint start/stop pair is disabled with log: false in ChesstrainerWeb.Endpoint so there is exactly one line.

ChesstrainerWeb.Plugs.PutRequestId copies Plug.RequestId's id into the session, and UserAuth.mount_current_scope/2 puts it into Logger.metadata/1, so logs from the LiveView process keep the id after the HTTP request ends.

Writing logs

Normal Logger calls need no changes. Atom metadata keys are flattened into the JSON object:

require Logger

Logger.info("tablebase cache miss", table: :tablebase, key: key)
Logger.warning("tablebase fetch failed", status: 503, reason: reason)

For a caught exception, pass the exception and stacktrace so the formatter can build the error object:

Logger.error("cache backup failed", exception: exception, stacktrace: __STACKTRACE__)

For background work, wrap the body so {:error, reason} returns, throws, exits and crashes all produce a log line:

Task.Supervisor.start_child(Chesstrainer.Cache.BackupSupervisor, fn ->
  Chesstrainer.Background.run("send digest", [user_id: user.id], fn ->
    send_digest(user)
  end)
end)

Prefer identifiers such as user_id or table over emails, and never log params or request bodies. The formatter is a safety net, not a licence to log PII.

Authentication events

ChesstrainerWeb.UserSessionController emits warning lines consumed by host-level tooling such as fail2ban:

Message Metadata Meaning
auth.login.failed reason (invalid_credentials / unconfirmed), ip A password login failed.
auth.login.throttled reason (ip / login), ip The address or login bucket blocked the attempt before bcrypt.

ip is a boot-scoped HMAC of the client address, not the address itself. The raw login identifier and password are never included. The login throttle is configured in config/config.exs and defaults to five failures per login and ten per address in a fifteen-minute window.

Querying logs

journalctl prints the raw line only with -o cat; the default output wraps it in journald fields. Install jq once on the VPS with sudo apt-get install -y jq, then:

# Pretty-print recent lines
sudo journalctl -u chesstrainer -o cat -n 50 --no-pager | jq .

# Errors only
sudo journalctl -u chesstrainer -o cat --since "1 hour ago" \
  | jq 'select(.level == "error")'

# Everything from one request id
sudo journalctl -u chesstrainer -o cat \
  | jq 'select(.request_id == "REQUEST_ID")'

# Crash reason and stacktrace
sudo journalctl -u chesstrainer -o cat \
  | jq -r 'select(.error) | "\(.timestamp) \(.error.kind): \(.error.reason)\n\((.error.stacktrace // []) | join("\n"))"'

# Slow requests
sudo journalctl -u chesstrainer -o cat \
  | jq 'select(.duration_ms > 1000) | {timestamp, method, path, status, duration_ms}'

# Most common error kinds in the current boot
sudo journalctl -u chesstrainer -o cat -b \
  | jq -r 'select(.error.kind) | .error.kind' | sort | uniq -c | sort -rn

Plug.RequestId also returns the id in the x-request-id response header, so a user can report the id from a failed request and you can filter straight to it.

Local checks

The formatter tests run without a production release:

mix test test/chesstrainer/json_log_formatter_test.exs \
  test/chesstrainer/background_test.exs \
  test/chesstrainer_web/request_logger_test.exs

To render one event without touching the Logger configuration:

mix run --no-start -e '
event = %{
  level: :error,
  msg: {:string, "boom"},
  meta: %{
    time: System.system_time(:microsecond),
    request_id: "req-1",
    email: "user@example.com",
    exception: %RuntimeError{message: "bad"},
    stacktrace: [{Foo, :bar, 1, [file: ~c"lib/foo.ex", line: 10]}]
  }
}
IO.puts(Chesstrainer.JSONLogFormatter.format(event, %{}))
'

That prints one JSON line with the email redacted and the exception unpacked into error.

Background work audit (#107)

Checked:

  • Chesstrainer.Cache scheduled backup: Task.Supervisor.start_child ignored the task's return value and Persist.backup/1 returns {:error, reason} without logging. Now wrapped in Chesstrainer.Background.run/3, which logs {:error, reason} returns and crashes (with exception, reason and stacktrace). A failure to start the task is logged too.
  • Chesstrainer.Cache.Persist.restore/1: an unreadable backup was silently skipped. It now logs a warning with the reason and starts with an empty cache.
  • Chesstrainer.Endgames.Tablebase.Parser: Lichess 5xx and network failures were returned to the caller (which shows a flash) but never logged. They now log a warning with the status or reason. 429 cooldowns stay quiet because they are expected.
  • Mix.Tasks.Tablebase.Fetch: already aborts loudly with Mix.raise on HTTP or network failure; no change.
  • Email delivery: sends are synchronous and their error is returned to the controller, so nothing is silent today. Future background sends should use Chesstrainer.Background.run/3.

Reading logs on the VPS

See the "Reading production logs" and "Keep the journal" sections in deploy/README.md.