Add structured production logging (#105, #107) (#141)

* Update README.md

* Add structured JSON production logging (#105)

* Log requests and background failures (#107)

* Document production logging and the VPS runbook
This commit is contained in:
2026-11-10 22:00:05 +01:00
committed by GitHub
parent f8897fa5ae
commit 202b9b1f6a
20 changed files with 1380 additions and 15 deletions
+8
View File
@@ -28,6 +28,14 @@ restart the service; the value is read at boot.
See [docs/tablebases.md](docs/tablebases.md) for the full reference — dev vs
VPS paths, `mix tablebase.check`, and boot logging.
## Logging
Production logs are one JSON object per line so they can be filtered with
`journalctl` and `jq`; development and test keep the readable formatter. See
[docs/logging.md](docs/logging.md) for the line shape, redaction rules and the
background-work audit, and [`deploy/README.md`](deploy/README.md) for the VPS
runbook.
## Learn more
* Official website: https://www.phoenixframework.org/
+5
View File
@@ -33,5 +33,10 @@ config :swoosh, local: false
# Do not print debug messages in production
config :logger, level: :info
# Emit one JSON object per line in production so journald output can be
# filtered and shipped without parsing prose. Dev and test keep the readable
# default formatter configured in config.exs.
config :logger, :default_handler, formatter: {Chesstrainer.JSONLogFormatter, %{}}
# Runtime production configuration, including reading
# of environment variables, is done on config/runtime.exs.
+202 -11
View File
@@ -310,6 +310,121 @@ sudo docker compose -f <UNIFIED_COMPOSE> pull chesstrainer-db
sudo docker compose -f <UNIFIED_COMPOSE> up -d chesstrainer-db
```
## Read-only database access
The app connects as `chesstrainer`, which is a superuser. Use a separate
read-only role for inspection, debugging, and dashboards.
Create it once per environment:
```sh
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
psql -U chesstrainer -d chesstrainer_prod <<'SQL'
CREATE ROLE chesstrainer_ro LOGIN PASSWORD '<generated-password>';
GRANT CONNECT ON DATABASE chesstrainer_prod TO chesstrainer_ro;
GRANT USAGE ON SCHEMA public TO chesstrainer_ro;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO chesstrainer_ro;
-- Cover tables created by future migrations. Migrations run as chesstrainer.
ALTER DEFAULT PRIVILEGES FOR ROLE chesstrainer IN SCHEMA public
GRANT SELECT ON TABLES TO chesstrainer_ro;
-- Optional: read other sessions and server settings in pg_stat_* views.
GRANT pg_read_all_stats TO chesstrainer_ro;
SQL
```
Generate the password with `openssl rand -hex 32` and keep it in a password
manager. The only access it has is `SELECT`; there are no `INSERT`, `UPDATE`,
`DELETE`, or object-creation grants. If the role already exists, skip the
`CREATE ROLE` line and re-run the grants.
Optional defence in depth:
```sql
ALTER ROLE chesstrainer_ro SET default_transaction_read_only = on;
```
This is not a hard boundary (the role can override it per session), but it
turns an accidental write into an error.
### Connect from the VPS
`docker exec` connects over the container's Unix socket, which the Postgres
image trusts locally, so no password is needed:
```sh
sudo docker exec -it chesstrainer-db \
psql -U chesstrainer_ro -d chesstrainer_prod
```
### Connect from a workstation over SSH
Postgres is published on `127.0.0.1:5432` only. OpenSSH also disables
forwarding by default in `/etc/ssh/sshd_config` (`AllowTcpForwarding no`).
Allow local forwarding for the deploy user only, restricted to Postgres, by
appending this to the **end** of that file:
```text
Match User <username>
AllowTcpForwarding local
PermitOpen 127.0.0.1:5432
```
Then validate and reload:
```sh
sudo sshd -t
sudo systemctl reload ssh
sudo sshd -T -C user=<username>,host=localhost,addr=127.0.0.1 | grep -iE 'allowtcpforwarding|permitopen'
```
Open the tunnel (local port `5433` because the dev database already binds
`5432`):
```sh
ssh -N -L 5433:127.0.0.1:5432 <VPS_USER>@<VPS_HOST>
```
Connect from a second terminal:
```sh
psql "postgres://chesstrainer_ro@127.0.0.1:5433/chesstrainer_prod"
```
Install `postgresql-client` if `psql` is not available on the workstation.
To avoid the password prompt, add this to `~/.pgpass` (mode `0600`):
```text
127.0.0.1:5433:chesstrainer_prod:chesstrainer_ro:<password>
```
DBeaver uses the same tunnel. Create a PostgreSQL connection with host
`127.0.0.1`, port `5432`, database `chesstrainer_prod`, user
`chesstrainer_ro`, then on the **SSH** tab enable the tunnel with host
`<VPS_HOST>`, port `22`, user `<VPS_USER>`, and a public key. The
`AllowTcpForwarding local` rule above is required for any tunnelled client.
Reset the role password:
```sh
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
psql -U chesstrainer -d chesstrainer_prod \
-c "ALTER ROLE chesstrainer_ro PASSWORD '<new-password>';"
```
Verify the role is read-only:
```sh
psql "postgres://chesstrainer_ro@127.0.0.1:5433/chesstrainer_prod" \
-c 'select current_user, current_database();'
psql "postgres://chesstrainer_ro@127.0.0.1:5433/chesstrainer_prod" \
-c 'create table should_fail (i int);'
```
The second command must fail with `permission denied for schema public`.
## systemd unit
From the deploy artifact copy:
@@ -438,7 +553,7 @@ sudo -E /opt/chesstrainer/src/deploy/activate /tmp/chesstrainer-release.tar.gz
## GitHub Actions CD
Merging to `main` (or running the workflow manually) tests, builds, uploads,
and deploys the release. The deploy job uses SSH as `owen`.
and deploys the release. The deploy job uses SSH as `<username>`.
### Deploy key
@@ -447,7 +562,7 @@ Generate a dedicated key, not your personal one:
```sh
ssh-keygen -t ed25519 -C "github-actions-chesstrainer" \
-f ~/.ssh/chesstrainer_deploy -N ""
ssh-copy-id -i ~/.ssh/chesstrainer_deploy.pub owen@<VPS_HOST>
ssh-copy-id -i ~/.ssh/chesstrainer_deploy.pub <username>@<VPS_HOST>
```
`VPS_HOST` must be the public IP or DNS name reachable from GitHub. Do not use
@@ -460,7 +575,7 @@ sudo visudo -f /etc/sudoers.d/chesstrainer-deploy
```
```text
owen ALL=(root) NOPASSWD: /opt/chesstrainer/src/deploy/activate, /opt/chesstrainer/src/deploy/rollback
<username> ALL=(root) NOPASSWD: /opt/chesstrainer/src/deploy/activate, /opt/chesstrainer/src/deploy/rollback
```
Validate:
@@ -481,7 +596,7 @@ add environment secrets:
| Secret | Value |
|---|---|
| `VPS_HOST` | public VPS IP or DNS name |
| `VPS_USER` | `owen` |
| `VPS_USER` | `<username>` |
| `VPS_SSH_KEY` | private deploy key |
| `VPS_KNOWN_HOSTS` | `ssh-keyscan -H <VPS_HOST>` output |
@@ -533,24 +648,100 @@ curl -i http://127.0.0.1:4100/health
It returns 200 for `ok`/`degraded` and 503 for `down`. A missing local tablebase
reports `degraded` but still 200 because Lichess fallback is available.
## Backups
### Reading production logs
Take a logical backup of the production database:
Production emits one JSON object per line. Use `-o cat` so journald prints the
raw line instead of its own field wrapper, then pipe to `jq`:
```sh
sudo docker compose -f <UNIFIED_COMPOSE> \
exec -T chesstrainer-db pg_dump -U chesstrainer chesstrainer_prod \
# Recent logs, pretty-printed
sudo journalctl -u chesstrainer -o cat -n 50 --no-pager | jq .
# Errors only
sudo journalctl -u chesstrainer -o cat --since "1 hour ago" | jq 'select(.level == "error")'
# Everything from one request
sudo journalctl -u chesstrainer -o cat | jq 'select(.request_id == "REQUEST_ID")'
# Crash reasons and stacktraces
sudo journalctl -u chesstrainer -o cat | jq -r 'select(.error) | .error.reason, (.error.stacktrace[]?)'
```
Each line has `timestamp`, `level` and `message`, plus `request_id` when one
exists. Access lines add `method`, `path`, `status` and `duration_ms`; crashes
add an `error` object with `kind`, `reason` and `stacktrace`. Query parameters
and request bodies are never logged, token-shaped path segments are redacted,
and email addresses or credentials in metadata render as `[REDACTED]`.
The full field table, redaction rules and examples for adding background logs
are in [`docs/logging.md`](../docs/logging.md).
Dev and test still use the readable `[level] message` formatter.
### Keep the journal
By default some distributions keep journald logs in memory only. On the VPS,
make sure they survive a reboot and have a retention budget:
```sh
sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
journalctl --disk-usage
```
Set `Storage=persistent` and a `SystemMaxUse=` value in
`/etc/systemd/journald.conf` if the distribution does not already. journald
rate-limits bursts; if a crash storm drops lines, raise `RateLimitBurst` or
`RateLimitIntervalSec` there.
## Backups
`docker exec` connects over the container's Unix socket, which the Postgres
image trusts locally, so `pg_dump` needs no password. Run it as the owner
(`chesstrainer`), not `chesstrainer_ro`: a full dump also needs sequence access
and ownership information.
Take a compressed logical backup (custom format, restorable with `pg_restore`):
```sh
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
pg_dump -U chesstrainer -d chesstrainer_prod -Fc \
> /var/backups/chesstrainer-$(date -u +%Y%m%d%H%M%S).dump
```
For a plain SQL file instead (human-readable, restorable with `psql`):
```sh
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
pg_dump -U chesstrainer -d chesstrainer_prod \
| gzip > /var/backups/chesstrainer-$(date -u +%Y%m%d%H%M%S).sql.gz
```
Check that a custom-format dump is readable before trusting it:
```sh
sudo docker exec -i chesstrainer-db pg_restore -l \
< /var/backups/<timestamp>.dump > /dev/null
```
Restore into a scratch database (do not restore over the live database without
stopping `chesstrainer` first):
```sh
sudo docker compose -f <UNIFIED_COMPOSE> exec -T chesstrainer-db \
createdb -U chesstrainer chesstrainer_restore
sudo docker exec -i chesstrainer-db pg_restore -U chesstrainer \
-d chesstrainer_restore < /var/backups/<timestamp>.dump
```
Example nightly entry in root's crontab (`sudo crontab -e`):
```cron
15 3 * * * docker compose -f /home/owen/docker-compose.yml exec -T chesstrainer-db pg_dump -U chesstrainer chesstrainer_prod | gzip > /var/backups/chesstrainer-$(date +\%Y\%m\%d).sql.gz
15 3 * * * docker compose -f /home/<username>/docker-compose.yml exec -T chesstrainer-db pg_dump -U chesstrainer -d chesstrainer_prod -Fc > /var/backups/chesstrainer-$(date +\%Y\%m\%d).dump
```
Keep `/var/backups` outside `/opt/chesstrainer/releases` and copy backups off
the VPS regularly.
the VPS regularly. A backup that is only on the VPS is not a backup.
## New VPS checklist
@@ -564,4 +755,4 @@ the VPS regularly.
7. `/health` and `/` verified on `127.0.0.1:4100`.
8. Caddy block added, validated, reloaded.
9. Public `/`, `/privacy`, `/terms-conditions`, and `/health` verified.
10. Backups scheduled and tested.
10. Read-only role created, tunnel verified, backups scheduled and tested.
+208
View File
@@ -0,0 +1,208 @@
# Logging
Production writes one JSON object per line to standard output, which systemd
captures in journald. Development and test keep the readable
`[level] message` formatter.
## Configuration
`config/prod.exs` swaps the default handler's formatter for
`Chesstrainer.JSONLogFormatter`; nothing else changes per environment. The
formatter adds no runtime dependency beyond `Jason`, which the app already uses
for Phoenix JSON.
## Line shape
```json
{
"timestamp": "2026-11-10T21:36:28.489000Z",
"level": "error",
"message": "GET /endgames 500 in 12.5ms",
"request_id": "F7x...",
"method": "GET",
"path": "/endgames",
"status": 500,
"duration_ms": 12.5,
"error": {
"kind": "RuntimeError",
"reason": "bad thing",
"stacktrace": ["lib/foo.ex:10: Foo.bar/1"]
}
}
```
`timestamp`, `level` and `message` are always present. Metadata keys are
flattened into the object; reserved internal keys (`time`, `gl`, `report_cb`,
`error_logger`, `exception`, `stacktrace`, `crash_reason`, `reason`, `kind`)
are either unpacked into the line or omitted. Fixed fields win over metadata
with the same name.
| Field | When | Meaning |
|---|---|---|
| `timestamp` | always | ISO-8601 UTC, microsecond precision |
| `level` | always | `debug`, `info`, `notice`, `warning`, `error`, … |
| `message` | always | Rendered log message, truncated to 8 KiB |
| `request_id` | request-scoped logs | `Plug.RequestId` value, including LiveView processes |
| `method`, `path`, `status`, `duration_ms` | access lines | One HTTP request; no query string or body |
| `error.kind` | exceptions/crashes | Exception module, or the crash reason's kind |
| `error.reason` | exceptions/crashes | `Exception.message/1` or an inspected reason |
| `error.stacktrace` | exceptions/crashes | Array of formatted frames, capped at 50 |
| everything else | as supplied | Flattened atom metadata, sanitised and redacted |
Exceptions (`Logger.error(..., exception: ..., stacktrace: ...)`) and OTP crash
reports (`meta.crash_reason`, including Bandit request crashes and GenServer
terminations) become the `error` object. Stacktrace frames are an array of
strings so `jq -r '.error.stacktrace[]'` prints them readably.
## Redaction
The formatter redacts:
- metadata keys named `email`, `*_email`, `email_address`, `contact_email` or
`recipient`;
- any email-shaped string anywhere in metadata, nested `params`, or the rendered
message;
- credentials in `password`, `secret`, `token`, `authorization`, `cookie`,
`api_key`, `private_key` and `credential` keys;
- `token=...`/`password=...` assignments in messages.
Path segments of 20 or more URL-safe characters (reset/confirmation tokens,
UUIDs) are replaced with `[REDACTED]` in access lines. Query parameters and
request bodies are never logged.
## Access lines
`ChesstrainerWeb.RequestLogger` listens to `[:phoenix, :endpoint, :stop]` and
emits one `:info` line per HTTP request that reaches the endpoint, including a
LiveView's initial request. Phoenix's own endpoint start/stop pair is disabled
with `log: false` in `ChesstrainerWeb.Endpoint` so there is exactly one line.
`ChesstrainerWeb.Plugs.PutRequestId` copies `Plug.RequestId`'s id into the
session, and `UserAuth.mount_current_scope/2` puts it into `Logger.metadata/1`,
so logs from the LiveView process keep the id after the HTTP request ends.
## Writing logs
Normal `Logger` calls need no changes. Atom metadata keys are flattened into the
JSON object:
```elixir
require Logger
Logger.info("tablebase cache miss", table: :tablebase, key: key)
Logger.warning("tablebase fetch failed", status: 503, reason: reason)
```
For a caught exception, pass the exception and stacktrace so the formatter can
build the `error` object:
```elixir
Logger.error("cache backup failed", exception: exception, stacktrace: __STACKTRACE__)
```
For background work, wrap the body so `{:error, reason}` returns, throws, exits
and crashes all produce a log line:
```elixir
Task.Supervisor.start_child(Chesstrainer.Cache.BackupSupervisor, fn ->
Chesstrainer.Background.run("send digest", [user_id: user.id], fn ->
send_digest(user)
end)
end)
```
Prefer identifiers such as `user_id` or `table` over emails, and never log
params or request bodies. The formatter is a safety net, not a licence to log
PII.
## Querying logs
`journalctl` prints the raw line only with `-o cat`; the default output wraps it
in journald fields. Install `jq` once on the VPS with
`sudo apt-get install -y jq`, then:
```sh
# Pretty-print recent lines
sudo journalctl -u chesstrainer -o cat -n 50 --no-pager | jq .
# Errors only
sudo journalctl -u chesstrainer -o cat --since "1 hour ago" \
| jq 'select(.level == "error")'
# Everything from one request id
sudo journalctl -u chesstrainer -o cat \
| jq 'select(.request_id == "REQUEST_ID")'
# Crash reason and stacktrace
sudo journalctl -u chesstrainer -o cat \
| jq -r 'select(.error) | "\(.timestamp) \(.error.kind): \(.error.reason)\n\((.error.stacktrace // []) | join("\n"))"'
# Slow requests
sudo journalctl -u chesstrainer -o cat \
| jq 'select(.duration_ms > 1000) | {timestamp, method, path, status, duration_ms}'
# Most common error kinds in the current boot
sudo journalctl -u chesstrainer -o cat -b \
| jq -r 'select(.error.kind) | .error.kind' | sort | uniq -c | sort -rn
```
`Plug.RequestId` also returns the id in the `x-request-id` response header, so a
user can report the id from a failed request and you can filter straight to it.
## Local checks
The formatter tests run without a production release:
```sh
mix test test/chesstrainer/json_log_formatter_test.exs \
test/chesstrainer/background_test.exs \
test/chesstrainer_web/request_logger_test.exs
```
To render one event without touching the Logger configuration:
```sh
mix run --no-start -e '
event = %{
level: :error,
msg: {:string, "boom"},
meta: %{
time: System.system_time(:microsecond),
request_id: "req-1",
email: "user@example.com",
exception: %RuntimeError{message: "bad"},
stacktrace: [{Foo, :bar, 1, [file: ~c"lib/foo.ex", line: 10]}]
}
}
IO.puts(Chesstrainer.JSONLogFormatter.format(event, %{}))
'
```
That prints one JSON line with the email redacted and the exception unpacked
into `error`.
## Background work audit (#107)
Checked:
- `Chesstrainer.Cache` scheduled backup: `Task.Supervisor.start_child` ignored
the task's return value and `Persist.backup/1` returns `{:error, reason}`
without logging. Now wrapped in `Chesstrainer.Background.run/3`, which logs
`{:error, reason}` returns and crashes (with exception, reason and
stacktrace). A failure to start the task is logged too.
- `Chesstrainer.Cache.Persist.restore/1`: an unreadable backup was silently
skipped. It now logs a warning with the reason and starts with an empty cache.
- `Chesstrainer.Endgames.Tablebase.Parser`: Lichess 5xx and network failures
were returned to the caller (which shows a flash) but never logged. They now
log a warning with the status or reason. 429 cooldowns stay quiet because they
are expected.
- `Mix.Tasks.Tablebase.Fetch`: already aborts loudly with `Mix.raise` on HTTP or
network failure; no change.
- Email delivery: sends are synchronous and their error is returned to the
controller, so nothing is silent today. Future background sends should use
`Chesstrainer.Background.run/3`.
## Reading logs on the VPS
See the "Reading production logs" and "Keep the journal" sections in
[`deploy/README.md`](../deploy/README.md).
+63
View File
@@ -0,0 +1,63 @@
defmodule Chesstrainer.Background do
@moduledoc """
Runs background work and logs any failure.
Background failures otherwise disappear: a task exits, the supervisor keeps
the rest of the system running, and nobody sees why the work did not happen.
This wrapper turns crashes, throws, exits and `{:error, reason}` returns into
one structured error line with the reason and stacktrace.
The label is a short description of the work; metadata is added to the log
line unchanged (minus the formatter's own redaction).
"""
require Logger
@type result :: term()
@doc """
Runs `fun` and logs a failure. Returns the function's value, or `:error` when
it crashed or returned an error tuple.
"""
@spec run(String.t(), (-> result)) :: result() | :error
def run(label, fun), do: run(label, [], fun)
@doc """
Like `run/2`, with extra metadata for the error line.
"""
@spec run(String.t(), keyword(), (-> result)) :: result() | :error
def run(label, metadata, fun)
when is_binary(label) and is_list(metadata) and is_function(fun, 0) do
case fun.() do
{:error, reason} ->
Logger.error(
"#{label} failed: #{format_term(reason)}",
Keyword.merge(metadata, reason: reason)
)
:error
result ->
result
end
rescue
exception ->
Logger.error(
"#{label} crashed: #{Exception.message(exception)}",
Keyword.merge(metadata, exception: exception, stacktrace: __STACKTRACE__)
)
:error
catch
kind, reason ->
Logger.error(
"#{label} crashed: #{format_term(reason)}",
Keyword.merge(metadata, kind: kind, reason: reason, stacktrace: __STACKTRACE__)
)
:error
end
defp format_term(term) when is_binary(term), do: term
defp format_term(term), do: inspect(term, limit: 10, printable_limit: 500)
end
+16 -2
View File
@@ -11,7 +11,8 @@ defmodule Chesstrainer.Cache do
that were used heavily once but have not been used since eventually age out.
A daily backup runs off the cache process at a configurable UTC time, so reads
and writes stay responsive while the snapshot is written.
and writes stay responsive while the snapshot is written. Failures are logged
rather than lost.
## Examples
@@ -19,7 +20,9 @@ defmodule Chesstrainer.Cache do
"""
use GenServer
require Logger
alias Chesstrainer.Background
alias Chesstrainer.Cache.BackupSchedule
alias Chesstrainer.Cache.Persist
@@ -121,7 +124,18 @@ defmodule Chesstrainer.Cache do
table = state.table
backup_fun = state.backup_fun
Task.Supervisor.start_child(Chesstrainer.Cache.BackupSupervisor, fn -> backup_fun.(table) end)
task = fn ->
Background.run("cache backup", [table: table], fn -> backup_fun.(table) end)
end
case Task.Supervisor.start_child(Chesstrainer.Cache.BackupSupervisor, task) do
{:ok, _pid} ->
:ok
{:error, reason} ->
Logger.error("cache backup could not start", table: table, reason: reason)
:ok
end
end
defp schedule_backup(state) do
+7 -1
View File
@@ -5,8 +5,13 @@ defmodule Chesstrainer.Cache.Persist do
A backup is written to a temporary DETS file in the same directory and then
atomically renamed into place. A crash during the write therefore leaves the
previous backup untouched instead of a truncated file that fails to restore.
A backup that cannot be opened at boot is logged and skipped, so a damaged
file does not stop the application from starting with an empty cache.
"""
require Logger
@backup_path "priv/static/backups"
@format_version "v2"
@@ -54,7 +59,8 @@ defmodule Chesstrainer.Cache.Persist do
:dets.close(table)
{:error, _} ->
{:error, reason} ->
Logger.warning("cache backup restore failed", table: table, reason: reason)
:ok
end
end
@@ -11,6 +11,8 @@ defmodule Chesstrainer.Endgames.Tablebase.Parser do
import Chesstrainer.Ratelimiters
alias Chesstrainer.Endgames.Tablebase
require Logger
@lichess_tablebase_url "https://tablebase.lichess.ovh/standard?fen="
@spec from_fen(String.t()) ::
@@ -74,6 +76,7 @@ defmodule Chesstrainer.Endgames.Tablebase.Parser do
# Server errors
{:ok, %Req.Response{status: status, body: body}} when status in 500..599 ->
Logger.warning("tablebase fetch failed with HTTP #{status}", status: status)
{:error, {:server_error, status, body}}
# Catch‑all for other status codes
@@ -82,6 +85,7 @@ defmodule Chesstrainer.Endgames.Tablebase.Parser do
# Network / client‑side failure
{:error, reason} ->
Logger.warning("tablebase fetch failed: #{inspect(reason)}", reason: reason)
{:error, reason}
end
end
+307
View File
@@ -0,0 +1,307 @@
defmodule Chesstrainer.JSONLogFormatter do
@moduledoc """
An OTP logger formatter that emits one JSON object per log line.
Production uses this formatter (see `config/prod.exs`), while development and
test keep `Logger.Formatter`'s readable output. Every line carries a
timestamp, level and message, plus a request id when one is in the logger
metadata. Access lines add `method`, `path`, `status` and `duration_ms`;
exceptions and OTP crash reports add an `error` object with `kind`, `reason`
and `stacktrace`.
The formatter is defensive. Metadata is sanitised into JSON-safe values, and
an event that cannot be encoded still produces a single JSON line instead of
taking the logger down. Email addresses and credentials found in metadata,
params or the rendered message are replaced with `[REDACTED]`.
"""
@max_message_bytes 8_096
@max_stacktrace_entries 50
@redacted "[REDACTED]"
@email_regex ~r/[A-Za-z0-9.!#$%&'*+\/=?^_`{|}~-]+@[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?(?:\.[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?)+/
@secret_assignment_regex ~r/\b(token|password|secret|api[_-]?key|authorization)=([^&\s"']+)/i
@sensitive_key_fragments ~w(password secret token authorization cookie api_key private_key credential)
@sensitive_email_keys ~w(email email_address user_email contact_email recipient)
# These are unpacked into the line itself rather than copied as metadata.
@internal_keys [
:time,
:gl,
:report_cb,
:error_logger,
:exception,
:stacktrace,
:crash_reason,
:reason,
:kind,
:conn,
:socket
]
# Fixed fields always win over a metadata key with the same name.
@reserved_keys [
:timestamp,
:level,
:message,
:error,
:request_id,
:method,
:path,
:status,
:duration_ms
]
@doc """
Formats a `:logger` event as a newline-terminated JSON object.
"""
@spec format(:logger.log_event(), :logger.formatter_config()) :: iodata()
def format(event, _config) do
event
|> build()
|> Jason.encode!()
|> then(&[&1, ?\n])
rescue
exception -> fallback(exception)
catch
kind, reason -> fallback({kind, reason})
end
defp build(event) do
meta = event.meta || %{}
%{
"timestamp" => timestamp(meta),
"level" => to_string(event.level),
"message" => message(event)
}
|> put_access(meta)
|> put_metadata(meta)
|> put_error(meta)
end
defp timestamp(%{time: time}) when is_integer(time) do
time
|> DateTime.from_unix!(:microsecond)
|> DateTime.to_iso8601()
end
defp timestamp(_meta), do: DateTime.utc_now() |> DateTime.to_iso8601()
defp message(event) do
event
|> Logger.Formatter.format_event(@max_message_bytes)
|> Logger.Formatter.prune()
|> IO.iodata_to_binary()
|> redact_text()
rescue
_exception ->
event.msg
|> inspect(limit: 20, printable_limit: @max_message_bytes)
|> redact_text()
catch
_kind, reason ->
{event.msg, reason}
|> inspect(limit: 20, printable_limit: @max_message_bytes)
|> redact_text()
end
defp put_access(entry, meta) do
entry
|> put_if_present("request_id", meta[:request_id])
|> put_if_present("method", meta[:method])
|> put_if_present("path", meta[:path])
|> put_if_present("status", meta[:status])
|> put_if_present("duration_ms", meta[:duration_ms])
end
defp put_if_present(entry, _key, nil), do: entry
defp put_if_present(entry, key, value) do
Map.put(entry, key, sanitize(value, key))
end
defp put_metadata(entry, meta) do
Enum.reduce(meta, entry, fn
{key, _value}, acc when key in @internal_keys ->
acc
{key, _value}, acc when key in @reserved_keys ->
acc
{key, value}, acc ->
Map.put(acc, key |> key_string() |> redact_text(), metadata_value(key, value))
end)
end
defp metadata_value(:mfa, {module, function, arity})
when is_atom(module) and is_atom(function) and is_integer(arity) do
Exception.format_mfa(module, function, arity)
end
defp metadata_value(key, value), do: sanitize(value, key)
# Only the first exception/reason source is rendered. OTP crash reports carry
# `crash_reason`, while Logger.error(..., exception: ..., stacktrace: ...)
# carries the two keys directly.
defp put_error(entry, meta) do
case error(meta) do
nil -> entry
error when map_size(error) == 0 -> entry
error -> Map.put(entry, "error", error)
end
end
defp error(%{exception: %{__exception__: true} = exception} = meta) do
%{
"kind" => inspect(exception.__struct__),
"reason" => Exception.message(exception),
"stacktrace" => stacktrace(meta[:stacktrace])
}
|> drop_nil()
end
defp error(%{crash_reason: crash_reason} = meta) do
{reason, trace} = split_crash_reason(crash_reason, meta[:stacktrace])
%{
"kind" => error_kind(reason),
"reason" => error_reason(reason),
"stacktrace" => stacktrace(trace)
}
|> drop_nil()
end
defp error(meta) do
if meta[:reason] || meta[:stacktrace] do
%{
"kind" => meta[:kind] && to_string(meta[:kind]),
"reason" => error_reason(meta[:reason]),
"stacktrace" => stacktrace(meta[:stacktrace])
}
|> drop_nil()
end
end
defp split_crash_reason({reason, trace}, _fallback) when is_list(trace), do: {reason, trace}
defp split_crash_reason(reason, fallback), do: {reason, fallback}
defp error_kind(%{__exception__: true} = exception), do: inspect(exception.__struct__)
defp error_kind(reason) when is_atom(reason), do: Atom.to_string(reason)
defp error_kind(reason) when is_tuple(reason), do: reason |> elem(0) |> error_kind()
defp error_kind(_reason), do: nil
defp error_reason(nil), do: nil
defp error_reason(%{__exception__: true} = exception), do: Exception.message(exception)
defp error_reason(reason) when is_binary(reason), do: redact_text(reason)
defp error_reason(reason) do
reason
|> inspect(limit: 30, printable_limit: 2_000)
|> redact_text()
end
defp stacktrace(stacktrace) when is_list(stacktrace) and stacktrace != [] do
stacktrace
|> Enum.take(@max_stacktrace_entries)
|> Enum.map(&format_stacktrace_entry/1)
|> Enum.map(&redact_text/1)
end
defp stacktrace(_stacktrace), do: nil
defp format_stacktrace_entry(entry) do
Exception.format_stacktrace_entry(entry)
rescue
_exception -> inspect(entry, limit: 10, printable_limit: 1_000)
end
defp drop_nil(map), do: Map.reject(map, fn {_key, value} -> is_nil(value) end)
defp key_string(key) when is_atom(key), do: Atom.to_string(key)
defp key_string(key) when is_binary(key), do: key
defp key_string(key), do: inspect(key)
defp sanitize(nil, _key), do: nil
defp sanitize(value, _key) when is_boolean(value), do: value
defp sanitize(value, _key) when is_integer(value) or is_float(value), do: value
defp sanitize(value, key) when is_binary(value) do
cond do
sensitive_key?(key) -> @redacted
String.valid?(value) -> redact_text(value)
true -> inspect(value)
end
end
defp sanitize(value, _key) when is_atom(value), do: Atom.to_string(value)
defp sanitize(value, _key) when is_struct(value) do
value
|> inspect(limit: 20, printable_limit: 2_000)
|> redact_text()
end
defp sanitize(value, key) when is_list(value) do
cond do
sensitive_key?(key) -> @redacted
value != [] and List.ascii_printable?(value) -> value |> List.to_string() |> redact_text()
true -> Enum.map(value, &sanitize(&1, nil))
end
end
defp sanitize(value, _key) when is_tuple(value) do
value
|> Tuple.to_list()
|> Enum.map(&sanitize(&1, nil))
end
defp sanitize(value, _key) when is_map(value) do
Enum.into(value, %{}, fn {key, nested} ->
{key |> key_string() |> redact_text(), sanitize(nested, key)}
end)
end
defp sanitize(value, _key) do
value
|> inspect(limit: 20, printable_limit: 2_000)
|> redact_text()
end
defp sensitive_key?(nil), do: false
defp sensitive_key?(key) do
key = key |> key_string() |> String.downcase()
key in @sensitive_email_keys or String.ends_with?(key, "_email") or
Enum.any?(@sensitive_key_fragments, &String.contains?(key, &1))
end
defp redact_text(text) when is_binary(text) do
text = if String.valid?(text), do: text, else: inspect(text)
text
|> then(&Regex.replace(@email_regex, &1, @redacted))
|> then(&Regex.replace(@secret_assignment_regex, &1, "\\1=#{@redacted}"))
end
defp fallback(reason) do
entry = %{
"timestamp" => DateTime.utc_now() |> DateTime.to_iso8601(),
"level" => "error",
"message" => "could not format log event",
"error" => %{
"reason" => reason |> inspect(limit: 10, printable_limit: 1_000) |> redact_text()
}
}
json =
case Jason.encode(entry) do
{:ok, json} -> json
{:error, _reason} -> ~s({"level":"error","message":"could not format log event"})
end
[json, ?\n]
end
end
+4 -1
View File
@@ -41,7 +41,10 @@ defmodule ChesstrainerWeb.Endpoint do
cookie_key: "request_logger"
plug Plug.RequestId
plug Plug.Telemetry, event_prefix: [:phoenix, :endpoint]
# Access lines come from ChesstrainerWeb.RequestLogger (one structured line
# per request); disable Phoenix.Logger's endpoint start/stop pair.
plug Plug.Telemetry, event_prefix: [:phoenix, :endpoint], log: false
plug Plug.Parsers,
parsers: [:urlencoded, :multipart, :json],
@@ -0,0 +1,25 @@
defmodule ChesstrainerWeb.Plugs.PutRequestId do
@moduledoc """
Copies `Plug.RequestId`'s request id into the session.
LiveView processes handle later events after the original HTTP request has
finished. Keeping the id in the session lets `UserAuth.mount_current_scope/2`
put it into `Logger.metadata/1`, so LiveView logs remain correlatable with the
request that mounted them.
"""
@behaviour Plug
import Plug.Conn
@impl true
def init(opts), do: opts
@impl true
def call(conn, _opts) do
case conn.assigns[:request_id] do
nil -> conn
request_id -> put_session(conn, :request_id, request_id)
end
end
end
+66
View File
@@ -0,0 +1,66 @@
defmodule ChesstrainerWeb.RequestLogger do
@moduledoc """
Emits one structured access line per HTTP request.
The line carries the method, path, status, duration in milliseconds and the
request id when one is available. Query parameters and bodies are never
logged. Path segments that look like tokens or identifiers (20 or more
URL-safe characters) are replaced with `[REDACTED]`.
`Phoenix.Logger`'s own endpoint start/stop lines are disabled in
`ChesstrainerWeb.Endpoint` so each request produces exactly one line.
"""
require Logger
@event [:phoenix, :endpoint, :stop]
@handler_id {__MODULE__, :access_log}
@token_segment ~r/\A[A-Za-z0-9_.-]{20,}\z/
@doc false
def install do
case :telemetry.attach(@handler_id, @event, &__MODULE__.handle_event/4, :ok) do
:ok -> :ok
{:error, :already_exists} -> :ok
end
end
@doc false
def handle_event(_event, measurements, metadata, _config) do
conn = metadata.conn
duration_ms = duration_ms(measurements.duration)
path = redact_path(conn.request_path)
status = conn.status || 0
metadata =
[
method: conn.method,
path: path,
status: status,
duration_ms: duration_ms
]
|> put_request_id(conn.assigns[:request_id])
Logger.info("#{conn.method} #{path} #{status} in #{duration_ms}ms", metadata)
end
defp duration_ms(native) do
native
|> System.convert_time_unit(:native, :microsecond)
|> Kernel./(1000)
|> Float.round(1)
end
defp put_request_id(metadata, nil), do: metadata
defp put_request_id(metadata, request_id), do: Keyword.put(metadata, :request_id, request_id)
defp redact_path(path) do
path
|> String.split("/")
|> Enum.map_join("/", &redact_segment/1)
end
defp redact_segment(segment) do
if Regex.match?(@token_segment, segment), do: "[REDACTED]", else: segment
end
end
+1
View File
@@ -6,6 +6,7 @@ defmodule ChesstrainerWeb.Router do
pipeline :browser do
plug :accepts, ["html"]
plug :fetch_session
plug ChesstrainerWeb.Plugs.PutRequestId
plug :fetch_live_flash
plug :put_root_layout, html: {ChesstrainerWeb.Layouts, :root}
plug :protect_from_forgery
+2
View File
@@ -8,6 +8,8 @@ defmodule ChesstrainerWeb.Telemetry do
@impl true
def init(_arg) do
ChesstrainerWeb.RequestLogger.install()
children = [
# Telemetry poller will execute the given period measurements
# every 10_000ms. Learn more here: https://telemetry-metrics.hexdocs.pm
+4
View File
@@ -290,6 +290,10 @@ defmodule ChesstrainerWeb.UserAuth do
end
defp mount_current_scope(socket, session) do
if request_id = session["request_id"] do
Logger.metadata(request_id: request_id)
end
Phoenix.Component.assign_new(socket, :current_scope, fn ->
{user, _} =
if user_token = session["user_token"] do
+115
View File
@@ -0,0 +1,115 @@
defmodule Chesstrainer.BackgroundTest do
use ExUnit.Case, async: false
import ExUnit.CaptureLog
alias Chesstrainer.Background
alias Chesstrainer.JSONLogFormatter
setup do
handler_id = :"background_test_capture_#{System.unique_integer([:positive])}"
:ok =
:logger.add_handler(handler_id, :logger_std_h, %{
formatter: {Chesstrainer.TestLogHandler, %{pid: self()}},
level: :all
})
on_exit(fn -> :logger.remove_handler(handler_id) end)
:ok
end
describe "run/3" do
test "returns the function's value on success" do
assert :ok = Background.run("test task", fn -> :ok end)
assert {:ok, 42} = Background.run("test task", fn -> {:ok, 42} end)
end
test "logs an error tuple with its reason" do
capture_log(fn ->
assert :error =
Background.run("test task", [table: :demo], fn -> {:error, :disk_full} end)
end)
assert_receive {:log_event, event}
assert event.level == :error
assert event.meta[:table] == :demo
assert event.meta[:reason] == :disk_full
assert message(event) == "test task failed: :disk_full"
end
test "logs a crash with the exception, reason and stacktrace" do
capture_log(fn ->
assert :error = Background.run("test task", fn -> raise "boom" end)
end)
assert_receive {:log_event, event}
assert event.level == :error
assert %RuntimeError{message: "boom"} = event.meta[:exception]
assert is_list(event.meta[:stacktrace])
assert event.meta[:stacktrace] != []
assert message(event) =~ "test task crashed: boom"
decoded = event |> JSONLogFormatter.format(%{}) |> IO.iodata_to_binary() |> Jason.decode!()
assert decoded["error"]["kind"] == "RuntimeError"
assert decoded["error"]["reason"] == "boom"
assert decoded["error"]["stacktrace"] |> is_list()
assert decoded["error"]["stacktrace"] != []
end
test "logs a throw with its reason and stacktrace" do
capture_log(fn ->
assert :error = Background.run("test task", fn -> throw(:nope) end)
end)
assert_receive {:log_event, event}
assert event.level == :error
assert event.meta[:kind] == :throw
assert event.meta[:reason] == :nope
assert message(event) == "test task crashed: :nope"
end
end
describe "Chesstrainer.Cache scheduled backups" do
test "log a failed backup" do
table = :"test_failed_backup_log_#{System.unique_integer([:positive])}"
parent = self()
backup_fun = fn _table ->
send(parent, {:backup_started, self()})
{:error, :disk_full}
end
log =
capture_log([level: :info], fn ->
pid =
start_supervised!(
Supervisor.child_spec(
{Chesstrainer.Cache, {table, [backup_fun: backup_fun]}},
id: table
)
)
send(pid, :backup)
assert_receive {:backup_started, task_pid}, 1_000
ref = Process.monitor(task_pid)
assert_receive {:DOWN, ^ref, :process, ^task_pid, :normal}, 1_000
end)
assert log =~ "cache backup failed"
assert log =~ ":disk_full"
end
end
defp message(event) do
event
|> Logger.Formatter.format_event(8_096)
|> IO.iodata_to_binary()
end
end
@@ -0,0 +1,190 @@
defmodule Chesstrainer.JSONLogFormatterTest do
use ExUnit.Case, async: true
alias Chesstrainer.JSONLogFormatter
@time 1_760_000_000_123_456
describe "format/2" do
test "emits one JSON line with timestamp, level, message and request id" do
event = %{
level: :info,
msg: {:string, "hello"},
meta: %{time: @time, request_id: "request-1"}
}
decoded = format_and_decode(event)
assert decoded["timestamp"] ==
@time |> DateTime.from_unix!(:microsecond) |> DateTime.to_iso8601()
assert decoded["level"] == "info"
assert decoded["message"] == "hello"
assert decoded["request_id"] == "request-1"
end
test "includes access-line fields" do
event = %{
level: :info,
msg: {:string, "GET /health 200 in 1.2ms"},
meta: %{
time: @time,
request_id: "request-1",
method: "GET",
path: "/health",
status: 200,
duration_ms: 1.2
}
}
decoded = format_and_decode(event)
assert decoded["method"] == "GET"
assert decoded["path"] == "/health"
assert decoded["status"] == 200
assert decoded["duration_ms"] == 1.2
assert decoded["request_id"] == "request-1"
end
test "renders an exception's reason and stacktrace" do
event = %{
level: :error,
msg: {:string, "boom"},
meta: %{
time: @time,
exception: %RuntimeError{message: "bad thing"},
stacktrace: stacktrace()
}
}
decoded = format_and_decode(event)
assert decoded["error"] == %{
"kind" => "RuntimeError",
"reason" => "bad thing",
"stacktrace" => ["lib/foo.ex:10: Foo.bar/1"]
}
end
test "renders an OTP crash report's reason and stacktrace" do
event = %{
level: :error,
msg: {:report, %{label: :gen_server_terminate}},
meta: %{
time: @time,
crash_reason: {%RuntimeError{message: "call boom"}, stacktrace()}
}
}
decoded = format_and_decode(event)
assert decoded["error"]["kind"] == "RuntimeError"
assert decoded["error"]["reason"] == "call boom"
assert decoded["error"]["stacktrace"] == ["lib/foo.ex:10: Foo.bar/1"]
end
test "redacts email addresses and credentials in metadata, params and the message" do
event = %{
level: :info,
msg: {:string, "contact user@example.com"},
meta: %{
time: @time,
email: "user@example.com",
token: "secret-token",
params: %{
"user" => %{
"email" => "user@example.com",
"name" => "Ada"
},
"user@example.com" => %{"name" => "Keyed by email"}
}
}
}
decoded = format_and_decode(event)
assert decoded["email"] == "[REDACTED]"
assert decoded["token"] == "[REDACTED]"
assert decoded["params"]["user"]["email"] == "[REDACTED]"
assert decoded["params"]["user"]["name"] == "Ada"
assert decoded["params"]["[REDACTED]"]["name"] == "Keyed by email"
assert decoded["message"] == "contact [REDACTED]"
end
test "redacts credential assignments in the message" do
event = %{
level: :info,
msg: {:string, "GET /reset token=abc123&next=/home"},
meta: %{time: @time}
}
decoded = format_and_decode(event)
assert decoded["message"] == "GET /reset token=[REDACTED]&next=/home"
end
test "sanitises non-JSON metadata instead of crashing" do
event = %{
level: :info,
msg: {:string, "weird"},
meta: %{
time: @time,
pid: self(),
reference: make_ref(),
tuple: {:ok, 1},
charlist: ~c"hello",
domain: [:bandit],
invalid: <<0, 255>>
}
}
decoded = format_and_decode(event)
assert decoded["pid"] =~ "#PID"
assert is_binary(decoded["reference"])
assert decoded["tuple"] == ["ok", 1]
assert decoded["charlist"] == "hello"
assert decoded["domain"] == ["bandit"]
assert decoded["invalid"] == "<<0, 255>>"
end
test "falls back to a JSON line when the message cannot be formatted" do
event = %{
level: :error,
msg: {:report, %{label: :broken}},
meta: %{
time: @time,
report_cb: fn _report -> raise "report blew up" end
}
}
decoded = format_and_decode(event)
assert decoded["level"] == "error"
assert is_binary(decoded["message"])
assert decoded["message"] =~ "broken"
end
end
describe "production configuration" do
test "uses the JSON formatter only in production" do
config = Config.Reader.read!("config/prod.exs", env: :prod)
assert get_in(config, [:logger, :default_handler, :formatter]) ==
{Chesstrainer.JSONLogFormatter, %{}}
end
end
defp format_and_decode(event) do
line = event |> JSONLogFormatter.format(%{}) |> IO.iodata_to_binary()
assert String.ends_with?(line, "\n")
assert length(String.split(line, "\n", trim: true)) == 1
Jason.decode!(line)
end
defp stacktrace do
[{Foo, :bar, 1, [file: ~c"lib/foo.ex", line: 10]}]
end
end
+47
View File
@@ -0,0 +1,47 @@
defmodule ChesstrainerWeb.RequestIdTest do
use ExUnit.Case, async: false
import Plug.Conn
alias ChesstrainerWeb.Plugs.PutRequestId
alias ChesstrainerWeb.UserAuth
setup do
on_exit(fn -> Logger.metadata(request_id: nil) end)
end
test "PutRequestId copies the request id into the session" do
conn =
:get
|> Phoenix.ConnTest.build_conn("/")
|> Phoenix.ConnTest.init_test_session(%{})
|> assign(:request_id, "request-1")
|> PutRequestId.call([])
assert get_session(conn, :request_id) == "request-1"
end
test "PutRequestId leaves the session alone without a request id" do
conn =
:get
|> Phoenix.ConnTest.build_conn("/")
|> Phoenix.ConnTest.init_test_session(%{})
|> PutRequestId.call([])
assert get_session(conn, :request_id) == nil
end
test "mounting a LiveView puts the session's request id into logger metadata" do
socket = %Phoenix.LiveView.Socket{}
assert {:cont, _socket} =
UserAuth.on_mount(
:mount_current_scope,
%{},
%{"request_id" => "request-2", "user_token" => nil},
socket
)
assert Logger.metadata()[:request_id] == "request-2"
end
end
@@ -0,0 +1,97 @@
defmodule ChesstrainerWeb.RequestLoggerTest do
use ChesstrainerWeb.ConnCase, async: false
import ExUnit.CaptureLog
import Plug.Conn
alias ChesstrainerWeb.RequestLogger
setup do
level = Logger.level()
Logger.configure(level: :info)
on_exit(fn -> Logger.configure(level: level) end)
:ok
end
describe "handle_event/4" do
test "logs one access line with method, path, status, duration and request id" do
conn = access_conn("GET", "/endgames", 200, "request-1")
log =
capture_log([level: :info], fn ->
RequestLogger.handle_event(
[:phoenix, :endpoint, :stop],
%{duration: 12_500_000},
%{conn: conn},
:ok
)
end)
assert log =~ "GET /endgames 200 in 12.5ms"
assert log =~ "request_id=request-1"
end
test "redacts token-like path segments" do
token = "SFMyNTY." <> String.duplicate("a", 43)
conn = access_conn("GET", "/users/reset-password/#{token}", 200, "request-1")
log =
capture_log([level: :info], fn ->
RequestLogger.handle_event(
[:phoenix, :endpoint, :stop],
%{duration: 1_000_000},
%{conn: conn},
:ok
)
end)
assert log =~ "GET /users/reset-password/[REDACTED] 200 in 1.0ms"
refute log =~ token
end
test "never logs the query string" do
conn =
:get
|> build_conn("/endgames?email=user@example.com&secret=abc")
|> assign(:request_id, "request-1")
|> put_status(200)
log =
capture_log([level: :info], fn ->
RequestLogger.handle_event(
[:phoenix, :endpoint, :stop],
%{duration: 1_000_000},
%{conn: conn},
:ok
)
end)
assert log =~ "GET /endgames 200 in 1.0ms"
refute log =~ "user@example.com"
refute log =~ "secret=abc"
end
end
describe "endpoint integration" do
test "emits exactly one access line for a request", %{conn: conn} do
log = capture_log([level: :info], fn -> get(conn, ~p"/health") end)
assert length(Regex.scan(~r/GET \/health 200 in [\d.]+ms/, log)) == 1
assert log =~ "request_id="
end
test "emits an access line for a LiveView initial request", %{conn: conn} do
log = capture_log([level: :info], fn -> get(conn, ~p"/users/log-in") end)
assert length(Regex.scan(~r/GET \/users\/log-in 200 in [\d.]+ms/, log)) == 1
assert log =~ "request_id="
end
end
defp access_conn(method, path, status, request_id) do
method
|> build_conn(path)
|> assign(:request_id, request_id)
|> put_status(status)
end
end
+9
View File
@@ -0,0 +1,9 @@
defmodule Chesstrainer.TestLogHandler do
@moduledoc false
@doc false
def format(event, %{pid: pid}) do
send(pid, {:log_event, event})
""
end
end