docs(auth): bound crafty's post-fix ratio to its own pod; record the metrics gap
gate / gate (pull_request) Successful in 1m41s
gate / conformance (pull_request) Has been skipped

Coordinator raised crafty as counter-evidence: it carries the cookie fix and
showed the platform's worst ratio (81:1 over 24h from Loki), implying the fix
does not work. Bounded it exactly instead of in Loki -- crafty's pod started
2026-09-30T03:57:19Z with 0 restarts, so a 24h window is ~10h of the PRE-fix
process. Read off the pod's own log, whose boundary is the process start:

  crafty  3 'No valid authentication', 0 AuthSuccess, in 13h52m   <- not 81
          one client IP, 2 of the 3 on /ws (WebSocket reconnects), two
          isolated minutes (04:02, 15:22)

Post-fix crafty is idle, not broken. Correctly bounded it is evidence FOR the
fix. The ratio metric ranks CHATTINESS, not breakage: radarr's 156:1 is
/api/v3/command (62), /api/v3/queue/details (46), /signalr/negotiate (43) --
UI polling. Added that trap to the runbook, because ranking apps this way is
how a deployed fix gets written off.

That polling also sharpens the jackett mechanism: every poll arriving with a
cookie radarr cannot decrypt calls ClearSession again, so a left-open radarr
tab deletes jackett's cookie every few seconds. Explains the intermittency.

Re-weighted the diagnosis as asked: mechanisms 2 and 3 (5-minute session with
no --cookie-refresh; 302-instead-of-401 on XHR) are demonstrated from
jackett's own log and between them fully explain the report. Mechanism 1
(collision) is firing and jackett genuinely violates it, but causation is one
3-second adjacency, not a population -- necessary and firing, not proven
sufficient. Stated that way in the entry.

Metrics: confirmed no per-app sidecar exposes any (--metrics-address ABSENT,
ports ['http'] on all six); only the shared proxy has the flag and a metrics
port. Worth adding and DECLINED HERE for the structural reason only -- the
container's args AND ports both come from _oauth2_sidecar_container(), so it
hits the identical five-file blast radius. It belongs in the same single
renderer commit. Recorded, not silently skipped.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-30 12:52:12 -05:00
1 parent 6a6fea8151
commit 0a0adce4d8
2 files changed
+94 -5

No files matched your search

+27
View File
@@ -97,6 +97,27 @@ instances that genuinely cannot read each other. Compare **hashes**, never value
Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here
before.
### Do not rank apps by their failure ratio
Counting `No valid authentication` against `AuthSuccess` per app looks like a health metric and is
not one. **It ranks chattiness, not breakage.** An app whose UI polls or holds a WebSocket
manufactures one failure line per retry, so a single dead cookie in one idle browser tab becomes
scores of log lines, while an app nobody is using scores zero no matter how broken it is. Measured
2026-09-30: Radarr showed **156** unauthenticated requests against **1** `AuthSuccess`, but the top
paths were `/api/v3/command` (62), `/api/v3/queue/details` (46) and `/signalr/messages/negotiate`
(43) — polling, not 156 user-visible failures.
**Bound the window to the current pod, not to wall-clock.** A ratio measured over 24 h across a pod
that restarted 13 h ago is mostly about the *previous* configuration, and that is exactly how a
deployed fix gets written off as ineffective. The pod's own log has an exact boundary — its process
start — so read it there:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> get pod \
-l app.kubernetes.io/name=<app> -o jsonpath="{.items[*].status.startTime}{\"\n\"}"'
then count within that same pod's log. On 2026-09-30 this took Crafty from an apparent 81:1 over 24 h
to **3 failures in its fixed pod's entire 13 h 52 m life**, two of them `/ws` reconnects.
## Likely causes, most likely first
1. **Cookie collision between two per-app proxies.** Several instances emit the default
@@ -148,6 +169,12 @@ before.
2026-09-30. Coordinate before starting — this is a single change, not five.
- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in
`platform/crafty/deployment.yaml`, so it can be, and was, fixed independently.
- **You cannot currently alert on any of this**, and that is worth knowing before you promise
monitoring. Measured 2026-09-30: no per-app sidecar sets `--metrics-address` and none declares a
`metrics` container port, so oauth2-proxy's metrics listener never starts — the only oauth2-proxy
series in Prometheus come from the shared proxy in namespace `oauth2-proxy`, which does set both.
Until the renderer emits them for the per-app sidecars, every diagnosis on this page is
log-reading only.
- **Validate an argument change before you open the PR**, not after. The pinned image will tell
you what it understood, without touching the cluster: