docs(auth): bound crafty's post-fix ratio to its own pod; record the metrics gap
Coordinator raised crafty as counter-evidence: it carries the cookie fix and
showed the platform's worst ratio (81:1 over 24h from Loki), implying the fix
does not work. Bounded it exactly instead of in Loki -- crafty's pod started
2026-09-30T03:57:19Z with 0 restarts, so a 24h window is ~10h of the PRE-fix
process. Read off the pod's own log, whose boundary is the process start:
crafty 3 'No valid authentication', 0 AuthSuccess, in 13h52m <- not 81
one client IP, 2 of the 3 on /ws (WebSocket reconnects), two
isolated minutes (04:02, 15:22)
Post-fix crafty is idle, not broken. Correctly bounded it is evidence FOR the
fix. The ratio metric ranks CHATTINESS, not breakage: radarr's 156:1 is
/api/v3/command (62), /api/v3/queue/details (46), /signalr/negotiate (43) --
UI polling. Added that trap to the runbook, because ranking apps this way is
how a deployed fix gets written off.
That polling also sharpens the jackett mechanism: every poll arriving with a
cookie radarr cannot decrypt calls ClearSession again, so a left-open radarr
tab deletes jackett's cookie every few seconds. Explains the intermittency.
Re-weighted the diagnosis as asked: mechanisms 2 and 3 (5-minute session with
no --cookie-refresh; 302-instead-of-401 on XHR) are demonstrated from
jackett's own log and between them fully explain the report. Mechanism 1
(collision) is firing and jackett genuinely violates it, but causation is one
3-second adjacency, not a population -- necessary and firing, not proven
sufficient. Stated that way in the entry.
Metrics: confirmed no per-app sidecar exposes any (--metrics-address ABSENT,
ports ['http'] on all six); only the shared proxy has the flag and a metrics
port. Worth adding and DECLINED HERE for the structural reason only -- the
container's args AND ports both come from _oauth2_sidecar_container(), so it
hits the identical five-file blast radius. It belongs in the same single
renderer commit. Recorded, not silently skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
6a6fea8151
commit
0a0adce4d8
2 files changed
+94
-5
No files matched your search
@@ -97,6 +97,27 @@ instances that genuinely cannot read each other. Compare **hashes**, never value
|
||||
Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here
|
||||
before.
|
||||
|
||||
### Do not rank apps by their failure ratio
|
||||
|
||||
Counting `No valid authentication` against `AuthSuccess` per app looks like a health metric and is
|
||||
not one. **It ranks chattiness, not breakage.** An app whose UI polls or holds a WebSocket
|
||||
manufactures one failure line per retry, so a single dead cookie in one idle browser tab becomes
|
||||
scores of log lines, while an app nobody is using scores zero no matter how broken it is. Measured
|
||||
2026-09-30: Radarr showed **156** unauthenticated requests against **1** `AuthSuccess`, but the top
|
||||
paths were `/api/v3/command` (62), `/api/v3/queue/details` (46) and `/signalr/messages/negotiate`
|
||||
(43) — polling, not 156 user-visible failures.
|
||||
|
||||
**Bound the window to the current pod, not to wall-clock.** A ratio measured over 24 h across a pod
|
||||
that restarted 13 h ago is mostly about the *previous* configuration, and that is exactly how a
|
||||
deployed fix gets written off as ineffective. The pod's own log has an exact boundary — its process
|
||||
start — so read it there:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> get pod \
|
||||
-l app.kubernetes.io/name=<app> -o jsonpath="{.items[*].status.startTime}{\"\n\"}"'
|
||||
|
||||
then count within that same pod's log. On 2026-09-30 this took Crafty from an apparent 81:1 over 24 h
|
||||
to **3 failures in its fixed pod's entire 13 h 52 m life**, two of them `/ws` reconnects.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
1. **Cookie collision between two per-app proxies.** Several instances emit the default
|
||||
@@ -148,6 +169,12 @@ before.
|
||||
2026-09-30. Coordinate before starting — this is a single change, not five.
|
||||
- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in
|
||||
`platform/crafty/deployment.yaml`, so it can be, and was, fixed independently.
|
||||
- **You cannot currently alert on any of this**, and that is worth knowing before you promise
|
||||
monitoring. Measured 2026-09-30: no per-app sidecar sets `--metrics-address` and none declares a
|
||||
`metrics` container port, so oauth2-proxy's metrics listener never starts — the only oauth2-proxy
|
||||
series in Prometheus come from the shared proxy in namespace `oauth2-proxy`, which does set both.
|
||||
Until the renderer emits them for the per-app sidecars, every diagnosis on this page is
|
||||
log-reading only.
|
||||
- **Validate an argument change before you open the PR**, not after. The pinned image will tell
|
||||
you what it understood, without touching the cluster:
|
||||
|
||||
|
||||
Reference in new issue
Block a user