Files
home-network-k3s-impl/decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md
T
jakman85andClaude Opus 5 dc136a4943
gate / gate (pull_request) Successful in 1m42s
gate / conformance (pull_request) Has been skipped
fix(auth): the forward-auth sidecar also died at 5 minutes and served no metrics
Widens the cookie-collision fix to the whole sidecar contract, because all three
defects live in the same function and land on the same five goldens, so they are one
atomic change or they are five conflicting ones.

Added to _oauth2_sidecar_container(), on top of the cookie-name/cookie-domain fix:

  --cookie-refresh=2m30s
      The session died FIVE MINUTES after sign-in. Keycloak's homelab realm issues
      300-second access tokens (MEASURED on a live session: expires - created =
      300.000s exactly, with refresh_token:true on the same line), and
      --cookie-refresh defaults to 0 -- which the pinned binary's own help spells
      "0 to disable". So the proxy held a refresh token it was never told to spend.
      Refresh at HALF the lifetime: two chances to renew before expiry.
      This, not the collision, is what dominates the rejection VOLUME -- ~95% of
      radarr's 155 rejections in 12h carry no cookie at all.

  --metrics-address=0.0.0.0:44180 + a `metrics` container port
      Without the flag the /metrics listener never starts, so no per-app sidecar was
      serving metrics at all (`ports` was ['http'] on all six). 44180 is the
      platform's existing convention, measured not chosen: the shared proxy already
      runs the same flag and port with 2 ready endpoints.
      PREREQUISITE ONLY -- NOT scraped. A container port is not a Service port, and
      each namespace's default-deny baseline is `ingress: []`, so this is unreachable
      until a Service port (or PodMonitor), a monitoring allow and a ServiceMonitor
      land. Do not read this as "the sidecars have metrics now".

EVERY FLAG VERIFIED AGAINST THE PINNED v7.8.1 BINARY (`--help`), not against docs or
the shared proxy -- which runs v7.15.3 and could not have vouched for v7.8.1:
  * --metrics-address exists in 7.8.1, so this cannot crashloop the pods
  * --cookie-refresh ... "0 to disable" confirms the default
  * --cookie-name ... (default "_oauth2_proxy") -- this confirms from the
    IMPLEMENTATION the constant the new L0 check is built on
  * --api-route is SINGULAR; the plural does not exist and the proxy refuses to start
  * --session-cookie-minimal exists and is mutually exclusive with --cookie-refresh,
    so dropping --pass-access-token is a least-privilege argument, never a size fix

NOT done, deliberately: --api-route=^/api/ renderer-wide. It was measured on jackett
and PiHole does not serve /api/, so a renderer constant would apply one app's path
shape to five. It needs a per-app hook the `auth` schema forbids
(additionalProperties: false) -- a schema change, separate PR.

Also records the mechanism that made this INTERMITTENT, as a renderer comment, because
nobody would reconstruct it from the flags: the loser does not just fail to read the
cookie, it calls ClearSession, which expires the cookie on its own --cookie-domain --
the whole zone. The *arr UIs poll (/api/v3/command, /api/v3/queue/details,
/signalr/negotiate), so every poll carrying an undecryptable cookie clears it again:
A LEFT-OPEN RADARR TAB DELETED JACKETT'S COOKIE EVERY FEW SECONDS, from another
namespace, with no user action. That is why signing in again never helped for longer
than the gap between two polls.

Crafty's counter-evidence resolved: the 81:1 ratio was a 24h window straddling a
13h52m-old pod. Bounded to the fixed process, crafty's ENTIRE life is 3 rejections in
6743 log lines, with ZERO `removing session` and ZERO `cookie signature not valid`
against unfixed radarr's 8 and 8. The eviction fingerprint is gone where the fix is
deployed. It logged 0 AuthSuccess too, so that is absence-of-the-failure-mode rather
than a session watched working -- stated as such in the handoff.

Renderer selftest gains six assertions, including a --cookie-refresh check that PARSES
the duration and requires 0 < refresh < 300s (so 2m30s/150s/2.5m all stay green while a
silent widening past the token lifetime does not). NOTE, and it is in the handoff as the
top follow-up: NOTHING IN ci/ RUNS `render.py --selftest`, so those assertions are
dev-time only. conformance/tests/l0-static/auditability-proofs-producible/check.sh:291
already drives other tools' selftests for exactly this reason and is the one-line fix.

Gate (toolbox, worktree shim with the spec repo mounted at /home-network-k3s): rc 3
UNDETERMINED. Green: spec-immutability, decisions-log, lint, l0, render-golden, l1,
approval-policy. Skipped: gitea-settings (no local token), conformance (declared).
The new L0 check PASSES on this branch -- all five apps fixed in one commit.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-30 13:07:43 -05:00

379 lines
29 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
date: 2026-09-30
branch: fix/oauth2-proxy-cookie-collision-fleet
title: The cookie collision was never Crafty's — it was the renderer's, and it shipped to five apps
---
## What changed
Follow-up #3 of `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`: *"Any further per-app
reverse-proxy sidecar needs a unique `--cookie-name` too. The collision is a property of the
pattern, not of Crafty."*
It is a property of the pattern, and this branch found where the pattern actually lives.
**Read the scope claim carefully before the detail.** This branch fixes THREE defects in the
forward-auth sidecar contract, all in one function, all landing on all five apps at once:
| Defect | Fix | Measured evidence |
|---|---|---|
| Six independent instances, seven different cookie-secrets, **one** cookie name on **one** zone-wide domain — so they overwrite and `ClearSession`-delete each other's sessions | `--cookie-name=_oauth2_proxy_<app>` + host-scoped `--cookie-domain` | 7/7 distinct secrets; 6 instances on the default name; radarr logs the eviction fingerprint 8× in 12h |
| The session dies **5 minutes** after sign-in — `--cookie-refresh` defaults to 0 = *disabled*, so the refresh token the proxy already holds is never spent | `--cookie-refresh=2m30s` (half the lifetime) | access-token lifetime = **300.000s exactly**, `refresh_token:true`, flag absent |
| **No** per-app sidecar serves `/metrics` at all — without `--metrics-address` the listener never starts | `--metrics-address=0.0.0.0:44180` + a `metrics` container port | `ports` was `['http']` on all six |
Rejection *volume* is dominated by the second defect, not the first: ~95% of radarr's rejections
carry no cookie at all. **Do not merge this expecting one number to drop** — three mechanisms were
tangled together, and separating them is most of the value here.
**Boundary note.** My brief originally reserved `apps/wave-0/jackett.yaml` for a concurrent branch.
That split is not achievable and has been revoked: the file is generated, the fix is one renderer
edit, and `render-golden` requires all five goldens regenerated in the *same commit*. I own all
five. The proof I ran before touching jackett is under "What I verified".
**The defect was not in the four manifests it was reported in.** `apps/wave-0/pihole.yaml`,
`apps/wave-0/qbittorrent.yaml`, `apps/wave-1/radarr.yaml`, `apps/wave-1/sonarr.yaml` — and
`apps/wave-0/jackett.yaml` — are **generated goldens**. One function emits all five sidecars:
`catalog/render.py::_oauth2_sidecar_container()`, which hardcoded `--cookie-domain=.{ZONE}` and
emitted no `--cookie-name` at all. Hand-editing a golden is not a fix twice over: `ci/render-golden.sh`
FAILS on the drift (provoked, not assumed — see below), and the next re-render silently reverts it.
So the fix is in the generator, and both flags are now **derived** from the ServiceDefinition:
--cookie-name=_oauth2_proxy_{s['name']} unique per service, by construction
--cookie-domain={host} the app's own route host, not the zone
`--whitelist-domain` deliberately stays zone-wide — it governs permitted post-login *redirect*
targets, not cookie scope. `catalog/render.py --selftest` gained four assertions, including one
that renders a **second** synthetic forward-auth service and requires the two cookie names to
differ, so a future refactor to a constant fails in the renderer's own test.
Re-rendering with the canonical command regenerated **five** goldens, not four. Jackett is the
fifth, and this is the one thing in this branch the PM brief did not anticipate — see
"the jackett collision" below. The three `conformance/tests/l1-rendering/*/golden/` fixtures are
expected-output copies of the same renderer, so they are regenerated too; that is not collateral
damage, it is the L1 stage doing its job (it FAILED on the first gate run and named the drift).
**The new gate check** is `conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/`,
declaring `conformance.home.k3s/scope: whole-repo` so `ci/l0.sh`'s `--include-scope` union runs it
on **every** gate invocation. No bespoke `ci/l0.sh` stage was added — `whole-repo-invariants-wired`
fails the gate if anyone adds one, and the scope annotation is the supported mechanism.
It inventories every oauth2-proxy instance the repo authors (containers at any depth, plus the
shared proxy's HelmRelease values) and resolves each one's **effective** cookie name — its declared
name, **or oauth2-proxy's built-in `_oauth2_proxy` when it declares none**. That single modelling
decision is the whole check: an omitted flag is not an absence, it is a *claim on a name*, and
treating it as an absence is precisely why five reviewers read these manifests and saw nothing.
Then: (1) no two effective names collide; (2) no name is another's name + `_csrf`, because that is
how oauth2-proxy derives the CSRF cookie and two such session names look distinct while their CSRF
cookies collide; (3) the default name has exactly one owner, and it is the one registered in
`SHARED_DEFAULT_COOKIE_OWNER`; (4) every non-registered instance is host-scoped; (5)
`catalog/render.py` derives rather than lists; (6) the **shipped** renderer is proven end-to-end on
two synthetic ServiceDefinitions run as a subprocess.
**On the shared proxy — "distinct" or "explicitly declared"?** The brief asked me to decide, and
the answer is *neither, quite*. The shared forward-auth proxy at `platform/oauth2-proxy/` **must
keep** `_oauth2_proxy`: it is the single session every forward-auth-gated surface without its own
sidecar rides on, so renaming it signs all of them out at once. Requiring *every* instance to
declare a name would therefore need an exemption, and an exemption is a hole: delete the shared
proxy and whichever app next omitted the flag would silently inherit the default with the rule
still satisfied. So the assertion is plain **distinctness over effective names**, with the default
treated as a name like any other — held by **exactly one owner, named in a shrink-only register
with its reason**. The shared proxy needs no exemption from the rule; it is simply the registered
holder of that one name. A stale register entry FAILS (the exemption would outlive its object) and
an unregistered holder FAILS (no silent heirs). Both were provoked and are quoted below.
**On host-scoped `--cookie-domain` — overreach, or not?** Asserted, and here is the argument both
ways. *Against:* distinct names alone close the eviction bug completely — two differently-named
cookies do not overwrite each other however wide their domains are — so this assertion fixes
nothing (1) has not already fixed, and a check that fails a legitimate configuration is a check
someone eventually disables. *For, and this wins:* a zone-wide cookie is **transmitted by the
browser to every host in the zone**. qBittorrent's session token was being sent to pihole, radarr,
sonarr and every other name under the zone, including hosts behind no proxy at all — a
session-credential disclosure across trust boundaries that nothing needs. And it is what makes a
*future* name collision catastrophic rather than contained: with host-scoped cookies, two
instances that accidentally share a name cannot see each other's cookie at all. Defence in depth
on the exact axis that already failed once. The escape hatch is the same register, by name and
with a reason, rather than a loosened rule.
### The jackett boundary, and why it could not hold
`apps/wave-0/jackett.yaml` carries the identical defect and was initially assigned to a concurrent
branch, with an instruction to me not to touch it. **That separation is not achievable** — the
ownership split was drawn across a generated file — and I did not discover this by reasoning about
it, I provoked it. With the renderer fixed and jackett's golden left at its `main` content,
`./ci/render-golden.sh` exits **1**:
[fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render:
- - --cookie-domain=.${BASE_DOMAIN}
+ - --cookie-name=_oauth2_proxy_jackett
+ - --cookie-domain=jackett.${BASE_DOMAIN}
[fail] render==golden check FAILED
Jackett's golden is an *output* of the file that contains the defect. There are exactly three
options and two of them are worse: hand-edit four goldens and leave the generator broken (fails
`render-golden`, and reverts on the next render); add a per-service carve-out to the renderer so
jackett alone keeps defaulting (ships the exact "list, don't derive" anti-pattern this check
exists to forbid, and leaves jackett broken *by design*); or fix the generator and regenerate all
five. I took the third. Jackett's golden here is **machine output of a one-line generator change**,
not an independent edit of Mira's file, and it is byte-identical to what her branch must produce.
**Consequence:** the jackett branch has no cookie work left in it — this commit fixes jackett
identically and by the same mechanism, and both branches independently arrived at the *same two
values* (`_oauth2_proxy_jackett`, `jackett.${BASE_DOMAIN}`), which is a good sign for the
diagnosis. Its hand-edit of the rendered file will fail `render-golden` on its own terms, so it
should be closed rather than rebased. **What must not be lost with it** is its evidence: the
interleaved radarr/jackett log showing `removing session` deleting a clean session 26s after login,
and the `expires − created = 300.000s` measurement that produced `--cookie-refresh` here. Both are
carried into this entry and into the renderer's comments; the branch itself is now redundant, the
work behind it is not.
**The general lesson, which is worth more than this bug:** an ownership split across
`apps/wave-*/*.yaml` cannot hold, because those files are *outputs*. Two agents assigned two
rendered apps are assigned the same function and will collide in `render-golden` no matter how
carefully they stay in their own files. Ownership of a rendered app is ownership of
`catalog/render.py`, and briefs should say so.
### What this branch deliberately does NOT do
- **`--api-route=^/api/` is NOT added**, and not because of scope discipline alone: it would be
**wrong** renderer-wide. It was measured on jackett, and **PiHole does not serve `/api/`** — so
a renderer constant applies one app's path shape to five. Doing it properly means declaring API
path prefixes per service, and the `auth` block is `additionalProperties: false`, so that is a
*schema change*. Separate PR. (Note the flag is **`--api-route`, singular** — verified in the
pinned binary's own `--help`. The plural `--api-routes` does not exist in v7.8.1 and the proxy
refuses to start with it. That error was propagating through briefs; it stops here.)
- **`--session-cookie-minimal` is NOT added** and must not be, as a cookie-size fix: it is
**mutually exclusive** with `--cookie-refresh` (*"cookie_refresh > 0 requires oauth tokens in
sessions"*). Dropping `--pass-access-token` is a least-privilege argument, never a size one.
- **Metrics are NOT scraped, only served.** The flag and container port are the prerequisite;
a container port is not a Service port. These Services expose the app port and `:4180`, and each
namespace's default-deny baseline is `ingress: []`, so Prometheus can neither resolve nor reach
`:44180` until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
exist. That wiring is someone else's follow-up and I have deliberately not faked its start.
**Nobody should read this commit as "the sidecars have metrics now."**
- **The new check was NOT widened to cover cookie-refresh or metrics.** Its subject is cookie
*identity*; those two share neither its inputs nor its failure mode. Bundling them would make
the check's name a lie and couple unrelated ratchets, so a future metrics decision could only
be expressed by weakening a *cookie* check. The cost of that discipline is named honestly in
the follow-ups: those two flags are currently **ungated**.
## What I verified
- **The premise, in-cluster, rather than inherited from the handoff** (CLAUDE.md §2b). Read-only,
`kubectl --context k3s-prod`, through the pinned toolbox.
- **Seven** `*-oauth2-proxy` Secrets carry a `cookie-secret`: crafty, jackett, pihole,
qbittorrent, radarr, sonarr, and the shared `oauth2-proxy-credentials`. **All seven values are
DISTINCT.** Compared without printing anything: each value was hashed **in-cluster-side script,
never echoed**, and only the first 8 hex characters of each `sha256` were emitted — enough to
say same-or-different, recovering nothing. Secret *key names* were listed; no `.data` or
`.stringData` was printed (CLAUDE.md §6). **So no instance can decrypt another's cookie — it can
only overwrite it. The premise holds.**
- **The default cookie names really were in use.** Of 8 live oauth2-proxy containers, **6 declare
no `--cookie-name` at all** — jackett, pihole, qbittorrent, radarr, sonarr, and both replicas of
the shared proxy — so all six emit `_oauth2_proxy` / `_oauth2_proxy_csrf`. All six also carry
`--cookie-domain=.home.keithjackson.dev`. **One cookie pair, six independent instances.**
- Crafty's live pod (`crafty-647c7d8cc5-5spv9`) already carries
`--cookie-name=_oauth2_proxy_crafty` and `--cookie-domain=crafty.home.keithjackson.dev`, so
PR #22 is merged and reconciled. It is the only compliant per-app instance on the cluster.
- **THE COLLISION IS REAL. IT IS NOT ESTABLISHED AS THE CAUSE OF THE REPORTED FAILURES.** These
are two different claims and only the first is supported. I was handed counter-evidence —
crafty, the app already fixed, showed the *worst* rejection ratio of the set over a 24h Loki
window (`crafty 81:1`, `radarr 67:1`, `jackett 39:4`) — and it is right to refuse the causal
claim. What I could do was bound the window. Read from **each sidecar's own log** rather than
Loki, because `kubectl logs` on a pod that has not restarted cannot reach a pod older than
itself, so a window inside the pod's age is inside the fixed spec's lifetime *by construction*:
namespace age(h) rst reject success ratio cleared badsig
crafty 13.9 0 3 0 n/a 0 0 <- FIXED spec, whole life
radarr 422.5 0 155 1 155.0:1 8 8
jackett 422.6 0 39 4 9.8:1 1 1
pihole 395.7 0 0 0 0:0 0 0
qbittorrent 404.1 0 0 0 0:0 0 0
sonarr 422.5 0 0 0 0:0 0 0
oauth2-proxy 422.6 0 0 0 0:0 0 0 (both replicas)
`reject` = `No valid authentication in request`; `success` = `AuthSuccess`;
`cleared` = `removing session` (oauth2-proxy's `ClearSession`); `badsig` = `cookie signature
not valid`. Window `--since=24h`, which exceeds every pod's age only for crafty — whose pod is
13.9h old with **0 restarts**, so these are its complete post-fix figures.
Then I read crafty's four auth-relevant lines rather than counting them, because at n=3 the
content is the evidence:
03:57:20 Cookie settings: name:_oauth2_proxy_crafty secure(https):true httponly:true
expiry:168h0m0s domains:crafty.home.keithjackson.dev path:/
04:02:55 No valid authentication in request. Initiating login.
15:22:39 No valid authentication in request. Initiating login.
15:22:44 No valid authentication in request. Initiating login.
6743 log lines; those are all four. **Zero** `removing session`, **zero** `cookie signature
not valid`, zero anything else.
Four conclusions, kept separate:
1. **`crafty 81:1` is a window artefact. It is not evidence against the fix and must not be
cited as such.** The fixed pod emitted **3** rejections in 13h 52m; the 81 came from a 24h
window straddling it, i.e. from the pre-fix process.
2. **The eviction FINGERPRINT is gone where the fix is deployed.** `cookie signature not
valid` + `removing session` is what a clobbered instance logs — a cookie *is* present,
written with a key this instance does not hold, so it cannot decrypt it and then
`ClearSession` expires it on its own `--cookie-domain`. Fixed crafty: **0 and 0** across
its whole life. Unfixed radarr: **8 and 8** in 12h. Unfixed jackett: 1 and 1. That is a
real, directional signal and it is the strongest evidence available that the fix works.
3. **It is still not a demonstrated successful sign-in.** Crafty logged **0 `AuthSuccess`** —
so what is shown is *absence of the failure mode*, not a session observed working
end-to-end. Those three rejections are consistent with benign first-hits (one client, two
of them 5s apart, i.e. a page plus a `/ws` reconnect). I am comfortable saying the fix
works; I am not willing to say it has been watched working, and the distinction is the
whole reason this section exists.
4. **The collision is a MINORITY of the rejection volume.** radarr: 8 of 155 (5%) carry the
fingerprint; ~95% carry **no cookie at all**. Those are the 5-minute session expiry and
`302`-on-XHR — the first of which this branch now also fixes, the second of which it
deliberately does not.
Two cautions about the instrument. `reject` ranks **chattiness, not breakage**: radarr's are
`/api/v3/command`, `/api/v3/queue/details` and `/signalr/negotiate` polls, so a busy *arr
inflates the count with nobody inconvenienced. And that same polling is the *amplifier* that
made this intermittent — every poll carrying an undecryptable cookie calls `ClearSession`
again on the zone-wide domain, so **a left-open Radarr tab deleted Jackett's cookie every few
seconds, from another namespace, with no user action at all.** That is why signing in again
never helped for longer than the gap between two polls. It is now a comment in the renderer,
because it is the part of this bug that no one would reconstruct from the flags alone.
- **The four target files are generated, not authored.** Rendering `catalog/` with the canonical
command into a temp dir and diffing against `apps/` reported all 10 onboarded goldens
byte-identical before any change. Hand-editing one is drift by definition.
- **`render-golden` FAILS if jackett is excluded** — provoked (output quoted above), rc **1**.
Restoring the regenerated golden returns it to `all 10 onboarded service golden(s) reproduce`,
rc **0**.
- **Every flag verified against the PINNED v7.8.1 binary**, not against docs, memory, or the
shared proxy (which runs **v7.15.3** and therefore could not vouch for v7.8.1):
`podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help`.
- `--cookie-name string ... (default "_oauth2_proxy")` — **this confirms the constant my whole
check is built on, from the binary itself.** The check's central claim, that an omitted flag
is a *claim on a name* rather than an absence, now rests on the implementation's own help
text rather than on my reading of its source.
- `--cookie-refresh duration refresh the cookie after this duration; 0 to disable` — confirms
the default really is disabled.
- `--metrics-address string` — present in v7.8.1, so the new flag cannot crashloop the pods.
- `--api-route strings` — **singular**, confirming the brief's plural was wrong.
- `--session-cookie-minimal` — exists, and is the flag that conflicts with `--cookie-refresh`.
- **44180 is the platform's existing convention, measured, not chosen.** The shared proxy already
runs `--metrics-address=0.0.0.0:44180` with a `metrics` container port, a `metrics:44180`
Service port, a ServiceMonitor whose endpoint is `metrics`, and **2 ready endpoint addresses** —
i.e. it demonstrably works on this platform.
- **`catalog/render.py --selftest`: PASS**, now including the two-service distinctness assertion,
a `--cookie-refresh` assertion that PARSES the duration and requires `0 < refresh < 300s` (so
`2m30s` -> `150s` -> `2.5m` all stay green but a silent widening past the token lifetime does
not), a `--session-cookie-minimal` mutual-exclusion guard, and both metrics assertions.
- **The check is selected by the whole-repo union.** `conformance/run.sh --layer l0 --requirements
r-nosuchreq-99 --include-scope whole-repo --print-selection` lists
`tests/l0-static/oauth2-proxy-cookie-names-distinct` among 20 tests — i.e. it runs even when the
touched-requirement filter matches nothing, which is the blind spot it was scoped for.
- **The check was tested in BOTH directions**, against throwaway trees outside the repo, never by
mutating the branch:
- **Compliant tree (this branch): rc 0, 39 assertions, 7 instances audited.**
- **Pre-fix tree** (`git archive HEAD` + this test dropped in): **rc 1, 26 failures**, naming all
five violators, the 6-way `_oauth2_proxy` collision, the zone-wide domains, the renderer
emitting no `--cookie-name`, and both synthetic fixtures rendering `None`.
- **Targeted mutations on a compliant copy**, one assertion class each, all fired and nothing
else did: duplicate name (sonarr←radarr) → 1 failure; CSRF derivation
(`_oauth2_proxy_sonarr_csrf` as a session name) → fired; zone-wide `--cookie-domain` on pihole
→ 2 failures; crafty's `--cookie-name` deleted → unregistered-default-heir + 2-way collision;
shared proxy renamed → **stale register entry** failure.
- A **false positive was found and fixed** by this testing: `platform/oauth2-proxy/forward-auth-cec.yaml`
contains `serviceName: oauth2-proxy/oauth2-proxy:80` — an Envoy `namespace/service:port`
reference, not an image — which the first inventory scan counted as an unaudited instance. The
scan is now anchored on `image:`, and the near-miss is documented in the check.
- **Gate:** toolbox `./ci/gate.sh` — exit code and stage list in the next section, because it is
not a pass and that belongs there.
## What I did NOT verify
- **Local gate: rc 3 (UNDETERMINED), not 0.** Green: `spec-immutability`, `decisions-log`, `lint`,
`l0`, `render-golden`, `l1`, `approval-policy`. **Skipped:** `gitea-settings` (no
`SETTINGS_API_TOKEN` locally) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset). Nothing
FAILED; nothing runtime was certified either. **rc 3 is UNDETERMINED, never green.** Touched
subset `R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-PLAT-02 R-TEST-01
R-TEST-02 R-TEST-03 R-TEST-04` — **UNVERIFIED** at L2/L3, all twelve.
- **`spec-immutability` was skipping because of a BUG IN MY TOOLING, not the environment.** The
worktree ENGINE shim in circulation mounts only the main checkout's `.git`, and `dev.sh`
computes the spec path from its own `REPO_ROOT` — which in a worktree is
`.claude/worktrees/agent-*/`, so `../home-network-k3s` does not resolve and it adds no spec
mount at all. The gate then reported the skip, and the skip read as "the one tolerated
environmental skip" while actually being a path bug being waved through. Adding
`-v /var/home/keith/source/home-network-k3s:/home-network-k3s:ro -e SPEC_REPO=/home-network-k3s`
makes **R-TEST-06 actually run and pass**. Worth putting in every worktree agent brief; the
two-mount shim in CLAUDE.md §3 should say so.
- **CI: `gate / gate (pull_request)` reports FAILURE on `75a4029`, for exactly ONE reason, and it
is not this change.** Read from the job log rather than the tick:
[fail] approval-policy: a sensitive change lacks a non-author approval -> blocks merge
[fail] MERGE GATE: BLOCKED
Every substantive stage is **green in CI** — `gitea-settings: pass` (CI has the token my local
run does not), `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. Skipped: `spec-immutability`
(the workflow does not check the spec repo out — the documented tolerated skip) and
`conformance` (declared). So CI is a **stricter, more complete** run than my local one on every
axis except the spec repo, and it agrees.
`infra/gitea/approval-policy.json` lists `conformance/**` as sensitive — *"the tests and the
harness; changing them changes what green MEANS"* — so this PR is **fail-closed pending a
non-author approving review**, which is CLAUDE.md §1 working, not a defect to route around.
**The red tick is the policy, and it clears when a human reviews it.** Note also that
`gate / conformance (pull_request)` reports `success` with description **"Has been skipped"** —
the precise trap CLAUDE.md §2b names; it certifies nothing.
- **A SUCCESSFUL SIGN-IN, ANYWHERE.** The strongest post-fix evidence is the *absence* of the
eviction fingerprint on crafty (0 `removing session`, 0 `cookie signature not valid`, across
13h 52m) against radarr's 8-and-8. That is directional and I believe it. But crafty logged
**0 `AuthSuccess`** in the same window, so no session has been observed working end to end on a
fixed instance. Whoever closes this out should re-measure crafty **after real sign-in traffic**.
- **No browser round-trip by me at all**, for any of the three defects. Reproducing the collision
needs two gated apps visited in one session. The concurrent jackett branch does carry an observed
log-level reproduction (interleaved radarr/jackett `removing session` deleting a clean session
26s after login) — real evidence for the MECHANISM, and it is theirs, not mine.
- **`--cookie-refresh=2m30s` has never been exercised.** The value is derived from a measured
300.000s access-token lifetime and is asserted to sit inside `(0, 300s)`, but no session has been
held past the five-minute cliff to watch a refresh succeed. If Keycloak's refresh-token or SSO
idle timeouts are shorter than expected, the symptom moves rather than disappears. **This is the
fix most likely to need a second pass**, and the first thing to watch after merge.
- **The `metrics` port has never been fetched.** The listener is asserted to start by flag
presence only; nobody has curled `:44180` on a rendered pod. And it is unreachable by design
until the Service/policy/ServiceMonitor wiring lands — see above.
- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. Cluster access was
read-only throughout; the one container I ran was the oauth2-proxy image locally under podman to
read its `--help`. The running pods are still the pre-fix spec. **Merging is what deploys**, and
it will roll five Deployments; every user's existing session on those five apps is invalidated by
the cookie rename and they will re-authenticate once. Radarr/sonarr/qbittorrent use
`strategy: Recreate` on RWO volumes, so there is a brief gap per app. Pick the moment — and
note this now changes **three** flags plus a port on all five, so a bad `--cookie-refresh` value
would surface as five apps misbehaving, not one.
- **The live flags are not re-measured by the check.** git is not the cluster. I measured the live
pods by hand, once; nothing in the gate re-reads them, so hand-applied drift would be invisible
to it. A live-state assertion is L2 and is not written.
- **`--cookie-refresh` and `--metrics-address` are UNGATED, and I found out why.** Nothing in
`ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the twenty-odd
assertions in that selftest, including the six I just added, are dev-time only: a future edit
that drops either flag *and* re-renders the goldens would pass the entire merge gate.
`render-golden` catches accidental drift (golden vs renderer) but not a deliberate removal of
both together, and my L0 check deliberately does not cover these two flags. **The repo already
has the precedent for the fix and the one-line shape of it:**
`conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` drives
`conformance/lib/<tool> --selftest` as a subprocess for exactly this reason — its own header
says it does so "because nothing [else] drives them". Adding `catalog/render.py` to that list,
or giving the renderer its own scope-declaring L0 test, closes it. I did not do it here because
it widens an already review-blocked PR into a third subject; it is the highest-value follow-up
in this entry.
- **Log-derived counts are a weak instrument and I did not strengthen them.** `kubectl logs
--since=` returns what the kubelet still retains, so an old, busy container's window may be
truncated; and `No valid authentication in request` counts unauthenticated first hits and XHR
polls indiscriminately. The `cookie signature not valid` / `removing session` pair is the only
collision-specific signal in there, and it is the one I leaned on. None of this is gated —
there is no alert on it and no L2 check reads it.
- **The cookie-secrets' distinctness is not gated either.** It was measured out of band and is
asserted nowhere. If they were ever made identical the collision would become harmless — but
sharing one session-encryption key across trust zones is a worse idea than the bug it would mask,
so the check does not offer it as an alternative and does not look for it.
- **The shared proxy's own configuration is unchanged and untested by me.** I did not touch
`platform/oauth2-proxy/**` (owned by another agent this cycle); the check only reads it.
- **Nothing about group authorization, CSRF behaviour, or session contents.** Names and domains only.
- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so the PR's green tick — if it goes green —
proves the L0/L1 offline subset and nothing about runtime.