It is a property of the pattern, and this branch found where the pattern actually lives.
**Read the scope claim carefully before the detail.** This branch fixes a *measured collision*: six
independent oauth2-proxy instances, seven different cookie-secrets, one cookie name, one zone-wide
domain. It does **not** claim to fix the operator's reported sign-in failures. Those are two
separate claims, only the first is supported by anything here, and the evidence for why the second
is *not* supported — including a fair reading of counter-evidence that looked bad for this fix — is
in "What I verified". Do not merge this expecting the auth-rejection volume to drop.
**Read the scope claim carefully before the detail.** This branch fixes THREE defects in the
forward-auth sidecar contract, all in one function, all landing on all five apps at once:
| Defect | Fix | Measured evidence |
|---|---|---|
| Six independent instances, seven different cookie-secrets, **one** cookie name on **one** zone-wide domain — so they overwrite and `ClearSession`-delete each other's sessions | `--cookie-name=_oauth2_proxy_<app>` + host-scoped `--cookie-domain` | 7/7 distinct secrets; 6 instances on the default name; radarr logs the eviction fingerprint 8× in 12h |
| The session dies **5 minutes** after sign-in — `--cookie-refresh` defaults to 0 = *disabled*, so the refresh token the proxy already holds is never spent | `--cookie-refresh=2m30s` (half the lifetime) | access-token lifetime = **300.000s exactly**, `refresh_token:true`, flag absent |
| **No** per-app sidecar serves `/metrics` at all — without `--metrics-address` the listener never starts | `--metrics-address=0.0.0.0:44180` + a `metrics` container port | `ports` was `['http']` on all six |
Rejection *volume* is dominated by the second defect, not the first: ~95% of radarr's rejections
carry no cookie at all. **Do not merge this expecting one number to drop** — three mechanisms were
tangled together, and separating them is most of the value here.
**Boundary note.** My brief originally reserved `apps/wave-0/jackett.yaml` for a concurrent branch.
That split is not achievable and has been revoked: the file is generated, the fix is one renderer
edit, and `render-golden` requires all five goldens regenerated in the *same commit*. I own all
five. The proof I ran before touching jackett is under "What I verified".
**The defect was not in the four manifests it was reported in.**`apps/wave-0/pihole.yaml`,
`apps/wave-0/qbittorrent.yaml`, `apps/wave-1/radarr.yaml`, `apps/wave-1/sonarr.yaml` — and
@@ -84,12 +95,13 @@ instances that accidentally share a name cannot see each other's cookie at all.
on the exact axis that already failed once. The escape hatch is the same register, by name and
with a reason, rather than a loosened rule.
### The jackett collision — read this
### The jackett boundary, and why it could not hold
`apps/wave-0/jackett.yaml` carries the identical defect and was assigned to **Mira Vance** on
`fix/jackett-oauth2-proxy-session`, with an instruction to me not to touch it. **That separation is
not achievable**, and I did not discover this by reasoning about it — I provoked it. With the
renderer fixed and jackett's golden left at its `main` content,`./ci/render-golden.sh` exits **1**:
`apps/wave-0/jackett.yaml` carries the identical defect and was initially assigned to a concurrent
branch, with an instruction to me not to touch it. **That separation is not achievable** — the
ownership split was drawn across a generated file — and I did not discover this by reasoning about
it, I provoked it. With the renderer fixed and jackett's golden left at its `main` content,
`./ci/render-golden.sh` exits **1**:
[fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render:
- - --cookie-domain=.${BASE_DOMAIN}
@@ -105,37 +117,45 @@ exists to forbid, and leaves jackett broken *by design*); or fix the generator a
five. I took the third. Jackett's golden here is **machine output of a one-line generator change**,
not an independent edit of Mira's file, and it is byte-identical to what her branch must produce.
**Consequence for the PM:**`fix/jackett-oauth2-proxy-session` has no work left in it. As of this
writing that branch has **zero commits ahead of `main`** and is not pushed, so nothing is lost —
but if Mira has since hand-edited `apps/wave-0/jackett.yaml`, her branch will fail `render-golden`
on its own terms and should be closed rather than rebased.
**Consequence:**the jackett branch has no cookie work left in it — this commit fixes jackett
identically and by the same mechanism, and both branches independently arrived at the *same two
values* (`_oauth2_proxy_jackett`, `jackett.${BASE_DOMAIN}`), which is a good sign for the
diagnosis. Its hand-edit of the rendered file will fail `render-golden` on its own terms, so it
should be closed rather than rebased. **What must not be lost with it** is its evidence: the
interleaved radarr/jackett log showing `removing session` deleting a clean session 26s after login,
and the `expires − created = 300.000s` measurement that produced `--cookie-refresh` here. Both are
carried into this entry and into the renderer's comments; the branch itself is now redundant, the
work behind it is not.
### Scope: what this branch deliberately does NOT fix
**The general lesson, which is worth more than this bug:** an ownership split across
`apps/wave-*/*.yaml` cannot hold, because those files are *outputs*. Two agents assigned two
rendered apps are assigned the same function and will collide in `render-golden` no matter how
carefully they stay in their own files. Ownership of a rendered app is ownership of
`catalog/render.py`, and briefs should say so.
Three further defects were raised against these sidecars while this branch was in flight. All
three are real; none is fixed here, and each omission is a decision, not an oversight.
### What this branch deliberately does NOT do
- **No `--cookie-refresh`, so the session dies in 5 minutes.** The session cookie's `expires` is
the Keycloak access-token lifetime; without `--cookie-refresh` oauth2-proxy never spends the
refresh token it already holds. **All four of my apps need this exactly as much as jackett
does** — same realm, same renderer, same function. It is not here because it is a *different
mechanism* (session lifetime, not cookie identity) that would be un-revertable independently
of the collision fix; because the correct VALUE is a judgement call belonging to whoever
measured it; and because nothing in my check guards it, so I would be shipping an unguarded
behaviour change. **When it lands it belongs in `_oauth2_sidecar_container()`, not in
jackett's golden** — one line there fixes all five; a hand-edit fixes one and fails
`render-golden`.
- **Unauthenticated XHR gets a 302, not a 401.** Same reasoning, plus one of its own: the fix
(`--api-routes`) is *path-shaped and per-app* (`^/api/` for the *arr and qBittorrent; pihole's
admin API is shaped differently), so doing it properly means declaring API path prefixes in the
ServiceDefinition — a schema change, well outside this branch.
- **No per-app sidecar exposes metrics at all** (no `--metrics-address`, no `metrics` port, so
the listener never starts and nothing is scrapable). I considered asserting this in the new
check and **rejected it as overreach**: the check's subject is "no two instances can share a
cookie", and a metrics assertion shares neither its inputs nor its failure mode. Bundling them
makes the check's name a lie and couples two unrelated ratchets, so a future metrics decision
could only be expressed by weakening a *cookie* check. `servicemonitor-ports-resolve` and
`alert-metrics-have-scrape-targets` already exist and are the right homes.
- **`--api-route=^/api/` is NOT added**, and not because of scope discipline alone: it would be
**wrong** renderer-wide. It was measured on jackett, and **PiHole does not serve `/api/`** — so
a renderer constant applies one app's path shape to five. Doing it properly means declaring API
path prefixes per service, and the `auth` block is `additionalProperties: false`, so that is a
*schema change*. Separate PR. (Note the flag is **`--api-route`, singular** — verified in the
pinned binary's own `--help`. The plural `--api-routes` does not exist in v7.8.1 and the proxy
refuses to start with it. That error was propagating through briefs; it stops here.)
- **`--session-cookie-minimal` is NOT added** and must not be, as a cookie-size fix: it is
**mutually exclusive** with `--cookie-refresh` (*"cookie_refresh > 0 requires oauth tokens in
sessions"*). Dropping `--pass-access-token` is a least-privilege argument, never a size one.
- **Metrics are NOT scraped, only served.** The flag and container port are the prerequisite;
a container port is not a Service port. These Services expose the app port and `:4180`, and each
namespace's default-deny baseline is `ingress: []`, so Prometheus can neither resolve nor reach
`:44180` until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
exist. That wiring is someone else's follow-up and I have deliberately not faked its start.
**Nobody should read this commit as "the sidecars have metrics now."**
- **The new check was NOT widened to cover cookie-refresh or metrics.** Its subject is cookie
*identity*; those two share neither its inputs nor its failure mode. Bundling them would make
the check's name a lie and couple unrelated ratchets, so a future metrics decision could only
be expressed by weakening a *cookie* check. The cost of that discipline is named honestly in
the follow-ups: those two flags are currently **ungated**.
## What I verified
@@ -177,32 +197,73 @@ three are real; none is fixed here, and each omission is a decision, not an over
not valid`. Window `--since=24h`, which exceeds every pod's age only for crafty — whose pod is
13.9h old with **0 restarts**, so these are its complete post-fix figures.
Three conclusions, kept separate:
1.**`crafty 81:1` is a window artefact and must not be used as evidence against the fix.**
The fixed pod has emitted **3** rejections in its entire life. The 81 came from a 24h Loki
window straddling a 13.9h-old pod, i.e. overwhelmingly from the PRE-fix pod.
2.**The fix is still NOT shown to work.** Crafty logged **0 `AuthSuccess`** in that window —
nobody signed in — so there is no post-fix ratio to evaluate at all. Absence of failures
against absence of traffic proves nothing. This remains the open gap, not a resolved one.
3. **The collision mechanism is independently confirmed, and it is a MINORITY of the
rejections.** `cookie signature not valid` + `removing session` is the collision's
fingerprint — a cookie *is* present, written by an instance whose key this one does not
have, so it cannot decrypt it and then `ClearSession` expires it on its own
`--cookie-domain` (the whole zone). radarr: **8 of 155** rejections (5%) carry it; jackett
1 of 39; crafty, fixed, **0**. So the collision is real and this branch addresses it — and
~95% of radarr's rejections carry no cookie at all, which points at the session-lifetime
and 302-vs-401 mechanisms above, not at this fix. **Anyone expecting these five renames to
fix the rejection volume will be disappointed, and should not be.**
Then I read crafty's four auth-relevant lines rather than counting them, because at n=3 the
content is the evidence:
`reject` is also not a clean SLI in either direction: it counts every unauthenticated first
hit and every XHR poll, so a busy *arr inflates it without any user being inconvenienced.
`conformance/lib/<tool> --selftest` as a subprocess for exactly this reason — its own header
says it does so "because nothing [else] drives them". Adding `catalog/render.py` to that list,
or giving the renderer its own scope-declaring L0 test, closes it. I did not do it here because
it widens an already review-blocked PR into a third subject; it is the highest-value follow-up
in this entry.
- **Log-derived counts are a weak instrument and I did not strengthen them.** `kubectl logs
--since=` returns what the kubelet still retains, so an old, busy container's window may be
truncated; and `No valid authentication in request` counts unauthenticated first hits and XHR
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.