chore(decisions): record the CI reading, and why the metrics policy is NOT here

Follow-up to the branch's one substantive commit; no code change.

CI on 99560c2d, read from the commit's /statuses rather than from a tick:
  gate / gate (pull_request)         state=failure   desc='Failing after 1m44s'
  gate / conformance (pull_request)  state=pending   desc='Blocked by required conditions'
The second line is PENDING, not the `success` + "Has been skipped" trap -- the
conformance job is gated on the gate job, which failed, so it never ran.

MY LOCAL `approval-policy: green` WAS MEANINGLESS AND I NEARLY REPORTED IT AS
EVIDENCE. ci/approval-policy.sh:47 reads GITHUB_EVENT_NAME, unset outside CI, so
it short-circuits to "N/A": a clean local gate STRUCTURALLY cannot catch a
sensitive-path problem. Verified by re-running with
GITHUB_EVENT_NAME=pull_request PR_NUMBER=34, where the stage evaluates and fails
CLOSED on the missing token while every other stage stays green -- which bounds
the CI failure to that one stage. I could NOT read CI's job log to confirm it
from CI's own output: this Gitea answers 404 on /actions/runs and on
/actions/jobs/{task}/logs, so only /actions/tasks is available. Attribution
therefore rests on the local reproduction plus the sensitive-path rule, not on
CI's log -- weaker evidence than PR #28 had, named rather than papered over.

Branch protection and the runner re-measured against the live API (CLAUDE.md
2b), not inherited from the CLAUDE.md table: protected=True,
user_can_push=False, enable_status_check=True, contexts=['gate / gate
(pull_request)'], required_approvals=1; runner id=1 incluster-dind online. So a
human is required on every PR here regardless of path. I did NOT provoke a
rejected push to main, so user_can_push is the API's word only.

DECLINED, with reasons: bundling Ines Dubois's metrics-ingress NetworkPolicy.
The coordinator asked for my judgement and the answer is to queue it as the next
PR, not to drop it. Principally because it is a MODEL question, not a policy to
bolt on: schema 1.6.0's observability.metricsEndpoint already renders the
monitoring-namespace ingress allow, and what the model cannot express is a
SECOND metrics endpoint belonging to a sidecar the RENDERER injected. Keying a
bespoke policy off the pod label auth-mode=forward-auth hardcodes "forward-auth
implies 44180" instead of deriving it -- the "list, don't derive" anti-pattern
oauth2-proxy-cookie-names-distinct assertion (5) forbids, one directory from the
check that forbids it. Also: it is a trust-boundary ingress allow and a
different requirement set (R-NET-01/02, R-OBS-01 vs R-IDP-02), so it should not
be reviewed in the shadow of an --api-route headline; the conflict argument does
not apply to my diff, which regenerates TWO goldens not five, and a
CiliumNetworkPolicy is a different document in files git merges independently;
every fact in the finding is hers and I have measured none of it; and the
PodMonitor it serves is in HER PR #29, so shipping the policy alone would open
an ingress allow for a scraper that selects nothing.

Also records that the sidecars' metrics remain UNSCRAPABLE and that this branch
does not change that.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/wave-0/jackett.yaml

@@ -100,6 +100,7 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
+        - --api-route=^/api/
         - --cookie-name=_oauth2_proxy_jackett
         - --cookie-domain=jackett.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}

apps/wave-1/radarr.yaml

@@ -153,6 +153,8 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
+        - --api-route=^/api/
+        - --api-route=^/signalr/
         - --cookie-name=_oauth2_proxy_radarr
         - --cookie-domain=radarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}

catalog/jackett.yaml

@@ -51,6 +51,23 @@ auth:
     # This grants no real privilege: platform-admins already maps to cluster-admin via
     # k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
     - platform-admins           # operators of these services must not be locked out of them
+  # Answer Jackett's OWN API with 401 instead of a 302 to Keycloak (schema 1.9.0; renders
+  # --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # MEASURED on this pod's access log, 2026-09-30: on /api/ paths, 33 x 302 against 2 x 401 —
+  # and 4 of 6 POSTs were answered BY THE PROXY and never reached Jackett (access-log upstream
+  # field `-`, not `/`), including both `/api/v2.0/server/update` calls. Jackett's jQuery does
+  # not send the exact `Accept: application/json` oauth2-proxy's AJAX guard requires, so the
+  # 302 was followed invisibly, satisfied silently by the live Keycloak SSO session, and
+  # resolved 200 carrying the dashboard's HTML where the JavaScript expected JSON. A browser
+  # following a 302 also re-issues a POST as a GET and DROPS THE BODY, which is why "adding an
+  # indexer silently does nothing" was the user-visible symptom and nothing logged a failure.
+  #
+  # SCOPE: `/api/` only. Jackett serves its UI at /UI/ and its static assets elsewhere, and a
+  # browser never NAVIGATES to /api/ — so a plain page load is still 302'd into a silent SSO
+  # sign-in, which is the behaviour that must not change.
+  apiRoutes:
+    - ^/api/
 update:
   channel: stable               # R-UPD-01
   range: ">=0.22.0"             # stay on the Jackett 0.22.x stable line

catalog/pihole.yaml

@@ -107,6 +107,13 @@ auth:
   mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
   groups:
     - platform-admins         # sensitive admin UI: authenticated AND in group (R-IDP-06)
+  # NO `apiRoutes` HERE, DELIBERATELY. PiHole's admin UI serves no /api/ path, so the `^/api/`
+  # that jackett and radarr need would match nothing on a good day — and a BROADER guess would
+  # answer PiHole's own page loads with 401, leaving a logged-out operator on a bare error page
+  # with no sign-in offered. This app is the reason auth.apiRoutes is per-service (schema 1.9.0)
+  # rather than a renderer constant: the measurement that justifies the flag on two apps is
+  # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
+  # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
 update:
   channel: stable             # R-UPD-01
   range: ">=2024.0.0"

catalog/qbittorrent.yaml

@@ -448,6 +448,12 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE: UNMEASURED, not cleared. qBittorrent's WebUI drives everything through
+  # /api/v2/, so it is plausibly affected in the same way jackett is — but plausibility is what
+  # put a renderer constant on the table in the first place. MEASURED 2026-10-01 on the live pod
+  # (qbittorrent-5547bdb9bd-zq557, 0 restarts, 23h): 13,649 log lines, ZERO unauthenticated
+  # requests, nothing but health probes. No traffic means no evidence either way. Add `^/api/`
+  # when someone opens the WebUI and records the 302/401 split, as was done for jackett.
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the qBittorrent 5.x stable line

catalog/radarr.yaml

@@ -112,6 +112,32 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # Answer Radarr's machine endpoints with 401 instead of a 302 to Keycloak (schema 1.9.0;
+  # renders --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # `^/signalr/` — MEASURED ON THE LIVE POD, 2026-10-01, 23h after the cookie fix deployed
+  # (radarr-7d55cbffb9-42hmh, 0 restarts): 232 unauthenticated requests in the pod's whole
+  # lifetime and ALL 232 are the same call — `POST /signalr/messages/negotiate` from a browser
+  # UA, answered 302 by the proxy with upstream `-`, i.e. Radarr never saw one of them. That is
+  # a left-open tab whose session the cookie rename orphaned, polling SignalR forever, silently
+  # redirected every time and NEVER re-prompting the operator. It is the reported symptom
+  # ("reloading never fixes it and I am never re-prompted") still running in production on a
+  # service nobody was watching. With 401 the UI's connection breaks LOUDLY, the operator
+  # reloads, and `/` is outside both regexes so the reload signs them back in.
+  #
+  # `^/api/` — measured on radarr, but on the PRE-FIX pod and NOT by me: the 2026-09-30
+  # diagnosis (decisions/2026-09-30-jackett-oauth2-proxy-session.md) recorded radarr's 156
+  # unauthenticated requests as `/api/v3/command` (62), `/api/v3/queue/details` (46) and
+  # `/signalr/messages/negotiate` (43). The current pod has logged ZERO /api/ traffic, so I did
+  # not re-measure it; it is included because 108 of 156 measured rejections on this app were
+  # /api/v3 XHRs, which is this app's own evidence rather than an analogy to another app.
+  #
+  # SCOPE: both are JSON/RPC namespaces a browser never NAVIGATES to. Radarr's UI is served at
+  # `/` with assets under /Content/ and /MediaCover/, none of which match — so a plain page load
+  # is still 302'd into a silent SSO sign-in.
+  apiRoutes:
+    - ^/api/
+    - ^/signalr/
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the Radarr v5 stable line

catalog/render.py

@@ -172,6 +172,45 @@ OAUTH2_PROXY_METRICS_PORT = 44180
 DRI_VOLUME_NAME = "dri"
 DRI_DEVICE_DIR = "/dev/dri"
 
+# --- API routes: 401 instead of 302, PER APP ------------------------------------------
+# oauth2-proxy answers an unauthenticated request with a 302 to Keycloak. Its built-in AJAX
+# guard would send a 401 instead, but it fires only on an EXACT `Accept: application/json`,
+# which these apps' JavaScript does not consistently send. So an unauthenticated XHR gets a
+# 302, the browser follows it INVISIBLY, the live Keycloak SSO session satisfies it silently,
+# and the call resolves 200 carrying the app's HTML where the JavaScript expected JSON — no
+# prompt, no console error, nothing logged as a failure. On a WRITE it is worse: a browser
+# following a 302 re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
+# discarded and the UI reports success. MEASURED, twice:
+#   jackett  2026-09-30: 33 x 302 against 2 x 401 on /api/ paths, and 4 of 6 POSTs answered by
+#            the proxy and never reaching the app (access-log upstream field `-`, not `/`).
+#   radarr   2026-10-01, 23h after the cookie fix rolled: 232 x 302, EVERY ONE a
+#            `POST /signalr/messages/negotiate` from the operator's own browser, upstream `-`.
+#            A left-open tab whose cookie the rename orphaned, polling forever, silently
+#            redirected every time, NEVER re-prompting. That is the reported symptom
+#            ("reloading never fixes it and I am never re-prompted") still happening in
+#            production, on a service nobody was looking at.
+# `--api-route=<regex>` makes matching paths answer 401.
+#
+# WHY THIS IS NOT A RENDERER CONSTANT, AND MUST NEVER BECOME ONE. The path shape is a property
+# of the APP, not of the forward-auth pattern: radarr's broken path is `/signalr/`, jackett's is
+# `/api/`, and PIHOLE SERVES NO `/api/` AT ALL. One constant would apply one app's path shape to
+# five, and the failure mode of a wrong one is not "no improvement" but "this service is
+# permanently broken": a path a BROWSER NAVIGATES to, answered 401, shows a logged-out operator
+# a bare error page and never offers a sign-in. So the routes are declared per service in
+# `auth.apiRoutes` (schema 1.9.0) and rendered one `--api-route` per entry — the flag is
+# repeatable, and it is SINGULAR: `--api-routes` does not exist in v7.8.1 and the proxy refuses
+# to start with it (verified against the pinned binary's own --help, 2026-10-01:
+# `--api-route strings   return HTTP 401 instead of redirecting to authentication server...`).
+#
+# BREADTH RATCHET. Every entry must be anchored at `^/` and continue with literal path
+# characters. That rejects `^/`, `^/.*`, `.*`, an unanchored `/api/`, and any alternation — the
+# shapes that turn a page load into a 401. Alternation is refused rather than supported because
+# the flag is repeatable, so one regex per route costs nothing and stays readable. The same
+# pattern is in the schema (so ci/l0.sh's schema stage rejects it at the authoring surface) and
+# enforced HERE (so ci/render-golden.sh fails even if the schema stage is skipped) — a breadth
+# mistake must not need two independent gates to both be running.
+OAUTH2_API_ROUTE_PATTERN = re.compile(r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$")
+
 
 # =============================================================================
 # Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
@@ -197,6 +236,12 @@ def apply_defaults(sd: dict) -> dict:
     if "mode" not in auth:
         auth["mode"] = "forward-auth" if s["ingressRoutes"] else "none"
     auth.setdefault("groups", [])
+    # apiRoutes default posture is EMPTY, deliberately, and that is the whole point of the field
+    # (schema 1.9.0): an app gets --api-route only where someone MEASURED that it serves a machine
+    # API, so a newly-onboarded app renders no --api-route at all rather than inheriting another
+    # app's path shape. See OAUTH2_API_ROUTE_PATTERN above for why a default of `^/api/` would be
+    # wrong rather than merely generous.
+    auth.setdefault("apiRoutes", [])
     s["auth"] = auth
 
     update = dict(s.get("update") or {})
@@ -1054,7 +1099,12 @@ def _oauth2_sidecar_container(s: dict) -> dict:
 
     --cookie-name and --cookie-domain are DERIVED per app (see the block comment on those two
     args): every instance holds a different cookie-secret, so sharing one cookie name across the
-    zone makes the instances evict each other's sessions."""
+    zone makes the instances evict each other's sessions.
+
+    --api-route is DECLARED per app, from auth.apiRoutes, and is the one flag here that is
+    neither a constant nor derivable: it names the paths THIS app serves as a machine API, and
+    those differ per app (radarr `/signalr/`, jackett `/api/`, pihole none). An app that declares
+    none renders no --api-route at all."""
     route = s["ingressRoutes"][0]
     host = route["host"]
     upstream_port = route["port"]
@@ -1074,6 +1124,27 @@ def _oauth2_sidecar_container(s: dict) -> dict:
     args += [
         "--skip-provider-button=true",
         f"--redirect-url=https://{host}/oauth2/callback",
+    ]
+    # ── API ROUTES: 401 instead of 302, ON THIS APP'S OWN PATHS ONLY ───────────────────────
+    # One flag per declared route, in declared order. An app that declares NONE renders NO
+    # --api-route, which is the correct and common case — see OAUTH2_API_ROUTE_PATTERN above for
+    # the measurements, for why this cannot be a renderer constant, and for the breadth ratchet.
+    # Validated here as well as in the schema so a breadth mistake is caught by render-golden
+    # even on a run where the schema stage did not execute.
+    for route in s["auth"].get("apiRoutes", []):
+        if not OAUTH2_API_ROUTE_PATTERN.match(str(route)):
+            raise SystemExit(
+                f"render.py: {s['name']}: auth.apiRoutes entry {route!r} is not an anchored path "
+                f"regex. Each entry must start with '^/' and continue with literal path "
+                f"characters (e.g. '^/api/', '^/signalr/', '^/api/v3/.*'). Rejected shapes: '^/' "
+                f"and '^/.*' (match every page load), an unanchored '/api/' (matches mid-path), "
+                f"and alternation (the flag is repeatable — declare one entry per route). A "
+                f"too-broad --api-route answers a PAGE LOAD with 401, so a logged-out operator "
+                f"sees a bare error page and is never offered a sign-in: the service looks "
+                f"permanently broken rather than merely unauthenticated."
+            )
+        args.append(f"--api-route={route}")
+    args += [
         # ── COOKIE ISOLATION (fleet fix 2026-09-30; crafty's 2026-09-29 fix, generalized) ──────
         # EVERY oauth2-proxy on this platform — this sidecar, its four siblings, and the SHARED
         # forward-auth proxy (platform/oauth2-proxy) — is an INDEPENDENT instance holding its OWN
@@ -2666,15 +2737,27 @@ def _selftest() -> int:
             "auth": {"mode": "forward-auth", "groups": ["media-users"]},
         }
     )
-    fa_dep = next(d[1] for d in fa_docs if d[1]["kind"] == "Deployment")
-    fa_containers = fa_dep["spec"]["template"]["spec"]["containers"]
+    fa_dep = next((d[1] for d in fa_docs if d[1]["kind"] == "Deployment"), None)
+    check(fa_dep is not None,
+          "a forward-auth app rendered NO Deployment, so there is no pod to gate at all")
+    fa_containers = (((fa_dep or {}).get("spec") or {}).get("template") or {}) \
+        .get("spec", {}).get("containers") or []
     check(
         len(fa_containers) == 2
         and fa_containers[0]["name"] == "fa"
         and fa_containers[1]["name"] == "oauth2-proxy",
-        "forward-auth app must render the app container FIRST + an oauth2-proxy sidecar SECOND",
+        f"forward-auth app must render the app container FIRST + an oauth2-proxy sidecar SECOND; "
+        f"rendered {[c.get('name') for c in fa_containers]!r}",
     )
-    side = fa_containers[1]
+    # EVERY assertion below reads `side`, so an absent sidecar must NOT be allowed to raise here.
+    # The whole point of the None-tolerant lookups in this block is that a missing sidecar (or a
+    # missing flag on it) reports as NAMED ASSERTIONS rather than as a traceback: an exception
+    # escaping _selftest() exits non-zero with the right verdict and ZERO diagnostics, and it
+    # hides every other assertion the same mutation broke. Measured by Desmond Oyelaran while
+    # wiring this selftest into the render-golden stage: deleting the --cookie-name emission gave
+    # rc 1 via an uncaught StopIteration and printed not one `SELFTEST FAIL:` line. Provoked
+    # again here with the sidecar removed entirely, which raised IndexError at this line.
+    side = fa_containers[1] if len(fa_containers) > 1 else {"args": [], "env": [], "ports": []}
     check(
         "--upstream=http://127.0.0.1:9117" in side["args"],
         "sidecar upstream must be the co-located app UI over http on 127.0.0.1:<uiport>",
@@ -2743,12 +2826,101 @@ def _selftest() -> int:
         "sidecar must declare BOTH the http and metrics container ports — an undeclared port "
         "cannot be named by a Service port or a PodMonitor, so the flag alone is inert",
     )
+    # API ROUTES (schema 1.9.0). Three properties, and the FIRST is the one that keeps the field
+    # honest: an app that declares nothing must render NO --api-route. `fa` above declares none.
+    check(
+        not any(a.startswith("--api-route") for a in side["args"]),
+        "a forward-auth app that declares no auth.apiRoutes must render NO --api-route. A "
+        "default or a renderer constant would answer PiHole's page loads with 401 — PiHole "
+        "serves no /api/ at all — so the service would look permanently broken to a "
+        "logged-out operator instead of signing them in",
+    )
+    # Same `next(..., None)` discipline as _cookie_name() below, for the same reason: if the
+    # sidecar stops being rendered at all, this must report as a named assertion rather than as a
+    # StopIteration that swallows every other SELFTEST FAIL line in the run.
+    _faapi_dep = next(
+        (
+            d[1]
+            for d in render(
+                {
+                    "name": "faapi",
+                    "source": {"image": "a:1"},
+                    "ingressRoutes": [{"host": "faapi.home.keithjackson.dev", "port": 9119}],
+                    "auth": {
+                        "mode": "forward-auth",
+                        "groups": ["media-users"],
+                        "apiRoutes": ["^/api/", "^/signalr/"],
+                    },
+                }
+            )
+            if d[1]["kind"] == "Deployment"
+        ),
+        None,
+    )
+    fa_api_side = next(
+        (
+            c
+            for c in ((_faapi_dep or {}).get("spec", {}).get("template", {})
+                      .get("spec", {}).get("containers") or [])
+            if c.get("name") == "oauth2-proxy"
+        ),
+        None,
+    )
+    check(
+        fa_api_side is not None,
+        "a forward-auth app declaring auth.apiRoutes rendered NO oauth2-proxy sidecar, so the "
+        "routes have nowhere to land and the app is not gated at all (R-IDP-02)",
+    )
+    check(
+        fa_api_side is not None
+        and [a for a in fa_api_side["args"] if a.startswith("--api-route")]
+        == ["--api-route=^/api/", "--api-route=^/signalr/"],
+        "a declared auth.apiRoutes must render as one repeatable --api-route per entry, in "
+        "declared order, and NOTHING else. Note the flag is SINGULAR: --api-routes does not "
+        "exist in v7.8.1 and the proxy refuses to start with it",
+    )
+    # BREADTH. The renderer must REFUSE a regex that would match a page load, not merely
+    # discourage one, because the schema stage can be skipped and render-golden cannot.
+    for _bad in ("^/", "^/.*", ".*", "/api/", "^/api/|^/"):
+        try:
+            render(
+                {
+                    "name": "fabad",
+                    "source": {"image": "a:1"},
+                    "ingressRoutes": [{"host": "fabad.home.keithjackson.dev", "port": 9120}],
+                    "auth": {
+                        "mode": "forward-auth",
+                        "groups": ["media-users"],
+                        "apiRoutes": [_bad],
+                    },
+                }
+            )
+            _refused = False
+        except SystemExit:
+            _refused = True
+        check(
+            _refused,
+            f"the renderer ACCEPTED auth.apiRoutes entry {_bad!r}. A too-broad --api-route "
+            f"answers a plain page load with 401, so a logged-out operator sees a bare error "
+            f"page and is never offered a sign-in — the service looks permanently broken "
+            f"rather than merely unauthenticated, which is a worse failure than the 302 bug "
+            f"this flag exists to fix",
+        )
+    for _good in ("^/api/", "^/signalr/", "^/api/v3/.*", "^/api/v2.0/server/update$"):
+        check(
+            bool(OAUTH2_API_ROUTE_PATTERN.match(_good)),
+            f"the breadth ratchet rejected the legitimate route {_good!r}. A ratchet that "
+            f"blocks real API prefixes is a ratchet someone eventually deletes",
+        )
+
     # Two forward-auth apps must not be able to collide. Renders a SECOND one and compares, so a
     # future refactor to a constant (or to a per-service table with a copy-paste slip) fails here
     # rather than silently in a browser six weeks later.
-    fa2_side = next(
-        c
-        for c in next(
+    # Same None-tolerant shape as the `faapi` fixture above and for the same reason: a missing
+    # sidecar must surface as a named assertion, never as a traceback that suppresses the
+    # collected SELFTEST FAIL lines.
+    _fa2_dep = next(
+        (
             d[1]
             for d in render(
                 {
@@ -2759,24 +2931,55 @@ def _selftest() -> int:
                 }
             )
             if d[1]["kind"] == "Deployment"
-        )["spec"]["template"]["spec"]["containers"]
-        if c["name"] == "oauth2-proxy"
+        ),
+        None,
+    )
+    fa2_side = next(
+        (
+            c
+            for c in ((_fa2_dep or {}).get("spec", {}).get("template", {})
+                      .get("spec", {}).get("containers") or [])
+            if c.get("name") == "oauth2-proxy"
+        ),
+        {"args": []},
     )
 
+    # `next(..., None)` — the DEFAULT IS LOAD-BEARING, not defensive style. With a bare
+    # `next(genexpr)` this raises StopIteration the moment a sidecar renders no --cookie-name,
+    # which is precisely the mutation this assertion exists to catch. The traceback then escapes
+    # _selftest() before its collected `SELFTEST FAIL:` lines are printed, so the run exits
+    # non-zero with the RIGHT VERDICT AND NO DIAGNOSTICS — and it masks every other assertion the
+    # same mutation broke. Measured by Desmond Oyelaran while wiring the selftest into the
+    # render-golden stage: deleting the --cookie-name emission gave rc 1 via an uncaught
+    # StopIteration and suppressed all of the messages. A missing flag must report AS AN
+    # ASSERTION, with its name in the output, which is what the None + explicit check below do.
     def _cookie_name(container):
-        return next(a for a in container["args"] if a.startswith("--cookie-name="))
+        return next((a for a in container["args"] if a.startswith("--cookie-name=")), None)
 
+    _fa_cn, _fa2_cn = _cookie_name(side), _cookie_name(fa2_side)
     check(
-        _cookie_name(side) != _cookie_name(fa2_side),
+        _fa_cn is not None and _fa2_cn is not None,
+        f"a forward-auth sidecar rendered NO --cookie-name at all (fa={_fa_cn!r}, "
+        f"fa2={_fa2_cn!r}). Omitting the flag is not neutral: oauth2-proxy then claims its "
+        f"built-in default `_oauth2_proxy`, which every other instance on the platform also "
+        f"claims, so they overwrite and ClearSession-delete each other's sessions",
+    )
+    check(
+        _fa_cn != _fa2_cn,
         "two forward-auth apps rendered IDENTICAL cookie names — they would overwrite each "
         "other's session on every sign-in (the 2026-09-29 crafty defect, fleet-wide)",
     )
+    # Index-safe for the same reason as `side` above: an absent sidecar has no env, and an
+    # IndexError here would exit non-zero with no diagnostics at all.
+    _fa_env = (side.get("env") or [{}])[0]
     check(
-        side["env"][0]["valueFrom"]["secretKeyRef"]["name"] == "fa-oauth2-proxy",
+        (((_fa_env.get("valueFrom") or {}).get("secretKeyRef") or {}).get("name")
+         == "fa-oauth2-proxy"),
         "sidecar creds must come by name from the per-app <app>-oauth2-proxy Secret (R-IDP-04)",
     )
-    fa_svc = next(d[1] for d in fa_docs if d[1]["kind"] == "Service")
-    fa_svc_ports = {p["port"] for p in fa_svc["spec"]["ports"]}
+    fa_svc = next((d[1] for d in fa_docs if d[1]["kind"] == "Service"), None)
+    check(fa_svc is not None, "a forward-auth app must render a Service for the Gateway to reach")
+    fa_svc_ports = {p["port"] for p in (((fa_svc or {}).get("spec") or {}).get("ports") or [])}
     check(
         fa_svc_ports == {9117, 4180},
         "forward-auth Service must expose the app port (east-west) AND :4180 (gateway->sidecar)",

catalog/sonarr.yaml

@@ -112,6 +112,14 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE, AND NOT BECAUSE IT NEEDS NONE — because nothing measures that it does.
+  # Sonarr is nearly the same codebase as Radarr, which is exactly what makes `^/api/` +
+  # `^/signalr/` a tempting guess and still a guess. MEASURED 2026-10-01 on the live pod
+  # (sonarr-ddf485775-8ckwt, 0 restarts, 23h): its oauth2-proxy log contains 13,643 lines and
+  # ZERO unauthenticated requests — no /api/, no /signalr/, nothing but health probes. Nobody
+  # has opened this UI since the cookie fix rolled, so the measurement is empty rather than
+  # negative: it is evidence of no TRAFFIC, not evidence of no need. Add the routes when someone
+  # reproduces the 302-on-XHR here, with the numbers.
 update:
   channel: stable               # R-UPD-01
   range: ">=4.0.0 <5.0.0"       # stay on the Sonarr v4 stable line

conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/chainsaw-test.yaml

@@ -0,0 +1,134 @@
+# =============================================================================
+# L0 STATIC — `--api-route` is PER APP, ANCHORED, and DECLARED
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. oauth2-proxy answers an unauthenticated request with a 302 to
+# Keycloak. Its built-in AJAX guard would send a 401 instead, but it fires only on an EXACT
+# `Accept: application/json`, which these apps' JavaScript does not consistently send. So a
+# browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied silently by the live Keycloak
+# SSO session, and resolves 200 carrying the app's HTML where the JavaScript expected JSON — no
+# prompt, no console error, nothing logged as a failure. On a WRITE it is worse: a browser
+# following a 302 re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
+# discarded and the UI reports success.
+#
+# MEASURED, twice, on two different apps and two different paths:
+#   jackett 2026-09-30 — on /api/ paths: 33 x 302 against 2 x 401, and 4 of 6 POSTs answered BY
+#           THE PROXY, never reaching Jackett (access-log upstream field `-`, not `/`). That is
+#           the operator's whole report: "GET works, most PUT/POST to add indexers fail,
+#           reloading never fixes it, and I am never re-prompted for login".
+#   radarr  2026-10-01, 23h after the cookie fix rolled (pod radarr-7d55cbffb9-42hmh, 0
+#           restarts) — 232 unauthenticated requests in the pod's whole lifetime and ALL 232 are
+#           the same call: `POST /signalr/messages/negotiate` from a browser UA, 302, upstream
+#           `-`. A left-open tab whose session the cookie rename orphaned, polling SignalR
+#           forever, silently redirected every time, NEVER re-prompting. The reported symptom,
+#           still running in production, on a service nobody was watching.
+#
+# WHY IT CANNOT BE A RENDERER CONSTANT — AND WHY THAT NEEDS A GATE. The path shape belongs to
+# the APP, not to the forward-auth pattern: radarr's broken path is `/signalr/`, jackett's is
+# `/api/`, and PIHOLE SERVES NO `/api/` AT ALL. One function renders all five sidecars
+# (catalog/render.py::_oauth2_sidecar_container), so the cheap fix is a constant — and a
+# constant applies one app's measurement to four apps nobody measured. The gate that already
+# exists cannot see that mistake: `ci/render-golden.sh` compares the render to the goldens, and
+# a constant plus regenerated goldens AGREE WITH EACH OTHER while both being wrong. Hence
+# assertion (2) below, which compares the rendered flag against each service's OWN
+# ServiceDefinition rather than against the renderer's output.
+#
+# WHY BREADTH IS THE DANGEROUS DIRECTION, AND IT IS NOT SYMMETRIC. A MISSING --api-route is a
+# silent 302 — bad, invisible, recoverable. A TOO BROAD one answers a plain PAGE LOAD with 401:
+# a logged-out operator gets a bare error page and is NEVER OFFERED A SIGN-IN, so the service
+# looks permanently broken and no reload recovers it. That is a worse outcome than the defect
+# the flag fixes, which is why the anchoring rule is asserted on three independent surfaces
+# (schema items.pattern, the renderer, and this check) and why assertion (5) fails if they drift
+# apart — if they do, the loosest one wins silently.
+#
+# GIVEN every oauth2-proxy sidecar this repo DEPLOYS (a container in any manifest at any depth:
+#       the rendered sidecars in apps/, crafty's hand-authored one in platform/), the
+#       ServiceDefinitions in catalog/ that generate five of them, and the renderer itself,
+# WHEN  each sidecar's `--api-route` values are collected (the flag is REPEATABLE, so all of
+#       them, in order) and each onboarded forward-auth service is paired with its golden,
+# THEN  (1) every rendered --api-route is anchored at `^/` and continues with literal path
+#           characters — `^/`, `^/.*`, `.*`, an unanchored `/api/` and any alternation FAIL;
+#       (2) every onboarded forward-auth service's rendered route list EQUALS its own
+#           ServiceDefinition's `auth.apiRoutes`, in order — which fails both a hand-edited
+#           golden and a renderer constant, neither of which render-golden can see;
+#       (3) BOTH cases are live in this tree: at least one service declares routes and at least
+#           one declares none and renders none (the pihole case), so neither path is an
+#           unexercised seam;
+#       (4) `_oauth2_sidecar_container()` reads `auth.apiRoutes` and contains NO literal path
+#           regex — a constant is the defect, back in the one place that reintroduces it
+#           fleet-wide in a single edit;
+#       (5) the anchoring pattern in the renderer and in the schema are the SAME pattern this
+#           check enforces;
+#       (6) the SHIPPED renderer, run as a subprocess on synthetic ServiceDefinitions, emits
+#           exactly the declared routes in order, emits NOTHING for a service declaring none or
+#           an empty list, and REFUSES all five breadth shapes.
+#
+# WHY ALSO A SCHEMA PATTERN, GIVEN THIS CHECK. The schema is the only surface that rejects a
+# too-broad regex WHERE A HUMAN TYPES IT — `ci/l0.sh` validates `catalog/*.yaml` against it on
+# every gate run, so the operator is told at the authoring surface rather than finding out from
+# a 401 on a page load. The renderer's own guard is there because `render-golden` runs even on a
+# gate invocation where the schema stage skipped. This check is there because neither of those
+# can see a renderer constant. Three surfaces, three different blind spots.
+#
+# WHY THIS IS A SIBLING AND NOT PART OF oauth2-proxy-cookie-names-distinct. That check's subject
+# is cookie IDENTITY: its inputs are names and domains, its failure mode is mutual session
+# eviction, its register is the shared proxy's claim on the default name. `--api-route` shares
+# none of those. Bundling them would make that check's name a lie and couple two unrelated
+# ratchets, so a future api-route decision could only be expressed by weakening a COOKIE check —
+# the same discipline that check's own header argues for, applied to this change.
+#
+# WHAT THIS DOES NOT ESTABLISH — read with the handoff entry, not instead of it:
+#   - That a 401 FIXES anything in a browser. No browser round-trip has been made, by anyone,
+#     for this flag. The 302/401 split, the dropped POST bodies and the orphaned SignalR poll
+#     are all read from access logs; the repair is reasoned from HTTP semantics.
+#   - That the regexes are RIGHT for these two apps beyond the measured paths. `^/api/` on
+#     radarr is justified by a measurement of radarr taken on its PRE-FIX pod by another
+#     engineer (decisions/2026-09-30-jackett-oauth2-proxy-session.md: 62 x /api/v3/command,
+#     46 x /api/v3/queue/details); the current pod has logged zero /api/ traffic.
+#   - Anything about sonarr, qbittorrent or pihole. Their logs were read (2026-10-01) and
+#     contain ZERO unauthenticated requests — no traffic, which is not the same as no need.
+#     Assertion (3) therefore proves the "declares none" PATH works; it does not claim those
+#     three apps are correctly configured.
+#   - That the LIVE pods carry what the tree says. git is not the cluster (CLAUDE.md §2b).
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: oauth2-proxy-api-routes-per-app
+  labels:
+    conformance.home.k3s/R-IDP-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-02"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — the subject spans apps/wave-0, apps/wave-1, platform/crafty, every
+    # ServiceDefinition in catalog/, the renderer that generates five of the sidecars, and the
+    # ServiceDefinition schema. No single requirement path-maps to that set, and the renderer
+    # maps to no identity requirement at all, so a requirement-scoped selection would skip this
+    # check on precisely the change that can reintroduce the defect fleet-wide in one edit.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      An unauthenticated XHR is answered 401, not 302, on exactly the paths each app was
+      MEASURED to serve as a machine API: every rendered --api-route is anchored at ^/ and
+      continues with literal path characters, every onboarded forward-auth service's rendered
+      routes equal its own auth.apiRoutes in order (so neither a hand-edited golden nor a
+      renderer constant can pass), an app declaring none renders none, the anchoring pattern is
+      the same in the schema and the renderer, and the shipped renderer is proven on synthetic
+      ServiceDefinitions in both directions — including refusing the five breadth shapes that
+      would answer a page load with 401.
+spec:
+  steps:
+    - name: assert-api-route-is-per-app-anchored-and-declared
+      description: >-
+        Inventory every oauth2-proxy sidecar's repeatable --api-route values, assert each is an
+        anchored path regex, cross-check each onboarded forward-auth service's rendered routes
+        against its own ServiceDefinition, require both the declares-routes and the
+        declares-none cases to be live in the tree, assert the renderer derives from
+        auth.apiRoutes and hardcodes no path regex, assert the schema and renderer share this
+        check's anchoring pattern, and prove the shipped renderer on synthetic
+        ServiceDefinitions in both directions.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/check.sh

@@ -0,0 +1,433 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — `--api-route` is PER APP, ANCHORED, and DECLARED.
+# See chainsaw-test.yaml's header for WHY this exists and what it measured. In one paragraph:
+#
+# oauth2-proxy answers an unauthenticated request with a 302 to Keycloak. A browser XHR follows
+# that redirect invisibly, the live SSO session satisfies it silently, and the call resolves 200
+# carrying the app's HTML where its JavaScript expected JSON — and a POST following a 302 is
+# re-issued as a GET with its BODY DROPPED. `--api-route=<regex>` makes matching paths answer
+# 401 instead. The flag is right for jackett (`/api/`) and radarr (`/signalr/`, `/api/`) and
+# WRONG for pihole, which serves no `/api/` at all — so it cannot be a renderer constant, and
+# the point of this check is that it never becomes one again.
+#
+# The breadth direction is the dangerous one and it is NOT symmetric with the bug. A missing
+# `--api-route` is a silent 302; a TOO BROAD one answers a plain PAGE LOAD with 401, so a
+# logged-out operator gets a bare error page and is never offered a sign-in — the service looks
+# permanently broken. That is a worse outcome than the defect, which is why the anchoring rule
+# is asserted here as well as in the schema.
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
+# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if the
+# tree contains no forward-auth oauth2-proxy sidecar at all — genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if the forward-auth pattern has left the
+# repo. Anchored on `image:` for the same reason as the sibling cookie check: a bare
+# `oauth2-proxy/oauth2-proxy:` also matches platform/oauth2-proxy/forward-auth-cec.yaml's Envoy
+# cluster reference (`serviceName: oauth2-proxy/oauth2-proxy:80`), which runs no proxy.
+if ! grep -rlE 'image:[[:space:]]*[^[:space:]]*oauth2-proxy' \
+       "${ROOT}/apps" "${ROOT}/platform" >/dev/null 2>&1; then
+  echo "check SKIPPED: no oauth2-proxy container image found under apps/ or platform/ — the forward-auth sidecar pattern is gone and there is nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: --api-route is per-app, anchored, and declared in the ServiceDefinition.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
+line-based parsing: a container's args live in a nested structure, and a line-based reader of
+one is precisely the approximate parser that silently stops matching.
+"""
+from __future__ import annotations
+
+import json
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+# THE BREADTH RATCHET, stated once and asserted against three independent surfaces below (the
+# rendered tree, the authored catalog, the shipped renderer). Each entry must be anchored at
+# `^/` and continue with LITERAL path characters. This rejects:
+#   `^/`      and `^/.*`   — match every page load, so the UI answers 401 and never signs in
+#   `/api/`               — unanchored; oauth2-proxy would match it mid-path
+#   `.*`                  — everything
+#   `^/api/|^/`           — alternation smuggles breadth past a prefix-only reading
+# Alternation is REFUSED rather than supported because `--api-route` is repeatable: one entry
+# per route costs nothing and each entry stays readable. Kept byte-identical to
+# catalog/render.py's OAUTH2_API_ROUTE_PATTERN and to the schema's items.pattern; assertion (5)
+# below fails if they drift apart.
+ANCHORED = re.compile(r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$")
+SCHEMA_PATTERN = r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$"
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+def is_test_fixture(p: Path) -> bool:
+    """conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests."""
+    parts = p.parts
+    return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
+
+
+def iter_containers(node):
+    """Yield every container mapping under any `containers`/`initContainers` list, at any depth.
+
+    Depth-agnostic on purpose: the same container shape appears under a Deployment, a DaemonSet,
+    a CronJob's jobTemplate or a bare Pod, and a reader that hardcoded one path would silently
+    stop seeing an instance the day one moved.
+    """
+    if isinstance(node, dict):
+        for key, val in node.items():
+            if key in ("containers", "initContainers") and isinstance(val, list):
+                for c in val:
+                    if isinstance(c, dict):
+                        yield c
+            else:
+                yield from iter_containers(val)
+    elif isinstance(node, list):
+        for item in node:
+            yield from iter_containers(item)
+
+
+def api_routes_of(args: list) -> list:
+    """Every `--api-route=<v>` / `--api-route <v>` value, in order. The flag is REPEATABLE, so
+    this must collect ALL of them — a single-value reader would audit the first and wave the
+    rest through, which is the shape of reader bug this file's header warns about."""
+    out = []
+    for i, a in enumerate(args):
+        if not isinstance(a, str):
+            continue
+        if a.startswith("--api-route="):
+            out.append(a.split("=", 1)[1])
+        elif a == "--api-route" and i + 1 < len(args) and isinstance(args[i + 1], str):
+            out.append(args[i + 1])
+    return out
+
+
+# ============================================================ (0) INVENTORY: THE RENDERED TREE
+# Every oauth2-proxy container this repo DEPLOYS, keyed by file + owner, with its api-routes.
+rendered: dict[str, list] = {}
+for p in sorted(ROOT.rglob("*.yaml")):
+    relparts = p.relative_to(ROOT).parent.parts
+    # NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+    # as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs,
+    # so a bare rglob audits up to 18 duplicates of every file — which both invents FAILs and,
+    # worse, can satisfy a "something declares X" check from a copy rather than from real source.
+    if any(part.startswith(".") for part in relparts) or is_test_fixture(p) or ".sops." in p.name:
+        continue
+    txt = read(p)
+    if "oauth2-proxy" not in txt:
+        continue
+    rel = str(p.relative_to(ROOT))
+    try:
+        docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
+    except yaml.YAMLError as exc:
+        check(False, f"{rel} is not parseable YAML ({exc}) — any oauth2-proxy instance in it "
+                     f"went unaudited, so a too-broad --api-route could be hiding there")
+        continue
+    for d in docs:
+        for c in iter_containers(d):
+            image = str(c.get("image") or "")
+            name = str(c.get("name") or "")
+            if "oauth2-proxy" not in image and name != "oauth2-proxy":
+                continue
+            owner = (d.get("metadata") or {}).get("name") or "?"
+            args = list(c.get("args") or []) + list(c.get("command") or [])
+            rendered[f"{rel}:{owner}/{name or 'oauth2-proxy'}"] = api_routes_of(args)
+
+check(len(rendered) >= 5,
+      f"only {len(rendered)} oauth2-proxy container(s) parsed out of the tree. This repo "
+      f"deploys more than that (five rendered sidecars plus crafty's hand-authored one), so the "
+      f"inventory has stopped seeing instances and every assertion below would pass vacuously "
+      f"while an unanchored --api-route went unaudited")
+
+# ============================================================ (1) EVERY ROUTE IS ANCHORED
+for key, routes in sorted(rendered.items()):
+    for r in routes:
+        check(bool(ANCHORED.match(r)),
+              f"{key} declares --api-route={r!r}, which is not an anchored path regex.\n"
+              f"       WHY THIS IS WORSE THAN THE BUG IT FIXES: --api-route makes matching paths "
+              f"answer 401 instead of 302ing to Keycloak. A regex that also matches a path a "
+              f"BROWSER NAVIGATES to — `^/`, `^/.*`, `.*`, or an alternation that reaches the "
+              f"UI root — answers a plain PAGE LOAD with 401. A logged-out operator then sees a "
+              f"bare error page and is NEVER OFFERED A SIGN-IN: the service looks permanently "
+              f"broken rather than merely unauthenticated, and no reload can recover it.\n"
+              f"       FIX: anchor it at '^/' followed by literal path characters "
+              f"(e.g. '^/api/', '^/signalr/', '^/api/v3/.*'), one entry per route — the flag is "
+              f"repeatable, so alternation is never needed. For a service rendered from catalog/, "
+              f"do NOT hand-edit the golden: declare auth.apiRoutes and re-render "
+              f"(ci/render-golden.sh's header has the canonical command)")
+
+# ============================================================ (2) DERIVED FROM THE CATALOG
+# The five rendered sidecars are OUTPUTS. Auditing only them would let someone "fix" a golden by
+# hand (reverted by the next render) or make the renderer emit a CONSTANT and regenerate every
+# golden to match — which render-golden cannot see, because render and golden would agree.
+# So: every onboarded forward-auth service's rendered route list must EQUAL its own
+# ServiceDefinition's auth.apiRoutes, in order.
+catalog_dir = ROOT / "catalog"
+declared: dict[str, list] = {}
+forward_auth: list[str] = []
+for sd_path in sorted(catalog_dir.glob("*.yaml")):
+    try:
+        sd = yaml.safe_load(read(sd_path))
+    except yaml.YAMLError as exc:
+        check(False, f"catalog/{sd_path.name} is not parseable YAML ({exc})")
+        continue
+    if not isinstance(sd, dict) or not sd.get("name"):
+        continue
+    auth = sd.get("auth") or {}
+    routes = auth.get("apiRoutes") or []
+    declared[sd["name"]] = list(routes)
+    if auth.get("mode") == "forward-auth" and (sd.get("ingressRoutes") or []):
+        forward_auth.append(sd["name"])
+    for r in routes:
+        check(bool(ANCHORED.match(str(r))),
+              f"catalog/{sd_path.name}: auth.apiRoutes entry {r!r} is not anchored at '^/' "
+              f"followed by literal path characters. See assertion (1) for why breadth here is "
+              f"the dangerous direction. The schema rejects this too (items.pattern); this "
+              f"assertion exists because the schema stage can be SKIPPED and this one cannot")
+    if routes:
+        check(auth.get("mode") == "forward-auth",
+              f"catalog/{sd_path.name} declares auth.apiRoutes with auth.mode="
+              f"{auth.get('mode')!r}. --api-route renders onto the per-app oauth2-proxy sidecar, "
+              f"and no sidecar is rendered for oidc/native/none — so this is a SILENT NO-OP: the "
+              f"operator believes the app's XHRs now answer 401 and nothing changed")
+
+matched = 0
+for name in sorted(forward_auth):
+    golden = next((p for p in sorted(ROOT.glob(f"apps/wave-*/{name}.yaml"))), None)
+    if golden is None:
+        continue                      # catalog service not onboarded to apps/ yet — no golden
+    key = next((k for k in rendered if k.startswith(str(golden.relative_to(ROOT)) + ":")), None)
+    check(key is not None,
+          f"apps/{golden.name} is the golden for forward-auth service {name!r} but no "
+          f"oauth2-proxy container was parsed out of it. Its --api-route configuration is "
+          f"UNAUDITED")
+    if key is None:
+        continue
+    matched += 1
+    check(rendered[key] == declared.get(name, []),
+          f"{name}: the rendered sidecar declares --api-route {rendered[key]!r} but "
+          f"catalog/{name}.yaml declares auth.apiRoutes {declared.get(name, [])!r}.\n"
+          f"       These must be IDENTICAL and in order. A mismatch means one of two things, "
+          f"both bad: a golden was hand-edited (the next re-render silently reverts it), or the "
+          f"renderer is emitting --api-route from somewhere other than this service's own "
+          f"ServiceDefinition — i.e. a constant or a per-service table. A constant is the defect "
+          f"this field exists to prevent: it applies ONE app's path shape to all five, and "
+          f"pihole serves no /api/ at all, so it would answer pihole's page loads with 401. "
+          f"render-golden cannot catch that, because render and golden would agree with each "
+          f"other while both being wrong")
+
+check(matched >= 4,
+      f"only {matched} onboarded forward-auth service(s) were cross-checked against their "
+      f"ServiceDefinition. This repo onboards five, so the catalog<->golden pairing has stopped "
+      f"resolving and assertion (2) is passing vacuously")
+
+# ============================================================ (3) BOTH CASES EXIST
+# Once the tree is consistent, (2) is satisfied by any pairing — including one where NOTHING
+# declares a route (vacuous) or one where EVERYTHING does (so the "renders nothing" path is
+# never exercised). Both halves are asserted to be live, because the per-app field only earns
+# its complexity if both outcomes actually occur in this tree.
+with_routes = [n for n in forward_auth if declared.get(n)]
+without_routes = [n for n in forward_auth if not declared.get(n)]
+check(bool(with_routes),
+      "no onboarded forward-auth service declares auth.apiRoutes, so assertion (2) cannot "
+      "distinguish a working renderer from one that emits the flag nowhere. If the flag is "
+      "genuinely not needed anywhere, delete the field rather than leaving an unexercised seam")
+check(bool(without_routes),
+      "EVERY onboarded forward-auth service declares auth.apiRoutes, so nothing in this tree "
+      "exercises the 'declares none => renders no flag' path. That path is the whole reason the "
+      "field is per-app: pihole serves no /api/, and a renderer-wide default would answer its "
+      "page loads with 401")
+for name in sorted(without_routes):
+    golden = next((p for p in sorted(ROOT.glob(f"apps/wave-*/{name}.yaml"))), None)
+    if golden is None:
+        continue
+    key = next((k for k in rendered if k.startswith(str(golden.relative_to(ROOT)) + ":")), None)
+    if key is None:
+        continue
+    check(rendered[key] == [],
+          f"{name} declares NO auth.apiRoutes but its rendered sidecar carries "
+          f"--api-route {rendered[key]!r}. An app that declares nothing must render NOTHING")
+
+# ============================================================ (4) THE RENDERER DERIVES
+render_py = ROOT / "catalog" / "render.py"
+check(render_py.is_file(),
+      "catalog/render.py is missing — the generator of five of these sidecars cannot be audited")
+if render_py.is_file():
+    src = read(render_py)
+    m = re.search(r"def _oauth2_sidecar_container\(.*?\n(?=\n*def |\n*# =)", src, re.S)
+    fn = m.group(0) if m else ""
+    check(bool(fn),
+          "could not locate _oauth2_sidecar_container() in catalog/render.py. It was renamed or "
+          "removed, so assertion (4) cannot run and the generator is unaudited")
+    if fn:
+        check("--api-route=" in fn,
+              "catalog/render.py's _oauth2_sidecar_container() emits no --api-route at all, so "
+              "every forward-auth app's XHRs are answered with a 302 the browser follows "
+              "invisibly — measured as 33x302-vs-2x401 on jackett and 232x302 on radarr, with "
+              "POST bodies dropped. auth.apiRoutes would then be a field with no consumer")
+        check(not re.search(r'--api-route=\^/[A-Za-z0-9]', fn),
+              "catalog/render.py hardcodes a literal path regex into --api-route. Whatever path "
+              "that is, EVERY forward-auth app now renders it: one app's path shape applied to "
+              "five. pihole serves no /api/, so a constant answers its page loads with 401 and "
+              "the service looks permanently broken to a logged-out operator. Render the routes "
+              "declared in each service's own auth.apiRoutes instead")
+        check("apiRoutes" in fn,
+              "catalog/render.py's _oauth2_sidecar_container() does not read auth.apiRoutes, so "
+              "the per-app declaration has no effect on what is rendered")
+
+# ============================================================ (5) ONE PATTERN, THREE SURFACES
+# The ratchet is written in three places (this check, the renderer, the schema) because each
+# guards a surface the others cannot. If they drift apart, the loosest one wins silently.
+if render_py.is_file():
+    # The renderer writes the pattern as a raw string literal, so the source text contains it
+    # verbatim. A substring test is deliberate: it also catches a rename of the constant.
+    check(SCHEMA_PATTERN in read(render_py),
+          f"catalog/render.py no longer contains the anchoring pattern {SCHEMA_PATTERN!r} that "
+          f"this check and the schema enforce. Three surfaces guard --api-route breadth (the "
+          f"authoring surface via the schema, the render via the renderer, the tree via this "
+          f"check); if they drift apart the loosest wins and the ratchet is decorative")
+
+# The schema's pattern is read from the PARSED JSON, not from the file text: in the file its
+# backslashes are JSON-escaped, so a substring test against the raw bytes would compare the
+# wrong string and pass or fail for the wrong reason.
+schema_file = ROOT / "schemas" / "servicedefinition.schema.json"
+schema_pat = None
+if schema_file.is_file():
+    try:
+        schema_pat = (((((json.loads(read(schema_file)).get("properties") or {})
+                         .get("auth") or {}).get("properties") or {})
+                       .get("apiRoutes") or {}).get("items") or {}).get("pattern")
+    except (ValueError, AttributeError) as exc:
+        check(False, f"schemas/servicedefinition.schema.json could not be read as JSON ({exc}) — "
+                     f"the schema stage that rejects a too-broad --api-route cannot be audited")
+check(schema_pat == SCHEMA_PATTERN,
+      f"schemas/servicedefinition.schema.json constrains auth.apiRoutes items with "
+      f"{schema_pat!r}, not {SCHEMA_PATTERN!r}. The schema is the only surface that rejects a "
+      f"too-broad regex WHERE A HUMAN TYPES IT (ci/l0.sh validates catalog/*.yaml on every gate "
+      f"run); if it is absent or looser than the renderer's, the operator finds out from a 401 "
+      f"on a page load")
+
+# ============================================================ (6) PROVE THE SHIPPED RENDERER
+# Once the tree is compliant, (1)-(3) have nothing left to catch and their passing is evidence
+# of nothing. This runs the SHIPPED renderer as a SUBPROCESS on synthetic ServiceDefinitions and
+# asserts the behaviour directly, in BOTH directions, so the guard stays falsifiable.
+def _sd(name: str, port: int, routes) -> str:
+    auth = {"mode": "forward-auth", "groups": ["platform-admins"]}
+    if routes is not None:
+        auth["apiRoutes"] = routes
+    return yaml.safe_dump({
+        "name": name,
+        "kind": "container",
+        "source": {"image": "example/app:1"},
+        "exposure": "internal",
+        "ingressRoutes": [{"host": f"{name}.home.keithjackson.dev", "port": port}],
+        "auth": auth,
+    }, sort_keys=False)
+
+
+def _render(name: str, port: int, routes):
+    with tempfile.TemporaryDirectory() as td:
+        f = Path(td) / f"{name}.yaml"
+        f.write_text(_sd(name, port, routes))
+        return subprocess.run([sys.executable, str(render_py), str(f)],
+                              capture_output=True, text=True)
+
+
+def _sidecar_args(rendered_yaml: str) -> list:
+    for doc in yaml.safe_load_all(rendered_yaml):
+        if not isinstance(doc, dict) or doc.get("kind") != "Deployment":
+            continue
+        for c in iter_containers(doc):
+            if c.get("name") == "oauth2-proxy":
+                return list(c.get("args") or [])
+    return []
+
+
+if render_py.is_file():
+    # (6a) DECLARED ROUTES RENDER, IN ORDER, AND NOTHING ELSE DOES.
+    proc = _render("apirtdecl", 8121, ["^/api/", "^/signalr/"])
+    check(proc.returncode == 0,
+          f"the shipped catalog/render.py failed on a synthetic forward-auth fixture declaring "
+          f"two apiRoutes: rc={proc.returncode} {proc.stderr.strip()[:300]}")
+    if proc.returncode == 0:
+        got = api_routes_of(_sidecar_args(proc.stdout))
+        check(got == ["^/api/", "^/signalr/"],
+              f"the shipped renderer turned auth.apiRoutes ['^/api/', '^/signalr/'] into "
+              f"--api-route {got!r}. It must emit one repeatable --api-route per entry, in "
+              f"declared order. Note the flag is SINGULAR: --api-routes does not exist in "
+              f"oauth2-proxy v7.8.1 and the proxy REFUSES TO START with it, so a plural would "
+              f"crashloop every sidecar it rendered")
+
+    # (6b) AN APP THAT DECLARES NOTHING RENDERS NOTHING. The pihole case, proven on the
+    #      generator rather than inferred from one committed golden.
+    for label, routes in (("absent", None), ("empty", [])):
+        proc = _render(f"apirtnone{label}", 8122, routes)
+        check(proc.returncode == 0,
+              f"the shipped renderer failed on a synthetic forward-auth fixture with apiRoutes "
+              f"{label}: rc={proc.returncode} {proc.stderr.strip()[:300]}")
+        if proc.returncode == 0:
+            got = api_routes_of(_sidecar_args(proc.stdout))
+            check(got == [],
+                  f"the shipped renderer emitted --api-route {got!r} for a synthetic service "
+                  f"whose auth.apiRoutes is {label}. A newly-onboarded forward-auth app must "
+                  f"render NO --api-route: pihole serves no /api/, so a default would answer "
+                  f"its page loads with 401 and the service would look permanently broken to a "
+                  f"logged-out operator — a worse failure than the 302 bug the flag fixes")
+
+    # (6c) THE RENDERER REFUSES BREADTH. Asserted on the generator because render-golden runs
+    #      on every gate invocation while the schema stage can be skipped.
+    for bad in ("^/", "^/.*", ".*", "/api/", "^/api/|^/"):
+        proc = _render("apirtbad", 8123, [bad])
+        check(proc.returncode != 0,
+              f"the shipped renderer ACCEPTED auth.apiRoutes {bad!r} and rendered a sidecar. "
+              f"That regex matches paths a browser NAVIGATES to, so a plain page load answers "
+              f"401 and the operator is never offered a sign-in. The renderer must refuse it, "
+              f"so a breadth mistake fails render-golden even on a run where the schema stage "
+              f"did not execute")
+
+# ============================================================ VERDICT
+inv = " | ".join(f"{k} -> {v or '(none)'}" for k, v in sorted(rendered.items()))
+print(f"oauth2-proxy sidecars audited ({len(rendered)}): {inv}")
+if FAILURES:
+    print(f"\nCHECK FAIL (oauth2-proxy-api-routes-per-app): {len(FAILURES)} failed, "
+          f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  FAIL {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: oauth2-proxy-api-routes-per-app — {CHECKS} assertions (R-IDP-02)")
+PY

conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh

@@ -129,22 +129,6 @@ KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
         "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
         "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
         "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
-    "platform/oauth2-proxy/podmonitor.yaml:oauth2-proxy-sidecars":
-        "a PodMonitor, so it resolves against POD ports, not Service ports. Unlike the other "
-        "entries here the target is NOT chart-owned — the `metrics` (44180) container port is "
-        "authored in this repo, on the oauth2-proxy sidecar in apps/wave-0/{jackett,qbittorrent,"
-        "pihole}.yaml and apps/wave-1/{radarr,sonarr}.yaml. It is unauditable here because NO "
-        "Service exposes 44180 BY DESIGN: adding one would mean changing catalog/render.py and "
-        "every apps/wave-* render and golden, to publish an endpoint only Prometheus should dial. "
-        "A PodMonitor needs no Service, which is the whole reason it was chosen. What assertion "
-        "(2) would have checked was instead MEASURED on prod 2026-10-01, read-only: the pods "
-        "declare ports [('http', 4180), ('metrics', 44180)], the selector label "
-        "homelab.keithjackson.dev/auth-mode=forward-auth is on exactly those five gate pods, and "
-        "port-forwarding 44180 returns http=200 with 10208 bytes of oauth2_proxy_* metrics. "
-        "CAVEAT, deliberately recorded: those targets are DOWN on prod, because the app "
-        "namespaces' default-deny NetworkPolicy drops the scrape from ns monitoring (same bug and "
-        "fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents). The port name "
-        "resolves; the network path does not yet.",
 }
 
 MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")

decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md

@@ -1,289 +0,0 @@
----
-date: 2026-09-30
-branch: feat/oauth2-proxy-auth-failure-visible
-title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
----
-
-## What changed
-
-Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
-operator's report named the property that matters — *"reloading the page never fixes the issue and
-the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
-about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
-the alert rules. It is an absence of data.
-
-**The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
-
-    count by (__name__) ({__name__=~"oauth2_proxy.*"})
-      oauth2_proxy_requests_in_flight => 2
-
-Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
-That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
-jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
-`--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
-starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
-cluster selects them.
-
-So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
-a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
-exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
-permanently silent, and indistinguishable from health. It was not written.
-
-What was written instead alerts on the condition that made the incident invisible, from families
-that **are** scraped:
-
-- **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
-
-      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
-        unless
-      count by (namespace) (up{container="oauth2-proxy"})
-
-  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
-  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
-  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
-  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
-  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
-  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
-  clears per-namespace as each gate gains a real scrape.
-- **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
-  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
-  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
-  prod, not invented.
-- **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
-  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
-  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
-  `configMapGenerator` and listed in `runbooks/README.md`.
-- **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
-  per-app sidecars, plus the measured reason.
-- **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
-  three-part change that would close the gap.
-
-**Why the obvious alternative was rejected.** An alert on the *shared* proxy's
-`oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
-need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
-and — decisively — **it would not have fired on this incident**: the shared instance is not in the
-data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
-at all, having proxied nothing since they started (the counter exists in the TSDB only from an
-older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
-Shipping it would have looked like coverage of the reported failure and been none.
-
-## What I verified
-
-Everything below was measured against live prod through the pinned container, read-only. No
-`apply`, no `edit`, no `patch`, no `flux reconcile`.
-
-- **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
-  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
-  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
-  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
-  `1`, `namespace="oauth2-proxy"`.
-- **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
-  `oauth2-proxy` container, with its declared ports: the two shared replicas have
-  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
-  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
-  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
-  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
-- **No existing alert is inert *for this reason*, because no alert references these metrics at
-  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
-  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
-  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
-  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
-  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
-- **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
-  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
-  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
-  observable from the IdP side.
-- **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
-  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
-  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
-  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
-- **The alert expression fires against live data.** Run verbatim on prod:
-  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
-  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
-  radarr`, value 1 each; `oauth2-proxy` correctly absent.
-- **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
-  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
-  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
-  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
-  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
-  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
-  runbook as the thing to compare against.
-- **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
-  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
-  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
-- **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
-  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
-  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
-  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
-  `runbook_url` points at exactly that name.
-- **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
-  per-stage outcomes in the next section — read them there, not here.
-
-## What I did NOT verify
-
-- **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
-  only in git on this branch. What is proven is that its expression returns the expected vector
-  against live prod data and that promtool agrees about its behaviour over synthetic series.
-  **Merging is what deploys it.**
-- **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
-  any useful sense** — it describes a standing condition that has been true since the sidecars
-  were deployed, so it would have been firing before, during and after. That is deliberate and it
-  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
-  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
-  instead of discovering it during the investigation. **No alert on this platform could have
-  detected the incident itself, and none can today.**
-- **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
-  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
-  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
-  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
-  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
-  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
-  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
-  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
-- **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
-  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
-  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
-  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
-  directory from the one the sidecar writes), and the rendered config contains **no
-  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
-  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
-  caught this incident, and is the single highest-value follow-up here — **would evaluate and
-  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
-  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
-  inert rule ConfigMap to look productive.**
-- **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
-  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
-- **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
-  snapshot; anything older than three days is already gone. This is stated in the runbook because
-  it bounds what an operator can still investigate.
-- **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
-  requirement touched by this branch is certified. See the exit code below — **rc 3 is
-  UNDETERMINED, not green.**
-
-### Follow-ups, in order of value
-
-1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
-   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
-   already being collected. The rule is one expression:
-   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
-   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
-2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
-   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
-   and it unlocks the ratio rule in Prometheus.
-3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
-   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
-   observable, and it belongs with whoever owns the sidecar args.
-
----
-
-# Round 2 — 2026-10-01: the scrape got wired, and found a bug in this branch's own alert
-
-Added to the same PR (#29) rather than a fourth concurrent branch. PR #28 merged in the meantime
-and `main` moved three PRs ahead; `origin/main` is merged in here, and the `runbooks/kustomization.yaml`
-conflict was resolved by **keeping both** runbook entries (mine and `oauth2-proxy-session`).
-
-## What changed
-
-**`platform/oauth2-proxy/podmonitor.yaml` (new).** PR #28 gave the five fleet sidecars
-`--metrics-address=0.0.0.0:44180` and a named `metrics` port. No Service exposes 44180 on any of
-them, so a ServiceMonitor would have nothing to select — a **PodMonitor** targets pod ports directly
-and needs no renderer change. Selects on the pod label `homelab.keithjackson.dev/auth-mode:
-forward-auth`, which is on exactly the five fleet gate pods and absent from crafty's: the property,
-not an app list, so a sixth forward-auth app needs no edit.
-
-**`auth-visibility-alerts.yaml`: `up{container="oauth2-proxy"}` → `up{container="oauth2-proxy"} == 1`.
-This is a real bug fix, and it is the most important thing in this round.** A target that is
-discovered but failing still publishes `up`, with value **0**. The series exists, so
-`count by (namespace)` returned 1, `unless` removed the namespace, and `OAuth2ProxyGateUnobserved`
-would have **gone quiet while nothing was being scraped at all** — this branch's own disease,
-reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets
-whose scrape is dropped, so the bare form would have silently cleared on this very commit.
-
-**Two new unit-test scenarios** pinning that: a discovered-but-failing target must still fire, and a
-target that works and then breaks must re-fire rather than latch cleared.
-
-**Docs corrected rather than appended to.** The 2026-09-30 claim "the sidecars export no metrics at
-all" is now only half true and was rewritten in `runbooks/README.md`, `platform/oauth2-proxy/README.md`
-and the alert header, each showing the two-step history, because each step looked like the end of it.
-The runbook gains the policy-vs-no-target triage (step 1b) and the **radarr `/signalr` worked case**.
-
-**Deliberately NOT shipped: the ratio alert.** `oauth2_proxy_requests_total` now exists on the
-endpoint, but not in the TSDB, because the scrape is blocked. Writing the ratio rule today would make
-it inert for precisely the reason it was refused on 2026-09-30. It becomes writable the day a scrape
-succeeds, not the day a listener exists.
-
-## What I verified
-
-- **The endpoint serves — by payload, not by flag.** Port-forwarded to the jackett pod's 44180:
-  `http=200 bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total`
-  (`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. `--metrics-address`
-  being in `args` and `/metrics` returning a body are different claims; both now hold.
-- **The scrape path is DROPPED, measured from the right place.** From grafana in ns `monitoring`
-  (beside Prometheus, same policy position) to the jackett pod IP: `:44180` → `wget: download timed
-  out`; `:4180` → **also** timed out; the shared proxy's `:44180` → 200 with `oauth2_proxy_*` present.
-  Cause, read from the objects: ns jackett's `default-deny` is `podSelector: {}`,
-  `policyTypes: [Ingress, Egress]`, with an `egress` rule and **no `ingress` key** (deny-all ingress);
-  the only allowance is CNP `jackett-allow-gateway-ingress`, `fromEntities: [ingress]` to port 4180.
-  The shared proxy is scrapable only because ns `oauth2-proxy` has **no NetworkPolicy at all**.
-- **No Service exposes 44180** on any of the five (enumerated).
-- **The pod label selector is exact:** `auth-mode=forward-auth` on all five fleet gate pods, absent
-  from crafty's. Crafty's sidecar still has no `--metrics-address` (args read in full).
-- **Flux ordering forbids shipping the fix from here:** `oauth2-proxy dependsOn=['platform-config']`
-  and `apps-wave-0 dependsOn=['platform-config']` — siblings, no ordering. A NetworkPolicy naming ns
-  `jackett` from this tier would apply today and fail on a cold rebuild.
-- **The `== 1` guard is non-vacuous — proven by mutation.** Reverting it to the bare `up` makes the
-  two new scenarios FAIL (`got: nil` at 1h5m and 2h5m) and leaves every other assertion passing;
-  restoring it returns the suite to SUCCESS under pinned promtool v3.14.0.
-- **Gate:** see the next section for the exit code and skips.
-
-## What I did NOT verify
-
-- **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from the
-  measured pod labels and port name, not observed — no target list has been read with it in place,
-  because applying it is an operator action. **Expect `up == 0` on all five when it lands**, which is
-  the documented state, not a regression.
-- **I did not fix the NetworkPolicy, and this PR therefore does not make the gates observable.** It
-  makes them *discovered*, and makes the remaining gap loud instead of silent. The one narrow rule
-  per app (ns monitoring → `auth-mode=forward-auth` → TCP 44180, per
-  `platform/longhorn/networkpolicy-metrics.yaml`) has to be rendered with each app's own policy, in
-  the catalog/renderer that is in flight on another branch. **It needs an owner.**
-- **Whether Prometheus will actually select a PodMonitor in ns `oauth2-proxy` targeting other
-  namespaces.** `namespaceSelector.any: true` plus `podMonitorSelectorNilUsesHelmValues: false` and
-  the fact that cross-namespace ServiceMonitors/PrometheusRules already work here say yes; it is not
-  measured with this object.
-- **No session-lifecycle claim can be made from the current window.** `AuthSuccess` is **0** on both
-  jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s`
-  has never executed a refresh. The cookie fix is unproven in the one dimension that matters most.
-- **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither
-  the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported.
-- **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified.
-
-### Round 2 addendum — CI is RED, by design, and the local gate cannot see why
-
-After pushing round 2, `gate / gate (pull_request)` reported **failure after 1m42s** while the local
-toolbox run was rc 3 with nothing failed. Measured cause, singular:
-
-    sensitive paths touched:
-      conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh   (matched conformance/**)
-    approving reviews: (no reviews at all)        author=gitea-admin
-
-`ci/approval-policy.sh` is fail-closed and exits 1 when a PR touches a `sensitive_paths` glob with no
-`APPROVED` review from a non-author — "a change to the gate cannot be gated by the gate". The single
-`KNOWN_EXTERNAL_MONITORS` entry is what trips it. **Nothing in L0/L1 failed.**
-
-**The local gate structurally cannot catch this, and said `approval-policy: green (N/A)`.** The stage
-short-circuits to N/A when `GITHUB_EVENT_NAME` is unset, which it always is off-CI. So for any branch
-touching a sensitive path, a clean local gate is a FALSE REASSURANCE — the PR run is the only honest
-signal. That is a general trap, not specific to this change, and it is worth knowing before someone
-concludes a red PR means broken code.
-
-To clear: a non-author approving review, then **re-run the gate** — approval does not re-trigger the
-status check, so the required context stays red until the job runs again. `required_approvals=1`
-already applied to every PR here, so the human was needed regardless; the conformance entry only
-turns the tick red while that is pending.
-
-The alternative is to drop the register entry, which means dropping `podmonitor.yaml` with it (the
-check fails without the declaration). I did not take it: the entry is the "declare the limit"
-mechanism the check explicitly asks for, it is shrink-only so it cannot rot quietly, and
-`flux-podmonitor.yaml` is the same case already in the register. But it is a real trade and the
-reviewer should make it knowingly.

decisions/2026-10-01-oauth2-proxy-api-route-per-app.md

@@ -0,0 +1,482 @@
+---
+date: 2026-10-01
+branch: feat/oauth2-proxy-api-route-per-app
+title: The config model had no way to say "this app's API lives here" — so --api-route could not ship. Now it can, per app.
+---
+
+## What changed
+
+Second half of PR #28 (`decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md`), which
+fixed the cookie collision, the 5-minute session and the missing metrics listener in one renderer
+function and **deliberately declined `--api-route`** for a reason I have preserved rather than
+worked around: that flag was measured on **one** app, and `PiHole serves no /api/ at all`, so a
+renderer-wide constant would apply one app's path shape to five. The `auth` block is
+`additionalProperties: false`, so there was no per-app override hook. **Creating the hook was the
+work; setting the flag is the easy part.**
+
+**Three surfaces, in order of what they protect:**
+
+| Surface | Change | What it guards that the others cannot |
+|---|---|---|
+| `schemas/servicedefinition.schema.json` (**1.8.0 → 1.9.0**) | new optional `auth.apiRoutes`: array of anchored path regexes, `uniqueItems`, default `[]` | rejects a too-broad regex **where a human types it** — `ci/l0.sh` validates `catalog/*.yaml` against this schema on every gate run |
+| `catalog/render.py` | emits one `--api-route=<r>` per declared entry, in order; **raises** on an unanchored entry | `ci/render-golden.sh` runs on every gate invocation even when the schema stage skips |
+| `conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/` (new) | cross-checks each rendered sidecar against its own ServiceDefinition | catches a **renderer constant**, which neither of the above can see |
+
+**The `auth` block stays `additionalProperties: false`.** One named, documented key was added; the
+door is not opened. That property is the reason the gap existed, and it is worth keeping — it is
+what made "there is no override hook" a *fact* rather than a *guess* for the two engineers who hit
+it before me.
+
+**Rebased mid-flight.** `origin/main` advanced from `08deb20` to `cdf3e14` while I was working
+(PR #27 and PR #32, the `hardware.gpu: intel-quicksync` capability). **That change also took schema
+`1.8.0`**, so mine is renumbered to **1.9.0** and chained behind it; both version entries are
+present in the description, and `catalog/render.py` had a two-sided conflict in the constants block
+which is resolved by keeping both (the `DRI_*` constants and `OAUTH2_API_ROUTE_PATTERN` are
+unrelated). Re-rendering after the rebase still changes **only** jackett and radarr — plex's new
+golden reproduces byte-identically, so the two changes do not interact.
+
+**Shape:** a **list** of regexes, not a single string, because Radarr needs two
+(`^/api/` and `^/signalr/`) and `--api-route` is natively repeatable. A single string would have
+forced alternation into the value, and alternation is exactly how breadth gets smuggled past a
+prefix-only reading of a regex.
+
+**The anchoring ratchet.** Each entry must match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$` — anchored at
+`^/`, then literal path characters, with an optional trailing `.*` or `$`. The same pattern is in
+the schema, in the renderer, and in the new check, and assertion (5) of the check **fails if they
+drift apart**, because if they do the loosest one wins silently. Rejected shapes, all five
+provoked: `^/`, `^/.*`, `.*`, `/api/` (unanchored), `^/api/|^/` (alternation). Alternation is
+**refused rather than supported**: the flag is repeatable, so one entry per route costs nothing.
+
+**Why breadth is the dangerous direction, and it is not symmetric.** A *missing* `--api-route` is a
+silent 302 — bad, invisible, recoverable. A *too broad* one answers a plain **page load** with 401:
+a logged-out operator gets a bare error page and is **never offered a sign-in**, so the service
+looks permanently broken and no reload recovers it. That is a worse outcome than the defect the
+flag fixes, which is why it is gated on three surfaces rather than documented in one.
+
+### Also folded in: the selftest could exit 1 with ZERO diagnostics
+
+Not my change originally — Desmond Oyelaran found it while wiring `catalog/render.py --selftest`
+into the `render-golden` stage (his PR #33, which touches only `ci/render-golden.sh` and is **not**
+in my base; `origin/main` at `cdf3e14` does not yet drive the selftest from any gate stage). The
+coordinator asked me to fold the fix into this PR because the defect is in the file I own.
+
+`_selftest()` reached `_cookie_name()` through a bare `next(genexpr)` with **no default**. Delete
+the `--cookie-name` emission — exactly the mutation that assertion exists to catch — and it raises
+`StopIteration`, which escapes `_selftest()` **before its collected `SELFTEST FAIL:` lines are
+printed**. Right verdict, no diagnostics, and it masks every other assertion the same mutation
+broke. Fixed with `next(..., None)` plus an explicit `check()` so a missing flag reports **as a
+named assertion**.
+
+Provoking it turned up **three more instances of the same failure class** in the same block, which
+is why this is not literally a one-line change: `side = fa_containers[1]`,
+`side["env"][0][...]`, and the `fa_svc` lookup all raise rather than assert when the sidecar is
+absent entirely. All four are now None-tolerant, and the two fixtures I added for `--api-route`
+were written the same way from the start. Measured, on throwaway copies:
+
+| Mutation | Before | After |
+|---|---|---|
+| `--cookie-name` emission deleted (origin/main's renderer) | rc 1, **0** `SELFTEST FAIL` lines, 1 `StopIteration` | rc 1, **3** `SELFTEST FAIL` lines, 0 tracebacks |
+| the oauth2-proxy sidecar not appended at all | rc 1, **0** lines, `IndexError` | rc 1, **20** `SELFTEST FAIL` lines, 0 tracebacks |
+
+Note the consequence of his stage change for anyone reading this later: once PR #33 lands,
+`.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]`, so **rc 1 is not tolerated
+in CI the way rc 3 is** — a selftest failure will block. Which is the right outcome, and it is why
+a selftest that prints nothing was worth fixing before that lands rather than after.
+
+### Declined, with reasons: the metrics-ingress NetworkPolicy
+
+The coordinator asked for my judgement on bundling Ines Dubois's finding — that #28's metrics
+listener genuinely serves (she measured `http=200`, 10,208 bytes,
+`oauth2_proxy_requests_total{code="302"} 2` via port-forward) but the scrape will be **dropped**,
+because namespace `jackett`'s default-deny has **no `ingress` key at all** and
+`jackett-allow-gateway-ingress` admits `fromEntities: [ingress]` only. The fix belongs in each
+app's own rendered policy, i.e. in my file and my goldens. **I declined to bundle it**, and asked
+for it as the immediate next PR. Five reasons, in the order that decided it:
+
+1. **The real reason is that it is a MODEL question, not a policy to bolt on.** Schema 1.6.0
+   already owns this: `observability.metricsEndpoint` renders the Service port **and** the
+   monitoring-namespace ingress allow. What the model cannot currently express is a **second**
+   metrics endpoint on the same pod belonging to a *sidecar the renderer injected*, rather than to
+   the workload the operator declared. Keying a bespoke policy off the pod label
+   `homelab.keithjackson.dev/auth-mode: forward-auth` works today, but it hardcodes "forward-auth
+   implies a metrics port on 44180" into a policy instead of **deriving** it from the sidecar
+   declaration that is already the single source of that port — the exact "list, don't derive"
+   anti-pattern `oauth2-proxy-cookie-names-distinct` assertion (5) exists to forbid, and which I
+   would be introducing one directory away from the check that forbids it. That deserves its own
+   review, and it is squarely the kind of decision I was brought in to make deliberately.
+2. **Different subject, and it is a trust-boundary change.** Mine is the auth *response code* on
+   API paths (R-IDP-02). That is an ingress allow (R-NET-01/02, R-OBS-01) admitting a new
+   namespace to a new port. Bundling a security-relevant ingress rule into a PR whose headline is
+   `--api-route` is how such a rule gets waved through on the strength of the other half — and this
+   PR already needs a human for a *different* sensitive-path reason, so the second subject would be
+   reviewed in the shadow of the first.
+3. **The conflict argument does not actually apply to my diff.** It assumes I am regenerating five
+   goldens; **I am regenerating two** (jackett, radarr), because the flag renders only where
+   declared. A metrics-ingress policy renders a *different document* — a CiliumNetworkPolicy — in
+   all five. Different hunks, in files git merges independently; and goldens regenerate
+   deterministically from the renderer, so the follow-up rebases onto `main` cleanly. The N-1
+   rebase problem `decisions/README.md` records was two branches appending to **the same hunk**,
+   which this is not. I checked my own diff before answering rather than accepting the premise.
+4. **Every fact in the finding is hers and I have measured none of it** — not the 200/10,208 bytes,
+   not the `wget: download timed out` on `:44180` *and* `:4180`, not the Flux sibling ordering that
+   makes `oauth2-proxy` and `apps-wave-0` unordered. Shipping a security-adjacent rule on wholly
+   inherited evidence, as a footnote in someone else's PR, is precisely what CLAUDE.md §2b exists
+   to stop. In its own PR I would measure it first.
+5. **Sequencing: the PodMonitor is in HER PR #29, not mine.** If I ship the policy and her
+   PodMonitor is reworked or delayed, I have opened an ingress allow for a scraper that selects
+   nothing — a hole with no consumer. The policy is only useful paired with the PodMonitor, which
+   argues for it landing adjacent to that, not here. It also drags in a
+   `KNOWN_EXTERNAL_MONITORS` entry in `conformance/tests/l0-static/servicemonitor-ports-resolve/`,
+   a second conformance interaction on top of the new check this PR already adds.
+
+Diff size is the tiebreaker rather than the argument: **1,245 insertions across 13 files**,
+including a ~400-line new conformance check. I would rather hand a reviewer one subject they can
+actually falsify. **Queued, not dropped** — and if the next person takes it, the model note in (1)
+is the part worth arguing about, not the YAML.
+
+### Which apps got which routes, and the measurement behind each
+
+| App | `auth.apiRoutes` | Evidence |
+|---|---|---|
+| **jackett** | `^/api/` | 2026-09-30, its own access log: **33 × 302 against 2 × 401** on `/api/` paths, and **4 of 6 POSTs answered by the proxy and never reaching Jackett** (upstream field `-`, not `/`), including both `/api/v2.0/server/update` calls |
+| **radarr** | `^/api/`, `^/signalr/` | `^/signalr/`: measured **by me, on the live post-fix pod** — see below. `^/api/`: measured on radarr by the PR #30 author, on the **pre-fix** pod |
+| **pihole** | *(none — deliberate)* | serves no `/api/`; **0** unauthenticated requests in its current pod's 23 h |
+| **sonarr** | *(none — not measured)* | 13,643 log lines, **0** unauthenticated requests. No traffic, not a negative result |
+| **qbittorrent** | *(none — not measured)* | 13,649 log lines, **0** unauthenticated requests. Same |
+
+**`^/signalr/` on radarr is the live proof this change was needed**, and it is the strongest
+evidence in this entry. 23 h after PR #28 deployed, on `radarr-7d55cbffb9-42hmh` (0 restarts), the
+sidecar logged **232 unauthenticated requests across the pod's entire lifetime, and all 232 are the
+same call**: `POST /signalr/messages/negotiate`, browser UA, **302**, upstream `-` (Radarr never saw
+one of them). That is a **left-open browser tab** whose session the cookie rename orphaned, polling
+SignalR forever, silently redirected every time, and **never re-prompting the operator**. It is the
+reported symptom — "reloading never fixes it and I am never re-prompted" — still happening in
+production, on a service nobody was looking at. With 401 the UI's connection breaks **loudly**; the
+operator reloads; `/` matches neither regex, so the reload signs them back in.
+
+**`^/api/` on radarr: included, and here is the honest basis.** The current pod has logged **zero**
+`/api/` traffic, so I did **not** re-measure it. It rests on a measurement of **radarr** recorded in
+the merged PR #30 handoff — its 156 unauthenticated requests were `/api/v3/command` (62),
+`/api/v3/queue/details` (46) and `/signalr/messages/negotiate` (43), i.e. **108 of 156 were
+`/api/v3` XHRs**. That is this app's own evidence, by another engineer, not an analogy to another
+app. I include it because `/api/v3/` is a pure JSON namespace a browser never navigates to (Radarr
+serves its UI at `/`, assets under `/Content/` and `/MediaCover/`), so the breadth risk is nil and
+the measured benefit is 70% of that app's rejection volume.
+
+**Sonarr and qBittorrent were measured and deliberately left alone.** Sonarr is nearly the same
+codebase as Radarr, which is exactly what makes `^/api/` + `^/signalr/` a tempting guess and still a
+guess. I read both pods' logs: **zero** unauthenticated requests, nothing but health probes, because
+nobody has opened either UI since the cookie fix rolled. **That is evidence of no traffic, not
+evidence of no need.** Each catalog file now carries a comment saying so, with the numbers and what
+to measure before adding the routes.
+
+### Goldens and fixtures regenerated
+
+Rendered with the canonical command
+(`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`).
+**Two goldens changed, not five** — and that is the per-app property proving itself on the real
+tree rather than in a fixture:
+
+    apps/wave-0/jackett.yaml   + --api-route=^/api/
+    apps/wave-1/radarr.yaml    + --api-route=^/api/
+                               + --api-route=^/signalr/
+
+`apps/wave-0/pihole.yaml`, `apps/wave-0/qbittorrent.yaml` and `apps/wave-1/sonarr.yaml` are
+**byte-identical to `main`**. **No L1 fixture needed regenerating**: all three
+`conformance/tests/l1-rendering/*/fixtures/svcdef*.yaml` declare `auth.mode: forward-auth` with no
+`apiRoutes`, so they render no flag and their goldens are unchanged. PR #28 touched 5 goldens and 3
+L1 fixtures because it changed flags on *every* sidecar; this one changes flags only where a
+measurement says so, which is the whole point.
+
+### The L0 decision (and it is a decision, not a default)
+
+**Not** in `oauth2-proxy-cookie-names-distinct`. That check's subject is cookie **identity** — its
+inputs are names and domains, its failure mode is mutual session eviction, its register is the
+shared proxy's claim on the default name. `--api-route` shares none of those. Bundling them would
+make that check's name a lie and couple two unrelated ratchets, so a future api-route decision could
+only be expressed by weakening a *cookie* check. That is PR #28's own stated discipline, applied to
+my change rather than quietly dropped.
+
+**A sibling check, `oauth2-proxy-api-routes-per-app`** (whole-repo scope, so it runs on every gate
+invocation via `ci/l0.sh`'s `--include-scope` union — no bespoke stage, which
+`whole-repo-invariants-wired` would fail). **Nowhere was the wrong answer for exactly one reason:**
+the existing gates cannot see a renderer constant. `render-golden` compares the render to the
+goldens, so a constant **plus regenerated goldens agree with each other while both being wrong** —
+I provoked that and it regenerates with **rc 0**. The new check's assertion (2) compares each
+rendered sidecar against its **own ServiceDefinition**, which is the only reading that fails it.
+
+## What I verified
+
+- **`--api-route` is SINGULAR, read off the PINNED binary rather than taken from the brief.**
+  `podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help`:
+  `--api-route strings   return HTTP 401 instead of redirecting to authentication server if token
+  is not valid. Format: path_regex`. There is **no `--api-routes`** in the help output. The PM asked
+  me to verify rather than trust, and that is the result — the singular form is confirmed and the
+  help text also confirms the *semantics* I am relying on (401 instead of a redirect).
+- **The premise, in-cluster, read-only, through the pinned toolbox, `--context k3s-prod` named
+  explicitly** (CLAUDE.md §2b and §5). PR #28 **is** deployed: all five sidecars carry
+  `--cookie-name=_oauth2_proxy_<app>`, `--cookie-domain=<app>.home.keithjackson.dev`,
+  `--cookie-refresh=2m30s` and `--metrics-address=0.0.0.0:44180`. Pods, all **0 restarts**:
+  `jackett-6877fcd9f5-ht9d9`, `pihole-…`, `qbittorrent-5547bdb9bd-zq557`,
+  `radarr-7d55cbffb9-42hmh`, `sonarr-ddf485775-8ckwt` (started 2026-09-30T21:41–21:42Z).
+- **The full post-fix log census, parsed rather than grepped** (oauth2-proxy's access-log format,
+  with `/ping` health probes excluded), over each pod's whole life:
+
+      app            lines   unauth  AuthSuccess  removing session  cookie sig invalid
+      jackett       13,649        0            0                 0                   0
+      pihole        13,649        0            0                 0                   0
+      qbittorrent   13,649        0            0                 0                   0
+      radarr        14,099      232            0                 0                   0
+      sonarr        13,643        0            0                 0                   0
+
+  Three findings, kept separate:
+  1. **The collision fix works.** Jackett was 39 unauthenticated before; it is **0** in 13,612
+     non-probe lines. The eviction fingerprint (`removing session` / `cookie signature not valid`)
+     is **0 on all five**, where radarr previously had 8 and 8.
+  2. **Radarr's 232 are one request.** 100% `POST /signalr/messages/negotiate`, 100% **302**, 100%
+     upstream `-`, browser UA (`Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537…`). **Zero `/api/`
+     traffic of any kind** on this pod.
+  3. **0 `AuthSuccess` anywhere**, so nobody has signed in since the restart and `--cookie-refresh`
+     is **still** unexercised — unchanged from PR #28's own caveat.
+- **Only two goldens change, and three are byte-identical to `main`.** `git status` after the
+  canonical render lists exactly `apps/wave-0/jackett.yaml` and `apps/wave-1/radarr.yaml`.
+  `render-golden` is green.
+- **`catalog/render.py --selftest`: PASS**, with new assertions that (a) the `fa` fixture, which
+  declares nothing, renders **no** `--api-route`; (b) a fixture declaring two routes renders exactly
+  those two, in order; (c) the renderer **refuses** all five breadth shapes; (d) four legitimate
+  routes (`^/api/`, `^/signalr/`, `^/api/v3/.*`, `^/api/v2.0/server/update$`) are **accepted** — a
+  ratchet that blocks real API prefixes is a ratchet someone eventually deletes.
+- **The new check: rc 0, 43 assertions, 6 sidecars audited**, with its inventory printed so the
+  reviewer can see what it read:
+  `jackett -> ['^/api/'] | pihole -> (none) | qbittorrent -> (none) | radarr -> ['^/api/',
+  '^/signalr/'] | sonarr -> (none) | crafty -> (none)`.
+- **The check was tested in BOTH directions, on throwaway copies outside the repo**, never by
+  mutating the branch. Each mutation fired its own assertion class and nothing else:
+  - **Pre-change tree** (`git archive origin/main` + the test dropped in): **rc 1, 11 failures** —
+    no service declares routes, the renderer emits none, the renderer does not read `auth.apiRoutes`,
+    the pattern is absent from both renderer and schema, and the shipped renderer accepts all five
+    breadth shapes.
+  - **pihole's golden hand-edited** to carry a route it never declared: **rc 1, 2 failures** (the
+    catalog↔golden mismatch, and "declares nothing must render nothing").
+  - **radarr's golden widened** to `--api-route=^/`: **rc 1, 2 failures** (unanchored, plus the
+    mismatch).
+  - **Renderer turned into a constant and EVERY golden regenerated to match: the regen exits 0** —
+    i.e. `render-golden` is **happy** — and the new check exits **1 with 15 failures**, naming
+    pihole, qbittorrent, radarr and sonarr individually. **This is the case that justifies the
+    check's existence** and I provoked it rather than argued it.
+  - **Schema `items.pattern` deleted**: **rc 1, 1 failure** — the three-surface drift guard.
+  - **Unmutated control: rc 0.**
+- **The selftest diagnostics fix was PROVOKED, not reasoned.** Both mutations above were run
+  against throwaway copies of the renderer outside the repo (`origin/main`'s `render.py` for the
+  "before" column, mine for the "after"), never by mutating the branch. The pre-fix StopIteration
+  and the post-fix 3-and-20 named assertions are quoted from those runs. `--selftest` on the
+  unmutated branch: **PASS**. Goldens after the fix: **unchanged** (it touches only `_selftest()`).
+- **The check is selected by the whole-repo union.** `./conformance/run.sh --layer l0 --requirements
+  r-nosuchreq-99 --include-scope whole-repo --print-selection` lists
+  `tests/l0-static/oauth2-proxy-api-routes-per-app` among **21** tests (it was 20), i.e. it runs
+  even when the touched-requirement filter matches nothing.
+- **THE SCHEMA GUARD IS PROVEN WITH THE PINNED VALIDATOR `ci/l0.sh` ACTUALLY DRIVES**, not by
+  reading the pattern. `./schemas/validate.sh` (`check-jsonschema, version 0.29.4`) against seven
+  throwaway ServiceDefinitions written outside the repo:
+
+      good            ["^/api/", "^/signalr/"]   rc 0
+      bad-root        ["^/"]                     rc 1  apiRoutes ... does not match
+      bad-dotstar     ["^/.*"]                   rc 1  apiRoutes ... does not match
+      bad-unanchored  ["/api/"]                  rc 1  apiRoutes ... does not match
+      bad-alt         ["^/api/|^/"]              rc 1  apiRoutes ... does not match
+      bad-everything  [".*"]                     rc 1  apiRoutes ... does not match
+      bad-mode        ["^/api/"] on mode: oidc   rc 1  const 'forward-auth'
+
+  So a too-broad regex is rejected **where a human types it**, on every gate run, by the real tool
+  — and the shape invariant fires too.
+- **THE RENDERER'S REFUSAL REACHES `render-golden` AS A READABLE FAILURE, NOT A TRACEBACK.** With
+  `^/signalr/` swapped for `^/` in a *copy* of the catalog (outside the repo), both invocation
+  shapes exit **rc 1 with 0 tracebacks** — the single-SD form and `--all <dir> -o <dir>
+  --wave-layout --no-image-automation`, which is the one `ci/render-golden.sh` drives:
+
+      render.py: radarr: auth.apiRoutes entry '^/' is not an anchored path regex. Each entry
+      must start with '^/' and continue with literal path characters (e.g. '^/api/',
+      '^/signalr/', '^/api/v3/.*'). ... A too-broad --api-route answers a PAGE LOAD with 401, so
+      a logged-out operator sees a bare error page and is never offered a sign-in: the service
+      looks permanently broken rather than merely unauthenticated.
+
+  So the breadth guard is gated by `render-golden` even on a run where the schema stage skipped,
+  and the operator is told *why* rather than handed a stack trace.
+- **The schema's new shape invariant.** A non-empty `auth.apiRoutes` requires
+  `auth.mode: forward-auth`, because the flag renders onto the per-app sidecar and no sidecar exists
+  for `oidc`/`native`/`none` — declaring it elsewhere would be a **silent no-op**, which is the
+  failure class this repo keeps paying for.
+- **Gate:** toolbox `./ci/gate.sh` with the two-mount ENGINE shim — exit code and stage list in the
+  next section, because it is not a pass and that belongs there.
+- **No mutating cluster access.** `kubectl get/logs/jsonpath` only, `--context k3s-prod` named on
+  every call. No `apply`/`edit`/`patch`, no `flux reconcile`. No Secret `.data` or `.stringData` was
+  read or printed; I did not need to touch a Secret at all this session. The one container I ran
+  locally was the pinned oauth2-proxy image, to read its `--help`.
+- **`~/.git-credentials` was never read.** PR creation used `git credential fill`, which does not
+  echo the value.
+
+## What I did NOT verify
+
+- **NO BROWSER ROUND-TRIP. BY ANYONE. FOR THIS FLAG.** This is the most important line here. The
+  302/401 split, the dropped POST bodies and the orphaned SignalR poll are all read from access
+  logs; that a **401 repairs** the operator's experience is reasoned from HTTP semantics, not
+  watched. Nobody has added a Jackett indexer or recovered a Radarr tab through this
+  configuration. The human test below is the first real evidence this change will ever have.
+- **Local gate: rc 3 (UNDETERMINED), not 0.** Green: `spec-immutability`, `decisions-log`, `lint`,
+  `l0`, `render-golden`, `l1`. **Skipped:** `gitea-settings` (no `SETTINGS_API_TOKEN` locally) and
+  **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset). Nothing FAILED; nothing runtime was certified
+  either. **rc 3 is UNDETERMINED, never green.**
+- **MY LOCAL `approval-policy: green` WAS MEANINGLESS, AND I ALMOST REPORTED IT AS EVIDENCE.**
+  `ci/approval-policy.sh:47` reads `GITHUB_EVENT_NAME`, which is unset outside CI, so the stage
+  short-circuits at line 50: `ok "not a pull_request event (event='none') — approval policy N/A"`.
+  **A clean local gate therefore CANNOT catch a sensitive-path problem — structurally, not by
+  accident.** Flagged to me by the coordinator from Ines Dubois's round, and I verified it by
+  re-running the gate with `GITHUB_EVENT_NAME=pull_request PR_NUMBER=34`: the stage then
+  **evaluates** and fails closed on the missing token —
+
+      [fail] no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews.
+      [fail] stage approval-policy: FAIL
+      [fail] approval-policy: a sensitive change lacks a non-author approval -> blocks merge
+      [fail] MERGE GATE: BLOCKED        (rc 1, not 3)
+
+  — while **every other stage stays green** (`spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1`). That is the useful result: it bounds the CI failure to this one stage
+  rather than leaving "something is red" as the whole report. Fail-closed is correct here and
+  deliberate: unlike `gitea-settings`, where a missing credential is a lock-out, a skip of this
+  stage would be a bypass (CLAUDE.md §1).
+- **CI, read from each commit's `/statuses` rather than from a tick.** Both contexts report, and
+  **only one of them runs**. Identical on `99560c2d`, `df1b25b6` and `f1d16f44`:
+
+      gate / gate (pull_request)         state=failure   desc='Failing after ~1m43-1m45s'
+      gate / conformance (pull_request)  state=success   desc='Has been skipped'
+
+  **That second line is §2b's trap in the wild**: a *skipped* Gitea job reports
+  `state=success`. The conformance job did not execute, so its green tick certifies nothing —
+  read the description, not the colour. The duration is given as a range on purpose: it differs
+  by a second or two per run, and pinning one number inside the file whose commit triggers the
+  next run is a self-reference, not a measurement. The load-bearing fields are the **state** and
+  the **description**, and those are stable across all three.
+
+  Worth recording that the **same context changed its answer over time**: on the first push it
+  briefly read `state=pending desc='Blocked by required conditions'` — still waiting on the gate
+  job it depends on — and resolved to `success` only once that job settled. So sampling a status
+  once can show either a pending **or** a green tick for a job that never executed.
+  **I could NOT read the CI job log to confirm the failing stage from CI's
+  own output**: this Gitea's `/api/v1/repos/{owner}/{repo}/actions/runs` endpoint answers
+  **404** (only `/actions/tasks` exists, which lists tasks without logs), and
+  `/actions/jobs/{task_id}/logs` answers `404 Not found` for the task id. So my attribution of the
+  CI failure to `approval-policy` rests on the local event-set reproduction above plus the
+  sensitive-path rule, **not** on CI's own log. That is weaker evidence than PR #28 had and I am
+  naming it rather than borrowing its wording.
+- **Branch protection and the runner, RE-MEASURED this session against the live API** (§2b),
+  not inherited from CLAUDE.md §2's table:
+
+      GET /repos/Home-Network/home-network-k3s-impl/branches/main
+        protected              True
+        user_can_push          False
+        enable_status_check    True
+        status_check_contexts  ['gate / gate (pull_request)']
+        required_approvals     1
+      GET /admin/actions/runners
+        id=1 name=incluster-dind status=online
+
+  So `required_approvals=1` applies to **every** PR here regardless of path — this needs a human
+  even without the sensitive-path rule. I did **not** provoke a rejected push to `main` this
+  session; `user_can_push: False` is the API's word, which is exactly the weaker form of evidence
+  CLAUDE.md §2 warns about, and I did not strengthen it.
+- **`^/api/` ON RADARR IS NOT MY MEASUREMENT.** It is PR #30's, taken on the **pre-fix** pod. The
+  current pod has logged **zero** `/api/` traffic in 23 h, so I could not reproduce it. I judge the
+  breadth risk nil (`/api/v3/` is pure JSON; the UI is at `/`), but if someone wants a strictly
+  self-measured configuration, drop `^/api/` and keep `^/signalr/` — the 232 are entirely SignalR.
+- **SONARR AND QBITTORRENT ARE UNMEASURED, NOT CLEARED.** Zero traffic in 23 h is zero evidence.
+  qBittorrent's WebUI drives everything through `/api/v2/` and is *plausibly* affected the same way
+  Jackett is; Sonarr is nearly Radarr's codebase. **Both are left alone on purpose** — plausibility
+  is what put a renderer constant on the table in the first place. Expect to add `^/api/` to
+  qBittorrent and `^/api/` + `^/signalr/` to Sonarr once someone opens those UIs and records the
+  split. **This PR does not fix them.**
+- **PiHole is asserted to serve no `/api/` on inherited evidence, not mine.** Its log shows 0
+  unauthenticated requests, which says nothing about what paths it serves. The claim comes from
+  PR #28/#30. Note the pinned `update.range` is `>=2024.0.0`, and PiHole v6 *does* introduce a REST
+  API at `/api/` — so if PiHole is ever upgraded across that line, this needs re-measuring. It is
+  still correct to declare nothing today, because declaring an unmeasured route is the mistake.
+- **`--cookie-refresh=2m30s` has STILL never been exercised** — 0 `AuthSuccess` on all five pods.
+  PR #28 named this as the fix most likely to need a second pass and it remains unobserved. My
+  change does not touch it, but a sign-in during the human test below will exercise both at once,
+  so a failure there must be attributed carefully.
+- **The regexes are not validated against the apps' real route tables.** I assert `/api/v3/` and
+  `/signalr/` are JSON/RPC namespaces a browser never navigates to from Radarr's and Jackett's
+  observed traffic and from how those UIs are built — **not** from reading their routing code or
+  enumerating their endpoints. A path I have not thought of that matches `^/api/` and *is*
+  browser-navigated would 401.
+- **`OPTIONS` preflight and non-browser clients are unconsidered.** `--skip-auth-preflight` exists
+  and is not set; nothing in these logs shows a CORS preflight, and Sonarr/Radarr talk to Jackett
+  **directly on `:9117`**, bypassing the proxy entirely — so I do not expect impact. Unverified.
+- **The live pods do not carry this change and nothing re-measures that they will.** git is not the
+  cluster (§2b). **Merging is what deploys**, and it will roll **jackett** and **radarr** only
+  (`strategy: Recreate` on RWO Longhorn volumes, so a brief gap per app, and radarr/sonarr lose
+  direct `:9117` access to Jackett while its pod restarts). Existing sessions survive — no cookie
+  flag changed — so unlike PR #28 nobody is signed out.
+- **No L2/L3 assertion reads any of this.** There is no runtime check that a `/api/` request to a
+  gated app answers 401, and I did not write one; `CLUSTER_AVAILABLE` is unset so it could not run
+  in CI anyway. The whole guard is static.
+- **`catalog/render.py --selftest` is UNGATED IN MY BASE, and I did not wire it.** Desmond's PR #33
+  does that (`ci/render-golden.sh`) and is in review, not merged — `origin/main` at `cdf3e14`
+  contains no stage that drives it, which I checked rather than assumed (`grep -n selftest
+  ci/render-golden.sh` → no match). **So until #33 lands, my new selftest assertions are
+  dev-time only**, and the *gated* half of my breadth ratchet is the schema `pattern` (via
+  `ci/l0.sh`'s schema stage), the renderer's own `raise` (via `render-golden`), and the new L0
+  check. I deliberately did not wire the selftest myself: #33 already does it, and duplicating it
+  here would collide with his branch in the one file he owns. Note the two PRs are **independent
+  but complementary** — his makes the stage real, mine makes its output readable when it fires.
+- **I did NOT verify PR #33's own claims.** The 106-assertion count, the
+  selftest→stage→`gate.sh`→`BLOCKED` propagation chain, and the `--cookie-refresh 2m30s → 10m`
+  mutation being caught by the selftest but not by `render-golden` are all his measurements,
+  relayed to me by the coordinator. I reproduced only the StopIteration defect itself, because that
+  is the part I was asked to fix. If his branch is reworked, nothing in mine depends on it.
+- **The new check does not read the spec repo** and asserts nothing about `../home-network-k3s`
+  (R-TEST-06, read-only). `schemas/renderer-contract.md` is updated in **this** repo only; the
+  canonical spec `docs/13-service-catalog.md` is **not** updated, because it is immutable from here.
+  Someone with spec write access should mirror `auth.apiRoutes` into docs/13 and docs/15.
+- **THE OAUTH2-PROXY SIDECARS' METRICS ARE STILL UNSCRAPABLE, AND THIS PR DOES NOT FIX IT.** See
+  the declined-bundle section above. #28 shipped `--metrics-address` as a prerequisite and said
+  plainly that it was not scraped; Ines has now measured that the listener **serves** and that the
+  scrape is **dropped by each app namespace's default-deny**. Nothing in my change touches that.
+  Anyone reading this entry as "the fleet gates are observable now" is wrong, and the next PR is
+  where that changes.
+- **Alternation is refused, and that is a judgement I may have to revisit.** No app in this catalog
+  needs it, and the flag is repeatable — but if some future app genuinely needs a regex the ratchet
+  rejects, the fix is to widen the pattern **with a reason** on all three surfaces at once, not to
+  delete the ratchet on one.
+- **CI status: two contexts, and only one of them ran.** Quoted in the PR body and in the report to
+  the PM. `approval-policy` **fails by design** until a non-author approving review exists, because
+  the diff touches `conformance/**`, which `infra/gitea/approval-policy.json` lists as sensitive
+  ("the tests and the harness; changing them changes what green MEANS"). That is CLAUDE.md §1
+  working, not a defect. `required_approvals=1` applies to every PR here regardless of path, so this
+  needs a human either way.
+
+### The human test — and how to tell a too-broad regex from a working one
+
+After merge, wait for **new** pod names at 2/2 in `jackett` and `radarr`.
+
+1. **The breadth test FIRST, before anything else, from a signed-out browser** (private window).
+   Open `https://radarr.home.keithjackson.dev/` and `https://jackett.home.keithjackson.dev/UI/Dashboard`.
+   **Each must bounce through Keycloak and land on the app's UI.** If either shows a bare
+   `401 Unauthorized` instead of a sign-in, **the regex is too broad — stop and revert.** That is
+   the one failure this change can introduce, and it is the only test that can see it. A working
+   regex is invisible on a page load; a broken one is total and immediate.
+2. **Jackett, the original report:** signed in, *+ Add indexer* → pick one → *Configure* → *Okay*.
+   **It must save.** Previously 4 of 6 POSTs never reached Jackett at all.
+3. **Radarr, the live symptom:** open Radarr and leave the tab open. DevTools → Network, filter
+   `signalr`. Today those negotiate calls are **302** and the UI never complains. They must now be
+   **200** while signed in — and if the session is ever lost, **401**, with the UI showing a visible
+   connection error instead of polling silently forever.
+4. **The loud-failure test:** in DevTools, delete the `_oauth2_proxy_radarr` cookie and let the tab
+   poll. The negotiate call must answer **401** and Radarr must *show* a problem. Then **reload** —
+   `/` matches neither regex, so the reload must sign you back in silently. That round trip is the
+   entire fix: fail loudly on the API, recover on the page load.
+5. **The six-minute test (PR #28's open item, still open):** stay on a dashboard for >6 minutes,
+   then save something. That exercises `--cookie-refresh=2m30s` past the 300 s access-token cliff.
+   If it fails, suspect the realm's refresh/SSO-idle timeouts — not this change.
+6. **If something still fails, it must now fail LOUDLY.** A silent success-looking no-op means the
+   change did not take: check the running pod's args for `--api-route` before assuming anything
+   else.

platform/oauth2-proxy/README.md

@@ -88,72 +88,3 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
    protected UIs).
-
-## Observability: what this component can and cannot see (measured 2026-09-30)
-
-`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
-two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
-in the TSDB is:
-
-    count by (__name__) ({__name__=~"oauth2_proxy.*"})
-      oauth2_proxy_requests_in_flight => 2
-
-**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
-`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
-each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
-listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
-their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
-written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
-
-This matters because the gate's characteristic failure is invisible by construction: an
-unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
-against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
-where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
-spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
-into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
-
-`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
-`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
-`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
-a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
-one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
-and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
-
-**To close the gap for an app**, three things are needed together, on that app's own manifest:
-`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
-Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
-`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
-session lifetime — become writable rather than decorative.
-
-### Update 2026-10-01: the listener exists; the scrape path does not
-
-PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
-fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
-discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
-rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
-renderer.
-
-The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
-10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
-`code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
-flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
-
-**The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
-the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
-`oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
-`policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
-`fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
-clause.
-
-So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
-and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
-and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
-ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
-(both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
-cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
-
-Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
-present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
-would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
-as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
-fails it and nothing else.

platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml

@@ -1,177 +0,0 @@
----
-# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
-#
-# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
-# failure modes of that shape are both silent: a label that does not survive `count by` makes the
-# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
-# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
-# visible by reading the expression, and neither is visible on a live cluster where the answer
-# happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
-# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
-#
-# HOW TO RUN. The one repo-wide harness, by the companion convention:
-#   ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
-# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
-#
-# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
-# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
-# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
-# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
-# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
-# pass while the real expression collapsed differently.
-#
-# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
-# comparing it verbatim would couple this file to every wording change while proving nothing the
-# ALERTS series does not already prove.
-rule_files:
-  - auth-visibility-alerts.rules.yaml
-evaluation_interval: 1m
-
-tests:
-  # ---------------------------------------------------------------------------------------------
-  # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
-  #    the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
-  #    jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
-  - interval: 1m
-    name: an unscraped gate namespace fires after 1h; the scraped one never does
-    input_series:
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
-        values: '1x130'
-      # The shared instance: kube-state-metrics sees both replicas...
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
-        values: '1x130'
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
-        values: '1x130'
-      # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
-        values: '1x130'
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
-        values: '1x130'
-    promql_expr_test:
-      # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
-      # the hold away and turning every rollout into a page.
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
-        eval_time: 59m
-        exp_samples: []
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
-        eval_time: 59m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
-            value: 1
-      # Firing for jackett past the hold...
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
-        eval_time: 65m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
-            value: 1
-      # ...and never for the namespace that IS scraped. This is the assertion that catches an
-      # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
-      # every gate namespace would pass the test above.
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
-        eval_time: 65m
-        exp_samples: []
-      # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
-      # that lost its `by` would collapse to a single unlabelled series.
-      - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
-        eval_time: 65m
-        exp_samples:
-          - labels: '{namespace="jackett"}'
-            value: 1
-
-  # ---------------------------------------------------------------------------------------------
-  # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
-  #    `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
-  #    name than the shared proxy's. The alert must stop. This is what makes the rule a gap
-  #    tracker rather than a permanent nag, and it pins that the match is on `container`, not on
-  #    a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
-  - interval: 1m
-    name: the alert clears once the sidecar is actually scraped
-    input_series:
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
-        values: '1x130'
-      # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
-      # exist yet.
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
-        values: '_x70 1x60'
-    promql_expr_test:
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
-        eval_time: 65m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
-            value: 1
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
-        eval_time: 80m
-        exp_samples: []
-
-  # ---------------------------------------------------------------------------------------------
-  # 3. THE REGRESSION GUARD FOR `== 1` — the scenario that exposed a real bug on 2026-10-01.
-  #
-  #    A target is DISCOVERED but every scrape FAILS: `up` exists with value 0. This is not
-  #    hypothetical — it is the exact state podmonitor.yaml creates today, because the app
-  #    namespaces' default-deny drops the scrape (measured; see podmonitor.yaml).
-  #
-  #    With the original `unless count by (namespace) (up{container="oauth2-proxy"})`, the `up`
-  #    SERIES EXISTS, so `count by (namespace)` returns 1, `unless` removes the namespace, and the
-  #    alert GOES QUIET while nothing is being scraped at all — a green signal standing in for an
-  #    absent one, which is the precise disease this whole rule was written to cure. Deleting the
-  #    `== 1` makes THIS test fail and every other test in this file still pass, which is the
-  #    property that makes it worth having.
-  - interval: 1m
-    name: a discovered-but-FAILING target does not count as observed (up==0 must still fire)
-    input_series:
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
-        values: '1x130'
-      # The PodMonitor target exists and never succeeds: NetworkPolicy drops the packets, so the
-      # scrape times out and `up` is published as 0 rather than going absent.
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.0.189:44180",job="oauth2-proxy-sidecars",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9"}'
-        values: '0x130'
-    promql_expr_test:
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
-        eval_time: 65m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
-            value: 1
-      # And the inner clause itself yields NOTHING while the scrape is failing — which is the
-      # mechanism the alert depends on, asserted directly so a future reader can see why the
-      # `== 1` matters without re-deriving it. (Scenario 4 below covers the other direction:
-      # a succeeding scrape really does clear the alert, so `== 1` has not made the rule
-      # impossible to satisfy.)
-      - expr: 'count by (namespace) (up{container="oauth2-proxy"} == 1)'
-        eval_time: 65m
-        exp_samples: []
-
-  # ---------------------------------------------------------------------------------------------
-  # 4. A FLAPPING target: up goes 1 -> 0 and stays there. The alert must come BACK, not latch
-  #    cleared. Covers the ordering bug where a once-observed namespace is permanently excluded.
-  - interval: 1m
-    name: a target that starts working and then fails re-fires
-    input_series:
-      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="radarr",pod="radarr-7d55cbffb9-42hmh",service="kube-prometheus-stack-kube-state-metrics",uid="3333cccc-0000-0000-0000-000000000001"}'
-        values: '1x250'
-      # Healthy for the first 60m, then every scrape fails for the rest.
-      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.77:44180",job="oauth2-proxy-sidecars",namespace="radarr",pod="radarr-7d55cbffb9-42hmh"}'
-        values: '1x60 0x180'
-    promql_expr_test:
-      # Quiet while it was working.
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="radarr"}'
-        eval_time: 55m
-        exp_samples: []
-      # Firing again an hour after it broke.
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}'
-        eval_time: 125m
-        exp_samples:
-          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr",severity="warning"}'
-            value: 1
-
-  # ---------------------------------------------------------------------------------------------
-  # 5. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
-  #    rule would page for every app on the platform the day someone widened the matcher.
-  - interval: 1m
-    name: namespaces without a gate sidecar are never reported
-    input_series:
-      - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
-        values: '1x130'
-    promql_expr_test:
-      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
-        eval_time: 65m
-        exp_samples: []

platform/oauth2-proxy/auth-visibility-alerts.yaml

@@ -1,137 +0,0 @@
----
-# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
-#
-# ============================================================================================
-# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
-# ============================================================================================
-# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
-# log in. The sidecar's own access log had the whole story:
-#
-#   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
-#   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
-#   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
-#   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
-#   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
-#
-# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
-# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
-# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
-# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
-# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
-# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
-# gate had rejected every request ever made.
-#
-# ============================================================================================
-# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
-# ============================================================================================
-# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
-# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
-# reason is this file's real subject:
-#
-#   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
-#
-#   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
-#     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
-#
-#   PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave
-#   the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the
-#   endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three
-#   `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml
-#   in this directory now discovers them. But the scrape is still DROPPED by the app namespaces'
-#   default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no
-#   metrics listener at all. The rule below is unchanged in purpose and still has work to do.
-#
-#   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with
-#   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
-#   was never started; the container declared only port `http:4180`, there was no `metrics` port on
-#   any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere
-#   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
-#   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
-#   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
-#   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
-#   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
-#   temptation was refused here rather than ratcheted into a register.
-#
-#   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
-#   path for any sidecar-gated app — each app gates itself — and its current pods export no
-#   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
-#   the shared instance would not have fired on this incident either. Saying so is the point.
-#
-# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
-# That condition is measurable today, from two already-scraped metric families, and it is the
-# precondition for ever being able to alert on the incident itself. It self-clears the moment a
-# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
-# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
-# by-hand equivalent an operator uses in the meantime.
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: oauth2-proxy-auth-visibility
-  namespace: oauth2-proxy
-  labels:
-    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
-    app.kubernetes.io/name: oauth2-proxy
-spec:
-  groups:
-    - name: oauth2-proxy.auth-visibility
-      rules:
-        - alert: OAuth2ProxyGateUnobserved
-          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
-          # target for that container in that namespace.
-          #
-          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
-          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
-          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
-          #     container exports anything, which is exactly the property needed here.
-          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
-          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
-          #     without knowing what job name it will pick.
-          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
-          # inert for the reason it is reporting.
-          #
-          # MEASURED ON PROD 2026-09-30 — this expression returned six namespaces:
-          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
-          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
-          # and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
-          # the state the cluster is in; it is not a rule waiting for a hypothetical.
-          #
-          # `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally
-          # a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters
-          # here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So
-          # the series exists, `count by (namespace)` returns 1, `unless` removes the namespace,
-          # and the alert GOES QUIET while nothing whatsoever is being scraped. That is this
-          # branch's own disease — a green signal standing in for an absent one — reproduced
-          # inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands
-          # five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so
-          # the bare form would have silently cleared on the very next commit. `== 1` means
-          # "observed" requires a scrape that actually succeeds.
-          #
-          # A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's
-          # generic TargetDown. That duplication is deliberate: TargetDown says "a target is
-          # down", this says "an authentication gate is unobserved", and during an incident those
-          # are different sentences. The overlap is the cheap half of the trade.
-          expr: |
-            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
-              unless
-            count by (namespace) (up{container="oauth2-proxy"} == 1)
-          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
-          # discovered yet, and this condition is a standing architectural gap, never an
-          # emergency. Long enough that no deploy can produce it.
-          for: 1h
-          labels:
-            severity: warning
-          annotations:
-            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
-            description: >-
-              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
-              Prometheus has NO scrape target for it, so every authentication decision that gate
-              makes — every redirect to Keycloak, every 403 on a group check, every session that
-              silently fails to hold — is invisible to metrics and to alerting. This is not a
-              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
-              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
-              seconds later, and the only record anywhere was a line in the sidecar's stdout.
-              No alert could have fired, because there is nothing to alert on. Until a scrape
-              exists the runbook's LogQL queries are the substitute; the fix is
-              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
-              ServiceMonitor, after which the auth-churn rule becomes writable.
-            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"

platform/oauth2-proxy/kustomization.yaml

@@ -14,12 +14,3 @@ resources:
   - oauth2-proxy-credentials.sops.yaml
   - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
   - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)
-  # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
-  # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
-  # tier, which is Ready before this tier's platform-config dependency completes.
-  - auth-visibility-alerts.yaml
-  # Discovers the PER-APP gate sidecars' :44180 metrics endpoints by pod label (PR #28 added the
-  # listener; no Service exposes it, so this is a PodMonitor, not a ServiceMonitor). The scrape is
-  # still denied by the app namespaces' default-deny ingress — read podmonitor.yaml's header
-  # before concluding the targets should be up.
-  - podmonitor.yaml

platform/oauth2-proxy/podmonitor.yaml

@@ -1,109 +0,0 @@
----
-# PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
-#
-# ============================================================================================
-# WHY A PodMonitor AND NOT A ServiceMonitor
-# ============================================================================================
-# PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
-# container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
-# Services exposes 44180:
-#
-#   jackett      jackett:     [('port-9117', 9117), ('authproxy', 4180)]
-#   radarr       radarr:      [('port-7878', 7878), ('authproxy', 4180)]
-#   sonarr       sonarr:      [('port-8989', 8989), ('authproxy', 4180)]
-#   qbittorrent  qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
-#   pihole       pihole:      [('port-80',     80), ('authproxy', 4180)]
-#
-# A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
-# pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
-# therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
-#
-# SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
-# Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
-# sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
-# (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
-# PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
-# picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
-# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
-#
-# ============================================================================================
-# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
-# ============================================================================================
-# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
-# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
-#
-#   THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
-#     GET /metrics -> http=200 bytes=10208
-#     # TYPE oauth2_proxy_requests_in_flight gauge
-#     # TYPE oauth2_proxy_requests_total counter
-#     # TYPE oauth2_proxy_response_duration_seconds histogram
-#     oauth2_proxy_requests_total{code="200"} 3
-#     oauth2_proxy_requests_total{code="302"} 2
-#
-#   THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
-#   Prometheus and so occupies the same policy position) to the jackett pod IP:
-#     http://10.0.0.189:44180/metrics   -> wget: download timed out
-#     http://10.0.0.189:4180/ping       -> wget: download timed out     (even the "allowed" port)
-#     http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
-#                                            has NO NetworkPolicy, which is the only reason the
-#                                            shared instance has ever been scrapable)
-#
-#   WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
-#   `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
-#   NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
-#   `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
-#   and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
-#
-# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
-# measured, documented state — NOT a regression introduced here, and NOT something to silence.
-# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
-# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
-# (see that file's header — this exact trap is why the `== 1` is there).
-#
-# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
-# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
-# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
-# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
-# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
-#
-# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
-# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
-#   oauth2-proxy  dependsOn=['platform-config']        <- this tier
-#   apps-wave-0   dependsOn=['platform-config']        <- SIBLING, no ordering between them
-#   apps-wave-1   dependsOn=['apps-wave-0']
-# They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
-# applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
-# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
-# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
-# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
-# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
-# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
-apiVersion: monitoring.coreos.com/v1
-kind: PodMonitor
-metadata:
-  name: oauth2-proxy-sidecars
-  namespace: oauth2-proxy
-  labels:
-    release: kube-prometheus-stack     # kube-prometheus-stack's podMonitorSelector
-    app.kubernetes.io/name: oauth2-proxy
-    homelab.keithjackson.dev/purpose: metrics-scrape
-spec:
-  namespaceSelector:
-    # The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
-    # a new forward-auth app needs no edit here — the pod label below is what scopes this.
-    any: true
-  selector:
-    matchLabels:
-      # The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
-      # 2026-10-01); absent from crafty, which has no metrics listener yet.
-      homelab.keithjackson.dev/auth-mode: forward-auth
-  podMetricsEndpoints:
-    # By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
-    # container, and a numeric port here would silently stop matching if the port were renumbered.
-    - port: metrics
-      path: /metrics
-      interval: 30s
-      # 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
-      # scrape times out. Leaving the default would make each failing target hold a scrape slot for
-      # the full interval.
-      scrapeTimeout: 10s

runbooks/README.md

@@ -51,36 +51,6 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
-| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — updated 2026-10-01 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
-
-The oauth2-proxy row is the newest, and it has **moved twice in two days** — worth reading in order,
-because each step looked like the end of it.
-
-**2026-09-30 — nothing to scrape.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})` returned
-exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
-forward-auth instance. All six per-app gate sidecars ran without `--metrics-address`, so their metrics
-listener never started, and no ServiceMonitor or PodMonitor selected them.
-
-**2026-10-01 — the listener exists, and the scrape is still blocked.** PR #28 added
-`--metrics-address=0.0.0.0:44180` and a named `metrics` port to the five fleet sidecars, and the
-endpoint genuinely serves (port-forwarded: `http=200`, 10208 bytes, all three `oauth2_proxy_*`
-families, `oauth2_proxy_requests_total{code="302"}` present). `platform/oauth2-proxy/podmonitor.yaml`
-now discovers them by pod label. **But the app namespaces' `default-deny` NetworkPolicy drops the
-scrape**: from a pod in ns `monitoring`, both `:44180` and `:4180` on the jackett pod time out, while
-the shared proxy's `:44180` answers — the only difference being that ns `oauth2-proxy` has no
-NetworkPolicy at all. So the targets are discovered and `up == 0`, and the TSDB is still empty for
-those namespaces. **This is the same bug Longhorn had**, with the same fingerprint; the fix idiom is
-`platform/longhorn/networkpolicy-metrics.yaml` (ns monitoring → one pod label → one port), and for
-these apps it has to be rendered alongside each app's own policy rather than shipped from the
-oauth2-proxy tier, which is a *sibling* of `apps-wave-0` in the Flux graph and so cannot rely on those
-namespaces existing yet.
-
-`OAuth2ProxyGateUnobserved` (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on the gap
-itself from `kube_pod_container_info` and `up` — two families that *are* scraped. Note its `up == 1`:
-a discovered-but-failing target publishes `up` with value **0**, so the earlier bare form would have
-gone quiet the moment the PodMonitor landed, while nothing was being scraped. That near-miss is the
-whole reason to distrust a cleared alert here. Crafty remains outside all of this — no metrics
-listener at all. See `oauth2-proxy-auth-failures`.
 
 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
 
@@ -138,14 +108,6 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
   every green signal missed.
-- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
-  `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
-  rather than a 401, the browser follows it silently against a live SSO session, and the call
-  resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
-  re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
-  from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
-  as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
-  so these logs (72h of them) are the only record this class of failure leaves.
 
 ### Operator procedures (no alert links here)
 - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only

runbooks/kustomization.yaml

@@ -85,17 +85,10 @@ configMapGenerator:
       - node-io=node-io.md
       - node-memory=node-memory.md
       - node-memory-swap=node-memory-swap.md
-      # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
-      # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
-      # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
-      - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
       # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
       # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
       # presents as random session loss in ONE app's log and is only legible with both side by side,
       # so the page has to be reachable at 3am without going looking for it in git.
-      # SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
-      # one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
-      # TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
       - oauth2-proxy-session=oauth2-proxy-session.md
       # Operator procedure, not an alert page (no alert links to it — the
       # alert-runbooks-resolvable check treats that as advisory). It is served

runbooks/oauth2-proxy-auth-failures.md

@@ -1,244 +0,0 @@
-# oauth2-proxy auth failures — telling the three layers apart
-
-Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
-does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
-stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
-page is the diagnostic procedure, and it works whether or not the alert ever fired.
-
-## What this means here
-
-Every app behind Keycloak on this platform has **three** places a request can be refused, and
-from a browser they look identical — because two of the three answer with **HTTP 200**.
-
-| Layer | What refuses | What the browser sees |
-|---|---|---|
-| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
-| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
-| 3. The app itself | the app's own login / API key | the app's own 401 or login page |
-
-**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
-redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
-JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
-Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
-dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
-nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
-failed.
-
-The worked example, jackett on 2026-09-30, straight from the sidecar log:
-
-    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
-    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
-    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
-    ... ten of these at 15:30:53
-    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
-                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
-    GET "/UI/Dashboard"        200 44379
-    GET "/api/v2.0/indexers?_=..." 200 1501229
-    ... and 26 seconds later, inside that same 5-minute session:
-    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
-
-A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
-signature of the session being **re-established over and over instead of held** — a cookie that
-another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
-down and not of the app being broken.
-
-**Read this before you trust any silence (updated 2026-10-01).** The five fleet gate sidecars now
-*do* run a metrics listener on `:44180`, and `platform/oauth2-proxy/podmonitor.yaml` discovers
-them — but **the scrape is dropped by the app namespaces' `default-deny` NetworkPolicy**, so those
-targets read `up == 0` and the TSDB holds nothing for them. Crafty has no listener at all. Only
-the shared `oauth2-proxy` namespace instance is genuinely scraped, and it is not in the data path
-for any of these apps. So there is still nothing to graph and nothing that could have alerted.
-**The logs below remain the only record that this class of failure happened.** Check which state
-you are in before you trust a quiet dashboard — "First commands" step 1 tells you in one query.
-
-## First commands
-
-`kubectl` does not exist on the workstation; everything runs in the pinned container, and always
-with an explicit context.
-
-**1. Which namespaces have an unobserved gate** (this is what the alert measures):
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
-     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
-     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"} == 1)"'
-```
-
-**1b. If a namespace is listed, is it "no target" or "target failing"?** The two need different
-fixes and the alert deliberately does not distinguish them:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
-     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
-     "up{container=\"oauth2-proxy\"}"'
-```
-
-No series for the namespace → nothing discovers it (no PodMonitor match; check the pod carries
-`homelab.keithjackson.dev/auth-mode=forward-auth` and a named `metrics` port). A series with
-value **0** → discovered but every scrape fails; get the reason from the target's `lastError`:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
-   curl -s localhost:19090/api/v1/targets | python3 -c "
-import json,sys
-for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
-    if t[\"labels\"].get(\"container\")==\"oauth2-proxy\":
-        print(t[\"labels\"].get(\"namespace\"), t[\"health\"], t.get(\"lastError\",\"\")[:90])"'
-```
-
-`context deadline exceeded` is a **drop**, not a refusal — that is NetworkPolicy, not a dead
-endpoint. Confirm the endpoint itself is fine by port-forwarding straight to the pod, which
-bypasses policy entirely:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n jackett port-forward \
-     $(kubectl --context k3s-prod -n jackett get pod -l app.kubernetes.io/name=jackett -o name | head -1) \
-     14418:44180 & sleep 5; curl -s -o /dev/null -w "http=%{http_code} bytes=%{size_download}\n" \
-     http://127.0.0.1:14418/metrics'
-```
-
-`http=200` with a few KB means the listener is healthy and the path is what is broken. As of
-2026-10-01 that is the state of all five fleet apps, and the fix is an ingress allowance from ns
-`monitoring` to port 44180 — see `platform/longhorn/networkpolicy-metrics.yaml` for the idiom.
-
-**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
-and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
-`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
-     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
-```
-
-Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
-
-**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
-you can ask about the whole fleet and about a window that has already scrolled out of
-`kubectl logs`. Port-forward Loki and query it:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
-   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
-     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
-```
-
-and the denominator, which is what makes the number mean something:
-
-```
-   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
-```
-
-**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
-lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
-ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
-
-| namespace | `No valid authentication` | `AuthSuccess` | ratio |
-|---|---|---|---|
-| crafty | 81 | 1 | 81:1 |
-| radarr | 67 | 1 | 67:1 |
-| jackett | 39 | 4 | ~10:1 |
-| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
-
-All three of those are the failure, not the baseline.
-
-**The cleanest worked case of all — radarr, measured 2026-10-01.** After the cookie fix landed,
-jackett dropped to **0** `No valid authentication` lines in 13,612 (from 39), and `removing
-session` / `cookie signature not valid` went to **0 on both apps** (radarr had 8 and 8 before).
-The eviction fingerprint is gone. But radarr still logged **206** rejections — and sampling 209 of
-them, **every single one was the same request**:
-
-    GET "/signalr/messages/negotiate" ... 302    <- browser UA, over and over
-
-That is one left-open browser tab. Its cookie is gone, its SignalR socket keeps trying to
-reconnect, and every attempt is silently 302'd to Keycloak. Nobody is at the keyboard, so nobody
-is ever re-prompted; the tab just sits there looking like a working page that has stopped
-updating. **This is the entire failure class in one line**: a non-navigation request answered with
-a redirect, forever, with no error surfaced anywhere.
-
-So when you read a rejection count, **always group by request path before concluding anything**:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n radarr logs deploy/radarr -c oauth2-proxy --since=24h \
-     | grep -v "/ping" | grep "No valid authentication" -A 1 \
-     | grep -oE "\"[^\"]+\" HTTP" | sort | uniq -c | sort -rn | head'
-```
-
-One path dominating means an orphaned client, not a broken gate — annoying but harmless, and it
-clears when the tab is closed. Many different paths from one session means the session itself is
-not surviving, which is the real defect. A per-app `--api-route` is the structural fix for the
-first shape: it makes those paths answer **401** instead of 302, so the client fails loudly
-instead of looping. Note that a count of **0 `AuthSuccess`** alongside a high rejection count
-means nobody has signed in since the last restart — so `--cookie-refresh` has never run, and no
-session-lifecycle conclusion can be drawn from that window at all.
-
-**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
-and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
-page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
-shape, and `runbooks/crafty-admin-credential.md` covers it.
-
-**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
-**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
-other's cookie and the loser logs exactly the `No valid authentication` line above. List what
-each gate is configured to emit (arguments only — never print a Secret's data):
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
-     kubectl --context k3s-prod -n $ns get deploy $ns \
-       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
-     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
-```
-
-A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
-share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
-evicting each other, and the symptom is intermittent and depends on which app you visited last.
-
-## Likely causes, most likely first
-
-1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
-   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
-   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
-   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
-   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
-2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
-   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
-   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
-   gate logs the same line. Look for numbered cookie names in the log, and for a session that
-   never survives a single navigation.
-3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
-   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
-   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
-   jackett trace above.
-4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
-   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
-   die early, the refresh is failing, not missing.
-5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
-   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
-   for the `groups:[...]` list on the `[AuthSuccess]` line.
-6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
-
-## When to escalate or stop
-
-- **Stop and hand over if the fix is a manifest change.** Everything above is read-only
-  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
-  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
-  reverts, and in the meantime you have an unreproducible cluster.
-- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
-  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
-  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
-  transcript here exactly this way.
-- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
-  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
-  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
-- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
-  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
-  report is older than three days, the evidence is gone and the only remaining move is to
-  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
-  the reason to close it rather than silence it.

schemas/renderer-contract.md

@@ -23,6 +23,7 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 | `dependencies` | `[]` → reaches **only DNS** (default-deny) | R-NET-01 |
 | `inClusterClients` | `[]` → no direct east-west client; only the Gateway may reach an exposed port | R-NET-01/02 |
 | `auth.mode` | `forward-auth` for any exposed UI | R-IDP-02 |
+| `auth.apiRoutes` | `[]` → the sidecar renders **no** `--api-route`; every unauthenticated request is 302'd to Keycloak | R-IDP-02 |
 | `update.channel` | `stable` | R-UPD-01 |
 | `update.autoRollbackOnFailedHealth` | `true` | R-UPD-03 |
 | `observability.metrics` | `true` | R-OBS-01 |
@@ -260,7 +261,7 @@ platform must render, and cites the requirement each mapping satisfies. Changing
   undeclared), **R-GITOPS-02** (the allowance is authored in the one reviewable object rather than
   surviving as a live-applied cluster object). Auditable via Hubble (R-SEC-03, R-OBS-04).
 
-### `auth` (`mode` / `groups[]`) — identity integration (Keycloak / oauth2-proxy)
+### `auth` (`mode` / `groups[]` / `apiRoutes[]`) — identity integration (Keycloak / oauth2-proxy)
 - `oidc` → register the app as a **Keycloak OIDC client** (declarative realm/client config, GitOps;
   client secret SOPS-encrypted). App does its own login redirect.
 - `forward-auth` → protect the service's `ingressRoutes` at the **Gateway** with **oauth2-proxy →
@@ -271,6 +272,20 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 - `groups[]` → the Keycloak groups permitted access, enforced by the OIDC client mappers /
   oauth2-proxy `allowed_groups`. **Schema requires ≥1 group for `oidc`/`forward-auth`** — access is
   "in an authorized group," not merely "logged in."
+- `apiRoutes[]` (**`forward-auth` only**) → one repeatable **`--api-route=<regex>`** per entry on
+  the per-app oauth2-proxy sidecar, making matching paths answer **401** instead of a **302** to
+  Keycloak. Default `[]` renders **no flag at all**, which is the common case. It exists because
+  oauth2-proxy's built-in AJAX guard fires only on an exact `Accept: application/json`, which
+  these apps' JavaScript does not consistently send: the XHR is 302'd, the browser follows it
+  invisibly, the live SSO session satisfies it silently, and the call resolves 200 carrying the
+  app's **HTML** where the JavaScript expected JSON — and a POST following a 302 is re-issued as
+  a GET with its **body dropped**. It is **per-service and must never become a renderer
+  constant**: the path shape belongs to the app (radarr `/signalr/`, jackett `/api/`, pihole
+  none), and a wrong constant does not merely fail to help — a path a **browser navigates to**,
+  answered 401, shows a logged-out operator a bare error page and never offers a sign-in. Each
+  entry must be **anchored at `^/`** and continue with literal path characters; `^/`, `^/.*`,
+  `.*`, an unanchored `/api/` and any alternation are **rejected** by the schema and by the
+  renderer (alternation is refused rather than supported because the flag is repeatable).
 - Keycloak, oauth2-proxy, and app→Keycloak / proxy→Keycloak / Keycloak→DB flows are ordinary
   workloads under default-deny with explicit allows (ties back to `dependencies`).
 - **Requirements:** **R-IDP-02** (protect OIDC-capable and bare UIs), **R-IDP-06** (group-based
@@ -329,6 +344,12 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 ## Cross-field invariants the renderer / L0-L1 tests rely on
 1. **kind ↔ source:** container⇒`image`, vm⇒`vmTemplate` (schema-enforced).
 2. **auth ↔ groups:** `oidc`/`forward-auth` ⇒ non-empty `groups` (schema-enforced; R-IDP-06).
+2a. **auth ↔ apiRoutes:** a non-empty `auth.apiRoutes` ⇒ `auth.mode: forward-auth`
+   (schema-enforced). It renders onto the per-app oauth2-proxy sidecar, and no sidecar exists for
+   `oidc`/`native`/`none`, so declaring it elsewhere would be a **silent no-op** — the operator
+   believes the app's XHRs now answer 401 and nothing changed. Every entry must be anchored at
+   `^/` (schema-enforced **and** enforced in `catalog/render.py`, so a breadth mistake is caught by
+   `render-golden` even on a run where the schema stage did not execute).
 3. **default-deny is unconditional:** every namespace gets the deny baseline regardless of
    `dependencies`; `dependencies` only *adds* allows (R-NET-01/02). The same holds for
    `inClusterClients`: it adds one namespace+port ingress allow and never relaxes the baseline.

schemas/servicedefinition.schema.json

@@ -2,7 +2,7 @@
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
   "title": "ServiceDefinition",
-  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 Line truncated
+  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked proLine truncated
   "type": "object",
   "additionalProperties": false,
   "required": ["name", "source"],
@@ -324,6 +324,16 @@
           "items": { "type": "string", "minLength": 1 },
           "uniqueItems": true,
           "default": []
+        },
+        "apiRoutes": {
+          "type": "array",
+          "description": "forward-auth ONLY: path regexes this app serves as a MACHINE API, which the oauth2-proxy sidecar must answer with 401 instead of a 302 to Keycloak (one --api-route per entry; the flag is repeatable and singular -- --api-routes does not exist in v7.8.1 and the proxy refuses to start with it). WHY THIS IS PER-APP AND NOT A RENDERER CONSTANT: oauth2-proxy's built-in AJAX guard fires only on an EXACT `Accept: application/json`, which these apps' JavaScript does not consistently send, so an unauthenticated XHR gets a 302 that the browser follows invisibly, the live Keycloak SSO session satisfies silently, and the call resolves 200 carrying the app's HTML where the JavaScript expected JSON -- no prompt, no error, and on a write it is worse, because a browser following a 302 re-issues a POST as a GET and DROPS THE BODY. Measured on jackett 2026-09-30: 33 x 302 against 2 x 401 on /api/ paths, with 4 of 6 POSTs answered by the proxy and never reaching the app (access-log upstream field `-`). Measured on radarr 2026-10-01, 23h after the cookie fix deployed: 232 x 302, ALL of them POST /signalr/messages/negotiate from the operator's own browser -- a left-open tab whose cookie the rename orphaned, polling forever, silently redirected every time and NEVER re-prompting. But the path shape is a property of the APP, not of the pattern: radarr's broken path is /signalr/ and jackett's is /api/, and pihole serves no /api/ at all, so a renderer-wide constant would apply one app's path shape to five. Declare only paths this app actually serves as JSON/RPC and that a browser never NAVIGATES to. BREADTH IS THE DANGER: too broad and a plain page load answers 401 instead of signing the operator in, i.e. the service looks permanently broken to a logged-out user and no sign-in is ever offered. The item pattern therefore REQUIRES each entry to be anchored at `^/` and to continue with literal path characters, which rejects `^/`, `^/.*`, `.*` and any unanchored or alternated regex; alternation is refused rather than supported because the flag is repeatable, so one regex per route costs nothing and keeps each entry readable. N/A for oidc/native/none (no sidecar is rendered).",
+          "items": {
+            "type": "string",
+            "pattern": "^\\^/[A-Za-z0-9._~/-]+(\\.\\*|\\$)?$"
+          },
+          "uniqueItems": true,
+          "default": []
         }
       }
     },
@@ -471,6 +481,26 @@
         }
       }
     },
+    {
+      "$comment": "auth.apiRoutes only has a consumer under forward-auth: it renders --api-route onto the per-app oauth2-proxy sidecar, and no sidecar is rendered for oidc/native/none. Declaring it elsewhere would be a SILENT NO-OP -- the operator believes the app's XHRs now answer 401 and nothing changed -- so it is rejected at the schema instead (R-IDP-02).",
+      "if": {
+        "required": ["auth"],
+        "properties": {
+          "auth": {
+            "required": ["apiRoutes"],
+            "properties": { "apiRoutes": { "minItems": 1 } }
+          }
+        }
+      },
+      "then": {
+        "properties": {
+          "auth": {
+            "required": ["mode"],
+            "properties": { "mode": { "const": "forward-auth" } }
+          }
+        }
+      }
+    },
     {
       "$comment": "public exposure must not be unauthenticated: an explicit auth.mode 'none' is disallowed on public services (docs/13 v1.1, docs/15, R-IDP-02). Absent auth defaults to forward-auth at render time, so it is permitted here.",
       "if": {