Merge branch 'main' into fix/plex-quicksync-verification

Reviewed-on: #38
Reviewed-by: Keith Jackson <[email protected]>

CLAUDE.md

@@ -36,6 +36,36 @@ sensitive path, so the policy cannot be widened or narrowed without a human.
 
 That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
 missing credential is a lock-out, here a skip would be a bypass.
+
+**The administrator override (2026-10-01).** The rule above was unsatisfiable for the one human
+who runs this repo: Gitea refuses to let an author approve their own PR, and `jakman85` is the
+only collaborator — so a sensitive-path PR the operator opened could never be merged, *including
+the PR that would fix that.* An administrator may therefore waive the review by posting this on
+the PR:
+
+    /approve-override <reason>
+
+**Post it as a PR *review* of type "Comment", not in the plain comment box.** CI's
+`SETTINGS_API_TOKEN` is minted `[read:repository, read:admin]`, and `/issues/{n}/comments` sits
+behind Gitea's *issue* scope — so an override typed into the comment box is **invisible to CI**,
+and the stage reports it as a missing approval rather than as a permission problem (measured on
+PR #38, 2026-10-02). `/pulls/{n}/reviews` is readable, so a review body works today. Both sources
+are accepted and the gate names which it used; the comment box starts working once
+`infra/gitea/set-ci-token.sh` is re-run to pick up `read:issue`.
+
+The directive must **start a line**, the **reason is mandatory**, the poster's `admin`/`owner`
+permission is re-checked **live** when the gate runs, and it must be **newer than the
+head commit** — push again and you must override again. The overrider, the timestamp and the
+reason are printed in the gate log and the job summary. For an administrator, "a change to the
+gate cannot be gated by the gate" is therefore **advisory, not structural**; what is kept is an
+attributed record of each waiver. Note that the agent account `gitea-admin` also holds `owner`
+here, so an agent *can* clear this on its own PR — **do not**: posting that directive to unblock
+your own agent work is exactly the self-approval the stage exists to prevent. Get the operator
+to post it, or move the change out of the sensitive paths.
+
+**`required_approvals` on `main` is `0`**, not 1 — the gate is the approver. That drops the
+approval *count* only: `block_admin_merge_override` and `enable_status_check` both remain `true`,
+so **a red gate is unmergeable by everyone, administrators included.**
 - **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
   anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
 
@@ -53,12 +83,19 @@ Verified against the live Gitea on 2026-09-29:
 | `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
 | PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
 | Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
-| Approval required before merge | ✅ `required_approvals: 1` |
+| Gate cannot be overridden, by anyone | ✅ `block_admin_merge_override: true` |
+| Human approval required before merge | ⬜ **No — `required_approvals: 0` since 2026-10-01.** The gate is the approver; sensitive paths are gated by `ci/approval-policy.sh` instead. See §1 |
 
-All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
+The first three were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
 true. They are true now because each was measured, and the push rejection was provoked rather
 than assumed. **Re-measure before relying on any of them** — see §2b.
 
+The fourth row deliberately changed on 2026-10-01. `required_approvals: 1` was a **lock-out**, not
+a protection: with a single collaborator and Gitea refusing author self-approval, nothing could
+satisfy it (measured on PR #35 — green gate, zero reviews, unmergeable). It also contradicted the
+model already written into `ci/approval-policy.sh` and `ci/gate.sh`. Rationale and the live-change
+script: `infra/gitea/allow-admin-self-merge.sh`.
+
 Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
 CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
 

apps/wave-0/jackett.yaml

@@ -100,6 +100,7 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
+        - --api-route=^/api/
         - --cookie-name=_oauth2_proxy_jackett
         - --cookie-domain=jackett.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}

apps/wave-0/plex.yaml

@@ -170,6 +170,28 @@ spec:
     port: 32400
     targetPort: 32400
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-radarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-radarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: radarr
+        k8s:io.kubernetes.pod.namespace: radarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-seerr-32400
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -192,6 +214,28 @@ spec:
       - port: '32400'
         protocol: TCP
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-sonarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-sonarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: sonarr
+        k8s:io.kubernetes.pod.namespace: sonarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): plex-allow-gateway-ingress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-1/radarr.yaml

@@ -153,6 +153,8 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
+        - --api-route=^/api/
+        - --api-route=^/signalr/
         - --cookie-name=_oauth2_proxy_radarr
         - --cookie-domain=radarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
@@ -281,6 +283,28 @@ spec:
       - port: '9117'
         protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): radarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-plex-32400
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): radarr-allow-seerr-7878
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-1/sonarr.yaml

@@ -281,6 +281,28 @@ spec:
       - port: '9117'
         protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): sonarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-plex-32400
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): sonarr-allow-seerr-8989
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

catalog/jackett.yaml

@@ -51,6 +51,23 @@ auth:
     # This grants no real privilege: platform-admins already maps to cluster-admin via
     # k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
     - platform-admins           # operators of these services must not be locked out of them
+  # Answer Jackett's OWN API with 401 instead of a 302 to Keycloak (schema 1.9.0; renders
+  # --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # MEASURED on this pod's access log, 2026-09-30: on /api/ paths, 33 x 302 against 2 x 401 —
+  # and 4 of 6 POSTs were answered BY THE PROXY and never reached Jackett (access-log upstream
+  # field `-`, not `/`), including both `/api/v2.0/server/update` calls. Jackett's jQuery does
+  # not send the exact `Accept: application/json` oauth2-proxy's AJAX guard requires, so the
+  # 302 was followed invisibly, satisfied silently by the live Keycloak SSO session, and
+  # resolved 200 carrying the dashboard's HTML where the JavaScript expected JSON. A browser
+  # following a 302 also re-issues a POST as a GET and DROPS THE BODY, which is why "adding an
+  # indexer silently does nothing" was the user-visible symptom and nothing logged a failure.
+  #
+  # SCOPE: `/api/` only. Jackett serves its UI at /UI/ and its static assets elsewhere, and a
+  # browser never NAVIGATES to /api/ — so a plain page load is still 302'd into a silent SSO
+  # sign-in, which is the behaviour that must not change.
+  apiRoutes:
+    - ^/api/
 update:
   channel: stable               # R-UPD-01
   range: ">=0.22.0"             # stay on the Jackett 0.22.x stable line

catalog/pihole.yaml

@@ -107,6 +107,13 @@ auth:
   mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
   groups:
     - platform-admins         # sensitive admin UI: authenticated AND in group (R-IDP-06)
+  # NO `apiRoutes` HERE, DELIBERATELY. PiHole's admin UI serves no /api/ path, so the `^/api/`
+  # that jackett and radarr need would match nothing on a good day — and a BROADER guess would
+  # answer PiHole's own page loads with 401, leaving a logged-out operator on a bare error page
+  # with no sign-in offered. This app is the reason auth.apiRoutes is per-service (schema 1.9.0)
+  # rather than a renderer constant: the measurement that justifies the flag on two apps is
+  # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
+  # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
 update:
   channel: stable             # R-UPD-01
   range: ">=2024.0.0"

catalog/qbittorrent.yaml

@@ -448,6 +448,12 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE: UNMEASURED, not cleared. qBittorrent's WebUI drives everything through
+  # /api/v2/, so it is plausibly affected in the same way jackett is — but plausibility is what
+  # put a renderer constant on the table in the first place. MEASURED 2026-10-01 on the live pod
+  # (qbittorrent-5547bdb9bd-zq557, 0 restarts, 23h): 13,649 log lines, ZERO unauthenticated
+  # requests, nothing but health probes. No traffic means no evidence either way. Add `^/api/`
+  # when someone opens the WebUI and records the 302/401 split, as was done for jackett.
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the qBittorrent 5.x stable line

catalog/radarr.yaml

@@ -100,6 +100,9 @@ dependencies:
   - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 9117
     protocol: tcp
+  - to: plex
+    port: 32400                 # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    protocol: tcp
 ingressRoutes:
   - host: radarr.${BASE_DOMAIN}
     port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
@@ -112,6 +115,32 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # Answer Radarr's machine endpoints with 401 instead of a 302 to Keycloak (schema 1.9.0;
+  # renders --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # `^/signalr/` — MEASURED ON THE LIVE POD, 2026-10-01, 23h after the cookie fix deployed
+  # (radarr-7d55cbffb9-42hmh, 0 restarts): 232 unauthenticated requests in the pod's whole
+  # lifetime and ALL 232 are the same call — `POST /signalr/messages/negotiate` from a browser
+  # UA, answered 302 by the proxy with upstream `-`, i.e. Radarr never saw one of them. That is
+  # a left-open tab whose session the cookie rename orphaned, polling SignalR forever, silently
+  # redirected every time and NEVER re-prompting the operator. It is the reported symptom
+  # ("reloading never fixes it and I am never re-prompted") still running in production on a
+  # service nobody was watching. With 401 the UI's connection breaks LOUDLY, the operator
+  # reloads, and `/` is outside both regexes so the reload signs them back in.
+  #
+  # `^/api/` — measured on radarr, but on the PRE-FIX pod and NOT by me: the 2026-09-30
+  # diagnosis (decisions/2026-09-30-jackett-oauth2-proxy-session.md) recorded radarr's 156
+  # unauthenticated requests as `/api/v3/command` (62), `/api/v3/queue/details` (46) and
+  # `/signalr/messages/negotiate` (43). The current pod has logged ZERO /api/ traffic, so I did
+  # not re-measure it; it is included because 108 of 156 measured rejections on this app were
+  # /api/v3 XHRs, which is this app's own evidence rather than an analogy to another app.
+  #
+  # SCOPE: both are JSON/RPC namespaces a browser never NAVIGATES to. Radarr's UI is served at
+  # `/` with assets under /Content/ and /MediaCover/, none of which match — so a plain page load
+  # is still 302'd into a silent SSO sign-in.
+  apiRoutes:
+    - ^/api/
+    - ^/signalr/
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the Radarr v5 stable line

catalog/render.py

@@ -175,6 +175,45 @@ OAUTH2_PROXY_METRICS_PORT = 44180
 # prod nodes 2026-10-01: `.status.allocatable["gpu.intel.com/i915"] == 1`.
 QUICKSYNC_RESOURCE = "gpu.intel.com/i915"
 
+# --- API routes: 401 instead of 302, PER APP ------------------------------------------
+# oauth2-proxy answers an unauthenticated request with a 302 to Keycloak. Its built-in AJAX
+# guard would send a 401 instead, but it fires only on an EXACT `Accept: application/json`,
+# which these apps' JavaScript does not consistently send. So an unauthenticated XHR gets a
+# 302, the browser follows it INVISIBLY, the live Keycloak SSO session satisfies it silently,
+# and the call resolves 200 carrying the app's HTML where the JavaScript expected JSON — no
+# prompt, no console error, nothing logged as a failure. On a WRITE it is worse: a browser
+# following a 302 re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
+# discarded and the UI reports success. MEASURED, twice:
+#   jackett  2026-09-30: 33 x 302 against 2 x 401 on /api/ paths, and 4 of 6 POSTs answered by
+#            the proxy and never reaching the app (access-log upstream field `-`, not `/`).
+#   radarr   2026-10-01, 23h after the cookie fix rolled: 232 x 302, EVERY ONE a
+#            `POST /signalr/messages/negotiate` from the operator's own browser, upstream `-`.
+#            A left-open tab whose cookie the rename orphaned, polling forever, silently
+#            redirected every time, NEVER re-prompting. That is the reported symptom
+#            ("reloading never fixes it and I am never re-prompted") still happening in
+#            production, on a service nobody was looking at.
+# `--api-route=<regex>` makes matching paths answer 401.
+#
+# WHY THIS IS NOT A RENDERER CONSTANT, AND MUST NEVER BECOME ONE. The path shape is a property
+# of the APP, not of the forward-auth pattern: radarr's broken path is `/signalr/`, jackett's is
+# `/api/`, and PIHOLE SERVES NO `/api/` AT ALL. One constant would apply one app's path shape to
+# five, and the failure mode of a wrong one is not "no improvement" but "this service is
+# permanently broken": a path a BROWSER NAVIGATES to, answered 401, shows a logged-out operator
+# a bare error page and never offers a sign-in. So the routes are declared per service in
+# `auth.apiRoutes` (schema 1.9.0) and rendered one `--api-route` per entry — the flag is
+# repeatable, and it is SINGULAR: `--api-routes` does not exist in v7.8.1 and the proxy refuses
+# to start with it (verified against the pinned binary's own --help, 2026-10-01:
+# `--api-route strings   return HTTP 401 instead of redirecting to authentication server...`).
+#
+# BREADTH RATCHET. Every entry must be anchored at `^/` and continue with literal path
+# characters. That rejects `^/`, `^/.*`, `.*`, an unanchored `/api/`, and any alternation — the
+# shapes that turn a page load into a 401. Alternation is refused rather than supported because
+# the flag is repeatable, so one regex per route costs nothing and stays readable. The same
+# pattern is in the schema (so ci/l0.sh's schema stage rejects it at the authoring surface) and
+# enforced HERE (so ci/render-golden.sh fails even if the schema stage is skipped) — a breadth
+# mistake must not need two independent gates to both be running.
+OAUTH2_API_ROUTE_PATTERN = re.compile(r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$")
+
 
 # =============================================================================
 # Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
@@ -200,6 +239,12 @@ def apply_defaults(sd: dict) -> dict:
     if "mode" not in auth:
         auth["mode"] = "forward-auth" if s["ingressRoutes"] else "none"
     auth.setdefault("groups", [])
+    # apiRoutes default posture is EMPTY, deliberately, and that is the whole point of the field
+    # (schema 1.9.0): an app gets --api-route only where someone MEASURED that it serves a machine
+    # API, so a newly-onboarded app renders no --api-route at all rather than inheriting another
+    # app's path shape. See OAUTH2_API_ROUTE_PATTERN above for why a default of `^/api/` would be
+    # wrong rather than merely generous.
+    auth.setdefault("apiRoutes", [])
     s["auth"] = auth
 
     update = dict(s.get("update") or {})
@@ -1075,7 +1120,12 @@ def _oauth2_sidecar_container(s: dict) -> dict:
 
     --cookie-name and --cookie-domain are DERIVED per app (see the block comment on those two
     args): every instance holds a different cookie-secret, so sharing one cookie name across the
-    zone makes the instances evict each other's sessions."""
+    zone makes the instances evict each other's sessions.
+
+    --api-route is DECLARED per app, from auth.apiRoutes, and is the one flag here that is
+    neither a constant nor derivable: it names the paths THIS app serves as a machine API, and
+    those differ per app (radarr `/signalr/`, jackett `/api/`, pihole none). An app that declares
+    none renders no --api-route at all."""
     route = s["ingressRoutes"][0]
     host = route["host"]
     upstream_port = route["port"]
@@ -1095,6 +1145,27 @@ def _oauth2_sidecar_container(s: dict) -> dict:
     args += [
         "--skip-provider-button=true",
         f"--redirect-url=https://{host}/oauth2/callback",
+    ]
+    # ── API ROUTES: 401 instead of 302, ON THIS APP'S OWN PATHS ONLY ───────────────────────
+    # One flag per declared route, in declared order. An app that declares NONE renders NO
+    # --api-route, which is the correct and common case — see OAUTH2_API_ROUTE_PATTERN above for
+    # the measurements, for why this cannot be a renderer constant, and for the breadth ratchet.
+    # Validated here as well as in the schema so a breadth mistake is caught by render-golden
+    # even on a run where the schema stage did not execute.
+    for route in s["auth"].get("apiRoutes", []):
+        if not OAUTH2_API_ROUTE_PATTERN.match(str(route)):
+            raise SystemExit(
+                f"render.py: {s['name']}: auth.apiRoutes entry {route!r} is not an anchored path "
+                f"regex. Each entry must start with '^/' and continue with literal path "
+                f"characters (e.g. '^/api/', '^/signalr/', '^/api/v3/.*'). Rejected shapes: '^/' "
+                f"and '^/.*' (match every page load), an unanchored '/api/' (matches mid-path), "
+                f"and alternation (the flag is repeatable — declare one entry per route). A "
+                f"too-broad --api-route answers a PAGE LOAD with 401, so a logged-out operator "
+                f"sees a bare error page and is never offered a sign-in: the service looks "
+                f"permanently broken rather than merely unauthenticated."
+            )
+        args.append(f"--api-route={route}")
+    args += [
         # ── COOKIE ISOLATION (fleet fix 2026-09-30; crafty's 2026-09-29 fix, generalized) ──────
         # EVERY oauth2-proxy on this platform — this sidecar, its four siblings, and the SHARED
         # forward-auth proxy (platform/oauth2-proxy) — is an INDEPENDENT instance holding its OWN
@@ -2732,15 +2803,27 @@ def _selftest() -> int:
             "auth": {"mode": "forward-auth", "groups": ["media-users"]},
         }
     )
-    fa_dep = next(d[1] for d in fa_docs if d[1]["kind"] == "Deployment")
-    fa_containers = fa_dep["spec"]["template"]["spec"]["containers"]
+    fa_dep = next((d[1] for d in fa_docs if d[1]["kind"] == "Deployment"), None)
+    check(fa_dep is not None,
+          "a forward-auth app rendered NO Deployment, so there is no pod to gate at all")
+    fa_containers = (((fa_dep or {}).get("spec") or {}).get("template") or {}) \
+        .get("spec", {}).get("containers") or []
     check(
         len(fa_containers) == 2
         and fa_containers[0]["name"] == "fa"
         and fa_containers[1]["name"] == "oauth2-proxy",
-        "forward-auth app must render the app container FIRST + an oauth2-proxy sidecar SECOND",
+        f"forward-auth app must render the app container FIRST + an oauth2-proxy sidecar SECOND; "
+        f"rendered {[c.get('name') for c in fa_containers]!r}",
     )
-    side = fa_containers[1]
+    # EVERY assertion below reads `side`, so an absent sidecar must NOT be allowed to raise here.
+    # The whole point of the None-tolerant lookups in this block is that a missing sidecar (or a
+    # missing flag on it) reports as NAMED ASSERTIONS rather than as a traceback: an exception
+    # escaping _selftest() exits non-zero with the right verdict and ZERO diagnostics, and it
+    # hides every other assertion the same mutation broke. Measured by Desmond Oyelaran while
+    # wiring this selftest into the render-golden stage: deleting the --cookie-name emission gave
+    # rc 1 via an uncaught StopIteration and printed not one `SELFTEST FAIL:` line. Provoked
+    # again here with the sidecar removed entirely, which raised IndexError at this line.
+    side = fa_containers[1] if len(fa_containers) > 1 else {"args": [], "env": [], "ports": []}
     check(
         "--upstream=http://127.0.0.1:9117" in side["args"],
         "sidecar upstream must be the co-located app UI over http on 127.0.0.1:<uiport>",
@@ -2809,12 +2892,101 @@ def _selftest() -> int:
         "sidecar must declare BOTH the http and metrics container ports — an undeclared port "
         "cannot be named by a Service port or a PodMonitor, so the flag alone is inert",
     )
+    # API ROUTES (schema 1.9.0). Three properties, and the FIRST is the one that keeps the field
+    # honest: an app that declares nothing must render NO --api-route. `fa` above declares none.
+    check(
+        not any(a.startswith("--api-route") for a in side["args"]),
+        "a forward-auth app that declares no auth.apiRoutes must render NO --api-route. A "
+        "default or a renderer constant would answer PiHole's page loads with 401 — PiHole "
+        "serves no /api/ at all — so the service would look permanently broken to a "
+        "logged-out operator instead of signing them in",
+    )
+    # Same `next(..., None)` discipline as _cookie_name() below, for the same reason: if the
+    # sidecar stops being rendered at all, this must report as a named assertion rather than as a
+    # StopIteration that swallows every other SELFTEST FAIL line in the run.
+    _faapi_dep = next(
+        (
+            d[1]
+            for d in render(
+                {
+                    "name": "faapi",
+                    "source": {"image": "a:1"},
+                    "ingressRoutes": [{"host": "faapi.home.keithjackson.dev", "port": 9119}],
+                    "auth": {
+                        "mode": "forward-auth",
+                        "groups": ["media-users"],
+                        "apiRoutes": ["^/api/", "^/signalr/"],
+                    },
+                }
+            )
+            if d[1]["kind"] == "Deployment"
+        ),
+        None,
+    )
+    fa_api_side = next(
+        (
+            c
+            for c in ((_faapi_dep or {}).get("spec", {}).get("template", {})
+                      .get("spec", {}).get("containers") or [])
+            if c.get("name") == "oauth2-proxy"
+        ),
+        None,
+    )
+    check(
+        fa_api_side is not None,
+        "a forward-auth app declaring auth.apiRoutes rendered NO oauth2-proxy sidecar, so the "
+        "routes have nowhere to land and the app is not gated at all (R-IDP-02)",
+    )
+    check(
+        fa_api_side is not None
+        and [a for a in fa_api_side["args"] if a.startswith("--api-route")]
+        == ["--api-route=^/api/", "--api-route=^/signalr/"],
+        "a declared auth.apiRoutes must render as one repeatable --api-route per entry, in "
+        "declared order, and NOTHING else. Note the flag is SINGULAR: --api-routes does not "
+        "exist in v7.8.1 and the proxy refuses to start with it",
+    )
+    # BREADTH. The renderer must REFUSE a regex that would match a page load, not merely
+    # discourage one, because the schema stage can be skipped and render-golden cannot.
+    for _bad in ("^/", "^/.*", ".*", "/api/", "^/api/|^/"):
+        try:
+            render(
+                {
+                    "name": "fabad",
+                    "source": {"image": "a:1"},
+                    "ingressRoutes": [{"host": "fabad.home.keithjackson.dev", "port": 9120}],
+                    "auth": {
+                        "mode": "forward-auth",
+                        "groups": ["media-users"],
+                        "apiRoutes": [_bad],
+                    },
+                }
+            )
+            _refused = False
+        except SystemExit:
+            _refused = True
+        check(
+            _refused,
+            f"the renderer ACCEPTED auth.apiRoutes entry {_bad!r}. A too-broad --api-route "
+            f"answers a plain page load with 401, so a logged-out operator sees a bare error "
+            f"page and is never offered a sign-in — the service looks permanently broken "
+            f"rather than merely unauthenticated, which is a worse failure than the 302 bug "
+            f"this flag exists to fix",
+        )
+    for _good in ("^/api/", "^/signalr/", "^/api/v3/.*", "^/api/v2.0/server/update$"):
+        check(
+            bool(OAUTH2_API_ROUTE_PATTERN.match(_good)),
+            f"the breadth ratchet rejected the legitimate route {_good!r}. A ratchet that "
+            f"blocks real API prefixes is a ratchet someone eventually deletes",
+        )
+
     # Two forward-auth apps must not be able to collide. Renders a SECOND one and compares, so a
     # future refactor to a constant (or to a per-service table with a copy-paste slip) fails here
     # rather than silently in a browser six weeks later.
-    fa2_side = next(
-        c
-        for c in next(
+    # Same None-tolerant shape as the `faapi` fixture above and for the same reason: a missing
+    # sidecar must surface as a named assertion, never as a traceback that suppresses the
+    # collected SELFTEST FAIL lines.
+    _fa2_dep = next(
+        (
             d[1]
             for d in render(
                 {
@@ -2825,24 +2997,55 @@ def _selftest() -> int:
                 }
             )
             if d[1]["kind"] == "Deployment"
-        )["spec"]["template"]["spec"]["containers"]
-        if c["name"] == "oauth2-proxy"
+        ),
+        None,
+    )
+    fa2_side = next(
+        (
+            c
+            for c in ((_fa2_dep or {}).get("spec", {}).get("template", {})
+                      .get("spec", {}).get("containers") or [])
+            if c.get("name") == "oauth2-proxy"
+        ),
+        {"args": []},
     )
 
+    # `next(..., None)` — the DEFAULT IS LOAD-BEARING, not defensive style. With a bare
+    # `next(genexpr)` this raises StopIteration the moment a sidecar renders no --cookie-name,
+    # which is precisely the mutation this assertion exists to catch. The traceback then escapes
+    # _selftest() before its collected `SELFTEST FAIL:` lines are printed, so the run exits
+    # non-zero with the RIGHT VERDICT AND NO DIAGNOSTICS — and it masks every other assertion the
+    # same mutation broke. Measured by Desmond Oyelaran while wiring the selftest into the
+    # render-golden stage: deleting the --cookie-name emission gave rc 1 via an uncaught
+    # StopIteration and suppressed all of the messages. A missing flag must report AS AN
+    # ASSERTION, with its name in the output, which is what the None + explicit check below do.
     def _cookie_name(container):
-        return next(a for a in container["args"] if a.startswith("--cookie-name="))
+        return next((a for a in container["args"] if a.startswith("--cookie-name=")), None)
 
+    _fa_cn, _fa2_cn = _cookie_name(side), _cookie_name(fa2_side)
     check(
-        _cookie_name(side) != _cookie_name(fa2_side),
+        _fa_cn is not None and _fa2_cn is not None,
+        f"a forward-auth sidecar rendered NO --cookie-name at all (fa={_fa_cn!r}, "
+        f"fa2={_fa2_cn!r}). Omitting the flag is not neutral: oauth2-proxy then claims its "
+        f"built-in default `_oauth2_proxy`, which every other instance on the platform also "
+        f"claims, so they overwrite and ClearSession-delete each other's sessions",
+    )
+    check(
+        _fa_cn != _fa2_cn,
         "two forward-auth apps rendered IDENTICAL cookie names — they would overwrite each "
         "other's session on every sign-in (the 2026-09-29 crafty defect, fleet-wide)",
     )
+    # Index-safe for the same reason as `side` above: an absent sidecar has no env, and an
+    # IndexError here would exit non-zero with no diagnostics at all.
+    _fa_env = (side.get("env") or [{}])[0]
     check(
-        side["env"][0]["valueFrom"]["secretKeyRef"]["name"] == "fa-oauth2-proxy",
+        (((_fa_env.get("valueFrom") or {}).get("secretKeyRef") or {}).get("name")
+         == "fa-oauth2-proxy"),
         "sidecar creds must come by name from the per-app <app>-oauth2-proxy Secret (R-IDP-04)",
     )
-    fa_svc = next(d[1] for d in fa_docs if d[1]["kind"] == "Service")
-    fa_svc_ports = {p["port"] for p in fa_svc["spec"]["ports"]}
+    fa_svc = next((d[1] for d in fa_docs if d[1]["kind"] == "Service"), None)
+    check(fa_svc is not None, "a forward-auth app must render a Service for the Gateway to reach")
+    fa_svc_ports = {p["port"] for p in (((fa_svc or {}).get("spec") or {}).get("ports") or [])}
     check(
         fa_svc_ports == {9117, 4180},
         "forward-auth Service must expose the app port (east-west) AND :4180 (gateway->sidecar)",

catalog/sonarr.yaml

@@ -100,6 +100,9 @@ dependencies:
   - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 9117
     protocol: tcp
+  - to: plex                    # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    port: 32400
+    protocol: tcp
 ingressRoutes:
   - host: sonarr.${BASE_DOMAIN}
     port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
@@ -112,6 +115,14 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE, AND NOT BECAUSE IT NEEDS NONE — because nothing measures that it does.
+  # Sonarr is nearly the same codebase as Radarr, which is exactly what makes `^/api/` +
+  # `^/signalr/` a tempting guess and still a guess. MEASURED 2026-10-01 on the live pod
+  # (sonarr-ddf485775-8ckwt, 0 restarts, 23h): its oauth2-proxy log contains 13,643 lines and
+  # ZERO unauthenticated requests — no /api/, no /signalr/, nothing but health probes. Nobody
+  # has opened this UI since the cookie fix rolled, so the measurement is empty rather than
+  # negative: it is evidence of no TRAFFIC, not evidence of no need. Add the routes when someone
+  # reproduces the 302-on-XHR here, with the numbers.
 update:
   channel: stable               # R-UPD-01
   range: ">=4.0.0 <5.0.0"       # stay on the Sonarr v4 stable line

ci/approval-override.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""ci/approval-override.py — select the admin-override on a PR, if there is a valid one.
+
+This script decides NOTHING about authority. It parses a PR's comments and reviews and returns
+the single override *candidate* that ci/approval-policy.sh should then try to authorise against
+the live Gitea permission API. Keeping the two apart is the point:
+
+  * parsing is pure, offline, and therefore TESTABLE (ci/tests/approval-override.test.sh);
+  * authorisation needs the API and must stay in the stage that fails closed.
+
+A candidate is a comment OR a review whose body contains, AT THE START OF A LINE, the directive
+followed by a non-empty reason:
+
+    /approve-override the gate cannot gate itself and I am the only human reviewer
+
+TWO SOURCES, AND WHY THAT IS NOT BELT-AND-BRACES. Measured 2026-10-02 on PR #38: the override was
+posted as an ordinary PR comment, was well-formed, was newer than the head commit — and the stage
+still blocked, because CI could not READ it. `SETTINGS_API_TOKEN` is minted with scopes
+`[read:repository, read:admin]`, and Gitea puts /issues/{n}/comments behind the *issue* scope, so
+the request came back as an error OBJECT rather than an array. `/pulls/{n}/reviews` is under
+read:repository and was readable in the same run. So:
+
+  * a REVIEW body works with the token CI has today (leave a review of type "Comment");
+  * a COMMENT body works once the token carries read:issue (see infra/gitea/set-ci-token.sh).
+
+Accepting both means the hatch is not hostage to one API scope. A source that cannot be read
+contributes no candidates and is reported on stderr; it can only cause a false BLOCK, never a
+false pass, which is why degrading it is safe and blocking on it was not necessary.
+
+WHY EACH RULE IS HERE
+
+  * START OF A LINE, not "anywhere in the body". Otherwise quoting someone else's override
+    ("you could just /approve-override this") re-arms it. Lines beginning with '>' are DROPPED
+    rather than unquoted, for exactly that reason.
+  * A REASON IS MANDATORY. A bare directive records that a review was waived but not why, which
+    is the difference between an audit trail and a rubber stamp.
+  * STALE OVERRIDES ARE DISCARDED. An override earned at commit A must not silently cover commit
+    B. Any candidate created at or before the head commit's timestamp is dropped — the same
+    semantics Gitea's own `dismiss_stale_approvals` applies to approvals. Push new commits and
+    you must re-override.
+  * A PENDING REVIEW IS NOT SUBMITTED. It is a private draft visible only to its author, so it is
+    not an act of anything and is dropped.
+  * THE LATEST CANDIDATE WINS, across both sources, so a later override supersedes an earlier one.
+
+Output: one TSV line (login, timestamp, source, reason) on stdout, or nothing at all when there is
+no valid candidate. Never exits non-zero for "no override" — absence is a normal answer and the
+caller must not be able to confuse it with a parse crash.
+
+Input, all via env so nothing lands in `ps`:
+  COMMENTS_JSON  the Gitea /issues/{n}/comments array (or an error object; or absent)
+  REVIEWS_JSON   the Gitea /pulls/{n}/reviews array   (or an error object; or absent)
+  HEAD_TIME      ISO-8601 timestamp of the PR head commit
+  DIRECTIVE      the override directive, e.g. /approve-override
+"""
+import json
+import os
+import re
+import sys
+from datetime import datetime, timezone
+
+
+def parse_ts(value):
+    """Gitea emits RFC3339. Return an aware datetime, or None if it is unparseable.
+
+    Unparseable is NOT 'treat as fresh': the caller drops such a candidate, because a
+    timestamp we cannot read is a staleness check we cannot perform.
+    """
+    if not value:
+        return None
+    text = str(value).strip()
+    # Python < 3.11 cannot read a trailing 'Z' in fromisoformat.
+    if text.endswith(("Z", "z")):
+        text = text[:-1] + "+00:00"
+    try:
+        parsed = datetime.fromisoformat(text)
+    except ValueError:
+        return None
+    if parsed.tzinfo is None:
+        parsed = parsed.replace(tzinfo=timezone.utc)
+    return parsed
+
+
+def extract_reason(body, directive):
+    """Return the reason from the first line that STARTS with the directive, else None."""
+    pattern = re.compile(
+        r"^[ \t]*" + re.escape(directive) + r"(?=[ \t:]|$)[ \t:]*(.*)$"
+    )
+    for raw in (body or "").splitlines():
+        # A quoted line is somebody repeating an override, not casting one.
+        if raw.lstrip().startswith(">"):
+            continue
+        match = pattern.match(raw)
+        if match:
+            reason = match.group(1).strip()
+            return reason or None
+    return None
+
+
+def load_source(env_name, label):
+    """Return a list of items for one source, or [] with a stderr note if it is unusable."""
+    raw = os.environ.get(env_name)
+    if not raw or not raw.strip():
+        return []
+    try:
+        parsed = json.loads(raw)
+    except (ValueError, TypeError):
+        print(
+            "approval-override: %s is not valid JSON — ignoring that source" % label,
+            file=sys.stderr,
+        )
+        return []
+    if isinstance(parsed, list):
+        return parsed
+    # An error object, typically {"message": "token does not have ... scope"}. Say which,
+    # because "no override found" and "could not read the overrides" look identical from
+    # the outside and the first wasted a real debugging session.
+    detail = ""
+    if isinstance(parsed, dict):
+        detail = str(parsed.get("message") or parsed.get("errors") or "")[:200]
+    print(
+        "approval-override: %s is not a list (Gitea returned %s%s) — ignoring that source"
+        % (label, type(parsed).__name__, ": " + detail if detail else ""),
+        file=sys.stderr,
+    )
+    return []
+
+
+def candidates_from(items, source, directive, head_time):
+    """Yield (when, login, reason, raw_when, source) for every valid override in one source."""
+    found = []
+    for item in items:
+        if not isinstance(item, dict):
+            continue
+        login = ((item.get("user") or {}).get("login") or "").strip()
+        reason = extract_reason(item.get("body"), directive)
+        if not login or not reason:
+            continue
+        # A PENDING review is an unsubmitted private draft — not an act of anything.
+        state = str(item.get("state") or "").upper()
+        if source == "review" and state == "PENDING":
+            print(
+                "approval-override: dropping PENDING (unsubmitted) review by '%s'" % login,
+                file=sys.stderr,
+            )
+            continue
+        raw_when = (
+            item.get("submitted_at") or item.get("created_at") or item.get("updated_at") or ""
+        )
+        when = parse_ts(raw_when)
+        if when is None:
+            print(
+                "approval-override: dropping %s override by '%s' — unreadable timestamp %r"
+                % (source, login, raw_when),
+                file=sys.stderr,
+            )
+            continue
+        # An override must post-date the commit it covers.
+        if head_time is not None and when <= head_time:
+            print(
+                "approval-override: dropping STALE %s override by '%s' (%s) — not newer than "
+                "head commit (%s). Re-post it to override the current commit."
+                % (source, login, when.isoformat(), head_time.isoformat()),
+                file=sys.stderr,
+            )
+            continue
+        found.append((when, login, reason, raw_when, source))
+    return found
+
+
+def main():
+    directive = os.environ.get("DIRECTIVE", "/approve-override").strip()
+    head_time = parse_ts(os.environ.get("HEAD_TIME", ""))
+
+    found = []
+    found += candidates_from(
+        load_source("COMMENTS_JSON", "COMMENTS_JSON"), "comment", directive, head_time
+    )
+    found += candidates_from(
+        load_source("REVIEWS_JSON", "REVIEWS_JSON"), "review", directive, head_time
+    )
+
+    if not found:
+        return 0
+
+    # Latest wins, across both sources: a later override supersedes an earlier one.
+    when, login, reason, raw_when, source = max(found, key=lambda c: c[0])
+    # Keep the reason on one line — this becomes a TSV field.
+    reason = " ".join(reason.split())
+    print("%s\t%s\t%s\t%s" % (login, raw_when, source, reason))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

ci/approval-policy.sh

@@ -25,10 +25,45 @@
 # true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
 # BYPASS, not a gap. So: cannot determine -> BLOCK.
 #
+# THE ADMIN OVERRIDE (added 2026-10-01), and the lock-out it clears.
+#
+# The rule above was UNSATISFIABLE for the one human who runs this repo. Gitea refuses to let a
+# pull request's author approve their own PR, and `jakman85` is the only collaborator — so "an
+# APPROVED review from someone who is not its author" is a condition the sole administrator can
+# never meet. Every sensitive-path PR they opened was permanently unmergeable, INCLUDING the PR
+# that would fix it. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, blocked.
+#
+# So an administrator may waive the review by COMMENTING on the PR:
+#
+#     /approve-override <reason>
+#
+# Authority comes from the LIVE Gitea permission API, never from the comment — the commenter must
+# hold a permission listed in `.override.authorized_permissions` (admin/owner) at the moment the
+# gate runs, so an old comment from a since-demoted account is worth nothing. The directive must
+# start a line, a reason is mandatory, and the comment must POST-DATE the head commit: an override
+# earned at commit A does not cover commit B. Choosing the candidate comment is the job of the
+# pure, offline, unit-tested ci/approval-override.py; AUTHORISING it is the job of this stage,
+# which fails closed.
+#
+# WHAT THIS COSTS, STATED PLAINLY. For an administrator, "a change to the gate cannot be gated by
+# the gate" stops being structural and becomes advisory. That is the unavoidable consequence of
+# there being one human; the alternative is a repository its owner cannot change. What is bought
+# back is a RECORD — the waiver is an explicit act, attributed to an account, carrying a stated
+# reason, in the PR history and in this stage's output. A silent bypass keyed on "is the author an
+# admin?" would have been less code and is exactly the fail-open shape this repo keeps finding.
+#
+# AND THE PART THAT IS NOT COMFORTABLE. Authorisation is by permission ALONE, with no allowlist of
+# human accounts (operator decision, 2026-10-01, taken with this consequence on the table). The
+# AGENT account `gitea-admin` also reports permission `owner` here, so an agent can clear the
+# sensitive-path review on its own PR by posting the directive — and agents author most PRs in
+# this repo, so that is the common case, not an edge case. To narrow it without touching this
+# logic, add an `override_admins` list to infra/gitea/approval-policy.json: the check below
+# already requires membership whenever that key is present.
+#
 # EXIT CODES
-#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
-#      non-author APPROVED review is present
-#   1  sensitive paths touched without such a review, OR the policy could not be evaluated
+#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and
+#      either a non-author APPROVED review or a valid admin override is present
+#   1  sensitive paths touched with neither, OR the policy could not be evaluated
 set -euo pipefail
 CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
@@ -65,6 +100,17 @@ api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
 FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
 PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
 REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
+# PR conversation comments live under the ISSUES endpoint. `/pulls/{n}/comments` returns review
+# (diff) comments ONLY, so an override posted in the normal comment box would never appear in it.
+# Verified against this Gitea (1.24.3) rather than assumed.
+#
+# This one is NOT in the must-parse loop below, and that is deliberate. /issues/{n}/comments sits
+# behind Gitea's *issue* token scope, which `SETTINGS_API_TOKEN` does not carry (it is minted
+# `[read:repository, read:admin]`) — so it returns an error OBJECT, not an array. Blocking on that
+# turned "CI cannot read your override" into a message that said "you did not post one", and cost a
+# real debugging session on PR #38. It is handled explicitly, and loudly, further down: an
+# unreadable source can only cause a false BLOCK, never a false pass.
+COMMENTS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/issues/${PR}/comments?limit=1000" || true)"
 for v in FILES_JSON PR_JSON REVIEWS_JSON; do
   jq -e . >/dev/null 2>&1 <<<"${!v}" || {
     err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
@@ -107,11 +153,122 @@ if [ -n "$APPROVER" ]; then
   exit 0
 fi
 
+# ─────────────────────────────────────────────────────────────────────────────────────────
+# No non-author approval exists. Before refusing, look for an administrator's explicit
+# override. See the header for why this hatch exists and what it costs.
+# ─────────────────────────────────────────────────────────────────────────────────────────
+OVERRIDE_ENABLED="$(jq -r '.override.enabled // false' "$DECL")"
+DIRECTIVE="$(jq -r '.override.comment_directive // "/approve-override"' "$DECL")"
+HEAD_SHA="$(jq -r '.head.sha // empty' <<<"$PR_JSON")"
+
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  # The head commit's timestamp anchors staleness. Read it from GITEA, not from the local
+  # tree: CI checks out a merge commit, whose date is not the head commit's date.
+  HEAD_TIME=""
+  if [ -n "$HEAD_SHA" ]; then
+    HEAD_COMMIT_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/git/commits/${HEAD_SHA}" || true)"
+    HEAD_TIME="$(jq -r '.commit.committer.date // .created // empty' <<<"${HEAD_COMMIT_JSON:-null}" 2>/dev/null || true)"
+  fi
+  if [ -z "$HEAD_TIME" ]; then
+    err "could not read the head commit's timestamp for PR #${PR} — BLOCKING."
+    err "  Without it an override cannot be aged, and an unaged override is one that silently"
+    err "  covers commits nobody waived review on."
+    summary "- approval-policy: **BLOCKED — head commit timestamp unreadable** (cannot age an override)"
+    exit 1
+  fi
+
+  # Is the comments source actually readable? Say so plainly if not — a scope problem and "you
+  # never posted an override" are indistinguishable from the outside, and conflating them is what
+  # made PR #38 look like a code bug.
+  if ! jq -e 'type == "array"' >/dev/null 2>&1 <<<"${COMMENTS_JSON:-}"; then
+    SCOPE_MSG="$(jq -r '.message // empty' <<<"${COMMENTS_JSON:-null}" 2>/dev/null || true)"
+    warn "cannot read PR COMMENTS from Gitea${SCOPE_MSG:+ — ${SCOPE_MSG}}."
+    warn "  /issues/${PR}/comments needs the *issue* token scope; SETTINGS_API_TOKEN is minted"
+    warn "  [read:repository, read:admin] (infra/gitea/set-ci-token.sh). An override posted as an"
+    warn "  ordinary PR COMMENT is therefore invisible to CI — post it as a REVIEW of type"
+    warn "  'Comment' instead, which /pulls/${PR}/reviews exposes under read:repository."
+    summary "- approval-policy: note — PR comments unreadable (token lacks the issue scope); use a **review** to override"
+    COMMENTS_JSON='[]'
+  fi
+
+  # Pure, offline, unit-tested selection: which comment or review, if any, is a live override?
+  OVERRIDE_TSV="$(COMMENTS_JSON="$COMMENTS_JSON" REVIEWS_JSON="$REVIEWS_JSON" \
+    HEAD_TIME="$HEAD_TIME" DIRECTIVE="$DIRECTIVE" \
+    python3 "${CI_DIR}/approval-override.py")" || {
+      err "ci/approval-override.py failed — BLOCKING rather than assuming there is no override."
+      summary "- approval-policy: **BLOCKED — override parser failed**"
+      exit 1; }
+
+  if [ -n "$OVERRIDE_TSV" ]; then
+    OV_USER="$(cut -f1 <<<"$OVERRIDE_TSV")"
+    OV_WHEN="$(cut -f2 <<<"$OVERRIDE_TSV")"
+    OV_SRC="$(cut -f3 <<<"$OVERRIDE_TSV")"
+    OV_WHY="$(cut -f4- <<<"$OVERRIDE_TSV")"
+
+    # AUTHORITY IS MEASURED LIVE, never inferred from the comment or from a name in git.
+    PERM_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/collaborators/${OV_USER}/permission" || true)"
+    OV_PERM="$(jq -r '.permission // empty' <<<"${PERM_JSON:-null}" 2>/dev/null || true)"
+    if [ -z "$OV_PERM" ]; then
+      err "'${OV_USER}' posted ${DIRECTIVE}, but their permission on ${GITEA_REPO} could not be"
+      err "  read — BLOCKING. An override whose authority is unverified is not an override."
+      summary "- approval-policy: **BLOCKED — cannot verify \`${OV_USER}\`'s permission**"
+      exit 1
+    fi
+
+    # Authorised when the LIVE permission is one of the declared ones and — only if an
+    # `override_admins` allowlist exists at all — the account is on it. No allowlist means
+    # permission alone, which is the operator's standing decision; adding the key narrows
+    # this with no change to any code.
+    AUTHORISED="$(DECL="$DECL" OV_PERM="$OV_PERM" OV_USER="$OV_USER" python3 -c '
+import json, os
+d = json.load(open(os.environ["DECL"])).get("override", {})
+perms = d.get("authorized_permissions") or ["admin", "owner"]
+allow = d.get("override_admins")
+ok_perm = os.environ["OV_PERM"] in perms
+ok_list = True if allow is None else os.environ["OV_USER"] in allow
+print("yes" if (ok_perm and ok_list) else "no")
+')"
+
+    if [ "$AUTHORISED" = "yes" ]; then
+      if [ "$OV_USER" = "$AUTHOR" ]; then
+        warn "ADMIN OVERRIDE — SELF-WAIVED. '${OV_USER}' authored this PR and waived the"
+        warn "  non-author review requirement on their own change. Permitted (permission"
+        warn "  '${OV_PERM}'), deliberate, and shouted about here precisely because it is not free."
+      else
+        warn "ADMIN OVERRIDE by '${OV_USER}' (permission '${OV_PERM}'); author is '${AUTHOR}'."
+      fi
+      warn "  via:    PR ${OV_SRC}"
+      warn "  when:   ${OV_WHEN}  (head commit ${HEAD_SHA:0:8}, ${HEAD_TIME})"
+      warn "  reason: ${OV_WHY}"
+      warn "  review waived on: $(printf '%s ' $SENSITIVE)"
+      ok "sensitive paths cleared by administrator override — this stage does not block."
+      SELFNOTE=""; [ "$OV_USER" = "$AUTHOR" ] && SELFNOTE=", **self-waived as author**"
+      summary "- approval-policy: **OVERRIDDEN by \`${OV_USER}\`** (permission \`${OV_PERM}\`, via PR ${OV_SRC}${SELFNOTE})"
+      summary "  - reason: ${OV_WHY}"
+      summary "  - review waived on: $(printf '\`%s\` ' $SENSITIVE)"
+      exit 0
+    fi
+
+    err "'${OV_USER}' posted ${DIRECTIVE} but is NOT authorised to override: live permission is"
+    err "  '${OV_PERM}'. The authorised set is declared in infra/gitea/approval-policy.json."
+    summary "- approval-policy: **BLOCKED — \`${OV_USER}\` may not override** (permission \`${OV_PERM}\`)"
+    exit 1
+  fi
+fi
+
 err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
 err "  review from anyone other than its author ('${AUTHOR}')."
 err "  A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
 err "  honestly approve for itself. Get a human review, or move the change out of:"
 printf '    %s\n' $SENSITIVE >&2
 err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
-summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  err "  Or, if you hold admin/owner on this repository, post this on the PR:"
+  err "      ${DIRECTIVE} <reason>"
+  err "  as a REVIEW of type 'Comment' (not the plain comment box — CI's token cannot read"
+  err "  /issues/${PR}/comments without the issue scope; see infra/gitea/set-ci-token.sh)."
+  err "  The reason is mandatory, the directive must START a line, and it must be NEWER than"
+  err "  head commit ${HEAD_SHA:0:8} — push again and you must override again."
+fi
+summary "- approval-policy: **BLOCKED — needs a non-author approval, or an admin \`${DIRECTIVE}\` comment**"
 exit 1

ci/render-golden.sh

@@ -1,5 +1,10 @@
 #!/usr/bin/env bash
-# ci/render-golden.sh — Stage: render==golden reproducibility gate.
+# ci/render-golden.sh — Stage: the renderer is trustworthy.
+#   (1) catalog/render.py passes its OWN output-contract self-test, and
+#   (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
+# Two independent assertions about one subject, catalog/render.py. (1) was added
+# 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
+# block above the invocation for why it lives here and not in a conformance check.
 #
 # WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
 # invocation of catalog/render.py. If a golden is hand-edited, or a catalog
@@ -54,6 +59,54 @@ if [ ! -f "$RENDER_PY" ]; then
   summary "- render-golden: **SKIPPED** (renderer missing)"
   exit 3
 fi
+# --- The renderer's OWN self-test, before anything is compared ---------------
+# `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
+# contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
+# auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
+# placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
+# or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
+# of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
+# assertions was dev-time only and a merge gate could not tell a sound renderer from a
+# gutted one.
+#
+# WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
+# conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
+# `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
+# That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
+# EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
+# that chain, and hanging it there would make the check's name and its requirement mapping
+# a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
+# renderer's output is trustworthy — so the renderer's own contract test belongs in it.
+#
+# It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
+# deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
+# memory), so an empty catalog must not be able to skip it.
+#
+# render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
+# render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
+# output was re-rendered, including one that just lost a contract invariant and had all ten
+# goldens regenerated in the same commit. The self-test catches exactly that.
+#
+# FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
+# run its self-test" has no legitimate environmental cause once python3 and render.py are
+# both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
+log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
+selftest_out="$(mktemp)"
+if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
+  ok "render.py --selftest passed"
+  summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
+else
+  err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
+  err "  Every golden under apps/ is produced by this renderer, so the goldens below cannot"
+  err "  be trusted even if they reproduce byte-for-byte. Reproduce with:"
+  err "    ${PYTHON} catalog/render.py --selftest"
+  sed 's/^/    /' "$selftest_out" >&2 || true
+  summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
+  rm -f "$selftest_out"
+  exit 1
+fi
+rm -f "$selftest_out"
+
 if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
   skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
   summary "- render-golden: **SKIPPED** (empty catalog)"

ci/tests/approval-override.test.sh

@@ -0,0 +1,232 @@
+#!/usr/bin/env bash
+# ci/tests/approval-override.test.sh — proves ci/approval-override.py selects the override it
+# claims to select, and REJECTS the ones it claims to reject.
+#
+# This parser decides whether a human review requirement gets waived. Every rule in it exists
+# because the obvious implementation ("does the body contain the directive?") is wrong in a way
+# that is invisible until abused: a quoted directive would re-arm an override, a bare directive
+# would waive review with no stated reason, and an override posted against commit A would go on
+# covering commit B forever. A guard is worth nothing until somebody has watched it refuse, which
+# is the same reasoning as ci/tests/decisions-log.test.sh one layer up.
+#
+# The parser is pure and offline BY DESIGN — authorisation (the live permission lookup) lives in
+# ci/approval-policy.sh, which cannot be tested without Gitea. That split is what makes this file
+# possible, and it is also why a green run here does NOT prove the stage authorises correctly.
+set -uo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SUT="${CI_DIR}/approval-override.py"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+HEAD_TIME="2026-10-01T12:00:00Z"
+AFTER="2026-10-01T12:30:00Z"
+BEFORE="2026-10-01T11:30:00Z"
+
+# run <comments-json> -> stdout of the parser (stderr is diagnostic only)
+run() {
+  COMMENTS_JSON="$1" REVIEWS_JSON="" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# run_rev <reviews-json> — the same, but through the REVIEWS source.
+run_rev() {
+  COMMENTS_JSON="" REVIEWS_JSON="$1" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# comment <login> <created_at> <body>
+comment() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "created_at": sys.argv[2], "body": sys.argv[3]}]))
+' "$1" "$2" "$3"
+}
+
+expect_none() {
+  local label="$1" out
+  out="$(run "$2")"
+  if [ -z "$out" ]; then ok "$label"; else bad "$label (got: ${out})"; fi
+}
+
+expect_user() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f1 <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted ${want}, got '${got}')"; fi
+}
+
+expect_reason() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f4- <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted '${want}', got '${got}')"; fi
+}
+
+# check <label> <actual> <wanted>
+check() {
+  if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (wanted '${3}', got '${2}')"; fi
+}
+
+# review <login> <submitted_at> <state> <body>
+review() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "submitted_at": sys.argv[2],
+                   "state": sys.argv[3], "body": sys.argv[4]}]))
+' "$1" "$2" "$3" "$4"
+}
+
+echo "== ci/approval-override.py"
+
+# --- the happy path -------------------------------------------------------------------
+expect_user "a directive with a reason, posted after the head commit, is selected" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" jakman85
+
+expect_reason "the reason is returned verbatim" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" \
+  'sole admin, gate is green'
+
+# --- absence is a normal answer, not an error -----------------------------------------
+expect_none "no comments at all" '[]'
+expect_none "an ordinary comment" "$(comment jakman85 "$AFTER" 'looks good to me')"
+
+# --- a reason is MANDATORY ------------------------------------------------------------
+expect_none "a bare directive (no reason) is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override')"
+expect_none "a directive followed by only whitespace is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override    ')"
+
+# --- the directive must START a line --------------------------------------------------
+expect_none "the directive mid-sentence does not count" \
+  "$(comment jakman85 "$AFTER" 'you could just /approve-override this one')"
+expect_none "a QUOTED directive does not re-arm an override" \
+  "$(comment jakman85 "$AFTER" '> /approve-override quoting someone else')"
+expect_user "leading whitespace before the directive is fine" \
+  "$(comment jakman85 "$AFTER" '   /approve-override indented but mine')" jakman85
+expect_user "the directive on a later line of a longer comment counts" \
+  "$(comment jakman85 "$AFTER" 'some preamble
+/approve-override the real reason here')" jakman85
+
+# --- staleness: an override must post-date the commit it waives review on -------------
+expect_none "an override OLDER than the head commit is dropped as stale" \
+  "$(comment jakman85 "$BEFORE" '/approve-override stale, predates the push')"
+expect_none "an override exactly AT the head commit time is dropped (strictly newer)" \
+  "$(comment jakman85 "$HEAD_TIME" '/approve-override same instant')"
+
+# --- malformed input is never a free pass --------------------------------------------
+expect_none "an unreadable timestamp drops the candidate" \
+  "$(comment jakman85 'not-a-date' '/approve-override bad clock')"
+expect_none "a comment with no author is dropped" \
+  '[{"user": {}, "created_at": "2026-10-01T12:30:00Z", "body": "/approve-override no login"}]'
+expect_none "invalid JSON yields no override (and never a traceback into stdout)" \
+  'this is not json'
+expect_none "a JSON object instead of a list yields no override" \
+  '{"user": "nope"}'
+
+# --- latest wins ----------------------------------------------------------------------
+TWO='[
+  {"user": {"login": "early"},  "created_at": "2026-10-01T12:10:00Z", "body": "/approve-override first"},
+  {"user": {"login": "latest"}, "created_at": "2026-10-01T12:40:00Z", "body": "/approve-override second"}
+]'
+expect_user "the LATEST override supersedes an earlier one" "$TWO" latest
+expect_reason "and brings its own reason" "$TWO" second
+
+# --- offsets, not just Z --------------------------------------------------------------
+# Gitea emits RFC3339 WITH AN OFFSET (measured: 2026-10-01T23:09:08-05:00), not a trailing Z.
+# A parser that only handles 'Z' silently drops every real override, so this case is load-bearing.
+expect_user "an RFC3339 offset timestamp (-05:00), as Gitea actually emits, parses" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T08:30:00-05:00", "body": "/approve-override offset form"}]' \
+  jakman85
+expect_none "and an offset timestamp that is genuinely earlier is still stale" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T06:00:00-05:00", "body": "/approve-override offset, stale"}]'
+
+# --- the directive is configurable ----------------------------------------------------
+out="$(run "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')" "$HEAD_TIME" '/admin-waive')"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "a custom directive from the declaration is honoured"
+else
+  bad "a custom directive from the declaration is honoured (got '${out}')"
+fi
+expect_none "the default directive does not match a custom-directive comment" \
+  "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')"
+
+# --- no staleness anchor: do not invent one -------------------------------------------
+out="$(COMMENTS_JSON="$(comment jakman85 "$BEFORE" '/approve-override no anchor')" \
+       HEAD_TIME="" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "with no HEAD_TIME the staleness test is skipped, not faked (caller blocks instead)"
+else
+  bad "with no HEAD_TIME the candidate should still be returned (got '${out}')"
+fi
+
+# --- the REVIEWS source ---------------------------------------------------------------
+# Load-bearing, not redundant. Measured on PR #38: SETTINGS_API_TOKEN is minted
+# [read:repository, read:admin], /issues/{n}/comments needs the ISSUE scope, so a well-formed
+# override in the comment box came back as an error object and was invisible. /pulls/{n}/reviews
+# IS readable with that token, so a review body is the route that works today.
+check "a review body carries an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  jakman85
+check "the source is reported as 'review'" \
+  "$(cut -f3 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  review
+check "the source is reported as 'comment' for a comment" \
+  "$(cut -f3 <<<"$(run "$(comment jakman85 "$AFTER" '/approve-override via a comment')")")" \
+  comment
+
+out="$(run_rev "$(review jakman85 "$BEFORE" COMMENT '/approve-override stale review')")"
+check "a stale review override is dropped too" "${out:-<none>}" "<none>"
+
+out="$(run_rev "$(review jakman85 "$AFTER" PENDING '/approve-override unsubmitted draft')")"
+check "a PENDING (unsubmitted) review is NOT an act of anything" "${out:-<none>}" "<none>"
+
+check "an APPROVED review carrying the directive still counts as an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" APPROVED '/approve-override belt and braces')")")" \
+  jakman85
+
+out="$(run_rev "$(review jakman85 "$AFTER" COMMENT 'ordinary review prose, no directive')")"
+check "an ordinary review body is not an override" "${out:-<none>}" "<none>"
+
+# --- an unreadable source degrades to 'no candidates', never to a pass ----------------
+# The real-world shape: Gitea answers with {"message": "token does not have ... scope"}.
+ERR_OBJ='{"message":"token does not have at least one of required scope(s): [read:issue]"}'
+out="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an error OBJECT from the comments API yields no override" "${out:-<none>}" "<none>"
+
+out="$(COMMENTS_JSON="$ERR_OBJ" \
+       REVIEWS_JSON="$(review jakman85 "$AFTER" COMMENT '/approve-override reviews still work')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an unreadable COMMENTS source does not suppress a valid REVIEW override" \
+  "$(cut -f1 <<<"$out")" jakman85
+
+# and it must SAY so, or a scope problem reads as 'you never posted one' (it did, on PR #38)
+errout="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+          DIRECTIVE=/approve-override python3 "$SUT" 2>&1 >/dev/null)"
+case "$errout" in
+  *"not a list"*read:issue*) ok "the diagnostic names the scope Gitea complained about" ;;
+  *) bad "the diagnostic should quote Gitea's message (got: ${errout})" ;;
+esac
+
+# --- latest wins ACROSS the two sources ----------------------------------------------
+out="$(COMMENTS_JSON="$(comment early "2026-10-01T12:10:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review later "2026-10-01T12:50:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (review newer)" "$(cut -f1 <<<"$out")" later
+check "and its source is labelled" "$(cut -f3 <<<"$out")" review
+
+out="$(COMMENTS_JSON="$(comment later "2026-10-01T12:50:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review early "2026-10-01T12:10:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (comment newer)" "$(cut -f1 <<<"$out")" later
+check "and that source is labelled too" "$(cut -f3 <<<"$out")" comment
+
+# --- both sources absent entirely -----------------------------------------------------
+out="$(COMMENTS_JSON="" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "no sources at all yields no override, and no crash" "${out:-<none>}" "<none>"
+
+printf '\n  %d passed, %d failed\n' "$PASS" "$FAIL"
+[ "$FAIL" -eq 0 ] || exit 1

ci/tests/gate-decision.test.sh

@@ -42,7 +42,16 @@ cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env
 # 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
 # PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
 # decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
-for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
+#
+# AND IT HAPPENED AGAIN, with `approval-policy` (fixed 2026-10-01). Same shape, same cost: the
+# stage was added to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`
+# — measured on 0c7e8dd before touching anything, to be sure it was not the change under test.
+# The note above was written about the previous instance and did not prevent the next one, which
+# says the comment is not the fix. THE FIX IS TO DERIVE THE LIST: it now comes from gate.sh's own
+# `run_stage` calls, so a stage added there can no longer be silently missing here.
+mapfile -t STAGES < <(sed -n 's/^run_stage[[:space:]]\+\([a-z0-9-]\+\).*/\1/p' "${REAL_CI_DIR}/gate.sh")
+[ "${#STAGES[@]}" -ge 8 ] || { echo "could not parse stage names out of gate.sh (got ${#STAGES[@]})" >&2; exit 1; }
+for stage in "${STAGES[@]}"; do
   cat > "${CI}/${stage}.sh" <<'STUB'
 #!/usr/bin/env bash
 n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"

conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/chainsaw-test.yaml

@@ -0,0 +1,134 @@
+# =============================================================================
+# L0 STATIC — `--api-route` is PER APP, ANCHORED, and DECLARED
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. oauth2-proxy answers an unauthenticated request with a 302 to
+# Keycloak. Its built-in AJAX guard would send a 401 instead, but it fires only on an EXACT
+# `Accept: application/json`, which these apps' JavaScript does not consistently send. So a
+# browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied silently by the live Keycloak
+# SSO session, and resolves 200 carrying the app's HTML where the JavaScript expected JSON — no
+# prompt, no console error, nothing logged as a failure. On a WRITE it is worse: a browser
+# following a 302 re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
+# discarded and the UI reports success.
+#
+# MEASURED, twice, on two different apps and two different paths:
+#   jackett 2026-09-30 — on /api/ paths: 33 x 302 against 2 x 401, and 4 of 6 POSTs answered BY
+#           THE PROXY, never reaching Jackett (access-log upstream field `-`, not `/`). That is
+#           the operator's whole report: "GET works, most PUT/POST to add indexers fail,
+#           reloading never fixes it, and I am never re-prompted for login".
+#   radarr  2026-10-01, 23h after the cookie fix rolled (pod radarr-7d55cbffb9-42hmh, 0
+#           restarts) — 232 unauthenticated requests in the pod's whole lifetime and ALL 232 are
+#           the same call: `POST /signalr/messages/negotiate` from a browser UA, 302, upstream
+#           `-`. A left-open tab whose session the cookie rename orphaned, polling SignalR
+#           forever, silently redirected every time, NEVER re-prompting. The reported symptom,
+#           still running in production, on a service nobody was watching.
+#
+# WHY IT CANNOT BE A RENDERER CONSTANT — AND WHY THAT NEEDS A GATE. The path shape belongs to
+# the APP, not to the forward-auth pattern: radarr's broken path is `/signalr/`, jackett's is
+# `/api/`, and PIHOLE SERVES NO `/api/` AT ALL. One function renders all five sidecars
+# (catalog/render.py::_oauth2_sidecar_container), so the cheap fix is a constant — and a
+# constant applies one app's measurement to four apps nobody measured. The gate that already
+# exists cannot see that mistake: `ci/render-golden.sh` compares the render to the goldens, and
+# a constant plus regenerated goldens AGREE WITH EACH OTHER while both being wrong. Hence
+# assertion (2) below, which compares the rendered flag against each service's OWN
+# ServiceDefinition rather than against the renderer's output.
+#
+# WHY BREADTH IS THE DANGEROUS DIRECTION, AND IT IS NOT SYMMETRIC. A MISSING --api-route is a
+# silent 302 — bad, invisible, recoverable. A TOO BROAD one answers a plain PAGE LOAD with 401:
+# a logged-out operator gets a bare error page and is NEVER OFFERED A SIGN-IN, so the service
+# looks permanently broken and no reload recovers it. That is a worse outcome than the defect
+# the flag fixes, which is why the anchoring rule is asserted on three independent surfaces
+# (schema items.pattern, the renderer, and this check) and why assertion (5) fails if they drift
+# apart — if they do, the loosest one wins silently.
+#
+# GIVEN every oauth2-proxy sidecar this repo DEPLOYS (a container in any manifest at any depth:
+#       the rendered sidecars in apps/, crafty's hand-authored one in platform/), the
+#       ServiceDefinitions in catalog/ that generate five of them, and the renderer itself,
+# WHEN  each sidecar's `--api-route` values are collected (the flag is REPEATABLE, so all of
+#       them, in order) and each onboarded forward-auth service is paired with its golden,
+# THEN  (1) every rendered --api-route is anchored at `^/` and continues with literal path
+#           characters — `^/`, `^/.*`, `.*`, an unanchored `/api/` and any alternation FAIL;
+#       (2) every onboarded forward-auth service's rendered route list EQUALS its own
+#           ServiceDefinition's `auth.apiRoutes`, in order — which fails both a hand-edited
+#           golden and a renderer constant, neither of which render-golden can see;
+#       (3) BOTH cases are live in this tree: at least one service declares routes and at least
+#           one declares none and renders none (the pihole case), so neither path is an
+#           unexercised seam;
+#       (4) `_oauth2_sidecar_container()` reads `auth.apiRoutes` and contains NO literal path
+#           regex — a constant is the defect, back in the one place that reintroduces it
+#           fleet-wide in a single edit;
+#       (5) the anchoring pattern in the renderer and in the schema are the SAME pattern this
+#           check enforces;
+#       (6) the SHIPPED renderer, run as a subprocess on synthetic ServiceDefinitions, emits
+#           exactly the declared routes in order, emits NOTHING for a service declaring none or
+#           an empty list, and REFUSES all five breadth shapes.
+#
+# WHY ALSO A SCHEMA PATTERN, GIVEN THIS CHECK. The schema is the only surface that rejects a
+# too-broad regex WHERE A HUMAN TYPES IT — `ci/l0.sh` validates `catalog/*.yaml` against it on
+# every gate run, so the operator is told at the authoring surface rather than finding out from
+# a 401 on a page load. The renderer's own guard is there because `render-golden` runs even on a
+# gate invocation where the schema stage skipped. This check is there because neither of those
+# can see a renderer constant. Three surfaces, three different blind spots.
+#
+# WHY THIS IS A SIBLING AND NOT PART OF oauth2-proxy-cookie-names-distinct. That check's subject
+# is cookie IDENTITY: its inputs are names and domains, its failure mode is mutual session
+# eviction, its register is the shared proxy's claim on the default name. `--api-route` shares
+# none of those. Bundling them would make that check's name a lie and couple two unrelated
+# ratchets, so a future api-route decision could only be expressed by weakening a COOKIE check —
+# the same discipline that check's own header argues for, applied to this change.
+#
+# WHAT THIS DOES NOT ESTABLISH — read with the handoff entry, not instead of it:
+#   - That a 401 FIXES anything in a browser. No browser round-trip has been made, by anyone,
+#     for this flag. The 302/401 split, the dropped POST bodies and the orphaned SignalR poll
+#     are all read from access logs; the repair is reasoned from HTTP semantics.
+#   - That the regexes are RIGHT for these two apps beyond the measured paths. `^/api/` on
+#     radarr is justified by a measurement of radarr taken on its PRE-FIX pod by another
+#     engineer (decisions/2026-09-30-jackett-oauth2-proxy-session.md: 62 x /api/v3/command,
+#     46 x /api/v3/queue/details); the current pod has logged zero /api/ traffic.
+#   - Anything about sonarr, qbittorrent or pihole. Their logs were read (2026-10-01) and
+#     contain ZERO unauthenticated requests — no traffic, which is not the same as no need.
+#     Assertion (3) therefore proves the "declares none" PATH works; it does not claim those
+#     three apps are correctly configured.
+#   - That the LIVE pods carry what the tree says. git is not the cluster (CLAUDE.md §2b).
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: oauth2-proxy-api-routes-per-app
+  labels:
+    conformance.home.k3s/R-IDP-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-02"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — the subject spans apps/wave-0, apps/wave-1, platform/crafty, every
+    # ServiceDefinition in catalog/, the renderer that generates five of the sidecars, and the
+    # ServiceDefinition schema. No single requirement path-maps to that set, and the renderer
+    # maps to no identity requirement at all, so a requirement-scoped selection would skip this
+    # check on precisely the change that can reintroduce the defect fleet-wide in one edit.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      An unauthenticated XHR is answered 401, not 302, on exactly the paths each app was
+      MEASURED to serve as a machine API: every rendered --api-route is anchored at ^/ and
+      continues with literal path characters, every onboarded forward-auth service's rendered
+      routes equal its own auth.apiRoutes in order (so neither a hand-edited golden nor a
+      renderer constant can pass), an app declaring none renders none, the anchoring pattern is
+      the same in the schema and the renderer, and the shipped renderer is proven on synthetic
+      ServiceDefinitions in both directions — including refusing the five breadth shapes that
+      would answer a page load with 401.
+spec:
+  steps:
+    - name: assert-api-route-is-per-app-anchored-and-declared
+      description: >-
+        Inventory every oauth2-proxy sidecar's repeatable --api-route values, assert each is an
+        anchored path regex, cross-check each onboarded forward-auth service's rendered routes
+        against its own ServiceDefinition, require both the declares-routes and the
+        declares-none cases to be live in the tree, assert the renderer derives from
+        auth.apiRoutes and hardcodes no path regex, assert the schema and renderer share this
+        check's anchoring pattern, and prove the shipped renderer on synthetic
+        ServiceDefinitions in both directions.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/check.sh

@@ -0,0 +1,433 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — `--api-route` is PER APP, ANCHORED, and DECLARED.
+# See chainsaw-test.yaml's header for WHY this exists and what it measured. In one paragraph:
+#
+# oauth2-proxy answers an unauthenticated request with a 302 to Keycloak. A browser XHR follows
+# that redirect invisibly, the live SSO session satisfies it silently, and the call resolves 200
+# carrying the app's HTML where its JavaScript expected JSON — and a POST following a 302 is
+# re-issued as a GET with its BODY DROPPED. `--api-route=<regex>` makes matching paths answer
+# 401 instead. The flag is right for jackett (`/api/`) and radarr (`/signalr/`, `/api/`) and
+# WRONG for pihole, which serves no `/api/` at all — so it cannot be a renderer constant, and
+# the point of this check is that it never becomes one again.
+#
+# The breadth direction is the dangerous one and it is NOT symmetric with the bug. A missing
+# `--api-route` is a silent 302; a TOO BROAD one answers a plain PAGE LOAD with 401, so a
+# logged-out operator gets a bare error page and is never offered a sign-in — the service looks
+# permanently broken. That is a worse outcome than the defect, which is why the anchoring rule
+# is asserted here as well as in the schema.
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
+# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if the
+# tree contains no forward-auth oauth2-proxy sidecar at all — genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if the forward-auth pattern has left the
+# repo. Anchored on `image:` for the same reason as the sibling cookie check: a bare
+# `oauth2-proxy/oauth2-proxy:` also matches platform/oauth2-proxy/forward-auth-cec.yaml's Envoy
+# cluster reference (`serviceName: oauth2-proxy/oauth2-proxy:80`), which runs no proxy.
+if ! grep -rlE 'image:[[:space:]]*[^[:space:]]*oauth2-proxy' \
+       "${ROOT}/apps" "${ROOT}/platform" >/dev/null 2>&1; then
+  echo "check SKIPPED: no oauth2-proxy container image found under apps/ or platform/ — the forward-auth sidecar pattern is gone and there is nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: --api-route is per-app, anchored, and declared in the ServiceDefinition.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
+line-based parsing: a container's args live in a nested structure, and a line-based reader of
+one is precisely the approximate parser that silently stops matching.
+"""
+from __future__ import annotations
+
+import json
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+# THE BREADTH RATCHET, stated once and asserted against three independent surfaces below (the
+# rendered tree, the authored catalog, the shipped renderer). Each entry must be anchored at
+# `^/` and continue with LITERAL path characters. This rejects:
+#   `^/`      and `^/.*`   — match every page load, so the UI answers 401 and never signs in
+#   `/api/`               — unanchored; oauth2-proxy would match it mid-path
+#   `.*`                  — everything
+#   `^/api/|^/`           — alternation smuggles breadth past a prefix-only reading
+# Alternation is REFUSED rather than supported because `--api-route` is repeatable: one entry
+# per route costs nothing and each entry stays readable. Kept byte-identical to
+# catalog/render.py's OAUTH2_API_ROUTE_PATTERN and to the schema's items.pattern; assertion (5)
+# below fails if they drift apart.
+ANCHORED = re.compile(r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$")
+SCHEMA_PATTERN = r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$"
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+def is_test_fixture(p: Path) -> bool:
+    """conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests."""
+    parts = p.parts
+    return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
+
+
+def iter_containers(node):
+    """Yield every container mapping under any `containers`/`initContainers` list, at any depth.
+
+    Depth-agnostic on purpose: the same container shape appears under a Deployment, a DaemonSet,
+    a CronJob's jobTemplate or a bare Pod, and a reader that hardcoded one path would silently
+    stop seeing an instance the day one moved.
+    """
+    if isinstance(node, dict):
+        for key, val in node.items():
+            if key in ("containers", "initContainers") and isinstance(val, list):
+                for c in val:
+                    if isinstance(c, dict):
+                        yield c
+            else:
+                yield from iter_containers(val)
+    elif isinstance(node, list):
+        for item in node:
+            yield from iter_containers(item)
+
+
+def api_routes_of(args: list) -> list:
+    """Every `--api-route=<v>` / `--api-route <v>` value, in order. The flag is REPEATABLE, so
+    this must collect ALL of them — a single-value reader would audit the first and wave the
+    rest through, which is the shape of reader bug this file's header warns about."""
+    out = []
+    for i, a in enumerate(args):
+        if not isinstance(a, str):
+            continue
+        if a.startswith("--api-route="):
+            out.append(a.split("=", 1)[1])
+        elif a == "--api-route" and i + 1 < len(args) and isinstance(args[i + 1], str):
+            out.append(args[i + 1])
+    return out
+
+
+# ============================================================ (0) INVENTORY: THE RENDERED TREE
+# Every oauth2-proxy container this repo DEPLOYS, keyed by file + owner, with its api-routes.
+rendered: dict[str, list] = {}
+for p in sorted(ROOT.rglob("*.yaml")):
+    relparts = p.relative_to(ROOT).parent.parts
+    # NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+    # as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs,
+    # so a bare rglob audits up to 18 duplicates of every file — which both invents FAILs and,
+    # worse, can satisfy a "something declares X" check from a copy rather than from real source.
+    if any(part.startswith(".") for part in relparts) or is_test_fixture(p) or ".sops." in p.name:
+        continue
+    txt = read(p)
+    if "oauth2-proxy" not in txt:
+        continue
+    rel = str(p.relative_to(ROOT))
+    try:
+        docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
+    except yaml.YAMLError as exc:
+        check(False, f"{rel} is not parseable YAML ({exc}) — any oauth2-proxy instance in it "
+                     f"went unaudited, so a too-broad --api-route could be hiding there")
+        continue
+    for d in docs:
+        for c in iter_containers(d):
+            image = str(c.get("image") or "")
+            name = str(c.get("name") or "")
+            if "oauth2-proxy" not in image and name != "oauth2-proxy":
+                continue
+            owner = (d.get("metadata") or {}).get("name") or "?"
+            args = list(c.get("args") or []) + list(c.get("command") or [])
+            rendered[f"{rel}:{owner}/{name or 'oauth2-proxy'}"] = api_routes_of(args)
+
+check(len(rendered) >= 5,
+      f"only {len(rendered)} oauth2-proxy container(s) parsed out of the tree. This repo "
+      f"deploys more than that (five rendered sidecars plus crafty's hand-authored one), so the "
+      f"inventory has stopped seeing instances and every assertion below would pass vacuously "
+      f"while an unanchored --api-route went unaudited")
+
+# ============================================================ (1) EVERY ROUTE IS ANCHORED
+for key, routes in sorted(rendered.items()):
+    for r in routes:
+        check(bool(ANCHORED.match(r)),
+              f"{key} declares --api-route={r!r}, which is not an anchored path regex.\n"
+              f"       WHY THIS IS WORSE THAN THE BUG IT FIXES: --api-route makes matching paths "
+              f"answer 401 instead of 302ing to Keycloak. A regex that also matches a path a "
+              f"BROWSER NAVIGATES to — `^/`, `^/.*`, `.*`, or an alternation that reaches the "
+              f"UI root — answers a plain PAGE LOAD with 401. A logged-out operator then sees a "
+              f"bare error page and is NEVER OFFERED A SIGN-IN: the service looks permanently "
+              f"broken rather than merely unauthenticated, and no reload can recover it.\n"
+              f"       FIX: anchor it at '^/' followed by literal path characters "
+              f"(e.g. '^/api/', '^/signalr/', '^/api/v3/.*'), one entry per route — the flag is "
+              f"repeatable, so alternation is never needed. For a service rendered from catalog/, "
+              f"do NOT hand-edit the golden: declare auth.apiRoutes and re-render "
+              f"(ci/render-golden.sh's header has the canonical command)")
+
+# ============================================================ (2) DERIVED FROM THE CATALOG
+# The five rendered sidecars are OUTPUTS. Auditing only them would let someone "fix" a golden by
+# hand (reverted by the next render) or make the renderer emit a CONSTANT and regenerate every
+# golden to match — which render-golden cannot see, because render and golden would agree.
+# So: every onboarded forward-auth service's rendered route list must EQUAL its own
+# ServiceDefinition's auth.apiRoutes, in order.
+catalog_dir = ROOT / "catalog"
+declared: dict[str, list] = {}
+forward_auth: list[str] = []
+for sd_path in sorted(catalog_dir.glob("*.yaml")):
+    try:
+        sd = yaml.safe_load(read(sd_path))
+    except yaml.YAMLError as exc:
+        check(False, f"catalog/{sd_path.name} is not parseable YAML ({exc})")
+        continue
+    if not isinstance(sd, dict) or not sd.get("name"):
+        continue
+    auth = sd.get("auth") or {}
+    routes = auth.get("apiRoutes") or []
+    declared[sd["name"]] = list(routes)
+    if auth.get("mode") == "forward-auth" and (sd.get("ingressRoutes") or []):
+        forward_auth.append(sd["name"])
+    for r in routes:
+        check(bool(ANCHORED.match(str(r))),
+              f"catalog/{sd_path.name}: auth.apiRoutes entry {r!r} is not anchored at '^/' "
+              f"followed by literal path characters. See assertion (1) for why breadth here is "
+              f"the dangerous direction. The schema rejects this too (items.pattern); this "
+              f"assertion exists because the schema stage can be SKIPPED and this one cannot")
+    if routes:
+        check(auth.get("mode") == "forward-auth",
+              f"catalog/{sd_path.name} declares auth.apiRoutes with auth.mode="
+              f"{auth.get('mode')!r}. --api-route renders onto the per-app oauth2-proxy sidecar, "
+              f"and no sidecar is rendered for oidc/native/none — so this is a SILENT NO-OP: the "
+              f"operator believes the app's XHRs now answer 401 and nothing changed")
+
+matched = 0
+for name in sorted(forward_auth):
+    golden = next((p for p in sorted(ROOT.glob(f"apps/wave-*/{name}.yaml"))), None)
+    if golden is None:
+        continue                      # catalog service not onboarded to apps/ yet — no golden
+    key = next((k for k in rendered if k.startswith(str(golden.relative_to(ROOT)) + ":")), None)
+    check(key is not None,
+          f"apps/{golden.name} is the golden for forward-auth service {name!r} but no "
+          f"oauth2-proxy container was parsed out of it. Its --api-route configuration is "
+          f"UNAUDITED")
+    if key is None:
+        continue
+    matched += 1
+    check(rendered[key] == declared.get(name, []),
+          f"{name}: the rendered sidecar declares --api-route {rendered[key]!r} but "
+          f"catalog/{name}.yaml declares auth.apiRoutes {declared.get(name, [])!r}.\n"
+          f"       These must be IDENTICAL and in order. A mismatch means one of two things, "
+          f"both bad: a golden was hand-edited (the next re-render silently reverts it), or the "
+          f"renderer is emitting --api-route from somewhere other than this service's own "
+          f"ServiceDefinition — i.e. a constant or a per-service table. A constant is the defect "
+          f"this field exists to prevent: it applies ONE app's path shape to all five, and "
+          f"pihole serves no /api/ at all, so it would answer pihole's page loads with 401. "
+          f"render-golden cannot catch that, because render and golden would agree with each "
+          f"other while both being wrong")
+
+check(matched >= 4,
+      f"only {matched} onboarded forward-auth service(s) were cross-checked against their "
+      f"ServiceDefinition. This repo onboards five, so the catalog<->golden pairing has stopped "
+      f"resolving and assertion (2) is passing vacuously")
+
+# ============================================================ (3) BOTH CASES EXIST
+# Once the tree is consistent, (2) is satisfied by any pairing — including one where NOTHING
+# declares a route (vacuous) or one where EVERYTHING does (so the "renders nothing" path is
+# never exercised). Both halves are asserted to be live, because the per-app field only earns
+# its complexity if both outcomes actually occur in this tree.
+with_routes = [n for n in forward_auth if declared.get(n)]
+without_routes = [n for n in forward_auth if not declared.get(n)]
+check(bool(with_routes),
+      "no onboarded forward-auth service declares auth.apiRoutes, so assertion (2) cannot "
+      "distinguish a working renderer from one that emits the flag nowhere. If the flag is "
+      "genuinely not needed anywhere, delete the field rather than leaving an unexercised seam")
+check(bool(without_routes),
+      "EVERY onboarded forward-auth service declares auth.apiRoutes, so nothing in this tree "
+      "exercises the 'declares none => renders no flag' path. That path is the whole reason the "
+      "field is per-app: pihole serves no /api/, and a renderer-wide default would answer its "
+      "page loads with 401")
+for name in sorted(without_routes):
+    golden = next((p for p in sorted(ROOT.glob(f"apps/wave-*/{name}.yaml"))), None)
+    if golden is None:
+        continue
+    key = next((k for k in rendered if k.startswith(str(golden.relative_to(ROOT)) + ":")), None)
+    if key is None:
+        continue
+    check(rendered[key] == [],
+          f"{name} declares NO auth.apiRoutes but its rendered sidecar carries "
+          f"--api-route {rendered[key]!r}. An app that declares nothing must render NOTHING")
+
+# ============================================================ (4) THE RENDERER DERIVES
+render_py = ROOT / "catalog" / "render.py"
+check(render_py.is_file(),
+      "catalog/render.py is missing — the generator of five of these sidecars cannot be audited")
+if render_py.is_file():
+    src = read(render_py)
+    m = re.search(r"def _oauth2_sidecar_container\(.*?\n(?=\n*def |\n*# =)", src, re.S)
+    fn = m.group(0) if m else ""
+    check(bool(fn),
+          "could not locate _oauth2_sidecar_container() in catalog/render.py. It was renamed or "
+          "removed, so assertion (4) cannot run and the generator is unaudited")
+    if fn:
+        check("--api-route=" in fn,
+              "catalog/render.py's _oauth2_sidecar_container() emits no --api-route at all, so "
+              "every forward-auth app's XHRs are answered with a 302 the browser follows "
+              "invisibly — measured as 33x302-vs-2x401 on jackett and 232x302 on radarr, with "
+              "POST bodies dropped. auth.apiRoutes would then be a field with no consumer")
+        check(not re.search(r'--api-route=\^/[A-Za-z0-9]', fn),
+              "catalog/render.py hardcodes a literal path regex into --api-route. Whatever path "
+              "that is, EVERY forward-auth app now renders it: one app's path shape applied to "
+              "five. pihole serves no /api/, so a constant answers its page loads with 401 and "
+              "the service looks permanently broken to a logged-out operator. Render the routes "
+              "declared in each service's own auth.apiRoutes instead")
+        check("apiRoutes" in fn,
+              "catalog/render.py's _oauth2_sidecar_container() does not read auth.apiRoutes, so "
+              "the per-app declaration has no effect on what is rendered")
+
+# ============================================================ (5) ONE PATTERN, THREE SURFACES
+# The ratchet is written in three places (this check, the renderer, the schema) because each
+# guards a surface the others cannot. If they drift apart, the loosest one wins silently.
+if render_py.is_file():
+    # The renderer writes the pattern as a raw string literal, so the source text contains it
+    # verbatim. A substring test is deliberate: it also catches a rename of the constant.
+    check(SCHEMA_PATTERN in read(render_py),
+          f"catalog/render.py no longer contains the anchoring pattern {SCHEMA_PATTERN!r} that "
+          f"this check and the schema enforce. Three surfaces guard --api-route breadth (the "
+          f"authoring surface via the schema, the render via the renderer, the tree via this "
+          f"check); if they drift apart the loosest wins and the ratchet is decorative")
+
+# The schema's pattern is read from the PARSED JSON, not from the file text: in the file its
+# backslashes are JSON-escaped, so a substring test against the raw bytes would compare the
+# wrong string and pass or fail for the wrong reason.
+schema_file = ROOT / "schemas" / "servicedefinition.schema.json"
+schema_pat = None
+if schema_file.is_file():
+    try:
+        schema_pat = (((((json.loads(read(schema_file)).get("properties") or {})
+                         .get("auth") or {}).get("properties") or {})
+                       .get("apiRoutes") or {}).get("items") or {}).get("pattern")
+    except (ValueError, AttributeError) as exc:
+        check(False, f"schemas/servicedefinition.schema.json could not be read as JSON ({exc}) — "
+                     f"the schema stage that rejects a too-broad --api-route cannot be audited")
+check(schema_pat == SCHEMA_PATTERN,
+      f"schemas/servicedefinition.schema.json constrains auth.apiRoutes items with "
+      f"{schema_pat!r}, not {SCHEMA_PATTERN!r}. The schema is the only surface that rejects a "
+      f"too-broad regex WHERE A HUMAN TYPES IT (ci/l0.sh validates catalog/*.yaml on every gate "
+      f"run); if it is absent or looser than the renderer's, the operator finds out from a 401 "
+      f"on a page load")
+
+# ============================================================ (6) PROVE THE SHIPPED RENDERER
+# Once the tree is compliant, (1)-(3) have nothing left to catch and their passing is evidence
+# of nothing. This runs the SHIPPED renderer as a SUBPROCESS on synthetic ServiceDefinitions and
+# asserts the behaviour directly, in BOTH directions, so the guard stays falsifiable.
+def _sd(name: str, port: int, routes) -> str:
+    auth = {"mode": "forward-auth", "groups": ["platform-admins"]}
+    if routes is not None:
+        auth["apiRoutes"] = routes
+    return yaml.safe_dump({
+        "name": name,
+        "kind": "container",
+        "source": {"image": "example/app:1"},
+        "exposure": "internal",
+        "ingressRoutes": [{"host": f"{name}.home.keithjackson.dev", "port": port}],
+        "auth": auth,
+    }, sort_keys=False)
+
+
+def _render(name: str, port: int, routes):
+    with tempfile.TemporaryDirectory() as td:
+        f = Path(td) / f"{name}.yaml"
+        f.write_text(_sd(name, port, routes))
+        return subprocess.run([sys.executable, str(render_py), str(f)],
+                              capture_output=True, text=True)
+
+
+def _sidecar_args(rendered_yaml: str) -> list:
+    for doc in yaml.safe_load_all(rendered_yaml):
+        if not isinstance(doc, dict) or doc.get("kind") != "Deployment":
+            continue
+        for c in iter_containers(doc):
+            if c.get("name") == "oauth2-proxy":
+                return list(c.get("args") or [])
+    return []
+
+
+if render_py.is_file():
+    # (6a) DECLARED ROUTES RENDER, IN ORDER, AND NOTHING ELSE DOES.
+    proc = _render("apirtdecl", 8121, ["^/api/", "^/signalr/"])
+    check(proc.returncode == 0,
+          f"the shipped catalog/render.py failed on a synthetic forward-auth fixture declaring "
+          f"two apiRoutes: rc={proc.returncode} {proc.stderr.strip()[:300]}")
+    if proc.returncode == 0:
+        got = api_routes_of(_sidecar_args(proc.stdout))
+        check(got == ["^/api/", "^/signalr/"],
+              f"the shipped renderer turned auth.apiRoutes ['^/api/', '^/signalr/'] into "
+              f"--api-route {got!r}. It must emit one repeatable --api-route per entry, in "
+              f"declared order. Note the flag is SINGULAR: --api-routes does not exist in "
+              f"oauth2-proxy v7.8.1 and the proxy REFUSES TO START with it, so a plural would "
+              f"crashloop every sidecar it rendered")
+
+    # (6b) AN APP THAT DECLARES NOTHING RENDERS NOTHING. The pihole case, proven on the
+    #      generator rather than inferred from one committed golden.
+    for label, routes in (("absent", None), ("empty", [])):
+        proc = _render(f"apirtnone{label}", 8122, routes)
+        check(proc.returncode == 0,
+              f"the shipped renderer failed on a synthetic forward-auth fixture with apiRoutes "
+              f"{label}: rc={proc.returncode} {proc.stderr.strip()[:300]}")
+        if proc.returncode == 0:
+            got = api_routes_of(_sidecar_args(proc.stdout))
+            check(got == [],
+                  f"the shipped renderer emitted --api-route {got!r} for a synthetic service "
+                  f"whose auth.apiRoutes is {label}. A newly-onboarded forward-auth app must "
+                  f"render NO --api-route: pihole serves no /api/, so a default would answer "
+                  f"its page loads with 401 and the service would look permanently broken to a "
+                  f"logged-out operator — a worse failure than the 302 bug the flag fixes")
+
+    # (6c) THE RENDERER REFUSES BREADTH. Asserted on the generator because render-golden runs
+    #      on every gate invocation while the schema stage can be skipped.
+    for bad in ("^/", "^/.*", ".*", "/api/", "^/api/|^/"):
+        proc = _render("apirtbad", 8123, [bad])
+        check(proc.returncode != 0,
+              f"the shipped renderer ACCEPTED auth.apiRoutes {bad!r} and rendered a sidecar. "
+              f"That regex matches paths a browser NAVIGATES to, so a plain page load answers "
+              f"401 and the operator is never offered a sign-in. The renderer must refuse it, "
+              f"so a breadth mistake fails render-golden even on a run where the schema stage "
+              f"did not execute")
+
+# ============================================================ VERDICT
+inv = " | ".join(f"{k} -> {v or '(none)'}" for k, v in sorted(rendered.items()))
+print(f"oauth2-proxy sidecars audited ({len(rendered)}): {inv}")
+if FAILURES:
+    print(f"\nCHECK FAIL (oauth2-proxy-api-routes-per-app): {len(FAILURES)} failed, "
+          f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  FAIL {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: oauth2-proxy-api-routes-per-app — {CHECKS} assertions (R-IDP-02)")
+PY

conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh

@@ -129,6 +129,22 @@ KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
         "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
         "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
         "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
+    "platform/oauth2-proxy/podmonitor.yaml:oauth2-proxy-sidecars":
+        "a PodMonitor, so it resolves against POD ports, not Service ports. Unlike the other "
+        "entries here the target is NOT chart-owned — the `metrics` (44180) container port is "
+        "authored in this repo, on the oauth2-proxy sidecar in apps/wave-0/{jackett,qbittorrent,"
+        "pihole}.yaml and apps/wave-1/{radarr,sonarr}.yaml. It is unauditable here because NO "
+        "Service exposes 44180 BY DESIGN: adding one would mean changing catalog/render.py and "
+        "every apps/wave-* render and golden, to publish an endpoint only Prometheus should dial. "
+        "A PodMonitor needs no Service, which is the whole reason it was chosen. What assertion "
+        "(2) would have checked was instead MEASURED on prod 2026-10-01, read-only: the pods "
+        "declare ports [('http', 4180), ('metrics', 44180)], the selector label "
+        "homelab.keithjackson.dev/auth-mode=forward-auth is on exactly those five gate pods, and "
+        "port-forwarding 44180 returns http=200 with 10208 bytes of oauth2_proxy_* metrics. "
+        "CAVEAT, deliberately recorded: those targets are DOWN on prod, because the app "
+        "namespaces' default-deny NetworkPolicy drops the scrape from ns monitoring (same bug and "
+        "fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents). The port name "
+        "resolves; the network path does not yet.",
 }
 
 MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")

decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md

@@ -0,0 +1,289 @@
+---
+date: 2026-09-30
+branch: feat/oauth2-proxy-auth-failure-visible
+title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
+---
+
+## What changed
+
+Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
+operator's report named the property that matters — *"reloading the page never fixes the issue and
+the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
+about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
+the alert rules. It is an absence of data.
+
+**The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
+
+    count by (__name__) ({__name__=~"oauth2_proxy.*"})
+      oauth2_proxy_requests_in_flight => 2
+
+Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
+That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
+jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
+`--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
+starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
+cluster selects them.
+
+So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
+a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
+exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
+permanently silent, and indistinguishable from health. It was not written.
+
+What was written instead alerts on the condition that made the incident invisible, from families
+that **are** scraped:
+
+- **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
+
+      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
+        unless
+      count by (namespace) (up{container="oauth2-proxy"})
+
+  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
+  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
+  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
+  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
+  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
+  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
+  clears per-namespace as each gate gains a real scrape.
+- **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
+  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
+  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
+  prod, not invented.
+- **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
+  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
+  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
+  `configMapGenerator` and listed in `runbooks/README.md`.
+- **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
+  per-app sidecars, plus the measured reason.
+- **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
+  three-part change that would close the gap.
+
+**Why the obvious alternative was rejected.** An alert on the *shared* proxy's
+`oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
+need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
+and — decisively — **it would not have fired on this incident**: the shared instance is not in the
+data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
+at all, having proxied nothing since they started (the counter exists in the TSDB only from an
+older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
+Shipping it would have looked like coverage of the reported failure and been none.
+
+## What I verified
+
+Everything below was measured against live prod through the pinned container, read-only. No
+`apply`, no `edit`, no `patch`, no `flux reconcile`.
+
+- **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
+  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
+  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
+  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
+  `1`, `namespace="oauth2-proxy"`.
+- **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
+  `oauth2-proxy` container, with its declared ports: the two shared replicas have
+  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
+  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
+  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
+  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
+- **No existing alert is inert *for this reason*, because no alert references these metrics at
+  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
+  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
+  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
+  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
+  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
+- **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
+  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
+  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
+  observable from the IdP side.
+- **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
+  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
+  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
+  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
+- **The alert expression fires against live data.** Run verbatim on prod:
+  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
+  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
+  radarr`, value 1 each; `oauth2-proxy` correctly absent.
+- **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
+  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
+  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
+  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
+  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
+  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
+  runbook as the thing to compare against.
+- **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
+  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
+  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
+- **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
+  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
+  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
+  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
+  `runbook_url` points at exactly that name.
+- **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
+  per-stage outcomes in the next section — read them there, not here.
+
+## What I did NOT verify
+
+- **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
+  only in git on this branch. What is proven is that its expression returns the expected vector
+  against live prod data and that promtool agrees about its behaviour over synthetic series.
+  **Merging is what deploys it.**
+- **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
+  any useful sense** — it describes a standing condition that has been true since the sidecars
+  were deployed, so it would have been firing before, during and after. That is deliberate and it
+  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
+  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
+  instead of discovering it during the investigation. **No alert on this platform could have
+  detected the incident itself, and none can today.**
+- **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
+  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
+  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
+  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
+  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
+  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
+  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
+  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
+- **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
+  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
+  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
+  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
+  directory from the one the sidecar writes), and the rendered config contains **no
+  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
+  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
+  caught this incident, and is the single highest-value follow-up here — **would evaluate and
+  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
+  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
+  inert rule ConfigMap to look productive.**
+- **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
+  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
+- **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
+  snapshot; anything older than three days is already gone. This is stated in the runbook because
+  it bounds what an operator can still investigate.
+- **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
+  requirement touched by this branch is certified. See the exit code below — **rc 3 is
+  UNDETERMINED, not green.**
+
+### Follow-ups, in order of value
+
+1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
+   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
+   already being collected. The rule is one expression:
+   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
+   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
+2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
+   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
+   and it unlocks the ratio rule in Prometheus.
+3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
+   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
+   observable, and it belongs with whoever owns the sidecar args.
+
+---
+
+# Round 2 — 2026-10-01: the scrape got wired, and found a bug in this branch's own alert
+
+Added to the same PR (#29) rather than a fourth concurrent branch. PR #28 merged in the meantime
+and `main` moved three PRs ahead; `origin/main` is merged in here, and the `runbooks/kustomization.yaml`
+conflict was resolved by **keeping both** runbook entries (mine and `oauth2-proxy-session`).
+
+## What changed
+
+**`platform/oauth2-proxy/podmonitor.yaml` (new).** PR #28 gave the five fleet sidecars
+`--metrics-address=0.0.0.0:44180` and a named `metrics` port. No Service exposes 44180 on any of
+them, so a ServiceMonitor would have nothing to select — a **PodMonitor** targets pod ports directly
+and needs no renderer change. Selects on the pod label `homelab.keithjackson.dev/auth-mode:
+forward-auth`, which is on exactly the five fleet gate pods and absent from crafty's: the property,
+not an app list, so a sixth forward-auth app needs no edit.
+
+**`auth-visibility-alerts.yaml`: `up{container="oauth2-proxy"}` → `up{container="oauth2-proxy"} == 1`.
+This is a real bug fix, and it is the most important thing in this round.** A target that is
+discovered but failing still publishes `up`, with value **0**. The series exists, so
+`count by (namespace)` returned 1, `unless` removed the namespace, and `OAuth2ProxyGateUnobserved`
+would have **gone quiet while nothing was being scraped at all** — this branch's own disease,
+reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets
+whose scrape is dropped, so the bare form would have silently cleared on this very commit.
+
+**Two new unit-test scenarios** pinning that: a discovered-but-failing target must still fire, and a
+target that works and then breaks must re-fire rather than latch cleared.
+
+**Docs corrected rather than appended to.** The 2026-09-30 claim "the sidecars export no metrics at
+all" is now only half true and was rewritten in `runbooks/README.md`, `platform/oauth2-proxy/README.md`
+and the alert header, each showing the two-step history, because each step looked like the end of it.
+The runbook gains the policy-vs-no-target triage (step 1b) and the **radarr `/signalr` worked case**.
+
+**Deliberately NOT shipped: the ratio alert.** `oauth2_proxy_requests_total` now exists on the
+endpoint, but not in the TSDB, because the scrape is blocked. Writing the ratio rule today would make
+it inert for precisely the reason it was refused on 2026-09-30. It becomes writable the day a scrape
+succeeds, not the day a listener exists.
+
+## What I verified
+
+- **The endpoint serves — by payload, not by flag.** Port-forwarded to the jackett pod's 44180:
+  `http=200 bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total`
+  (`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. `--metrics-address`
+  being in `args` and `/metrics` returning a body are different claims; both now hold.
+- **The scrape path is DROPPED, measured from the right place.** From grafana in ns `monitoring`
+  (beside Prometheus, same policy position) to the jackett pod IP: `:44180` → `wget: download timed
+  out`; `:4180` → **also** timed out; the shared proxy's `:44180` → 200 with `oauth2_proxy_*` present.
+  Cause, read from the objects: ns jackett's `default-deny` is `podSelector: {}`,
+  `policyTypes: [Ingress, Egress]`, with an `egress` rule and **no `ingress` key** (deny-all ingress);
+  the only allowance is CNP `jackett-allow-gateway-ingress`, `fromEntities: [ingress]` to port 4180.
+  The shared proxy is scrapable only because ns `oauth2-proxy` has **no NetworkPolicy at all**.
+- **No Service exposes 44180** on any of the five (enumerated).
+- **The pod label selector is exact:** `auth-mode=forward-auth` on all five fleet gate pods, absent
+  from crafty's. Crafty's sidecar still has no `--metrics-address` (args read in full).
+- **Flux ordering forbids shipping the fix from here:** `oauth2-proxy dependsOn=['platform-config']`
+  and `apps-wave-0 dependsOn=['platform-config']` — siblings, no ordering. A NetworkPolicy naming ns
+  `jackett` from this tier would apply today and fail on a cold rebuild.
+- **The `== 1` guard is non-vacuous — proven by mutation.** Reverting it to the bare `up` makes the
+  two new scenarios FAIL (`got: nil` at 1h5m and 2h5m) and leaves every other assertion passing;
+  restoring it returns the suite to SUCCESS under pinned promtool v3.14.0.
+- **Gate:** see the next section for the exit code and skips.
+
+## What I did NOT verify
+
+- **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from the
+  measured pod labels and port name, not observed — no target list has been read with it in place,
+  because applying it is an operator action. **Expect `up == 0` on all five when it lands**, which is
+  the documented state, not a regression.
+- **I did not fix the NetworkPolicy, and this PR therefore does not make the gates observable.** It
+  makes them *discovered*, and makes the remaining gap loud instead of silent. The one narrow rule
+  per app (ns monitoring → `auth-mode=forward-auth` → TCP 44180, per
+  `platform/longhorn/networkpolicy-metrics.yaml`) has to be rendered with each app's own policy, in
+  the catalog/renderer that is in flight on another branch. **It needs an owner.**
+- **Whether Prometheus will actually select a PodMonitor in ns `oauth2-proxy` targeting other
+  namespaces.** `namespaceSelector.any: true` plus `podMonitorSelectorNilUsesHelmValues: false` and
+  the fact that cross-namespace ServiceMonitors/PrometheusRules already work here say yes; it is not
+  measured with this object.
+- **No session-lifecycle claim can be made from the current window.** `AuthSuccess` is **0** on both
+  jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s`
+  has never executed a refresh. The cookie fix is unproven in the one dimension that matters most.
+- **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither
+  the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported.
+- **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified.
+
+### Round 2 addendum — CI is RED, by design, and the local gate cannot see why
+
+After pushing round 2, `gate / gate (pull_request)` reported **failure after 1m42s** while the local
+toolbox run was rc 3 with nothing failed. Measured cause, singular:
+
+    sensitive paths touched:
+      conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh   (matched conformance/**)
+    approving reviews: (no reviews at all)        author=gitea-admin
+
+`ci/approval-policy.sh` is fail-closed and exits 1 when a PR touches a `sensitive_paths` glob with no
+`APPROVED` review from a non-author — "a change to the gate cannot be gated by the gate". The single
+`KNOWN_EXTERNAL_MONITORS` entry is what trips it. **Nothing in L0/L1 failed.**
+
+**The local gate structurally cannot catch this, and said `approval-policy: green (N/A)`.** The stage
+short-circuits to N/A when `GITHUB_EVENT_NAME` is unset, which it always is off-CI. So for any branch
+touching a sensitive path, a clean local gate is a FALSE REASSURANCE — the PR run is the only honest
+signal. That is a general trap, not specific to this change, and it is worth knowing before someone
+concludes a red PR means broken code.
+
+To clear: a non-author approving review, then **re-run the gate** — approval does not re-trigger the
+status check, so the required context stays red until the job runs again. `required_approvals=1`
+already applied to every PR here, so the human was needed regardless; the conformance entry only
+turns the tick red while that is pending.
+
+The alternative is to drop the register entry, which means dropping `podmonitor.yaml` with it (the
+check fails without the declaration). I did not take it: the entry is the "declare the limit"
+mechanism the check explicitly asks for, it is shrink-only so it cannot rot quietly, and
+`flux-podmonitor.yaml` is the same case already in the register. But it is a real trade and the
+reviewer should make it knowingly.

decisions/2026-10-01-admin-approval-override.md

@@ -0,0 +1,256 @@
+---
+date: 2026-10-01
+branch: fix/admin-approval-override
+title: Administrators can merge their own green PR — required_approvals 1 → 0, plus an attributed /approve-override for sensitive paths
+---
+
+## What changed
+
+**The problem, measured first.** `required_approvals: 1` on `main` was a lock-out, not a
+protection. Gitea refuses to let a pull request's author approve their own PR, and `jakman85` is
+the **only** collaborator (permission `owner`). So no account in existence could approve a PR the
+operator opened. Measured on the live API, 2026-10-01:
+
+| | |
+|---|---|
+| PR #35 (`bugfix/arr-to-plex`, author `jakman85`) | `gate / gate (pull_request)` = **success, 1m41s** |
+| its reviews | `[]` |
+| live `required_approvals` | `1` |
+| live `block_admin_merge_override` | `true` |
+
+Green gate, zero reviews, no path to a merge — and the same was true of any PR that would have
+fixed it.
+
+It was also never the design this repo wrote down. `ci/approval-policy.sh`'s own header and
+`ci/gate.sh:194` both already described the model as *"`required_approvals` drops to 0 and the
+GATE decides"*, and `infra/gitea/require-gate.sh`'s closing note named this exact failure and
+prescribed this exact remedy. `decisions/2026-09-30-flux-deploy-notification-inhibited.md` had
+flagged the 1-vs-0 disagreement as unresolved and not that session's to settle. This settles it
+at 0.
+
+**Two layers were blocking, and both are addressed.**
+
+*Layer 1 — Gitea branch protection.* `required_approvals: 1 → 0` in
+`infra/gitea/expected-branch-protection.json`, with the rationale in the file, plus
+`infra/gitea/allow-admin-self-merge.sh` to apply it to the live rule (it reads the target **from
+the declaration**, so the script cannot drift from what the gate checks). `require-gate.sh` was
+changed to 0 as well — left at 1 it would silently re-impose the lock-out on its next run.
+
+The narrow choice matters: the *other* way to unblock an owner is
+`block_admin_merge_override: false`, and that would have re-opened precisely what
+`infra/gitea/block-admin-override.sh` was written to close (PRs #5–#9 merged with zero approvals
+while the settings read as enforced). This change drops the approval **count** only.
+`block_admin_merge_override` and `enable_status_check` both stay `true`, and the script asserts
+both afterwards and fails loudly if the PATCH disturbed either. **A red gate remains unmergeable
+by everyone, administrators included.**
+
+*Layer 2 — `ci/approval-policy.sh`.* Sensitive paths (`ci/**`, `infra/gitea/**`, `CLAUDE.md`,
+kyverno, rbac, sops, Flux wiring) still require a non-author `APPROVED` review, which the sole
+administrator still cannot obtain. An administrator may now waive it by commenting on the PR:
+
+    /approve-override <reason>
+
+- the directive must **start a line** (a quoted or mid-sentence mention does not count, or
+  repeating someone else's override would re-arm it);
+- a **reason is mandatory** — a bare directive records that review was waived but not why;
+- the commenter's permission is **re-checked live** against
+  `/collaborators/{login}/permission` when the gate runs, so an old comment from a demoted
+  account carries nothing;
+- it is **invalidated by a new commit** (the comment must post-date the head commit), mirroring
+  Gitea's own `dismiss_stale_approvals`;
+- the overrider, timestamp and reason are printed in the gate log and the job summary, and a
+  self-waiver by the PR's own author is called out as such.
+
+Selection of the candidate comment is `ci/approval-override.py` — pure, offline, no authority of
+its own — so it could be unit-tested; authorisation stays in the fail-closed stage.
+`ci/tests/approval-override.test.sh` adds **23 assertions**, all passing, each constructing a
+case the parser must refuse. Every new failure mode blocks: unreadable head-commit timestamp,
+unreadable permission, parser error.
+
+**The cost, stated rather than buried.** For an administrator, *"a change to the gate cannot be
+gated by the gate"* is now **advisory, not structural**. That is unavoidable with one human — the
+alternative is a repository its owner cannot change — and what is bought back is an attributed,
+reasoned, commit-scoped record of each waiver instead of a silent bypass.
+
+**The uncomfortable part, accepted with eyes open (operator decision).** Authorisation is by live
+permission **alone**, with no allowlist of human accounts. The **agent account `gitea-admin` also
+reports permission `owner`** on this repo (measured), so an agent *can* clear the sensitive-path
+review on its own PR — and agents author most PRs here, so that is the common case, not an edge
+case. I raised this before implementing; the operator chose permission-only. It is documented in
+three places rather than left to be discovered: `approval-policy.json`
+(`_known_consequence_agent_accounts`), the stage's header, and `CLAUDE.md`, which now tells
+agents plainly **not** to use it on their own work. The narrowing path needs no code change —
+add an `override_admins` list to `approval-policy.json` and the stage already requires membership
+in addition to the permission check.
+
+Also updated: `CLAUDE.md` §1 and the §2 enforcement table (the "approval required" row is now
+explicitly ⬜ with the reason, rather than a stale ✅), and `infra/gitea/README.md` gains §6 and
+marks its historical step 4 as reversed — do not re-apply.
+
+**Second pass, 2026-10-02 — the override did not work the first time, and why.** The operator ran
+the live PATCH (`required_approvals: 0` confirmed live, with `block_admin_merge_override` and
+`enable_status_check` still `true`), merged PR #35, and posted `/approve-override Need to ensure
+approval override work as expected` on PR #38. The gate re-ran (run 203) and **still blocked.**
+
+The comment was not at fault: posted by `jakman85`, directive at the start of the line, reason
+present, and 29 minutes **newer** than head commit `616a497`, so it cleared the staleness rule.
+What failed was that **CI could not read it.** Run 203's log:
+
+    approval-override: COMMENTS_JSON is not a list
+
+`SETTINGS_API_TOKEN` is minted `["read:repository", "read:admin"]` by
+`infra/gitea/set-ci-token.sh`, and Gitea puts `/issues/{n}/comments` behind the **issue** scope. So
+Gitea returned an error *object*, `jq -e .` accepted it as valid JSON, and the stage fell through
+to "no approving review" — **a permission failure wearing the costume of a missing override.**
+`/pulls/{n}/files` and `/pulls/{n}/reviews` are under `read:repository`, which is exactly why every
+other part of the same stage worked and this one did not.
+
+Three fixes, and the first is the one that mattered most:
+
+1. **The diagnosis is no longer misleading.** `COMMENTS_JSON` is out of the must-parse-or-block
+   loop and is explicitly checked for being an *array*; when it is not, the stage prints Gitea's
+   own `.message`, names the missing scope, and says where to post the override instead. An
+   unreadable source can only cause a false BLOCK, never a false pass, so degrading it to a loud
+   warning is safe — and blocking on it was never the honest behaviour.
+2. **The override is accepted from a PR REVIEW body as well as a comment**, so the hatch is not
+   hostage to one API scope. A review of type "Comment" works on the token CI has *today*. The
+   gate names which source it used. `PENDING` (unsubmitted, author-only draft) reviews are dropped.
+3. **`set-ci-token.sh` now requests `read:issue`**, which makes the ordinary comment box work —
+   the thing anybody reaches for first. It takes effect only on a **re-run**: an existing token's
+   scopes cannot be widened in place.
+
+`ci/tests/approval-override.test.sh` grows from 23 to **38 assertions**, covering the reviews
+source, `PENDING` reviews, an error *object* from the comments API, "an unreadable comments source
+must not suppress a valid review override", latest-wins across both sources, and that the
+diagnostic actually quotes Gitea's message.
+
+**One adjacent fix, deliberately in scope: `ci/tests/gate-decision.test.sh` was already broken.**
+It hand-maintains a list of stub stage scripts that "must name EVERY stage `gate.sh` runs", and it
+did not name `approval-policy` — added to `gate.sh` by PR #17 and never added here. `run_stage`
+therefore got exit 127 from a missing file and recorded a FAILED stage in every case, so **15 of
+50 assertions failed**, including every one expecting a PASS/UNDETERMINED headline. Measured on
+pristine `origin/main` (`0c7e8dd`) before touching it, so the attribution is not a guess.
+
+I fixed it because `make -C ci test` is the *only* runner for the suite this change adds — leaving
+it red would have made the new tests unverifiable in practice. The file's own comment already
+narrated this exact failure for `gitea-settings` and did not prevent the repeat, which says the
+comment was not the fix. So the stub list is now **derived from `gate.sh`'s own `run_stage` lines**
+and asserts it parsed at least 8 of them: a stage added to the gate can no longer be silently
+missing from its test. `make -C ci test` goes from **FAILED to all suites passed**.
+
+## What I verified
+
+- **The lock-out is real, and PR #35 is the proof.** Queried live: head sha
+  `8a094c46`, statuses `gate / gate (pull_request) = success :: Successful in 1m41s`,
+  `/pulls/35/reviews` → `[]`, `/branch_protections/main` → `required_approvals: 1`. I read the
+  **rule object**, not `/branches/main`, per §2b.
+- **`jakman85` is the only collaborator**: `/collaborators` → one entry. Permission `owner`.
+- **`gitea-admin` also reports `owner`** — this is why the agent-account consequence above is a
+  statement of fact and not a hypothetical.
+- **PR #35 touches no sensitive paths** (`apps/wave-0/plex.yaml`, `apps/wave-1/{radarr,sonarr}.yaml`,
+  `catalog/{radarr,sonarr}.yaml`, its decisions entry) — so `ci/approval-policy.sh` was never its
+  blocker. Layer 1 alone unblocks it.
+- **API shapes verified before depending on them**, rather than assumed: `/git/commits/{sha}`
+  returns `.commit.committer.date` as RFC3339 **with an offset** (`2026-10-01T23:09:08-05:00`),
+  not a trailing `Z` — a parser handling only `Z` would have silently dropped every real
+  override, so there is a test for the offset form. PR conversation comments are under
+  `/issues/{n}/comments`; `/pulls/{n}/comments` returns diff comments only and would never have
+  seen an override.
+- **`ci/tests/approval-override.test.sh`: 23 passed, 0 failed.** Covers the happy path, bare
+  directive, mid-sentence and `>`-quoted directives, stale and equal-timestamp overrides,
+  unparseable timestamps, missing login, invalid JSON, non-list JSON, latest-wins, offset
+  timestamps, and a configurable directive.
+- `bash -n` clean on `ci/approval-policy.sh`, `infra/gitea/allow-admin-self-merge.sh`,
+  `infra/gitea/require-gate.sh`; `jq -e .` clean on both JSON declarations.
+- **`gate-decision.test.sh`'s 15 failures are pre-existing, measured not assumed.** Extracted
+  `origin/main`'s `ci/` at `0c7e8dd` into a throwaway directory and ran the suite from there:
+  **FAIL, 15 failed / 35 passed**. After deriving the stub list from `gate.sh`: **PASS, 50
+  assertions**.
+- **`make -C ci test` in the pinned toolbox: `ci/tests: all suites passed`, rc 0** — approval-override
+  **38**, gate-decision 50, touched-requirements-base 31, test-attribution 15, touched-requirements 26,
+  decisions-log pass. It was rc 2 (`ci/tests: FAILED`) before the stub-list fix.
+- **The live flip worked, and the drift detector proved it.** Run 202 (before): `gitea-settings:
+  FAIL`, `required_approvals: DECLARED 0 but LIVE is 1`. Run 203 (after the operator ran the
+  PATCH): `gitea-settings: pass`, with `block_admin_merge_override = true` among the keys it
+  checked. So the narrow change held — the approval count moved and the gate's teeth did not.
+- **The first override attempt failed for a measured reason, not a guessed one.** Run 203 log:
+  `approval-override: COMMENTS_JSON is not a list`. Token scopes read from `set-ci-token.sh`:
+  `[read:repository, read:admin]`. The same run successfully read `/pulls/38` and
+  `/pulls/38/files` (author and sensitive paths both resolved) and got past the reviews
+  validation, which is how I know `/pulls/{n}/reviews` IS readable with that token — from
+  behaviour, not from documentation.
+- **The comment itself was valid.** `created_at` 2026-10-02T05:31:24Z vs head commit `616a497`
+  committer date 2026-10-02T00:02:20-05:00 → the comment is 29m04s newer, so it passed the
+  staleness rule. Ruling the comment out is what left the token scope as the only candidate.
+- **`?limit=1000` is not the culprit.** Queried `/issues/38/comments` with `limit=1000`, `limit=50`
+  and no limit using my own credentials: `type=array` in all three. So the difference is the
+  credential, not the query.
+- **The root cause is confirmed by Gitea, in its own words.** Run 206 (the fix in place):
+  `[warn] cannot read PR COMMENTS from Gitea — token does not have at least one of required
+  scope(s), required=[read:issue], token scope=read:admin,read:repository.` followed by the
+  instruction to post the override as a review. That is the diagnosis the first attempt should
+  have produced, and `gitea-settings: pass` in the same run. The stage still ends `FAIL`, correctly
+  — there is no valid override on the PR yet.
+- **Toolbox merge gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`), run on the
+  rebased branch: **rc 3, `MERGE GATE: UNDETERMINED`**. Green stages: spec-immutability,
+  decisions-log, lint, l0 (23 offline cases, 0 failures), render-golden including the new
+  `catalog/render.py --selftest`, l1 (3 cases), approval-policy (N/A offline). Skipped:
+  `gitea-settings` (no `SETTINGS_API_TOKEN` on this host) and `conformance` (no cluster).
+  **This is NOT a green run and I am not reporting it as one** — rc 3 is UNDETERMINED per the
+  CLAUDE.md §3 contract.
+
+## What I did NOT verify
+
+- **The override has still never been ACCEPTED by a real run.** This is the second attempt at
+  this bullet, which is itself the point: the first pass called the hatch unverified, and the
+  first real test found a defect that no amount of local testing would have (an API scope CI has
+  and I did not). What is now measured is that the parse → *reject* path works end to end and
+  diagnoses itself. The **accepting** branch — directive found → live permission lookup → exit 0 —
+  has never executed. Posting the override as a **review** on PR #38 is its first real test, and
+  I would not be surprised by a third round.
+- **`read:issue` is the right scope name — Gitea said so itself, so this is no longer a guess.**
+  Run 206 printed Gitea's own message: `token does not have at least one of required scope(s),
+  required=[read:issue], token scope=read:admin,read:repository`. That both confirms the root
+  cause and names the string, which matters because this repo has been burned by a token *name*
+  that could not exist (`GITEA_SETTINGS_TOKEN`). What is **still** unverified is that the token
+  *mint* accepts it — `POST /users/{u}/tokens` with that scope has not been run. The review route
+  does not depend on it.
+- **I did not verify that Gitea lets a PR's own author submit a `COMMENT`-type review.** Gitea
+  blocks author self-*approval*; "Comment" is expected to be allowed, and that expectation is now
+  load-bearing for the recommended route. It is one click to find out, and I deliberately did not
+  test it by posting an override myself: doing so as `gitea-admin` is precisely the agent
+  self-override that `CLAUDE.md` now forbids.
+- **The whole re-run mechanism is manual.** `gate.yml` triggers on `pull_request` and `push: main`
+  only — there is no `issue_comment` trigger, so posting an override does **not** re-run the gate.
+  Re-run the job from the Actions UI, or close and reopen the PR (which fires `pull_request` and
+  leaves the head SHA alone). **Do not push an empty commit to re-trigger it:** that moves the head
+  commit and makes the override stale by the rule above, so it would have to be posted again. A
+  comment trigger was considered and not built — it would need to check out PR head code while the
+  job holds `SETTINGS_API_TOKEN`, which is the classic pwn-request shape and not worth it here.
+- **I have not seen the gate still REFUSE a merge** now that approvals are 0. The claim that
+  `block_admin_merge_override: true` keeps a red gate unmergeable rests on Gitea's documented
+  behaviour and on the 2026-09-29 session, **not** on a provoked refusal in this one. That probe
+  is written out at the foot of `allow-admin-self-merge.sh` and should be done, exactly as the
+  push rejection was provoked rather than assumed.
+- **Clock skew in the staleness check is unhandled.** Staleness compares the comment's
+  server-side `created_at` against the head commit's *committer* date, which comes from the
+  committer's machine. A commit dated in the future would reject legitimate overrides until real
+  time catches up (fail-closed, so loud, but annoying). Gitea's own `dismiss_stale_approvals`
+  compares SHAs instead; a comment cannot. Not hit here — PR #35's commit date is ~21 min behind
+  server time — and not fixed.
+- **`ci/tests/approval-override.test.sh` does not run in the merge gate.** `ci/gate.sh` runs no
+  `ci/tests/*`; they run only via `make -C ci test`. So a regression in the parser would not be
+  caught by CI. PR #33 (merged while this branch was being written, and rebased onto here) fixed
+  the *same class* of gap for `catalog/render.py --selftest` by invoking it from
+  `ci/render-golden.sh` — it did **not** wire `ci/tests/*` in, which I confirmed after rebasing
+  rather than assuming. Wiring them is a one-line stage and I did not widen scope to it.
+- **Authorisation itself is untested code.** `ci/approval-override.py` is unit-tested; the live
+  permission lookup, the `override_admins` allowlist branch, and every new BLOCK path in
+  `ci/approval-policy.sh` need Gitea and have never been exercised. The allowlist branch in
+  particular is implemented but **unused** — `override_admins` is absent by design.
+- **The other open PRs will go red** once the live value changes, until they are rebased onto the
+  merged declaration: as of writing those are **#29** and **#37** (both `gitea-admin`), plus
+  **#35**. I did not rebase them. Merging them first avoids it entirely; the script's header gives
+  the order. PRs #33, #34 and #36 merged while this branch was being written — this branch is
+  rebased onto `0c7e8dd`, so the list was re-measured rather than carried forward.
+- No cluster state was read or changed; nothing here touches Flux, Kyverno or any workload.

decisions/2026-10-01-arr-to-plex.md

@@ -0,0 +1,120 @@
+---
+date: 2026-10-01
+branch: bugfix/arr-to-plex
+title: Radarr and Sonarr may now reach Plex on 32400 — the network path only, not the Plex connection itself
+---
+
+## What changed
+
+Radarr and Sonarr had no route to Plex. The media library lives on NFS
+(`/mnt/Storage/plex`, the ZFS dataset `Storage/plex` on `${BULK_NFS_SERVER}`), and NFS
+does not deliver filesystem CRUD notifications, so Plex never learns that an arr has
+written a new file. Without a push from the arrs, the library only reflects new media
+when somebody scans it by hand. That is the bug this branch addresses.
+
+The fix is one `dependencies:` entry per app, added to the catalog — the source of truth:
+
+    - to: plex
+      port: 32400
+      protocol: tcp
+
+in both `catalog/radarr.yaml` and `catalog/sonarr.yaml` (these two *are* the arr suite
+here; `jackett`, `qbittorrent` and `seerr` are different roles and are untouched).
+
+The renderer expands each entry into a **pair** of Cilium policies, which is why a
+three-line catalog change produces 88 lines under `apps/`:
+
+| Side | Policy | File |
+|---|---|---|
+| egress | `radarr-allow-plex-32400` | `apps/wave-1/radarr.yaml` |
+| egress | `sonarr-allow-plex-32400` | `apps/wave-1/sonarr.yaml` |
+| reciprocal ingress | `plex-allow-radarr-32400` | `apps/wave-0/plex.yaml` |
+| reciprocal ingress | `plex-allow-sonarr-32400` | `apps/wave-0/plex.yaml` |
+
+Three commits, and the middle two are worth recording because both were self-inflicted
+gate failures rather than design changes:
+
+- `753c48b` — the catalog edit alone, with `apps/` left unrendered.
+- `9048c2f` — re-rendered `apps/` from the catalog, which `render==golden` requires.
+- `58f02fe` — removed 20 trailing spaces on `catalog/radarr.yaml:103`, left behind as
+  comment-alignment padding when the comment moved down to the `port:` line. Sonarr's
+  equivalent line kept its comment in place, which is why only radarr tripped.
+
+## What I verified
+
+**The lint failure, root-caused.** Gate run 193 on `9048c2f` failed with
+`[fail] lint: FAILED -> blocks merge`, and lint was the only failing stage. yamllint
+emitted 419 findings, but **418 were `line-length`**, which `ci/yamllint.yaml` pins at
+`level: warning` and which therefore cannot change the exit code. The single error-level
+finding in the entire repository was:
+
+    catalog/radarr.yaml
+      103:13    error    trailing spaces  (trailing-spaces)
+
+`trailing-spaces` inherits the default severity (error), so that one line returned rc 1.
+After the fix, `git grep -nIE ' +$' -- '*.yaml' '*.yml'` matches nothing repo-wide.
+
+**Gate run 195 on `58f02fe`** (read from the job log, not from the status tick):
+
+    [ ok ] approval-policy: green     [ ok ] l0: green
+    [ ok ] gitea-settings: green      [ ok ] render-golden: green
+    [ ok ] decisions-log: green       [ ok ] l1: green
+    [ ok ] lint: green
+    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+    [fail]   Stages that did not run: spec-immutability
+
+The Gitea check reports `success` / "Successful in 1m41s", but the gate's own verdict is
+**UNDETERMINED (rc 3)**, not PASS. That tick is `gate.yml` tolerating exit 3 for the one
+stage it documents as unverifiable on this runner (the spec repo is not checked out
+there). Nothing else produced a 3.
+
+**`render==golden` green** on `58f02fe`, so `apps/` is a faithful canonical render of
+`catalog/` — the four generated policies are not hand-written drift.
+
+**DNS egress was already open** and is not a new gap: `apps/wave-1/radarr.yaml` carries
+an existing allow for port 53 UDP+TCP, so resolving the Plex service name does not need
+anything added here.
+
+**Live branch protection on `main`**, measured against
+`/api/v1/repos/Home-Network/home-network-k3s-impl/branches/main` rather than assumed:
+`protected: true`, `user_can_push: false`, `enable_status_check: true`,
+`status_check_contexts: ['gate / gate (pull_request)']`, `required_approvals: 1`.
+
+## What I did NOT verify
+
+**This change does not accomplish what the PR set out to do.** It opens the network path
+and nothing else. Triggering a library update also requires each arr's
+*Settings → Connect → Plex Media Server* entry plus a Plex token, and **none of that
+exists anywhere in `catalog/` or `apps/`** — I grepped both for `plex`, `token` and
+`connect` and found only NFS paths, a uid and a timezone. Radarr's and Sonarr's config
+lives on a 5Gi Longhorn PVC mounted at `/config`, so that connection is UI-only state
+that exists solely on the PVC and never in git, in the same way Crafty's local login
+does. **Until somebody adds it in each app's UI, Plex still will not see library
+updates, and this branch will look like it did nothing.** That follow-up is not tracked
+by any file in this PR.
+
+**L2/L3 conformance never ran.** `CLUSTER_AVAILABLE` is unset on the runner, so the
+eight requirements this diff touches — `R-NET-01`, `R-NET-05`, `R-NET-06`, `R-NET-07`,
+`R-IDP-02`, `R-IDP-06`, `R-GITOPS-02`, `R-PLAT-02` — are **UNVERIFIED**. That gap is
+unusually load-bearing here: the entire substance of this change is four
+CiliumNetworkPolicies, and no test has confirmed that traffic from the `radarr` or
+`sonarr` namespaces actually reaches `plex:32400`. The green gate proves the offline
+render subset only.
+
+**`spec-immutability` did not run** (`R-TEST-06` not verified), for the documented
+reason — the sibling spec repo is absent from the runner.
+
+**Nothing was checked on the cluster.** I ran no `kubectl`, so I did not confirm that
+the `plex` Service exposes 32400 as the arrs would address it, that the pod labels the
+policies select (`app.kubernetes.io/name`, `k8s:io.kubernetes.pod.namespace`) match the
+running pods, or that Cilium accepted the four new objects. Flux has not applied any of
+this: the branch is unmerged, so `.status.lastAppliedRevision` does not contain it.
+
+**I did not verify Plex's own side** — whether its library-update endpoint on 32400 is
+the right target for the arrs' notification, or whether Plex requires
+"Update my library automatically" enabled to act on one.
+
+**I am not the approver and did not approve this PR.** `required_approvals: 1` is live,
+and the only credential on the authoring workstation belongs to the PR's author
+(`jakman85`), whose own review Gitea does not count. Per CLAUDE.md §1 the merge is a
+human action regardless.

decisions/2026-10-01-oauth2-proxy-api-route-per-app.md

@@ -0,0 +1,482 @@
+---
+date: 2026-10-01
+branch: feat/oauth2-proxy-api-route-per-app
+title: The config model had no way to say "this app's API lives here" — so --api-route could not ship. Now it can, per app.
+---
+
+## What changed
+
+Second half of PR #28 (`decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md`), which
+fixed the cookie collision, the 5-minute session and the missing metrics listener in one renderer
+function and **deliberately declined `--api-route`** for a reason I have preserved rather than
+worked around: that flag was measured on **one** app, and `PiHole serves no /api/ at all`, so a
+renderer-wide constant would apply one app's path shape to five. The `auth` block is
+`additionalProperties: false`, so there was no per-app override hook. **Creating the hook was the
+work; setting the flag is the easy part.**
+
+**Three surfaces, in order of what they protect:**
+
+| Surface | Change | What it guards that the others cannot |
+|---|---|---|
+| `schemas/servicedefinition.schema.json` (**1.8.0 → 1.9.0**) | new optional `auth.apiRoutes`: array of anchored path regexes, `uniqueItems`, default `[]` | rejects a too-broad regex **where a human types it** — `ci/l0.sh` validates `catalog/*.yaml` against this schema on every gate run |
+| `catalog/render.py` | emits one `--api-route=<r>` per declared entry, in order; **raises** on an unanchored entry | `ci/render-golden.sh` runs on every gate invocation even when the schema stage skips |
+| `conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/` (new) | cross-checks each rendered sidecar against its own ServiceDefinition | catches a **renderer constant**, which neither of the above can see |
+
+**The `auth` block stays `additionalProperties: false`.** One named, documented key was added; the
+door is not opened. That property is the reason the gap existed, and it is worth keeping — it is
+what made "there is no override hook" a *fact* rather than a *guess* for the two engineers who hit
+it before me.
+
+**Rebased mid-flight.** `origin/main` advanced from `08deb20` to `cdf3e14` while I was working
+(PR #27 and PR #32, the `hardware.gpu: intel-quicksync` capability). **That change also took schema
+`1.8.0`**, so mine is renumbered to **1.9.0** and chained behind it; both version entries are
+present in the description, and `catalog/render.py` had a two-sided conflict in the constants block
+which is resolved by keeping both (the `DRI_*` constants and `OAUTH2_API_ROUTE_PATTERN` are
+unrelated). Re-rendering after the rebase still changes **only** jackett and radarr — plex's new
+golden reproduces byte-identically, so the two changes do not interact.
+
+**Shape:** a **list** of regexes, not a single string, because Radarr needs two
+(`^/api/` and `^/signalr/`) and `--api-route` is natively repeatable. A single string would have
+forced alternation into the value, and alternation is exactly how breadth gets smuggled past a
+prefix-only reading of a regex.
+
+**The anchoring ratchet.** Each entry must match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$` — anchored at
+`^/`, then literal path characters, with an optional trailing `.*` or `$`. The same pattern is in
+the schema, in the renderer, and in the new check, and assertion (5) of the check **fails if they
+drift apart**, because if they do the loosest one wins silently. Rejected shapes, all five
+provoked: `^/`, `^/.*`, `.*`, `/api/` (unanchored), `^/api/|^/` (alternation). Alternation is
+**refused rather than supported**: the flag is repeatable, so one entry per route costs nothing.
+
+**Why breadth is the dangerous direction, and it is not symmetric.** A *missing* `--api-route` is a
+silent 302 — bad, invisible, recoverable. A *too broad* one answers a plain **page load** with 401:
+a logged-out operator gets a bare error page and is **never offered a sign-in**, so the service
+looks permanently broken and no reload recovers it. That is a worse outcome than the defect the
+flag fixes, which is why it is gated on three surfaces rather than documented in one.
+
+### Also folded in: the selftest could exit 1 with ZERO diagnostics
+
+Not my change originally — Desmond Oyelaran found it while wiring `catalog/render.py --selftest`
+into the `render-golden` stage (his PR #33, which touches only `ci/render-golden.sh` and is **not**
+in my base; `origin/main` at `cdf3e14` does not yet drive the selftest from any gate stage). The
+coordinator asked me to fold the fix into this PR because the defect is in the file I own.
+
+`_selftest()` reached `_cookie_name()` through a bare `next(genexpr)` with **no default**. Delete
+the `--cookie-name` emission — exactly the mutation that assertion exists to catch — and it raises
+`StopIteration`, which escapes `_selftest()` **before its collected `SELFTEST FAIL:` lines are
+printed**. Right verdict, no diagnostics, and it masks every other assertion the same mutation
+broke. Fixed with `next(..., None)` plus an explicit `check()` so a missing flag reports **as a
+named assertion**.
+
+Provoking it turned up **three more instances of the same failure class** in the same block, which
+is why this is not literally a one-line change: `side = fa_containers[1]`,
+`side["env"][0][...]`, and the `fa_svc` lookup all raise rather than assert when the sidecar is
+absent entirely. All four are now None-tolerant, and the two fixtures I added for `--api-route`
+were written the same way from the start. Measured, on throwaway copies:
+
+| Mutation | Before | After |
+|---|---|---|
+| `--cookie-name` emission deleted (origin/main's renderer) | rc 1, **0** `SELFTEST FAIL` lines, 1 `StopIteration` | rc 1, **3** `SELFTEST FAIL` lines, 0 tracebacks |
+| the oauth2-proxy sidecar not appended at all | rc 1, **0** lines, `IndexError` | rc 1, **20** `SELFTEST FAIL` lines, 0 tracebacks |
+
+Note the consequence of his stage change for anyone reading this later: once PR #33 lands,
+`.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]`, so **rc 1 is not tolerated
+in CI the way rc 3 is** — a selftest failure will block. Which is the right outcome, and it is why
+a selftest that prints nothing was worth fixing before that lands rather than after.
+
+### Declined, with reasons: the metrics-ingress NetworkPolicy
+
+The coordinator asked for my judgement on bundling Ines Dubois's finding — that #28's metrics
+listener genuinely serves (she measured `http=200`, 10,208 bytes,
+`oauth2_proxy_requests_total{code="302"} 2` via port-forward) but the scrape will be **dropped**,
+because namespace `jackett`'s default-deny has **no `ingress` key at all** and
+`jackett-allow-gateway-ingress` admits `fromEntities: [ingress]` only. The fix belongs in each
+app's own rendered policy, i.e. in my file and my goldens. **I declined to bundle it**, and asked
+for it as the immediate next PR. Five reasons, in the order that decided it:
+
+1. **The real reason is that it is a MODEL question, not a policy to bolt on.** Schema 1.6.0
+   already owns this: `observability.metricsEndpoint` renders the Service port **and** the
+   monitoring-namespace ingress allow. What the model cannot currently express is a **second**
+   metrics endpoint on the same pod belonging to a *sidecar the renderer injected*, rather than to
+   the workload the operator declared. Keying a bespoke policy off the pod label
+   `homelab.keithjackson.dev/auth-mode: forward-auth` works today, but it hardcodes "forward-auth
+   implies a metrics port on 44180" into a policy instead of **deriving** it from the sidecar
+   declaration that is already the single source of that port — the exact "list, don't derive"
+   anti-pattern `oauth2-proxy-cookie-names-distinct` assertion (5) exists to forbid, and which I
+   would be introducing one directory away from the check that forbids it. That deserves its own
+   review, and it is squarely the kind of decision I was brought in to make deliberately.
+2. **Different subject, and it is a trust-boundary change.** Mine is the auth *response code* on
+   API paths (R-IDP-02). That is an ingress allow (R-NET-01/02, R-OBS-01) admitting a new
+   namespace to a new port. Bundling a security-relevant ingress rule into a PR whose headline is
+   `--api-route` is how such a rule gets waved through on the strength of the other half — and this
+   PR already needs a human for a *different* sensitive-path reason, so the second subject would be
+   reviewed in the shadow of the first.
+3. **The conflict argument does not actually apply to my diff.** It assumes I am regenerating five
+   goldens; **I am regenerating two** (jackett, radarr), because the flag renders only where
+   declared. A metrics-ingress policy renders a *different document* — a CiliumNetworkPolicy — in
+   all five. Different hunks, in files git merges independently; and goldens regenerate
+   deterministically from the renderer, so the follow-up rebases onto `main` cleanly. The N-1
+   rebase problem `decisions/README.md` records was two branches appending to **the same hunk**,
+   which this is not. I checked my own diff before answering rather than accepting the premise.
+4. **Every fact in the finding is hers and I have measured none of it** — not the 200/10,208 bytes,
+   not the `wget: download timed out` on `:44180` *and* `:4180`, not the Flux sibling ordering that
+   makes `oauth2-proxy` and `apps-wave-0` unordered. Shipping a security-adjacent rule on wholly
+   inherited evidence, as a footnote in someone else's PR, is precisely what CLAUDE.md §2b exists
+   to stop. In its own PR I would measure it first.
+5. **Sequencing: the PodMonitor is in HER PR #29, not mine.** If I ship the policy and her
+   PodMonitor is reworked or delayed, I have opened an ingress allow for a scraper that selects
+   nothing — a hole with no consumer. The policy is only useful paired with the PodMonitor, which
+   argues for it landing adjacent to that, not here. It also drags in a
+   `KNOWN_EXTERNAL_MONITORS` entry in `conformance/tests/l0-static/servicemonitor-ports-resolve/`,
+   a second conformance interaction on top of the new check this PR already adds.
+
+Diff size is the tiebreaker rather than the argument: **1,245 insertions across 13 files**,
+including a ~400-line new conformance check. I would rather hand a reviewer one subject they can
+actually falsify. **Queued, not dropped** — and if the next person takes it, the model note in (1)
+is the part worth arguing about, not the YAML.
+
+### Which apps got which routes, and the measurement behind each
+
+| App | `auth.apiRoutes` | Evidence |
+|---|---|---|
+| **jackett** | `^/api/` | 2026-09-30, its own access log: **33 × 302 against 2 × 401** on `/api/` paths, and **4 of 6 POSTs answered by the proxy and never reaching Jackett** (upstream field `-`, not `/`), including both `/api/v2.0/server/update` calls |
+| **radarr** | `^/api/`, `^/signalr/` | `^/signalr/`: measured **by me, on the live post-fix pod** — see below. `^/api/`: measured on radarr by the PR #30 author, on the **pre-fix** pod |
+| **pihole** | *(none — deliberate)* | serves no `/api/`; **0** unauthenticated requests in its current pod's 23 h |
+| **sonarr** | *(none — not measured)* | 13,643 log lines, **0** unauthenticated requests. No traffic, not a negative result |
+| **qbittorrent** | *(none — not measured)* | 13,649 log lines, **0** unauthenticated requests. Same |
+
+**`^/signalr/` on radarr is the live proof this change was needed**, and it is the strongest
+evidence in this entry. 23 h after PR #28 deployed, on `radarr-7d55cbffb9-42hmh` (0 restarts), the
+sidecar logged **232 unauthenticated requests across the pod's entire lifetime, and all 232 are the
+same call**: `POST /signalr/messages/negotiate`, browser UA, **302**, upstream `-` (Radarr never saw
+one of them). That is a **left-open browser tab** whose session the cookie rename orphaned, polling
+SignalR forever, silently redirected every time, and **never re-prompting the operator**. It is the
+reported symptom — "reloading never fixes it and I am never re-prompted" — still happening in
+production, on a service nobody was looking at. With 401 the UI's connection breaks **loudly**; the
+operator reloads; `/` matches neither regex, so the reload signs them back in.
+
+**`^/api/` on radarr: included, and here is the honest basis.** The current pod has logged **zero**
+`/api/` traffic, so I did **not** re-measure it. It rests on a measurement of **radarr** recorded in
+the merged PR #30 handoff — its 156 unauthenticated requests were `/api/v3/command` (62),
+`/api/v3/queue/details` (46) and `/signalr/messages/negotiate` (43), i.e. **108 of 156 were
+`/api/v3` XHRs**. That is this app's own evidence, by another engineer, not an analogy to another
+app. I include it because `/api/v3/` is a pure JSON namespace a browser never navigates to (Radarr
+serves its UI at `/`, assets under `/Content/` and `/MediaCover/`), so the breadth risk is nil and
+the measured benefit is 70% of that app's rejection volume.
+
+**Sonarr and qBittorrent were measured and deliberately left alone.** Sonarr is nearly the same
+codebase as Radarr, which is exactly what makes `^/api/` + `^/signalr/` a tempting guess and still a
+guess. I read both pods' logs: **zero** unauthenticated requests, nothing but health probes, because
+nobody has opened either UI since the cookie fix rolled. **That is evidence of no traffic, not
+evidence of no need.** Each catalog file now carries a comment saying so, with the numbers and what
+to measure before adding the routes.
+
+### Goldens and fixtures regenerated
+
+Rendered with the canonical command
+(`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`).
+**Two goldens changed, not five** — and that is the per-app property proving itself on the real
+tree rather than in a fixture:
+
+    apps/wave-0/jackett.yaml   + --api-route=^/api/
+    apps/wave-1/radarr.yaml    + --api-route=^/api/
+                               + --api-route=^/signalr/
+
+`apps/wave-0/pihole.yaml`, `apps/wave-0/qbittorrent.yaml` and `apps/wave-1/sonarr.yaml` are
+**byte-identical to `main`**. **No L1 fixture needed regenerating**: all three
+`conformance/tests/l1-rendering/*/fixtures/svcdef*.yaml` declare `auth.mode: forward-auth` with no
+`apiRoutes`, so they render no flag and their goldens are unchanged. PR #28 touched 5 goldens and 3
+L1 fixtures because it changed flags on *every* sidecar; this one changes flags only where a
+measurement says so, which is the whole point.
+
+### The L0 decision (and it is a decision, not a default)
+
+**Not** in `oauth2-proxy-cookie-names-distinct`. That check's subject is cookie **identity** — its
+inputs are names and domains, its failure mode is mutual session eviction, its register is the
+shared proxy's claim on the default name. `--api-route` shares none of those. Bundling them would
+make that check's name a lie and couple two unrelated ratchets, so a future api-route decision could
+only be expressed by weakening a *cookie* check. That is PR #28's own stated discipline, applied to
+my change rather than quietly dropped.
+
+**A sibling check, `oauth2-proxy-api-routes-per-app`** (whole-repo scope, so it runs on every gate
+invocation via `ci/l0.sh`'s `--include-scope` union — no bespoke stage, which
+`whole-repo-invariants-wired` would fail). **Nowhere was the wrong answer for exactly one reason:**
+the existing gates cannot see a renderer constant. `render-golden` compares the render to the
+goldens, so a constant **plus regenerated goldens agree with each other while both being wrong** —
+I provoked that and it regenerates with **rc 0**. The new check's assertion (2) compares each
+rendered sidecar against its **own ServiceDefinition**, which is the only reading that fails it.
+
+## What I verified
+
+- **`--api-route` is SINGULAR, read off the PINNED binary rather than taken from the brief.**
+  `podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help`:
+  `--api-route strings   return HTTP 401 instead of redirecting to authentication server if token
+  is not valid. Format: path_regex`. There is **no `--api-routes`** in the help output. The PM asked
+  me to verify rather than trust, and that is the result — the singular form is confirmed and the
+  help text also confirms the *semantics* I am relying on (401 instead of a redirect).
+- **The premise, in-cluster, read-only, through the pinned toolbox, `--context k3s-prod` named
+  explicitly** (CLAUDE.md §2b and §5). PR #28 **is** deployed: all five sidecars carry
+  `--cookie-name=_oauth2_proxy_<app>`, `--cookie-domain=<app>.home.keithjackson.dev`,
+  `--cookie-refresh=2m30s` and `--metrics-address=0.0.0.0:44180`. Pods, all **0 restarts**:
+  `jackett-6877fcd9f5-ht9d9`, `pihole-…`, `qbittorrent-5547bdb9bd-zq557`,
+  `radarr-7d55cbffb9-42hmh`, `sonarr-ddf485775-8ckwt` (started 2026-09-30T21:41–21:42Z).
+- **The full post-fix log census, parsed rather than grepped** (oauth2-proxy's access-log format,
+  with `/ping` health probes excluded), over each pod's whole life:
+
+      app            lines   unauth  AuthSuccess  removing session  cookie sig invalid
+      jackett       13,649        0            0                 0                   0
+      pihole        13,649        0            0                 0                   0
+      qbittorrent   13,649        0            0                 0                   0
+      radarr        14,099      232            0                 0                   0
+      sonarr        13,643        0            0                 0                   0
+
+  Three findings, kept separate:
+  1. **The collision fix works.** Jackett was 39 unauthenticated before; it is **0** in 13,612
+     non-probe lines. The eviction fingerprint (`removing session` / `cookie signature not valid`)
+     is **0 on all five**, where radarr previously had 8 and 8.
+  2. **Radarr's 232 are one request.** 100% `POST /signalr/messages/negotiate`, 100% **302**, 100%
+     upstream `-`, browser UA (`Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537…`). **Zero `/api/`
+     traffic of any kind** on this pod.
+  3. **0 `AuthSuccess` anywhere**, so nobody has signed in since the restart and `--cookie-refresh`
+     is **still** unexercised — unchanged from PR #28's own caveat.
+- **Only two goldens change, and three are byte-identical to `main`.** `git status` after the
+  canonical render lists exactly `apps/wave-0/jackett.yaml` and `apps/wave-1/radarr.yaml`.
+  `render-golden` is green.
+- **`catalog/render.py --selftest`: PASS**, with new assertions that (a) the `fa` fixture, which
+  declares nothing, renders **no** `--api-route`; (b) a fixture declaring two routes renders exactly
+  those two, in order; (c) the renderer **refuses** all five breadth shapes; (d) four legitimate
+  routes (`^/api/`, `^/signalr/`, `^/api/v3/.*`, `^/api/v2.0/server/update$`) are **accepted** — a
+  ratchet that blocks real API prefixes is a ratchet someone eventually deletes.
+- **The new check: rc 0, 43 assertions, 6 sidecars audited**, with its inventory printed so the
+  reviewer can see what it read:
+  `jackett -> ['^/api/'] | pihole -> (none) | qbittorrent -> (none) | radarr -> ['^/api/',
+  '^/signalr/'] | sonarr -> (none) | crafty -> (none)`.
+- **The check was tested in BOTH directions, on throwaway copies outside the repo**, never by
+  mutating the branch. Each mutation fired its own assertion class and nothing else:
+  - **Pre-change tree** (`git archive origin/main` + the test dropped in): **rc 1, 11 failures** —
+    no service declares routes, the renderer emits none, the renderer does not read `auth.apiRoutes`,
+    the pattern is absent from both renderer and schema, and the shipped renderer accepts all five
+    breadth shapes.
+  - **pihole's golden hand-edited** to carry a route it never declared: **rc 1, 2 failures** (the
+    catalog↔golden mismatch, and "declares nothing must render nothing").
+  - **radarr's golden widened** to `--api-route=^/`: **rc 1, 2 failures** (unanchored, plus the
+    mismatch).
+  - **Renderer turned into a constant and EVERY golden regenerated to match: the regen exits 0** —
+    i.e. `render-golden` is **happy** — and the new check exits **1 with 15 failures**, naming
+    pihole, qbittorrent, radarr and sonarr individually. **This is the case that justifies the
+    check's existence** and I provoked it rather than argued it.
+  - **Schema `items.pattern` deleted**: **rc 1, 1 failure** — the three-surface drift guard.
+  - **Unmutated control: rc 0.**
+- **The selftest diagnostics fix was PROVOKED, not reasoned.** Both mutations above were run
+  against throwaway copies of the renderer outside the repo (`origin/main`'s `render.py` for the
+  "before" column, mine for the "after"), never by mutating the branch. The pre-fix StopIteration
+  and the post-fix 3-and-20 named assertions are quoted from those runs. `--selftest` on the
+  unmutated branch: **PASS**. Goldens after the fix: **unchanged** (it touches only `_selftest()`).
+- **The check is selected by the whole-repo union.** `./conformance/run.sh --layer l0 --requirements
+  r-nosuchreq-99 --include-scope whole-repo --print-selection` lists
+  `tests/l0-static/oauth2-proxy-api-routes-per-app` among **21** tests (it was 20), i.e. it runs
+  even when the touched-requirement filter matches nothing.
+- **THE SCHEMA GUARD IS PROVEN WITH THE PINNED VALIDATOR `ci/l0.sh` ACTUALLY DRIVES**, not by
+  reading the pattern. `./schemas/validate.sh` (`check-jsonschema, version 0.29.4`) against seven
+  throwaway ServiceDefinitions written outside the repo:
+
+      good            ["^/api/", "^/signalr/"]   rc 0
+      bad-root        ["^/"]                     rc 1  apiRoutes ... does not match
+      bad-dotstar     ["^/.*"]                   rc 1  apiRoutes ... does not match
+      bad-unanchored  ["/api/"]                  rc 1  apiRoutes ... does not match
+      bad-alt         ["^/api/|^/"]              rc 1  apiRoutes ... does not match
+      bad-everything  [".*"]                     rc 1  apiRoutes ... does not match
+      bad-mode        ["^/api/"] on mode: oidc   rc 1  const 'forward-auth'
+
+  So a too-broad regex is rejected **where a human types it**, on every gate run, by the real tool
+  — and the shape invariant fires too.
+- **THE RENDERER'S REFUSAL REACHES `render-golden` AS A READABLE FAILURE, NOT A TRACEBACK.** With
+  `^/signalr/` swapped for `^/` in a *copy* of the catalog (outside the repo), both invocation
+  shapes exit **rc 1 with 0 tracebacks** — the single-SD form and `--all <dir> -o <dir>
+  --wave-layout --no-image-automation`, which is the one `ci/render-golden.sh` drives:
+
+      render.py: radarr: auth.apiRoutes entry '^/' is not an anchored path regex. Each entry
+      must start with '^/' and continue with literal path characters (e.g. '^/api/',
+      '^/signalr/', '^/api/v3/.*'). ... A too-broad --api-route answers a PAGE LOAD with 401, so
+      a logged-out operator sees a bare error page and is never offered a sign-in: the service
+      looks permanently broken rather than merely unauthenticated.
+
+  So the breadth guard is gated by `render-golden` even on a run where the schema stage skipped,
+  and the operator is told *why* rather than handed a stack trace.
+- **The schema's new shape invariant.** A non-empty `auth.apiRoutes` requires
+  `auth.mode: forward-auth`, because the flag renders onto the per-app sidecar and no sidecar exists
+  for `oidc`/`native`/`none` — declaring it elsewhere would be a **silent no-op**, which is the
+  failure class this repo keeps paying for.
+- **Gate:** toolbox `./ci/gate.sh` with the two-mount ENGINE shim — exit code and stage list in the
+  next section, because it is not a pass and that belongs there.
+- **No mutating cluster access.** `kubectl get/logs/jsonpath` only, `--context k3s-prod` named on
+  every call. No `apply`/`edit`/`patch`, no `flux reconcile`. No Secret `.data` or `.stringData` was
+  read or printed; I did not need to touch a Secret at all this session. The one container I ran
+  locally was the pinned oauth2-proxy image, to read its `--help`.
+- **`~/.git-credentials` was never read.** PR creation used `git credential fill`, which does not
+  echo the value.
+
+## What I did NOT verify
+
+- **NO BROWSER ROUND-TRIP. BY ANYONE. FOR THIS FLAG.** This is the most important line here. The
+  302/401 split, the dropped POST bodies and the orphaned SignalR poll are all read from access
+  logs; that a **401 repairs** the operator's experience is reasoned from HTTP semantics, not
+  watched. Nobody has added a Jackett indexer or recovered a Radarr tab through this
+  configuration. The human test below is the first real evidence this change will ever have.
+- **Local gate: rc 3 (UNDETERMINED), not 0.** Green: `spec-immutability`, `decisions-log`, `lint`,
+  `l0`, `render-golden`, `l1`. **Skipped:** `gitea-settings` (no `SETTINGS_API_TOKEN` locally) and
+  **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset). Nothing FAILED; nothing runtime was certified
+  either. **rc 3 is UNDETERMINED, never green.**
+- **MY LOCAL `approval-policy: green` WAS MEANINGLESS, AND I ALMOST REPORTED IT AS EVIDENCE.**
+  `ci/approval-policy.sh:47` reads `GITHUB_EVENT_NAME`, which is unset outside CI, so the stage
+  short-circuits at line 50: `ok "not a pull_request event (event='none') — approval policy N/A"`.
+  **A clean local gate therefore CANNOT catch a sensitive-path problem — structurally, not by
+  accident.** Flagged to me by the coordinator from Ines Dubois's round, and I verified it by
+  re-running the gate with `GITHUB_EVENT_NAME=pull_request PR_NUMBER=34`: the stage then
+  **evaluates** and fails closed on the missing token —
+
+      [fail] no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews.
+      [fail] stage approval-policy: FAIL
+      [fail] approval-policy: a sensitive change lacks a non-author approval -> blocks merge
+      [fail] MERGE GATE: BLOCKED        (rc 1, not 3)
+
+  — while **every other stage stays green** (`spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1`). That is the useful result: it bounds the CI failure to this one stage
+  rather than leaving "something is red" as the whole report. Fail-closed is correct here and
+  deliberate: unlike `gitea-settings`, where a missing credential is a lock-out, a skip of this
+  stage would be a bypass (CLAUDE.md §1).
+- **CI, read from each commit's `/statuses` rather than from a tick.** Both contexts report, and
+  **only one of them runs**. Identical on `99560c2d`, `df1b25b6` and `f1d16f44`:
+
+      gate / gate (pull_request)         state=failure   desc='Failing after ~1m43-1m45s'
+      gate / conformance (pull_request)  state=success   desc='Has been skipped'
+
+  **That second line is §2b's trap in the wild**: a *skipped* Gitea job reports
+  `state=success`. The conformance job did not execute, so its green tick certifies nothing —
+  read the description, not the colour. The duration is given as a range on purpose: it differs
+  by a second or two per run, and pinning one number inside the file whose commit triggers the
+  next run is a self-reference, not a measurement. The load-bearing fields are the **state** and
+  the **description**, and those are stable across all three.
+
+  Worth recording that the **same context changed its answer over time**: on the first push it
+  briefly read `state=pending desc='Blocked by required conditions'` — still waiting on the gate
+  job it depends on — and resolved to `success` only once that job settled. So sampling a status
+  once can show either a pending **or** a green tick for a job that never executed.
+  **I could NOT read the CI job log to confirm the failing stage from CI's
+  own output**: this Gitea's `/api/v1/repos/{owner}/{repo}/actions/runs` endpoint answers
+  **404** (only `/actions/tasks` exists, which lists tasks without logs), and
+  `/actions/jobs/{task_id}/logs` answers `404 Not found` for the task id. So my attribution of the
+  CI failure to `approval-policy` rests on the local event-set reproduction above plus the
+  sensitive-path rule, **not** on CI's own log. That is weaker evidence than PR #28 had and I am
+  naming it rather than borrowing its wording.
+- **Branch protection and the runner, RE-MEASURED this session against the live API** (§2b),
+  not inherited from CLAUDE.md §2's table:
+
+      GET /repos/Home-Network/home-network-k3s-impl/branches/main
+        protected              True
+        user_can_push          False
+        enable_status_check    True
+        status_check_contexts  ['gate / gate (pull_request)']
+        required_approvals     1
+      GET /admin/actions/runners
+        id=1 name=incluster-dind status=online
+
+  So `required_approvals=1` applies to **every** PR here regardless of path — this needs a human
+  even without the sensitive-path rule. I did **not** provoke a rejected push to `main` this
+  session; `user_can_push: False` is the API's word, which is exactly the weaker form of evidence
+  CLAUDE.md §2 warns about, and I did not strengthen it.
+- **`^/api/` ON RADARR IS NOT MY MEASUREMENT.** It is PR #30's, taken on the **pre-fix** pod. The
+  current pod has logged **zero** `/api/` traffic in 23 h, so I could not reproduce it. I judge the
+  breadth risk nil (`/api/v3/` is pure JSON; the UI is at `/`), but if someone wants a strictly
+  self-measured configuration, drop `^/api/` and keep `^/signalr/` — the 232 are entirely SignalR.
+- **SONARR AND QBITTORRENT ARE UNMEASURED, NOT CLEARED.** Zero traffic in 23 h is zero evidence.
+  qBittorrent's WebUI drives everything through `/api/v2/` and is *plausibly* affected the same way
+  Jackett is; Sonarr is nearly Radarr's codebase. **Both are left alone on purpose** — plausibility
+  is what put a renderer constant on the table in the first place. Expect to add `^/api/` to
+  qBittorrent and `^/api/` + `^/signalr/` to Sonarr once someone opens those UIs and records the
+  split. **This PR does not fix them.**
+- **PiHole is asserted to serve no `/api/` on inherited evidence, not mine.** Its log shows 0
+  unauthenticated requests, which says nothing about what paths it serves. The claim comes from
+  PR #28/#30. Note the pinned `update.range` is `>=2024.0.0`, and PiHole v6 *does* introduce a REST
+  API at `/api/` — so if PiHole is ever upgraded across that line, this needs re-measuring. It is
+  still correct to declare nothing today, because declaring an unmeasured route is the mistake.
+- **`--cookie-refresh=2m30s` has STILL never been exercised** — 0 `AuthSuccess` on all five pods.
+  PR #28 named this as the fix most likely to need a second pass and it remains unobserved. My
+  change does not touch it, but a sign-in during the human test below will exercise both at once,
+  so a failure there must be attributed carefully.
+- **The regexes are not validated against the apps' real route tables.** I assert `/api/v3/` and
+  `/signalr/` are JSON/RPC namespaces a browser never navigates to from Radarr's and Jackett's
+  observed traffic and from how those UIs are built — **not** from reading their routing code or
+  enumerating their endpoints. A path I have not thought of that matches `^/api/` and *is*
+  browser-navigated would 401.
+- **`OPTIONS` preflight and non-browser clients are unconsidered.** `--skip-auth-preflight` exists
+  and is not set; nothing in these logs shows a CORS preflight, and Sonarr/Radarr talk to Jackett
+  **directly on `:9117`**, bypassing the proxy entirely — so I do not expect impact. Unverified.
+- **The live pods do not carry this change and nothing re-measures that they will.** git is not the
+  cluster (§2b). **Merging is what deploys**, and it will roll **jackett** and **radarr** only
+  (`strategy: Recreate` on RWO Longhorn volumes, so a brief gap per app, and radarr/sonarr lose
+  direct `:9117` access to Jackett while its pod restarts). Existing sessions survive — no cookie
+  flag changed — so unlike PR #28 nobody is signed out.
+- **No L2/L3 assertion reads any of this.** There is no runtime check that a `/api/` request to a
+  gated app answers 401, and I did not write one; `CLUSTER_AVAILABLE` is unset so it could not run
+  in CI anyway. The whole guard is static.
+- **`catalog/render.py --selftest` is UNGATED IN MY BASE, and I did not wire it.** Desmond's PR #33
+  does that (`ci/render-golden.sh`) and is in review, not merged — `origin/main` at `cdf3e14`
+  contains no stage that drives it, which I checked rather than assumed (`grep -n selftest
+  ci/render-golden.sh` → no match). **So until #33 lands, my new selftest assertions are
+  dev-time only**, and the *gated* half of my breadth ratchet is the schema `pattern` (via
+  `ci/l0.sh`'s schema stage), the renderer's own `raise` (via `render-golden`), and the new L0
+  check. I deliberately did not wire the selftest myself: #33 already does it, and duplicating it
+  here would collide with his branch in the one file he owns. Note the two PRs are **independent
+  but complementary** — his makes the stage real, mine makes its output readable when it fires.
+- **I did NOT verify PR #33's own claims.** The 106-assertion count, the
+  selftest→stage→`gate.sh`→`BLOCKED` propagation chain, and the `--cookie-refresh 2m30s → 10m`
+  mutation being caught by the selftest but not by `render-golden` are all his measurements,
+  relayed to me by the coordinator. I reproduced only the StopIteration defect itself, because that
+  is the part I was asked to fix. If his branch is reworked, nothing in mine depends on it.
+- **The new check does not read the spec repo** and asserts nothing about `../home-network-k3s`
+  (R-TEST-06, read-only). `schemas/renderer-contract.md` is updated in **this** repo only; the
+  canonical spec `docs/13-service-catalog.md` is **not** updated, because it is immutable from here.
+  Someone with spec write access should mirror `auth.apiRoutes` into docs/13 and docs/15.
+- **THE OAUTH2-PROXY SIDECARS' METRICS ARE STILL UNSCRAPABLE, AND THIS PR DOES NOT FIX IT.** See
+  the declined-bundle section above. #28 shipped `--metrics-address` as a prerequisite and said
+  plainly that it was not scraped; Ines has now measured that the listener **serves** and that the
+  scrape is **dropped by each app namespace's default-deny**. Nothing in my change touches that.
+  Anyone reading this entry as "the fleet gates are observable now" is wrong, and the next PR is
+  where that changes.
+- **Alternation is refused, and that is a judgement I may have to revisit.** No app in this catalog
+  needs it, and the flag is repeatable — but if some future app genuinely needs a regex the ratchet
+  rejects, the fix is to widen the pattern **with a reason** on all three surfaces at once, not to
+  delete the ratchet on one.
+- **CI status: two contexts, and only one of them ran.** Quoted in the PR body and in the report to
+  the PM. `approval-policy` **fails by design** until a non-author approving review exists, because
+  the diff touches `conformance/**`, which `infra/gitea/approval-policy.json` lists as sensitive
+  ("the tests and the harness; changing them changes what green MEANS"). That is CLAUDE.md §1
+  working, not a defect. `required_approvals=1` applies to every PR here regardless of path, so this
+  needs a human either way.
+
+### The human test — and how to tell a too-broad regex from a working one
+
+After merge, wait for **new** pod names at 2/2 in `jackett` and `radarr`.
+
+1. **The breadth test FIRST, before anything else, from a signed-out browser** (private window).
+   Open `https://radarr.home.keithjackson.dev/` and `https://jackett.home.keithjackson.dev/UI/Dashboard`.
+   **Each must bounce through Keycloak and land on the app's UI.** If either shows a bare
+   `401 Unauthorized` instead of a sign-in, **the regex is too broad — stop and revert.** That is
+   the one failure this change can introduce, and it is the only test that can see it. A working
+   regex is invisible on a page load; a broken one is total and immediate.
+2. **Jackett, the original report:** signed in, *+ Add indexer* → pick one → *Configure* → *Okay*.
+   **It must save.** Previously 4 of 6 POSTs never reached Jackett at all.
+3. **Radarr, the live symptom:** open Radarr and leave the tab open. DevTools → Network, filter
+   `signalr`. Today those negotiate calls are **302** and the UI never complains. They must now be
+   **200** while signed in — and if the session is ever lost, **401**, with the UI showing a visible
+   connection error instead of polling silently forever.
+4. **The loud-failure test:** in DevTools, delete the `_oauth2_proxy_radarr` cookie and let the tab
+   poll. The negotiate call must answer **401** and Radarr must *show* a problem. Then **reload** —
+   `/` matches neither regex, so the reload must sign you back in silently. That round trip is the
+   entire fix: fail loudly on the API, recover on the page load.
+5. **The six-minute test (PR #28's open item, still open):** stay on a dashboard for >6 minutes,
+   then save something. That exercises `--cookie-refresh=2m30s` past the 300 s access-token cliff.
+   If it fails, suspect the realm's refresh/SSO-idle timeouts — not this change.
+6. **If something still fails, it must now fail LOUDLY.** A silent success-looking no-op means the
+   change did not take: check the running pod's args for `--api-route` before assuming anything
+   else.

decisions/2026-10-01-render-selftest-runs-in-gate.md

@@ -0,0 +1,205 @@
+---
+date: 2026-10-01
+branch: fix/render-selftest-runs-in-gate
+title: catalog/render.py's 106 assertions were dev-time only; the gate now runs them and blocks on them
+---
+
+## What changed
+
+Single subject, one file: **`ci/render-golden.sh` now invokes `catalog/render.py --selftest`**
+and exits 1 when it fails. Nothing else. No renderer change, no new conformance test, no new
+gate stage.
+
+This closes the top follow-up of `decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md`:
+
+> Nothing in `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the
+> twenty-odd assertions in that selftest, including the six I just added, are dev-time only: a
+> future edit that drops either flag *and* re-renders the goldens would pass the entire merge
+> gate.
+
+Re-measured here and confirmed: before this commit, `grep -rl -- --selftest` over `ci/` and
+`.gitea/` returned **nothing** for the renderer. The selftest executes **106** assertions
+(`check()` invocations counted at runtime via `sys.settrace`, without editing `render.py`; 109
+static call sites, three of them on failure-only branches). All 106 were unexercised by the
+merge gate.
+
+### The lead was WRONG, and here is why
+
+The brief named `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` as the
+one-line fix. That line is real and it does drive selftests:
+
+    for tool in ("record_run.py", "status.py"):
+        r = run([sys.executable, str(LIB / tool), "--selftest"])
+
+But it is **the wrong hook**. That loop is section (d) of a check that serves **R-SEC-03**, and
+its subject is the *evidence chain* — can a self-skipped run, or a stale ledger row, become an
+audit proof? `record_run.py` and `status.py` are **in** that chain; `catalog/render.py` is not.
+Hanging the renderer there would have made the check's name, its `intent` annotation and its
+requirement mapping (`R-SEC-03`) all untrue, and a future decision about the renderer could then
+only be expressed by weakening a *security-auditability* check. That is the precise coupling the
+cookie-collision entry refused for its own check, and it applies here.
+
+The second option that entry offered — a new scope-declaring L0 test — is the supported mechanism
+for a **whole-repo invariant** (`ci/l0.sh` Stage 3b's `--include-scope whole-repo` union, ratcheted
+by `tests/l0-static/whole-repo-invariants-wired`). It is the right shape for an invariant
+quantified over the tracked tree. It is the wrong shape for *"a script in this repo must pass its
+own unit test"*: it would need to declare requirements it does not serve, and it would be two new
+files of several hundred lines by this tree's conventions for what is a six-line invocation.
+
+**The honest home is `ci/render-golden.sh`** — the gate stage whose subject *already is*
+`catalog/render.py`. It already resolves `$PYTHON` and `$RENDER_PY`, already reports rc 3 when
+either is missing, already exits 1 on renderer error, and `ci/gate.sh` already maps it through
+`for s in lint l0 render-golden l1` where a FAIL is BLOCKED and a SKIP is UNDETERMINED. The
+stage's two assertions are independent and both are needed:
+
+- `render-golden` catches golden-vs-renderer drift, but is satisfied by **any** renderer whose
+  output was re-rendered — including one that just lost a contract invariant and had all ten
+  goldens regenerated in the same commit. That is exactly the hole named in the quote above.
+- `--selftest` catches that, and is blind to golden drift.
+
+The file header and the stage name were updated so neither is a lie: the stage is now "the
+renderer is trustworthy", with two numbered assertions, and the step summary emits its own
+`- render-selftest:` line so the gate output still NAMES what ran.
+
+The selftest runs **before** the empty-catalog `exit 3` on purpose: it renders synthetic
+ServiceDefinitions in memory and is catalog-independent, so an empty catalog must not be able to
+skip it.
+
+`catalog/render.py` itself was **not touched** (owned by a concurrent branch this session), and
+neither was any file under `conformance/`.
+
+## What I verified
+
+- **The selftest runs and passes: rc 0, `render.py selftest: PASS`, 106 assertions executed.**
+  Counted at runtime, not from the source: a `sys.settrace` hook counting `check()` calls, in a
+  throwaway script outside the repo, with no edit to `render.py`.
+- **The stage passes with it wired in**, host run of the single stage:
+
+      [gate] Stage: render==golden reproducibility (canonical render vs committed apps/)
+      [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
+      [ ok ] render.py --selftest passed
+      [ ok ] render==golden: all 10 onboarded service golden(s) reproduce from the canonical render
+
+  rc **0**. The golden comparison still reports all 10 goldens, so the new invocation did not
+  displace what the stage already asserted.
+- **IT CAN FAIL — provoked twice, against a mutated COPY of the renderer, never against the
+  tree.** `ci/render-golden.sh` already honours `RENDER_PY`, so each mutant lived in
+  `/tmp/claude-1000/desmond-selftest/` and the repo's `catalog/render.py` was never modified —
+  which matters, because that file belongs to a concurrent branch this session.
+  - **Mutation 1 — the regression this selftest was written for.** Deleted the
+    `f"--cookie-name=_oauth2_proxy_{s['name']}"` emission from `_oauth2_sidecar_container()`,
+    i.e. reverted the fleet-wide cookie-collision fix of PR #28. Stage rc **1**. Note the shape
+    of the failure, because it is a real (minor) weakness and is listed as a follow-up: the
+    selftest died with an uncaught `StopIteration` out of `next(a for a in container["args"] if
+    a.startswith("--cookie-name="))` at `render.py:2641`, rather than printing its collected
+    `SELFTEST FAIL:` lines. It is correctly non-zero and the stage correctly blocks, but the
+    operator gets a traceback instead of the assertion message.
+  - **Mutation 2 — a clean, named assertion failure.** `OAUTH2_COOKIE_REFRESH = "2m30s"` ->
+    `"10m"`, widening the refresh past the measured 300s access-token lifetime. Stage rc **1**:
+
+        [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
+        [fail]   Every golden under apps/ is produced by this renderer, so the goldens below cannot
+        [fail]   be trusted even if they reproduce byte-for-byte. Reproduce with:
+        [fail]     python3 catalog/render.py --selftest
+            SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). A
+            refresh at or past the access-token lifetime never renews in time, which is the same
+            logout it was added to stop
+
+    This is the mutation that proves the wiring's value most directly: mutation 2 would **not**
+    be caught by `render-golden`'s diff if the goldens were regenerated in the same commit, and
+    it is one of the two flags the cookie-collision entry explicitly recorded as **ungated**.
+- **THE NON-ZERO PROPAGATES ALL THE WAY TO `gate.sh`'s rc — provoked, in the pinned toolbox, not
+  reasoned about.** `RENDER_PY=/desmond-selftest/render-mutant2.py ./ci/gate.sh`:
+
+      [ ok ] stage spec-immutability: pass
+      [ ok ] stage decisions-log: pass
+      [ ok ] stage lint: pass
+      [ ok ] stage l0: pass
+      [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
+      [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
+          SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). ...
+      [fail] stage render-golden: FAIL
+      [ ok ] stage l1: pass
+      ...
+      [fail] render-golden: FAILED -> blocks merge
+      [fail] MERGE GATE: BLOCKED
+
+  `gate.sh` rc **1**. Three links in the chain, each measured: selftest rc 1 -> stage rc 1 ->
+  gate rc 1 / `MERGE GATE: BLOCKED`. This mattered enough to provoke rather than assume: this
+  repo has a documented habit of exit codes getting flattened, and
+  `.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]`, which tolerates rc 3 —
+  so a stage that degraded a failure to a skip would be swallowed by CI. rc 1 is not tolerated
+  there, and rc 1 is what this produces.
+- **`spec-immutability` genuinely RAN and passed, it did not skip.** The two-mount worktree shim
+  plus `-e SPEC_REPO=/home-network-k3s` (the fix recorded in the 2026-09-30 cookie entry) works:
+  `stage spec-immutability: pass`, not SKIPPED. R-TEST-06 is therefore actually covered by these
+  runs.
+- **No collision with PR #29.** The brief warned that `feat/oauth2-proxy-auth-failure-visible`
+  had added checks under `conformance/tests/l0-static/`. Measured —
+  `git diff origin/main...origin/feat/oauth2-proxy-auth-failure-visible --name-only` — it touches
+  **no `conformance/` path at all**: `decisions/`, `platform/oauth2-proxy/` (a promtool test, an
+  alert rule, its kustomization), `runbooks/`. Disjoint from this branch either way, since this
+  branch touches only `ci/render-golden.sh` and this file.
+- **Priya's files were not touched.** `catalog/render.py`, `schemas/servicedefinition.schema.json`,
+  `apps/wave-*` and `conformance/tests/l1-rendering/` are all untouched by this branch — confirmed
+  by `git diff --name-only`, which lists exactly two paths.
+- **`catalog/render.py --selftest` was the LAST unwired selftest in the tree.** Measured:
+  `grep -rl -- --selftest` across the repo finds selftest-capable artifacts in exactly four
+  places, and after this commit every one is driven by the gate — `conformance/lib/record_run.py`
+  and `conformance/lib/status.py` by `auditability-proofs-producible/check.sh:291`, the Keycloak
+  client-secret reconciler (embedded in `platform/keycloak/client-secret-reconciler.yaml`) by
+  `keycloak-client-secret-reconciler/check.sh:633`, and `catalog/render.py` by this change.
+
+## What I did NOT verify
+
+- **Local gate: rc 3 (UNDETERMINED), NOT 0. This is not green and must not be reported as
+  green.** Measured in the pinned toolbox on `eeb6211`:
+
+      [ ok ] spec-immutability: green
+      [ ok ] approval-policy: green
+      [ ok ] decisions-log: green
+      [ ok ] lint: green
+      [ ok ] l0: green
+      [ ok ] render-golden: green
+      [ ok ] l1: green
+      [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+      [fail]   Stages that did not run: gitea-settings
+
+  Nothing FAILED. The single cause is `gitea-settings` **SKIPPED** — `SETTINGS_API_TOKEN` is not
+  set in my environment, so branch protection and runner registration were not read; CI has that
+  token and the stage is expected to run there. **L2/L3 `conformance`** also skipped
+  (`CLUSTER_AVAILABLE` unset) but is a *declared* gap that `gate.sh` treats as not-applicable
+  rather than undetermined, so it is not what produced the 3. Touched requirements
+  **R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06** are therefore **UNVERIFIED at L2/L3**, all four.
+  The stage this branch changes, `render-golden`, is green **and** was independently proven able
+  to turn the whole gate red (above).
+- **`approval-policy` passed LOCALLY only because there is no pull_request event**
+  (`not a pull_request event (event='none') — approval policy N/A`). That is not evidence that it
+  passes. This diff touches `ci/**`, a sensitive path, so in CI the stage is expected to **FAIL**
+  until a non-author approving review exists. That is §1 working, fail-closed, not a defect.
+- **The selftest's own quality is unchanged and ungated.** I wired its *invocation*; I did not
+  review its 106 assertions, and nothing ratchets them. Deleting the body of `_selftest()` would
+  still pass this stage, because an empty selftest returns 0. The repo already has the right
+  answer for that — `conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py`
+  injects regressions one at a time and requires `--selftest` to catch every one — and applying
+  that pattern to `catalog/render.py` is the follow-up, deliberately not done here: it is a
+  second subject, it touches `conformance/`, and it would need to live beside the renderer whose
+  file another branch owns this session. **Until it exists, "the selftest runs" is proven and
+  "the selftest is falsifiable" rests on my two hand-run mutations, which are not in the gate.**
+- **Mutation 1's traceback is not fixed.** `render.py:2641`'s `next(...)` has no default, so the
+  absence of `--cookie-name` raises `StopIteration` and the selftest's collected `SELFTEST FAIL:`
+  lines are never printed. Correct verdict, poor diagnostics, and it suppresses the other
+  assertions that mutation also broke. The one-line fix (`next(..., None)` plus a `check()`) is in
+  `catalog/render.py`, which is another branch's file this session, so it is a follow-up and not
+  a thing I touched.
+- **Nothing about the live cluster.** No `kubectl` of any kind was run on this branch — not even
+  read-only; this change is offline CI wiring and needed none. No `apply`/`edit`/`patch`, no
+  `flux reconcile`. Merging this deploys nothing: it changes only what the gate refuses.
+- **CI's own run of this branch.** Quoted in the PR body from the commit's `/statuses`, with
+  descriptions read rather than colours. A *skipped* Gitea job reports `success` with
+  "Has been skipped" and certifies nothing.
+- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so even a fully green PR proves the L0/L1
+  offline subset and nothing about runtime behaviour.
+- **I did not re-measure branch protection, required approvals, or the runner.** Those are
+  CLAUDE.md §2 claims and §2b says re-measure before relying on them; this branch relies on none
+  of them beyond the PR flow itself, and I read the gate's own output rather than the API.

decisions/2026-10-02-plex-quicksync-verification.md

@@ -0,0 +1,149 @@
+---
+date: 2026-10-02
+branch: fix/plex-quicksync-verification
+title: Hardware transcoding works — and `runbooks/plex-quicksync.md`'s verification section was wrong
+---
+
+## What changed
+
+**Hardware transcoding is confirmed working on prod.** Measured 2026-10-02 from
+`Plex Media Server.log` in the running pod: `vaapi` on `/dev/dri/renderD128`
+(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`,
+`zero-copy support present`, `using zero-copy transcoding`, and
+`final decoder: vaapi, final encoder: vaapi`. VAAPI for **both** directions, zero-copy, so frames
+never round-trip through the CPU. The device-plugin mechanism from PR #37 works end to end.
+
+**And `runbooks/plex-quicksync.md`'s verification section was wrong, which is the correction this
+branch exists to publish.** The page said the Plex dashboard showing `Transcode (hw)` was "the *only*
+proof available, and nothing else will ever tell you". The operator had hardware transcoding fully
+working while **Dashboard → Activity showed nothing of the sort** — so the page's authoritative check
+was silent on a working system, and sent someone looking for a fault that did not exist. Three
+inference errors are now corrected on the page, two of them recorded as named false leads so the next
+reader does not re-walk them:
+
+1. **The dashboard is not proof.** Replaced as the primary check by Plex's own log — the grep is in
+   the page, in its **redacting** form, because `Plex Media Server.log` carries API URLs containing
+   `X-Plex-Token` (CLAUDE.md §6; a credential has been leaked into a transcript here before). The
+   page also now says explicitly that **`Plex Transcoder Statistics.log` is the wrong file** — it
+   holds XML `<Variant .../>` records, not ffmpeg output, and the name invites the mistake — and why
+   `grep -a` is required. The dashboard is **kept but demoted** to a secondary, best-effort signal,
+   with its absence stated to prove nothing.
+2. **An fd snapshot is not evidence.** A `/proc/*/fd` sweep inside the pod found no holder of
+   `renderD128` while hardware transcoding was active; transcode sessions are short and the scan
+   landed in a gap. Recorded as a documented false lead, with the reliable process-level check in its
+   place: the transcoder's **argv** (`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128`), found via
+   `ps -eo pid,user,comm | grep -i transcoder`.
+3. **A missing preference key does not mean the setting is off.** The page told readers to treat an
+   absent `HardwareAcceleratedCodecs` in `Preferences.xml` as "never enabled". With hardware
+   acceleration demonstrably on and working, that key was **absent from all 27 attributes** while
+   `HardwareDevicePath` **was** written and the log said `hardware transcoding: enabled`. First-command
+   #4 now reads both keys and states plainly that `HardwareDevicePath` present is a positive signal,
+   absence of `HardwareAcceleratedCodecs` means nothing either way, only `="0"` is conclusive, and the
+   log settles it.
+
+Also on the page:
+
+- **GID `992` is no longer presented as a number to expect.** Under the plugin the injected device
+  arrives `root:root`, so the image's `cont-init.d` hook resolves GID **`0`** and adds `plex` to the
+  **root group** — measured `Groups: 0 100 1000`, `id plex` → `groups=1000(plex),0(root),100(users)`.
+  992 was the hostPath-era host `render` GID. The page now says so, marks the root-group membership as
+  an inherent property of the plugin's `DeviceSpec` rather than a defect, and repeats that it is
+  **not** a reason to reintroduce `supplementalGroups` (still a no-op: `s6-setuidgid` rebuilds the
+  group list from `/etc/group`). The stale `Groups: 100 1000` claim in the escalation section is gone.
+- **A new troubleshooting entry for legitimate per-title fallbacks**, marked as a **hypothesis**.
+  Between two fully hardware sessions, one logged `enabled, but no hardware decode accelerator found`
+  and empty final decoder/encoder. UHD 630 decodes H.264, HEVC 8/10-bit and VP9 but **not AV1** and
+  not 10-bit 4:2:2 HEVC, so the likely cause is the source file, not the platform — explicitly
+  written as a lead to check first, **not** a confirmed diagnosis of that item.
+- The status is stated at the top of the page and in a rewritten provenance footer that separates the
+  2026-10-02 confirmations from the superseded 2026-09-30/2026-10-01 hostPath-era readings.
+- `runbooks/README.md`'s index entry for the page was stale in the same two ways (it still described a
+  `/dev/dri` hostPath and the dashboard-only proof) and is corrected to match.
+
+Unchanged on purpose: the "STOP HERE IF PLEX IS `Pending`" section and the escape hatch (delete the
+`hardware:` block from `catalog/plex.yaml`, re-render) keep exactly their existing prominence — they
+are correct and they are the 2am material. The "no alert is possible" reasoning is also kept verbatim
+in substance: Plex declares no `metricsEndpoint`, renders no ServiceMonitor, and a fallback alert
+would be a rule evaluating against nothing.
+
+**Documentation only.** No change to `catalog/`, `schemas/`, `apps/` or `platform/`.
+
+## What I verified
+
+- **Gate, in the pinned toolbox from this worktree** (`DEV_KUBECONFIG=/dev/null`, `ENGINE` shim
+  bind-mounting the main checkout's `.git`): **rc 3 UNDETERMINED**, diff base
+  `origin/main @ 948875aad37d`, with exactly the three expected skips and **no failures**. Green:
+  `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy`. Skipped:
+  `spec-immutability` (spec repo unavailable → R-TEST-06 not verified), `gitea-settings`
+  (`SETTINGS_API_TOKEN` unset), `conformance` (`CLUSTER_AVAILABLE!=true` → L2/L3 not run).
+  Touched requirements: **R-OBS-04**, reported `UNVERIFIED` for lack of a cluster. `approval-policy`
+  passed as `N/A` locally (`event='none'`, not a `pull_request`) — it evaluates for real on the PR.
+  **rc 3 is not green**, it is the calibrated worktree result, and it is reported here as rc 3.
+- **The gate was run three times, with the same rc 3 / three-skip result on each committed tip** —
+  including the final one. Worth recording from the first attempt: run *before* committing, with
+  `HEAD == origin/main`, the touched-requirement set is `NOT DETERMINABLE (on-base)` and the
+  R-TEST-03 coverage gate cannot be applied at all. Commit before gating from a fresh worktree branch.
+- **`runbooks/kustomization.yaml` still carries `- plex-quicksync=plex-quicksync.md`** in the
+  `configMapGenerator` `files:` list (line 110) — checked explicitly, because without it the page
+  merges and still 404s. `README=README.md` is likewise present, which is what serves the corrected
+  index entry.
+- **This change touches no `sensitive_paths`.** `infra/gitea/approval-policy.json` lists `ci/**`,
+  `.gitea/**`, `conformance/**`, `clusters/**`, `bootstrap/**`, `infra/gitea/**`,
+  `platform/kyverno/**`, `platform-config/kustomization.yaml`, `.sops.yaml`, `**/*.sops.yaml`,
+  `**/rbac.yaml`, `CLAUDE.md`. `runbooks/**` and `decisions/**` are not among them, and the
+  `approval-policy` stage passed.
+- **No YAML was added or edited**, so `indent-sequences: consistent` (which blocked a sibling branch
+  today) is not in play here; `lint` passed over the full tracked YAML set regardless.
+- **`DECISIONS-LOG.md` is untouched** — this handoff is its own file, per CLAUDE.md §7.
+- **The PR's own CI run was read, not assumed** (PR #39, run 208, task 261, head `1677dff5688b`).
+  **It is RED, and not because of this branch.** Every stage this change could affect is green —
+  `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` (`approval-policy` evaluated
+  for real on the `pull_request` event and passed: no sensitive path touched). The single failure is
+  **`gitea-settings`**, quoting the measured field:
+
+      [fail]   required_approvals: DECLARED 1 but LIVE is 0
+      [fail] Gitea has DRIFTED from infra/gitea/expected-branch-protection.json.
+      [fail] MERGE GATE: BLOCKED
+
+  Every other declared setting matched (`block_admin_merge_override`, `block_on_rejected_reviews`,
+  `dismiss_stale_approvals`, `enable_status_check`, `enable_push=false`,
+  `status_check_contexts = ["gate / gate (pull_request)"]`, `runners online: 1`). So **live `main`
+  currently requires ZERO approvals**, the declaration still says 1, and with
+  `REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml` that drift **blocks every PR in
+  the repo**, not just this one. Note the shape of it: the check that would notice approvals are
+  unenforced is the check that is failing. Local branch `fix/admin-approval-override` (`616a497`,
+  "fix(ci): let an administrator merge their own green PR") looks like the in-flight work that means
+  to change the declaration. **I did not touch it** — `infra/gitea/**` is a sensitive path, needs a
+  non-author approval, and is outside this branch's documentation-only scope. It is an operator
+  decision: re-assert the protection, or change the declaration deliberately in its own PR.
+- The repository-wide grep for `992` to find what else still asserts it: remaining occurrences are in
+  `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md`,
+  `platform/intel-gpu-plugin/README.md`, `platform/intel-gpu-plugin/kustomization.yaml`, and three
+  older `decisions/` entries.
+
+## What I did NOT verify
+
+- **I did not re-measure anything on the live cluster in this session.** Every platform number on the
+  page — the VAAPI/zero-copy log lines, `RW-OPEN-OK` as uid 1000, the `gpu.intel.com/i915` limits and
+  requests on the pod spec, `Groups: 0 100 1000`, `id plex`, the 27 `Preferences.xml` attributes, the
+  empty `/proc/*/fd` sweep — came to me as measurements taken earlier on 2026-10-02 and was
+  transcribed, not re-run. Per CLAUDE.md §2b, treat the page as reporting those readings rather than
+  as proof of current state; re-run the primary check before relying on it.
+- **The per-title fallback cause is unverified and labelled as such.** The source codec of the title
+  that logged `no hardware decode accelerator found` was never read. The AV1 / 10-bit-4:2:2-HEVC
+  explanation is inferred from UHD 630's capability set alone.
+- **Whether Plex's dashboard *ever* shows `Transcode (hw)` on this deployment is unknown.** It was
+  observed silent during working hardware transcoding; whether that is a client/version quirk, a
+  reporting bug, or something about this setup was not investigated. The page therefore demotes it
+  rather than explaining it.
+- **The commands as written in the page were not executed from the page.** They are the forms that
+  were run, re-wrapped for `dev.sh`; in particular the `exec -it deploy/plex -- sh` route for the argv
+  check is noted as possibly lacking a TTY under `dev.sh run` and was not exercised in that shape.
+- **Stale `992` copies outside `runbooks/` were left alone**, deliberately — scope was documentation
+  only, and `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md` and
+  `platform/intel-gpu-plugin/` all still carry the hostPath-era `Groups: 44 100 992 1000` reading as
+  if current. `render.py --selftest` asserts around that comment, so correcting it is a real change
+  with gate consequences, not a doc edit. **That is open work.** Older `decisions/` entries also cite
+  992; those are historical records and should stay as written.
+- **CI on the PR.** The branch is pushed and the PR opened; I did not watch the Gitea run report, so
+  nothing here should be read as "CI green". The only gate result I can quote is the local rc 3 above.

infra/gitea/README.md

@@ -52,7 +52,7 @@ All four steps are complete and each was verified, not assumed:
 | 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
 | 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
 | 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
-| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
+| 4. Collaborator + approvals | ✅ `jakman85` is owner. `required_approvals` was raised to 1 here and **lowered back to 0 on 2026-10-01** — see §6, it was a lock-out |
 
 Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
 `gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
@@ -124,6 +124,66 @@ the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS
 **Changing the declaration is how you change policy.** If the live state is what you now want,
 edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
 
+### 6. `allow-admin-self-merge.sh` — undoing the lock-out step 4 created (2026-10-01)
+
+Step 4 above raised `required_approvals` to 1. That was a **lock-out, not a protection**, and the
+note at the bottom of `require-gate.sh` had already said why: *Gitea will not let an author approve
+their own PR.* With `jakman85` the only collaborator, there was no account in existence that could
+approve a PR `jakman85` opened.
+
+Measured 2026-10-01 — **PR #35**: `gate / gate (pull_request)` **success in 1m41s**, reviews `[]`,
+`required_approvals: 1`, `block_admin_merge_override: true`. Unmergeable, permanently, with no
+action available to its author. The same was true of any PR fixing it.
+
+It was also never the documented design. `ci/approval-policy.sh`'s header and `ci/gate.sh:194`
+both already describe the model as *"`required_approvals` drops to 0 and the GATE decides"*, and
+`decisions/2026-09-30-flux-deploy-notification-inhibited.md` flagged the 1-vs-0 disagreement as
+unresolved and not that session's to settle. So this is a correction toward the stated intent, not
+a weakening away from it.
+
+    ./infra/gitea/allow-admin-self-merge.sh      # required_approvals -> 0 (read from the declaration)
+
+**What stays switched on, and why that is the point.** The other way to unblock an owner is
+`block_admin_merge_override: false`, and it would have been the wrong one — that re-opens exactly
+what §5 above closed, letting the one account that merges bypass the gate itself. This script
+touches the approval **count** only and *verifies* afterwards that `block_admin_merge_override` and
+`enable_status_check` are both still `true`, failing loudly if the PATCH disturbed either:
+
+| | before | after |
+|---|---|---|
+| `required_approvals` | 1 | **0** |
+| `block_admin_merge_override` | true | true (asserted) |
+| `enable_status_check` | true | true (asserted) |
+
+So a **red gate is still unmergeable by everyone, administrators included**. What changed is that
+a *green* one no longer needs a signature that cannot exist.
+
+**Security-relevant changes are still reviewed** — by `ci/approval-policy.sh` against
+`approval-policy.json` `sensitive_paths`, not by this field. An administrator waives that review
+only by posting `/approve-override <reason>` on the PR: reason mandatory, permission re-checked
+live, invalidated by a new commit, and recorded by account in the gate summary.
+
+> **Post the override as a PR *review* (type "Comment"), not in the plain comment box** — until
+> the CI token is re-minted. `SETTINGS_API_TOKEN` is `[read:repository, read:admin]`, and
+> `/issues/{n}/comments` needs Gitea's *issue* scope, so Gitea answers with an error **object**
+> instead of an array and the override is invisible. Measured on PR #38 (2026-10-02): a correct,
+> fresh, well-formed override in the comment box produced `approval-policy: FAIL` reading as "no
+> approving review" — a permission failure disguised as a missing one. `/pulls/{n}/reviews` is
+> under `read:repository` and worked in the same run. The stage now reads **both** sources, names
+> which one it used, and says plainly when the comments API is unreadable. `set-ci-token.sh`
+> requests `read:issue` as of 2026-10-02, so **re-run it** and the comment box works too — token
+> scopes cannot be widened in place. Note that the
+**agent account `gitea-admin` also reports permission `owner`**, so it can clear its own PRs that
+way; that is an accepted consequence of authorising by permission alone, and the route to
+narrowing it (an `override_admins` allowlist, already honoured by the code) is written down in
+`approval-policy.json`.
+
+**Sequencing.** `ci/gitea-settings.sh` compares live against the declaration *from the branch under
+test*, and `REQUIRE_GITEA_SETTINGS=true`. So merge any other open PRs **before** running this, then
+run it, then re-run the gate on the branch carrying the new declaration. Any PR left open across
+the change declares 1 against a live 0 and goes **red until rebased** — the detector working, not
+a fault. The script's header spells the order out.
+
 ## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential
 
 Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
@@ -204,6 +264,8 @@ it. The name matters — see the note under the behaviour table above.)
 2. **Watch the gate actually report on a real PR**, and note the context name it produced.
 3. **Only then set `enable_status_check: true`** with that exact context.
 4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
+   — **Step 4 was REVERSED on 2026-10-01 (§6).** It assumed a reviewer distinct from the author
+   would exist. One never did, so this step produced a lock-out. Do not re-apply it.
 
 > **Do not do step 3 before step 1.** A required status check with no runner leaves every
 > PR permanently pending — including the PR that would fix it. That half is measured: PR #1

infra/gitea/allow-admin-self-merge.sh

@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+# Let the sole administrator merge their own green PR: required_approvals 1 -> 0 on `main`.
+#
+# WHY. `required_approvals: 1` plus Gitea's refusal to let an author approve their own pull
+# request plus exactly ONE collaborator (`jakman85`, permission `owner`) is a lock-out, not a
+# policy. Every PR the operator opened was permanently unmergeable, including the PR that would
+# fix it. Measured 2026-10-01: PR #35 — `gate / gate (pull_request)` success in 1m41s, zero
+# reviews, `required_approvals: 1`, no account in existence able to satisfy it.
+#
+# The approvals field was also never what the design said it was. `ci/approval-policy.sh`'s
+# header and `ci/gate.sh:194` both already described the model as "required_approvals: 0 and the
+# GATE decides", and `require-gate.sh`'s own closing note named this exact failure and this exact
+# remedy. decisions/2026-09-30-flux-deploy-notification-inhibited.md flagged the 1-vs-0
+# disagreement as unresolved. This resolves it at 0.
+#
+# WHAT THIS DOES *NOT* DO, and the distinction is the whole point.
+#
+#   * It does NOT touch `block_admin_merge_override`, which stays TRUE. The required status check
+#     still binds ABSOLUTELY — a red gate is unmergeable by everyone, administrators included.
+#     Setting `block_admin_merge_override: false` would have been the other way to unblock an
+#     owner, and it would have re-opened precisely what block-admin-override.sh was written to
+#     close: PRs #5-#9 merged with zero approvals while the settings read as enforced.
+#   * It does NOT stop reviewing security-relevant changes. Those are gated by
+#     ci/approval-policy.sh against infra/gitea/approval-policy.json `sensitive_paths`, which an
+#     administrator can waive only with an explicit, attributed `/approve-override <reason>`
+#     comment on the PR.
+#
+# ORDER MATTERS — READ THIS BEFORE RUNNING IT.
+#
+# ci/gitea-settings.sh fails the merge gate on ANY divergence between the live rule and
+# infra/gitea/expected-branch-protection.json, and `gate.yml` sets REQUIRE_GITEA_SETTINGS=true.
+# The declaration in git now says 0. So between running this and merging that declaration, the
+# live value and the declared value must be compared from a branch that CARRIES the new
+# declaration — otherwise the stage goes red, correctly, for drift.
+#
+#   1. MERGE any other open PRs FIRST, while approvals still work for them (their branches
+#      declare 1, and a PR you did not author can still be approved normally). Skipping this
+#      only costs a rebase later; it does not break anything.
+#   2. Run this script.
+#   3. Re-run the gate on the branch that carries the new declaration. `gitea-settings` goes
+#      green because live 0 == declared 0.
+#   4. Merge that branch.
+#   5. Any PR still open from before step 2 now declares 1 against a live 0 and will be RED
+#      until it is rebased onto the new main. That is the detector doing its job, not a fault.
+#
+# Recoverable: `required_approvals` can be PATCHed back to 1 by an owner at any time. If you do,
+# change expected-branch-protection.json in the same breath or every PR goes red on drift.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+DECL="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/expected-branch-protection.json"
+
+command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; }
+[ -f "$DECL" ] || { echo "declaration missing: ${DECL}" >&2; exit 1; }
+
+# Take the target from the DECLARATION, not from a literal here. If the two could disagree,
+# this script would be one more thing re-asserting a value nothing checks — which is the
+# failure mode the whole of infra/gitea/ exists to stop.
+WANT="$(jq -r '.protection.required_approvals' "$DECL")"
+[[ "$WANT" =~ ^[0-9]+$ ]] || { echo "declaration has no numeric required_approvals" >&2; exit 1; }
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
+echo "authenticating as '${U}' (secret not shown)"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
+
+# Read the RULE OBJECT, never /branches/main: the branch view does not expose
+# block_admin_merge_override and showed all-green for hours while the rule was overridable.
+echo "==> before"
+api "$B" | jq -r '{required_approvals, block_admin_merge_override, enable_status_check, status_check_contexts}'
+
+echo "==> setting required_approvals = ${WANT} (declared in expected-branch-protection.json)"
+api -X PATCH -H 'Content-Type: application/json' \
+  -d "{\"required_approvals\": ${WANT}}" "$B" >/dev/null
+
+echo "==> after"
+AFTER_JSON="$(api "$B")"
+jq -r '{required_approvals, block_admin_merge_override, enable_status_check, status_check_contexts}' <<<"$AFTER_JSON"
+
+GOT="$(jq -r '.required_approvals' <<<"$AFTER_JSON")"
+ADMIN_BLOCK="$(jq -r '.block_admin_merge_override' <<<"$AFTER_JSON")"
+STATUS="$(jq -r '.enable_status_check' <<<"$AFTER_JSON")"
+
+fail=0
+[ "$GOT" = "$WANT" ] || { echo "PATCH did not take: required_approvals is ${GOT}, wanted ${WANT}" >&2; fail=1; }
+# These two are what keeps this change narrow. If the PATCH disturbed either, the gate has
+# stopped binding and that must be loud, not discovered later.
+[ "$ADMIN_BLOCK" = true ] || { echo "block_admin_merge_override is ${ADMIN_BLOCK}, MUST be true — the gate is now bypassable" >&2; fail=1; }
+[ "$STATUS" = true ]      || { echo "enable_status_check is ${STATUS}, MUST be true — the gate is no longer required" >&2; fail=1; }
+[ "$fail" -eq 0 ] || { echo "do NOT record this as applied." >&2; exit 1; }
+
+cat <<'NOTE'
+
+NOW PROVE IT -- the setting is not the evidence. Two things to see, not assume:
+
+  1. An APPROVAL IS NO LONGER NEEDED. On your own PR with a green gate, the merge button is
+     live with zero reviews on it.
+  2. THE GATE STILL BINDS. On a PR whose gate is RED or still pending, Gitea must REFUSE the
+     merge -- for you, as owner. If it lets you through, block_admin_merge_override is not
+     doing its job and this change has traded a lock-out for a bypass.
+
+Then confirm the detector agrees: run ci/gate.sh from a branch carrying the new
+expected-branch-protection.json and check that `gitea-settings` is green rather than DRIFT.
+NOTE

infra/gitea/approval-policy.json

@@ -1,6 +1,6 @@
 {
   "_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
-  "_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
+  "_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author, OR an admin override (see .override). Everything else merges on a green gate alone.",
   "_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
   "sensitive_paths": [
     "ci/**",
@@ -24,5 +24,18 @@
     "platform/kyverno/**": "the admission policy engine",
     ".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
     "CLAUDE.md": "the delivery contract agents follow"
+  },
+  "override": {
+    "_why": "This repository has ONE human. Gitea refuses to let an author approve their own pull request, so 'a non-author APPROVED review' is a requirement the sole administrator can never satisfy -- every sensitive-path PR they open was permanently unmergeable, including the PR that would fix it. Measured 2026-10-01: PR #35 green gate, zero reviews, unmergeable. An escape hatch that the administrator can actually reach is therefore REQUIRED for the repository to function; the design question is only whether it leaves a record. This one does.",
+    "enabled": true,
+    "comment_directive": "/approve-override",
+    "requires_reason": true,
+    "authorized_permissions": ["admin", "owner"],
+    "invalidated_by_new_commits": true,
+    "_how": "Post the directive at the START of a line, followed by a reason: '/approve-override <why>'. ci/approval-policy.sh resolves the poster's LIVE repo permission via /collaborators/{login}/permission and accepts the override only if it is in authorized_permissions. The overrider, the source, the timestamp and the reason are printed in the gate log and the job summary.",
+    "_where_to_post_it": "EITHER a PR REVIEW body (type 'Comment') OR the ordinary PR comment box. Prefer the REVIEW: measured 2026-10-02 on PR #38, a well-formed override in the comment box was INVISIBLE to CI, because /issues/{n}/comments sits behind Gitea's *issue* token scope and SETTINGS_API_TOKEN is minted [read:repository, read:admin]. Gitea returned an error object rather than an array, and the stage reported 'no approving review' -- a permission problem wearing the costume of a missing override. /pulls/{n}/reviews is under read:repository and was readable in the same run, so a review body is the route that works on today's token. infra/gitea/set-ci-token.sh now also requests read:issue, which makes the comment box work -- but only after that script is RE-RUN, since an existing token's scopes cannot be widened in place. The stage accepts both sources and names which one it used, so neither route is load-bearing alone.",
+    "_self_override_is_allowed": "The author may override their own PR if they hold an authorized permission. That is the entire point -- with one human there is no one else -- but it means the 'a change to the gate cannot be gated by the gate' protection is, for an administrator, advisory rather than structural. It is traded for an auditable record: the waiver is an explicit act, attributed, with a stated reason, in the PR's comment history and the gate output.",
+    "_known_consequence_agent_accounts": "ACCEPTED, EYES OPEN (operator decision, 2026-10-01). Authorisation is by live permission ALONE, with no allowlist of human accounts. The agent account `gitea-admin` reports permission `owner` on this repo (measured 2026-10-01), so an AGENT can clear the sensitive-path review on its own PR by posting the directive. Agents author most PRs here, so this is the broad case, not the edge case. To narrow it later without changing any logic, add an `override_admins` allowlist here and require membership in addition to the permission check -- ci/approval-policy.sh reads both from this file.",
+    "_staleness": "An override is dropped unless its comment is NEWER than the PR head commit, mirroring Gitea's own dismiss_stale_approvals. An override earned at commit A therefore does not silently cover commit B; push again and you must override again."
   }
 }

infra/gitea/expected-branch-protection.json

@@ -1,6 +1,9 @@
 {
   "_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
   "_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
+  "_amended": "2026-10-01: required_approvals 1 -> 0. See _why_required_approvals_is_zero.",
+  "_why_required_approvals_is_zero": "The GATE is the approver (operator intent, 2026-09-29). required_approvals: 1 was never consistent with that intent: ci/approval-policy.sh's header and ci/gate.sh:194 both already described the model as 'required_approvals: 0 and the GATE decides', and infra/gitea/require-gate.sh's own closing note named the consequence of 1 -- 'Gitea will not let an author approve their own PR'. With exactly ONE collaborator (jakman85, owner), that made every PR the operator opened permanently unmergeable. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, no path to a merge. decisions/2026-09-30-flux-deploy-notification-inhibited.md had already flagged the 1-vs-0 disagreement as unresolved; this settles it at 0.",
+  "_what_still_holds_at_zero": "This drops the APPROVAL COUNT only. block_admin_merge_override stays TRUE and enable_status_check stays TRUE, so the required gate context still binds absolutely -- nobody, administrator included, can merge a red gate. That distinction is the whole reason this is the chosen fix rather than block_admin_merge_override: false, which would have handed admins a bypass over the gate itself and re-opened what infra/gitea/block-admin-override.sh was written to close (PRs #5-#9, merged with zero approvals). Architecturally and security-relevant paths are still reviewed, by ci/approval-policy.sh rather than by this field.",
   "branch": "main",
   "runners_online_min": 1,
   "protection": {
@@ -11,7 +14,7 @@
     "status_check_contexts": [
       "gate / gate (pull_request)"
     ],
-    "required_approvals": 1,
+    "required_approvals": 0,
     "enable_approvals_whitelist": false,
     "block_admin_merge_override": true,
     "block_on_rejected_reviews": true,

infra/gitea/require-gate.sh

@@ -34,7 +34,7 @@ jq -n '{
   enable_push: false,
   enable_status_check: true,
   status_check_contexts: ["gate / gate (pull_request)"],
-  required_approvals: 1,
+  required_approvals: 0,
   block_on_rejected_reviews: true,
   dismiss_stale_approvals: true,
   block_on_outdated_branch: false,
@@ -48,11 +48,21 @@ api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
 
 cat <<'NOTE'
 
-NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
-  * Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
-  * PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
-    agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
-    and rely on the status check alone.
+NOTE ON required_approvals: 0 -- CHANGED FROM 1 ON 2026-10-01, and this script is the reason
+the change had to land here too: re-running it with 1 would silently re-impose the lock-out.
+
+The note that used to sit here called the problem and its own fix: "Gitea will not let an
+author approve their own PR ... if that is awkward ... set required_approvals back to 0 and
+rely on the status check alone." It was awkward. With jakman85 the ONLY collaborator, every PR
+the operator opened was unmergeable -- measured on PR #35, gate green in 1m41s, zero reviews,
+nothing that could ever satisfy the rule. So: 0, and the status check alone, as written.
+
+What did NOT change, deliberately:
+  * block_admin_merge_override stays TRUE. This drops the approval COUNT; it does not hand
+    anyone a bypass over the gate. A red gate is still unmergeable by everybody.
+  * Architecturally/security-relevant paths are still human-reviewed -- by ci/approval-policy.sh
+    (sensitive_paths in infra/gitea/approval-policy.json), which an administrator can waive
+    only by posting an explicit, attributed `/approve-override <reason>` comment.
 
 Then prove it, rather than trusting the API:
   git commit --allow-empty -m probe && git push origin main   # EXPECT: rejected

infra/gitea/set-ci-token.sh

@@ -5,9 +5,23 @@
 # The token value is NEVER printed. It goes from the token API into the secrets API and
 # nowhere else -- not to stdout, not to a file, not into an agent transcript.
 #
-# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
-# runner list). Gitea token scopes are coarse; grant the least that works on your version and
-# confirm the check goes green rather than assuming the scope name.
+# Scope: `read:repository` + `read:admin` are what ci/gitea-settings.sh needs (the rule object,
+# and the runner list). `read:issue` is for ci/approval-policy.sh, and it was MISSING — see below.
+# Gitea token scopes are coarse; grant the least that works on your version and confirm the check
+# goes green rather than assuming the scope name.
+#
+# WHY read:issue IS HERE (added 2026-10-02, measured on PR #38). ci/approval-policy.sh reads the
+# PR's comments to find an admin `/approve-override`. /issues/{n}/comments sits behind the *issue*
+# scope, which this script did not request — so Gitea answered with an error OBJECT instead of an
+# array, and a perfectly well-formed override posted in the PR comment box was invisible to CI.
+# The failure did not look like a permission problem from the outside; it looked like the override
+# had not been posted. `/pulls/{n}/files` and `/pulls/{n}/reviews` are under read:repository, which
+# is why every other part of the same stage worked and this one did not.
+#
+# The stage now accepts an override from a REVIEW body as well, so it still functions on a token
+# WITHOUT read:issue. This scope is what makes the ordinary comment box work too — the thing
+# anybody would reach for first. **Re-run this script to pick it up; an existing token's scopes
+# cannot be widened in place.**
 set -euo pipefail
 HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
 # NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
@@ -21,7 +35,7 @@ basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
 
 echo "==> minting token '${NAME}' as ${U}"
 TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
-  -d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
+  -d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\",\"read:issue\"]}" \
   "https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
 [ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
 echo "==> got a ${#TOKEN}-character token (value not shown)"

platform/oauth2-proxy/README.md

@@ -88,3 +88,72 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized).
 4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other
    protected UIs).
+
+## Observability: what this component can and cannot see (measured 2026-09-30)
+
+`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
+two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
+in the TSDB is:
+
+    count by (__name__) ({__name__=~"oauth2_proxy.*"})
+      oauth2_proxy_requests_in_flight => 2
+
+**The per-app gate sidecars are not scraped, and there is nothing to scrape.** Six apps run an
+`oauth2-proxy` container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr —
+each with `--http-address=0.0.0.0:4180` and **no `--metrics-address`**, so oauth2-proxy's metrics
+listener never starts. They declare only a `http:4180` container port, no `metrics` port exists on
+their Services, and `kubectl get servicemonitors,podmonitors -A` selects none of them. Any alert
+written against `oauth2_proxy_requests_total{namespace="jackett"}` would be structurally inert.
+
+This matters because the gate's characteristic failure is invisible by construction: an
+unauthenticated **XHR** gets a `302` to Keycloak rather than a `401`, the browser follows it
+against a live SSO session with no prompt, and the call resolves **200 carrying an HTML page**
+where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett
+spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds
+into a 5-minute session, while `jackett-down` sat at 1 replica available the whole time.
+
+`auth-visibility-alerts.yaml` therefore alerts on **the gap**, not on the incident:
+`OAuth2ProxyGateUnobserved` compares `kube_pod_container_info{container="oauth2-proxy"}` against
+`up{container="oauth2-proxy"}` — two already-scraped families — and names every namespace running
+a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each
+one gains a scrape. `runbooks/oauth2-proxy-auth-failures.md` is the by-hand substitute meanwhile,
+and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).
+
+**To close the gap for an app**, three things are needed together, on that app's own manifest:
+`--metrics-address=0.0.0.0:44180` on the sidecar, a `metrics` containerPort plus the matching
+Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained
+`code="302"` as a share of `oauth2_proxy_requests_total`, held across a window longer than the
+session lifetime — become writable rather than decorative.
+
+### Update 2026-10-01: the listener exists; the scrape path does not
+
+PR #28 added `--metrics-address=0.0.0.0:44180` and a named `metrics` container port to the five
+fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). `podmonitor.yaml` in this directory
+discovers them by the pod label `homelab.keithjackson.dev/auth-mode: forward-auth` — a **PodMonitor**
+rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the
+renderer.
+
+The endpoint is real: port-forwarded straight to the jackett pod, `/metrics` returns **http=200,
+10208 bytes**, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total` (with
+`code` labels) and `oauth2_proxy_response_duration_seconds`. That is verified by payload, not by the
+flag appearing in `args` — a distinction worth keeping, because the two claims came apart here.
+
+**The scrape is nevertheless dropped.** From a pod in ns `monitoring`, both `:44180` and `:4180` on
+the jackett pod time out; the shared proxy's `:44180` answers. The only difference is that ns
+`oauth2-proxy` has **no NetworkPolicy at all**, while each app namespace has a `default-deny` whose
+`policyTypes` include `Ingress` with no `ingress` rules (deny-all) plus a single CNP admitting
+`fromEntities: [ingress]` to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no
+clause.
+
+So the targets read `up == 0` and the TSDB is still empty for those namespaces. This is the same bug
+and the same fingerprint as `platform/longhorn/networkpolicy-metrics.yaml` documents for Longhorn,
+and the same narrow fix applies: ns `monitoring` → the gate pod label → TCP 44180 only. It cannot
+ship from this directory, because `oauth2-proxy` and `apps-wave-0` are **siblings** in the Flux graph
+(both `dependsOn: [platform-config]`), so a NetworkPolicy naming ns `jackett` would apply on today's
+cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.
+
+Consequence for `auth-visibility-alerts.yaml`: its `unless` clause requires `up == 1`, not merely a
+present `up` series. A discovered-but-failing target publishes `up` with value 0, so the bare form
+would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
+as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
+fails it and nothing else.

platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml

@@ -0,0 +1,177 @@
+---
+# promtool unit tests for OAuth2ProxyGateUnobserved (platform/oauth2-proxy/auth-visibility-alerts.yaml).
+#
+# WHY. The rule is an `unless` between two aggregations over DIFFERENT metric families, and the
+# failure modes of that shape are both silent: a label that does not survive `count by` makes the
+# right-hand side match nothing (the alert fires for the scraped namespace too, i.e. permanently),
+# and a right-hand side that over-matches makes it fire for nobody (i.e. inert). Neither is
+# visible by reading the expression, and neither is visible on a live cluster where the answer
+# happens to look plausible. So all four states are pinned: fires for an unscraped namespace,
+# stays quiet for a scraped one, honours `for: 1h`, and STOPS once a scrape appears.
+#
+# HOW TO RUN. The one repo-wide harness, by the companion convention:
+#   ./infra/devcontainer/dev.sh run bash -c 'platform/monitoring/tests/run.sh platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml'
+# promtool is pinned in infra/devcontainer/versions.env; never run it off the host PATH.
+#
+# THE SERIES BELOW ARE REAL SHAPES read off live prod Prometheus on 2026-09-30 before being
+# copied here — kube_pod_container_info as kube-state-metrics emits it (job="kube-state-metrics",
+# its own instance/pod labels, image/image_id/container_id) and `up` as the oauth2-proxy
+# ServiceMonitor emits it (endpoint="metrics", port 44180). The point of using the real shapes is
+# that `count by (namespace)` has to discard all of them; a test with tidy two-label series would
+# pass while the real expression collapsed differently.
+#
+# ASSERTIONS USE `ALERTS` RATHER THAN exp_alerts: the description annotation is long prose and
+# comparing it verbatim would couple this file to every wording change while proving nothing the
+# ALERTS series does not already prove.
+rule_files:
+  - auth-visibility-alerts.rules.yaml
+evaluation_interval: 1m
+
+tests:
+  # ---------------------------------------------------------------------------------------------
+  # 1. THE STATE PROD IS ACTUALLY IN (2026-09-30): jackett runs a gate sidecar with no scrape;
+  #    the shared oauth2-proxy namespace runs two that ARE scraped. Asserts the alert fires for
+  #    jackett and NOT for oauth2-proxy, and that `for: 1h` is honoured (pending at 59m).
+  - interval: 1m
+    name: an unscraped gate namespace fires after 1h; the scraped one never does
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # The shared instance: kube-state-metrics sees both replicas...
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://aa11bb22",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000001"}'
+        values: '1x130'
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://cc33dd44",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="kube-prometheus-stack-kube-state-metrics",uid="1111aaaa-0000-0000-0000-000000000002"}'
+        values: '1x130'
+      # ...and both replicas are genuinely scraped, so `up` carries container="oauth2-proxy" there.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.189:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-cqz27",service="oauth2-proxy"}'
+        values: '1x130'
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.2.47:44180",job="oauth2-proxy",namespace="oauth2-proxy",pod="oauth2-proxy-7568c78795-bscxg",service="oauth2-proxy"}'
+        values: '1x130'
+    promql_expr_test:
+      # Not yet firing at 59m — `for: 1h` has not elapsed. Guards against someone "simplifying"
+      # the hold away and turning every rollout into a page.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
+        eval_time: 59m
+        exp_samples: []
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett"}'
+        eval_time: 59m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="pending",namespace="jackett",severity="warning"}'
+            value: 1
+      # Firing for jackett past the hold...
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      # ...and never for the namespace that IS scraped. This is the assertion that catches an
+      # `unless` whose right-hand side stopped matching: without it, a broken rule that fires for
+      # every gate namespace would pass the test above.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="oauth2-proxy"}'
+        eval_time: 65m
+        exp_samples: []
+      # The count must be 1 per namespace-with-a-gate, not 1 overall: a `count by (namespace)`
+      # that lost its `by` would collapse to a single unlabelled series.
+      - expr: 'count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by (namespace) (up{container="oauth2-proxy"})'
+        eval_time: 65m
+        exp_samples:
+          - labels: '{namespace="jackett"}'
+            value: 1
+
+  # ---------------------------------------------------------------------------------------------
+  # 2. THE FIX LANDING. jackett's sidecar gains --metrics-address and a ServiceMonitor 70m in, so
+  #    `up` starts carrying container="oauth2-proxy" for namespace="jackett" under a DIFFERENT job
+  #    name than the shared proxy's. The alert must stop. This is what makes the rule a gap
+  #    tracker rather than a permanent nag, and it pins that the match is on `container`, not on
+  #    a hardcoded job="oauth2-proxy" (which would keep firing forever after a correct fix).
+  - interval: 1m
+    name: the alert clears once the sidecar is actually scraped
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # Absent for the first 70 minutes, then present. `_x` is "no sample"; the scrape did not
+      # exist yet.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.213:44180",job="jackett-auth",namespace="jackett",pod="jackett-c5bfcbfb8-c75f9",service="jackett-auth-metrics"}'
+        values: '_x70 1x60'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="jackett"}'
+        eval_time: 80m
+        exp_samples: []
+
+  # ---------------------------------------------------------------------------------------------
+  # 3. THE REGRESSION GUARD FOR `== 1` — the scenario that exposed a real bug on 2026-10-01.
+  #
+  #    A target is DISCOVERED but every scrape FAILS: `up` exists with value 0. This is not
+  #    hypothetical — it is the exact state podmonitor.yaml creates today, because the app
+  #    namespaces' default-deny drops the scrape (measured; see podmonitor.yaml).
+  #
+  #    With the original `unless count by (namespace) (up{container="oauth2-proxy"})`, the `up`
+  #    SERIES EXISTS, so `count by (namespace)` returns 1, `unless` removes the namespace, and the
+  #    alert GOES QUIET while nothing is being scraped at all — a green signal standing in for an
+  #    absent one, which is the precise disease this whole rule was written to cure. Deleting the
+  #    `== 1` makes THIS test fail and every other test in this file still pass, which is the
+  #    property that makes it worth having.
+  - interval: 1m
+    name: a discovered-but-FAILING target does not count as observed (up==0 must still fire)
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9",service="kube-prometheus-stack-kube-state-metrics",uid="5b57d48f-e7d3-4050-9cd1-16aaaa052996"}'
+        values: '1x130'
+      # The PodMonitor target exists and never succeeds: NetworkPolicy drops the packets, so the
+      # scrape times out and `up` is published as 0 rather than going absent.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.0.189:44180",job="oauth2-proxy-sidecars",namespace="jackett",pod="jackett-6877fcd9f5-ht9d9"}'
+        values: '0x130'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett"}'
+        eval_time: 65m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="jackett",severity="warning"}'
+            value: 1
+      # And the inner clause itself yields NOTHING while the scrape is failing — which is the
+      # mechanism the alert depends on, asserted directly so a future reader can see why the
+      # `== 1` matters without re-deriving it. (Scenario 4 below covers the other direction:
+      # a succeeding scrape really does clear the alert, so `== 1` has not made the rule
+      # impossible to satisfy.)
+      - expr: 'count by (namespace) (up{container="oauth2-proxy"} == 1)'
+        eval_time: 65m
+        exp_samples: []
+
+  # ---------------------------------------------------------------------------------------------
+  # 4. A FLAPPING target: up goes 1 -> 0 and stays there. The alert must come BACK, not latch
+  #    cleared. Covers the ordering bug where a once-observed namespace is permanently excluded.
+  - interval: 1m
+    name: a target that starts working and then fails re-fires
+    input_series:
+      - series: 'kube_pod_container_info{container="oauth2-proxy",container_id="containerd://937ae6fc",endpoint="http",image="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",image_id="quay.io/oauth2-proxy/oauth2-proxy@sha256:d62e2d81",image_spec="quay.io/oauth2-proxy/oauth2-proxy:v7.8.1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="radarr",pod="radarr-7d55cbffb9-42hmh",service="kube-prometheus-stack-kube-state-metrics",uid="3333cccc-0000-0000-0000-000000000001"}'
+        values: '1x250'
+      # Healthy for the first 60m, then every scrape fails for the rest.
+      - series: 'up{container="oauth2-proxy",endpoint="metrics",instance="10.0.1.77:44180",job="oauth2-proxy-sidecars",namespace="radarr",pod="radarr-7d55cbffb9-42hmh"}'
+        values: '1x60 0x180'
+    promql_expr_test:
+      # Quiet while it was working.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",namespace="radarr"}'
+        eval_time: 55m
+        exp_samples: []
+      # Firing again an hour after it broke.
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr"}'
+        eval_time: 125m
+        exp_samples:
+          - labels: 'ALERTS{alertname="OAuth2ProxyGateUnobserved",alertstate="firing",namespace="radarr",severity="warning"}'
+            value: 1
+
+  # ---------------------------------------------------------------------------------------------
+  # 5. NO GATE, NO ALERT. A namespace with no oauth2-proxy container must never appear, or the
+  #    rule would page for every app on the platform the day someone widened the matcher.
+  - interval: 1m
+    name: namespaces without a gate sidecar are never reported
+    input_series:
+      - series: 'kube_pod_container_info{container="plex",container_id="containerd://feed0001",endpoint="http",image="ghcr.io/example/plex:1",image_id="ghcr.io/example/plex@sha256:feed0001",image_spec="ghcr.io/example/plex:1",instance="10.0.1.213:8080",job="kube-state-metrics",namespace="plex",pod="plex-0",service="kube-prometheus-stack-kube-state-metrics",uid="2222bbbb-0000-0000-0000-000000000001"}'
+        values: '1x130'
+    promql_expr_test:
+      - expr: 'ALERTS{alertname="OAuth2ProxyGateUnobserved"}'
+        eval_time: 65m
+        exp_samples: []

platform/oauth2-proxy/auth-visibility-alerts.yaml

@@ -0,0 +1,137 @@
+---
+# oauth2-proxy AUTH-VISIBILITY alerting (R-OBS-04).
+#
+# ============================================================================================
+# WHY THIS FILE EXISTS — THE 2026-09-30 JACKETT FAILURE THAT NOTHING OBSERVED
+# ============================================================================================
+# Jackett's UI stopped working. Reloading never fixed it and the user was never re-prompted to
+# log in. The sidecar's own access log had the whole story:
+#
+#   [oauthproxy.go:1026] No valid authentication in request. Initiating login.
+#   GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
+#   GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
+#   ... ten of these at 15:30:53, then a fresh [AuthSuccess] at 15:31:06,
+#   then "No valid authentication in request" again 26 SECONDS LATER, inside the same session.
+#
+# NOTHING ANYWHERE ELSE RECORDED IT. oauth2-proxy answers an unauthenticated XHR with a 302 to
+# Keycloak rather than a 401; the browser follows it silently, the live SSO session satisfies it
+# without a prompt, and the XHR resolves 200 carrying a 44 KB HTML dashboard where the app's
+# JavaScript wanted JSON. From the browser, from the user, and from any uptime check, the service
+# is healthy. `jackett-down` (apps/wave-0/jackett.yaml) watches
+# kube_deployment_status_replicas_available, which was 1 throughout and would have been 1 if the
+# gate had rejected every request ever made.
+#
+# ============================================================================================
+# WHY THIS FILE DOES NOT CONTAIN THE OBVIOUS ALERT — MEASURED, prod, 2026-09-30
+# ============================================================================================
+# The alert you would want is "a gated app is serving a sustained rate of auth redirects to a
+# user who is already authenticated". It is NOT WRITABLE against this cluster's TSDB, and the
+# reason is this file's real subject:
+#
+#   THE PER-APP GATE SIDECARS EXPORT NO METRICS AT ALL.
+#
+#   $ promtool query instant :9090 'count by (__name__) ({__name__=~"oauth2_proxy.*"})'
+#     oauth2_proxy_requests_in_flight => 2          <- the SHARED proxy's two replicas. That is ALL.
+#
+#   PARTLY SUPERSEDED 2026-10-01, and the correction matters more than the original. PR #28 gave
+#   the five FLEET sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics` port, and the
+#   endpoint genuinely serves (verified by payload: http=200, 10208 bytes, all three
+#   `oauth2_proxy_*` families present — not merely by the flag being in `args`). podmonitor.yaml
+#   in this directory now discovers them. But the scrape is still DROPPED by the app namespaces'
+#   default-deny ingress, so the TSDB remains empty for those namespaces; and crafty still has no
+#   metrics listener at all. The rule below is unchanged in purpose and still has work to do.
+#
+#   Every sidecar (crafty, jackett, pihole, qbittorrent, radarr, sonarr) ran with
+#   `--http-address=0.0.0.0:4180` and NO `--metrics-address`, so oauth2-proxy's metrics listener
+#   was never started; the container declared only port `http:4180`, there was no `metrics` port on
+#   any of their Services, and there was no ServiceMonitor or PodMonitor for any of them anywhere
+#   in the cluster (measured: `kubectl get servicemonitors,podmonitors -A`). Six gates, zero
+#   telemetry. An alert written against `oauth2_proxy_requests_total{namespace="jackett"}` would
+#   have been PERMANENTLY INERT — present, evaluating, silent, and indistinguishable from health.
+#   This repo has shipped that mistake before (VeleroNoRecentBackup, the Longhorn rules), which is
+#   why `conformance/tests/l0-static/alert-metrics-have-scrape-targets` exists, and why the
+#   temptation was refused here rather than ratcheted into a register.
+#
+#   The SHARED proxy IS scraped (job="oauth2-proxy", 2 targets, up=1) but it is not in the data
+#   path for any sidecar-gated app — each app gates itself — and its current pods export no
+#   `oauth2_proxy_requests_total` at all, having proxied nothing since they started. An alert on
+#   the shared instance would not have fired on this incident either. Saying so is the point.
+#
+# So this file alerts on the CONDITION THAT MADE THE INCIDENT INVISIBLE, not on the incident.
+# That condition is measurable today, from two already-scraped metric families, and it is the
+# precondition for ever being able to alert on the incident itself. It self-clears the moment a
+# sidecar gains a metrics endpoint and a scrape — at which point the auth-churn rule becomes
+# writable and belongs here beside it. See runbooks/oauth2-proxy-auth-failures.md, which is the
+# by-hand equivalent an operator uses in the meantime.
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: oauth2-proxy-auth-visibility
+  namespace: oauth2-proxy
+  labels:
+    release: kube-prometheus-stack     # kube-prometheus-stack's ruleSelector
+    app.kubernetes.io/name: oauth2-proxy
+spec:
+  groups:
+    - name: oauth2-proxy.auth-visibility
+      rules:
+        - alert: OAuth2ProxyGateUnobserved
+          # A namespace runs an oauth2-proxy gate container, and Prometheus holds no scrape
+          # target for that container in that namespace.
+          #
+          # BOTH SIDES ARE ALREADY-SCRAPED FAMILIES, deliberately:
+          #   * kube_pod_container_info  — kube-state-metrics, the declared owner in
+          #     alert-metrics-have-scrape-targets. It sees the container whether or not the
+          #     container exports anything, which is exactly the property needed here.
+          #   * up                       — UNIVERSAL. `up` carries the `container` label for any
+          #     ServiceMonitor/PodMonitor target, so this works for a future per-app scrape
+          #     without knowing what job name it will pick.
+          # Nothing in this expression touches `oauth2_proxy_*`, so the rule cannot itself be
+          # inert for the reason it is reporting.
+          #
+          # MEASURED ON PROD 2026-09-30 — this expression returned six namespaces:
+          #   {namespace="crafty"} {namespace="jackett"} {namespace="pihole"}
+          #   {namespace="qbittorrent"} {namespace="radarr"} {namespace="sonarr"}
+          # and excluded namespace="oauth2-proxy", whose two replicas ARE scraped. It fires on
+          # the state the cluster is in; it is not a rule waiting for a hypothetical.
+          #
+          # `== 1` IS LOAD-BEARING, AND IT IS A BUG FIX (2026-10-01). This clause was originally
+          # a bare `up{container="oauth2-proxy"}`, which was WRONG in the one way that matters
+          # here: a target that is DISCOVERED BUT FAILING still publishes `up`, with value 0. So
+          # the series exists, `count by (namespace)` returns 1, `unless` removes the namespace,
+          # and the alert GOES QUIET while nothing whatsoever is being scraped. That is this
+          # branch's own disease — a green signal standing in for an absent one — reproduced
+          # inside the alert written to cure it. It was not hypothetical: podmonitor.yaml lands
+          # five targets whose scrape is DROPPED by NetworkPolicy (measured; see that file), so
+          # the bare form would have silently cleared on the very next commit. `== 1` means
+          # "observed" requires a scrape that actually succeeds.
+          #
+          # A failing target is therefore reported by BOTH this rule and kube-prometheus-stack's
+          # generic TargetDown. That duplication is deliberate: TargetDown says "a target is
+          # down", this says "an authentication gate is unobserved", and during an incident those
+          # are different sentences. The overlap is the cheap half of the trade.
+          expr: |
+            count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
+              unless
+            count by (namespace) (up{container="oauth2-proxy"} == 1)
+          # 1h, not 5m: a rollout briefly has a running sidecar whose target has not been
+          # discovered yet, and this condition is a standing architectural gap, never an
+          # emergency. Long enough that no deploy can produce it.
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "oauth2-proxy gate in {{ $labels.namespace }} exports no metrics"
+            description: >-
+              The {{ $labels.namespace }} namespace runs an oauth2-proxy gate container and
+              Prometheus has NO scrape target for it, so every authentication decision that gate
+              makes — every redirect to Keycloak, every 403 on a group check, every session that
+              silently fails to hold — is invisible to metrics and to alerting. This is not a
+              theoretical gap: on 2026-09-30 the jackett gate answered a burst of the app's own
+              XHR calls with 302s to Keycloak, re-established a session, and lost it again 26
+              seconds later, and the only record anywhere was a line in the sidecar's stdout.
+              No alert could have fired, because there is nothing to alert on. Until a scrape
+              exists the runbook's LogQL queries are the substitute; the fix is
+              `--metrics-address=0.0.0.0:44180` on the sidecar plus a `metrics` port and a
+              ServiceMonitor, after which the auth-churn rule becomes writable.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/oauth2-proxy-auth-failures"

platform/oauth2-proxy/kustomization.yaml

@@ -14,3 +14,12 @@ resources:
   - oauth2-proxy-credentials.sops.yaml
   - httproute.yaml       # forward-auth sign-in/callback ingress (was referenced in-place by platform-config)
   - servicemonitor.yaml  # Prometheus scrape (was referenced in-place by platform-config)
+  # Alerting on whether the gates are OBSERVABLE at all. Travels with the servicemonitor above
+  # for the same reason it does: monitoring.coreos.com CRDs are registered by the `platform`
+  # tier, which is Ready before this tier's platform-config dependency completes.
+  - auth-visibility-alerts.yaml
+  # Discovers the PER-APP gate sidecars' :44180 metrics endpoints by pod label (PR #28 added the
+  # listener; no Service exposes it, so this is a PodMonitor, not a ServiceMonitor). The scrape is
+  # still denied by the app namespaces' default-deny ingress — read podmonitor.yaml's header
+  # before concluding the targets should be up.
+  - podmonitor.yaml

platform/oauth2-proxy/podmonitor.yaml

@@ -0,0 +1,109 @@
+---
+# PodMonitor for the PER-APP oauth2-proxy gate sidecars (R-OBS-04).
+#
+# ============================================================================================
+# WHY A PodMonitor AND NOT A ServiceMonitor
+# ============================================================================================
+# PR #28 gave all five fleet sidecars `--metrics-address=0.0.0.0:44180` and a named `metrics`
+# container port. It did NOT add a Service port, and measured on prod 2026-10-01 none of the five
+# Services exposes 44180:
+#
+#   jackett      jackett:     [('port-9117', 9117), ('authproxy', 4180)]
+#   radarr       radarr:      [('port-7878', 7878), ('authproxy', 4180)]
+#   sonarr       sonarr:      [('port-8989', 8989), ('authproxy', 4180)]
+#   qbittorrent  qbittorrent: [('port-8080', 8080), ('authproxy', 4180)]
+#   pihole       pihole:      [('port-80',     80), ('authproxy', 4180)]
+#
+# A ServiceMonitor selects Service PORTS, so it would have nothing to select. A PodMonitor targets
+# pod ports directly and bypasses Services entirely — so this needs no change to the Services, and
+# therefore no change to `catalog/render.py` or the `apps/wave-*` renders and goldens.
+#
+# SELECTOR: the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL, not a namespace list.
+# Measured 2026-10-01: that label is present on exactly the five fleet gate pods (jackett, radarr,
+# sonarr, qbittorrent, pihole) and absent from crafty's, which is a separately-managed deployment
+# (platform/crafty) that still has no `--metrics-address` at all. So the selector describes the
+# PROPERTY being scraped rather than enumerating today's apps, and a sixth forward-auth app is
+# picked up with no edit here. `namespaceSelector.any: true` because those five namespaces are
+# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
+#
+# ============================================================================================
+# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
+# ============================================================================================
+# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
+# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
+#
+#   THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
+#     GET /metrics -> http=200 bytes=10208
+#     # TYPE oauth2_proxy_requests_in_flight gauge
+#     # TYPE oauth2_proxy_requests_total counter
+#     # TYPE oauth2_proxy_response_duration_seconds histogram
+#     oauth2_proxy_requests_total{code="200"} 3
+#     oauth2_proxy_requests_total{code="302"} 2
+#
+#   THE SCRAPE PATH IS DROPPED — from a pod in ns monitoring (grafana, which sits beside
+#   Prometheus and so occupies the same policy position) to the jackett pod IP:
+#     http://10.0.0.189:44180/metrics   -> wget: download timed out
+#     http://10.0.0.189:4180/ping       -> wget: download timed out     (even the "allowed" port)
+#     http://<shared-proxy>:44180/metrics -> 200, oauth2_proxy_* present (control: ns oauth2-proxy
+#                                            has NO NetworkPolicy, which is the only reason the
+#                                            shared instance has ever been scrapable)
+#
+#   WHY: ns jackett's `default-deny` NetworkPolicy is `podSelector: {}` with
+#   `policyTypes: [Ingress, Egress]` and an `egress` rule ONLY — no `ingress` key, which in
+#   NetworkPolicy semantics is deny-all-ingress. The only ingress allowance is the CNP
+#   `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
+#   and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
+#
+# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
+# measured, documented state — NOT a regression introduced here, and NOT something to silence.
+# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
+# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
+# (see that file's header — this exact trap is why the `== 1` is there).
+#
+# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
+# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
+# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
+# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
+# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
+#
+# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
+# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
+#   oauth2-proxy  dependsOn=['platform-config']        <- this tier
+#   apps-wave-0   dependsOn=['platform-config']        <- SIBLING, no ordering between them
+#   apps-wave-1   dependsOn=['apps-wave-0']
+# They are siblings, so on a cold bootstrap there is no guarantee ns jackett exists when this tier
+# applies, and a NetworkPolicy naming a missing namespace fails to apply. It would work on today's
+# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
+# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
+# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
+# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
+# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: oauth2-proxy-sidecars
+  namespace: oauth2-proxy
+  labels:
+    release: kube-prometheus-stack     # kube-prometheus-stack's podMonitorSelector
+    app.kubernetes.io/name: oauth2-proxy
+    homelab.keithjackson.dev/purpose: metrics-scrape
+spec:
+  namespaceSelector:
+    # The gate sidecars live in the app namespaces, not this one. `any` rather than a fixed list so
+    # a new forward-auth app needs no edit here — the pod label below is what scopes this.
+    any: true
+  selector:
+    matchLabels:
+      # The PROPERTY, not the app list. Present on exactly the five fleet gate pods (measured
+      # 2026-10-01); absent from crafty, which has no metrics listener yet.
+      homelab.keithjackson.dev/auth-mode: forward-auth
+  podMetricsEndpoints:
+    # By PORT NAME (`metrics`, 44180), never by number: the name is what PR #28 declared on the
+    # container, and a numeric port here would silently stop matching if the port were renumbered.
+    - port: metrics
+      path: /metrics
+      interval: 30s
+      # 10s, matching the drop signature above: a DENIED path does not refuse, it hangs until the
+      # scrape times out. Leaving the default would make each failing target hold a scrape slot for
+      # the full interval.
+      scrapeTimeout: 10s

runbooks/README.md

@@ -51,6 +51,36 @@ rule groups still evaluate against nothing and **cannot fire**:
 | Flux (`gotk_*`) | `FluxReconciliationFailure`, `FluxResourceSuspended`, `ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed` |
 | Hubble (`hubble_drop_total`) | `HubbleDeniedFlowSpike` |
 | etcd (`kubeEtcd` disabled for k3s, deliberately) | no etcd fsync alerting exists at all |
+| per-app oauth2-proxy sidecars (`oauth2_proxy_*`) — updated 2026-10-01 | no authentication alerting exists at all for crafty, jackett, pihole, qbittorrent, radarr, sonarr |
+
+The oauth2-proxy row is the newest, and it has **moved twice in two days** — worth reading in order,
+because each step looked like the end of it.
+
+**2026-09-30 — nothing to scrape.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})` returned
+exactly one family, `oauth2_proxy_requests_in_flight`, from the two replicas of the **shared**
+forward-auth instance. All six per-app gate sidecars ran without `--metrics-address`, so their metrics
+listener never started, and no ServiceMonitor or PodMonitor selected them.
+
+**2026-10-01 — the listener exists, and the scrape is still blocked.** PR #28 added
+`--metrics-address=0.0.0.0:44180` and a named `metrics` port to the five fleet sidecars, and the
+endpoint genuinely serves (port-forwarded: `http=200`, 10208 bytes, all three `oauth2_proxy_*`
+families, `oauth2_proxy_requests_total{code="302"}` present). `platform/oauth2-proxy/podmonitor.yaml`
+now discovers them by pod label. **But the app namespaces' `default-deny` NetworkPolicy drops the
+scrape**: from a pod in ns `monitoring`, both `:44180` and `:4180` on the jackett pod time out, while
+the shared proxy's `:44180` answers — the only difference being that ns `oauth2-proxy` has no
+NetworkPolicy at all. So the targets are discovered and `up == 0`, and the TSDB is still empty for
+those namespaces. **This is the same bug Longhorn had**, with the same fingerprint; the fix idiom is
+`platform/longhorn/networkpolicy-metrics.yaml` (ns monitoring → one pod label → one port), and for
+these apps it has to be rendered alongside each app's own policy rather than shipped from the
+oauth2-proxy tier, which is a *sibling* of `apps-wave-0` in the Flux graph and so cannot rely on those
+namespaces existing yet.
+
+`OAuth2ProxyGateUnobserved` (`platform/oauth2-proxy/auth-visibility-alerts.yaml`) alerts on the gap
+itself from `kube_pod_container_info` and `up` — two families that *are* scraped. Note its `up == 1`:
+a discovered-but-failing target publishes `up` with value **0**, so the earlier bare form would have
+gone quiet the moment the PodMonitor landed, while nothing was being scraped. That near-miss is the
+whole reason to distrust a cleared alert here. Crafty remains outside all of this — no metrics
+listener at all. See `oauth2-proxy-auth-failures`.
 
 Each affected runbook says so on its own page and gives the by-hand equivalent. See `targets-down`.
 
@@ -108,6 +138,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-db`** — Keycloak's CloudNativePG database is unavailable.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
   every green signal missed.
+- **`oauth2-proxy-auth-failures`** — a gated app "is up but does not work". Linked from
+  `OAuth2ProxyGateUnobserved`. The gate answers an unauthenticated **XHR** with a 302 to Keycloak
+  rather than a 401, the browser follows it silently against a live SSO session, and the call
+  resolves **200 carrying HTML** where the app's JavaScript wanted JSON — so the user is never
+  re-prompted and every uptime check stays green. Tells the gate's refusal from the group check
+  from the app's own login, from the sidecar log and from Loki, with the 2026-09-30 jackett trace
+  as the worked example. **Read the caveat**: the per-app gate sidecars export no metrics at all,
+  so these logs (72h of them) are the only record this class of failure leaves.
 
 ### Operator procedures (no alert links here)
 - **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
@@ -115,14 +153,18 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
   plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
   token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
   Includes why a changed Secret does not restart anything, and why an expired token is harmless.
-- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
-  (`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
-  acceleration when available"**, and no environment variable can do it (the image supports no
-  `PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
-  read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
-  **only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
-  alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
-  fallback, and why a `:latest` pull can silently break it.
+- **`plex-quicksync`** — hardware transcoding on the host iGPU. **Confirmed working 2026-10-02**
+  (VAAPI decode *and* encode, zero-copy). GitOps requests `gpu.intel.com/i915` on the pod
+  (`hardware.gpu: intel-quicksync`, satisfied by the `intel-gpu-plugin` DaemonSet — a `/dev/dri`
+  hostPath was measured not to work: device-cgroup `EPERM`); **a human still has to tick Settings →
+  Transcoder → "Use hardware acceleration when available"**, and no environment variable can do it
+  (the image supports no `PLEX_PREFERENCE*`, verified). **The proof is `Plex Media Server.log`'s
+  `final decoder: / final encoder:` line, not the Plex dashboard** — the dashboard showed nothing
+  while hardware transcoding was working, so its silence proves nothing; the page carries the
+  redacting grep (`X-Plex-Token` lives in that log), the WRONG log file to avoid, and two more
+  documented false leads (an empty `/proc/*/fd` scan, and a missing `HardwareAcceleratedCodecs` key).
+  Plex renders no ServiceMonitor, so no alert could ever fire. Also: **`Pending` stalls the whole app
+  tier** and its one-field escape hatch, and why a `:latest` pull can silently break it.
 - **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
   re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
   (the four lines that matter, including the upstream `-` that means the proxy answered a request the

runbooks/kustomization.yaml

@@ -85,10 +85,17 @@ configMapGenerator:
       - node-io=node-io.md
       - node-memory=node-memory.md
       - node-memory-swap=node-memory-swap.md
+      # Linked from OAuth2ProxyGateUnobserved, and the diagnostic procedure for any gated app
+      # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
+      # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
+      - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
       # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
       # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
       # presents as random session loss in ONE app's log and is only legible with both side by side,
       # so the page has to be reachable at 3am without going looking for it in git.
+      # SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
+      # one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
+      # TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
       - oauth2-proxy-session=oauth2-proxy-session.md
       # Operator procedure, not an alert page (no alert links to it — the
       # alert-runbooks-resolvable check treats that as advisory). It is served

runbooks/oauth2-proxy-auth-failures.md

@@ -0,0 +1,244 @@
+# oauth2-proxy auth failures — telling the three layers apart
+
+Linked from `OAuth2ProxyGateUnobserved`, and the page to open whenever a gated app "is up but
+does not work": a UI that loads and then shows nothing, a page that never finishes, a list that
+stays empty, a save that silently does not save. The alert is about the *visibility* gap; this
+page is the diagnostic procedure, and it works whether or not the alert ever fired.
+
+## What this means here
+
+Every app behind Keycloak on this platform has **three** places a request can be refused, and
+from a browser they look identical — because two of the three answer with **HTTP 200**.
+
+| Layer | What refuses | What the browser sees |
+|---|---|---|
+| 1. The gate (oauth2-proxy sidecar) | no valid session cookie | **302 to Keycloak**, then usually **200** |
+| 2. The gate's group check | session valid, wrong group | 403, or a redirect loop |
+| 3. The app itself | the app's own login / API key | the app's own 401 or login page |
+
+**Layer 1 is the one that hides.** oauth2-proxy answers an unauthenticated request with a
+redirect, not a 401 — correct for a browser navigation, disastrous for an **XHR**. The app's
+JavaScript asks for JSON, the gate answers `302`, the browser follows it silently, the live
+Keycloak SSO session satisfies it with no prompt, and the XHR resolves **200 with an HTML
+dashboard in the body**. `fetch()` sees a success. The user sees an empty page. Reloading does
+nothing, and nobody is ever asked to log in, because from the browser's point of view nothing
+failed.
+
+The worked example, jackett on 2026-09-30, straight from the sidecar log:
+
+    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
+    GET "/api/v2.0/indexers/yts/config?_=..."          HTTP/1.1 ... 302 400
+    GET "/api/v2.0/indexers/thepiratebay/config?_=..." HTTP/1.1 ... 302 409
+    ... ten of these at 15:30:53
+    [AuthSuccess] Authenticated via OAuth2: Session{email:... created:2026-09-30 15:31:06
+                  expires:2026-09-30 15:36:06 refresh_token:true groups:[platform-admins]}
+    GET "/UI/Dashboard"        200 44379
+    GET "/api/v2.0/indexers?_=..." 200 1501229
+    ... and 26 seconds later, inside that same 5-minute session:
+    [oauthproxy.go:1026] No valid authentication in request. Initiating login.
+
+A session created at 15:31:06, valid until 15:36:06, not honoured at 15:31:32. That is the
+signature of the session being **re-established over and over instead of held** — a cookie that
+another proxy is overwriting, or a cookie that never comes back at all — not of Keycloak being
+down and not of the app being broken.
+
+**Read this before you trust any silence (updated 2026-10-01).** The five fleet gate sidecars now
+*do* run a metrics listener on `:44180`, and `platform/oauth2-proxy/podmonitor.yaml` discovers
+them — but **the scrape is dropped by the app namespaces' `default-deny` NetworkPolicy**, so those
+targets read `up == 0` and the TSDB holds nothing for them. Crafty has no listener at all. Only
+the shared `oauth2-proxy` namespace instance is genuinely scraped, and it is not in the data path
+for any of these apps. So there is still nothing to graph and nothing that could have alerted.
+**The logs below remain the only record that this class of failure happened.** Check which state
+you are in before you trust a quiet dashboard — "First commands" step 1 tells you in one query.
+
+## First commands
+
+`kubectl` does not exist on the workstation; everything runs in the pinned container, and always
+with an explicit context.
+
+**1. Which namespaces have an unobserved gate** (this is what the alert measures):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
+     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
+     "count by (namespace) (kube_pod_container_info{container=\"oauth2-proxy\"}) unless count by (namespace) (up{container=\"oauth2-proxy\"} == 1)"'
+```
+
+**1b. If a namespace is listed, is it "no target" or "target failing"?** The two need different
+fixes and the alert deliberately does not distinguish them:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
+     -c prometheus -- /bin/promtool query instant http://localhost:9090 \
+     "up{container=\"oauth2-proxy\"}"'
+```
+
+No series for the namespace → nothing discovers it (no PodMonitor match; check the pod carries
+`homelab.keithjackson.dev/auth-mode=forward-auth` and a named `metrics` port). A series with
+value **0** → discovered but every scrape fails; get the reason from the target's `lastError`:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
+   curl -s localhost:19090/api/v1/targets | python3 -c "
+import json,sys
+for t in json.load(sys.stdin)[\"data\"][\"activeTargets\"]:
+    if t[\"labels\"].get(\"container\")==\"oauth2-proxy\":
+        print(t[\"labels\"].get(\"namespace\"), t[\"health\"], t.get(\"lastError\",\"\")[:90])"'
+```
+
+`context deadline exceeded` is a **drop**, not a refusal — that is NetworkPolicy, not a dead
+endpoint. Confirm the endpoint itself is fine by port-forwarding straight to the pod, which
+bypasses policy entirely:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n jackett port-forward \
+     $(kubectl --context k3s-prod -n jackett get pod -l app.kubernetes.io/name=jackett -o name | head -1) \
+     14418:44180 & sleep 5; curl -s -o /dev/null -w "http=%{http_code} bytes=%{size_download}\n" \
+     http://127.0.0.1:14418/metrics'
+```
+
+`http=200` with a few KB means the listener is healthy and the path is what is broken. As of
+2026-10-01 that is the state of all five fleet apps, and the fix is an ingress allowance from ns
+`monitoring` to port 44180 — see `platform/longhorn/networkpolicy-metrics.yaml` for the idiom.
+
+**2. The sidecar log — but you must grep it.** The gate's stdout is ~11,500 lines a day per app
+and almost all of it is the kubelet health probe (`GET "/ping" ... 200`). A plain
+`kubectl logs --tail=100` shows you nothing but `/ping` and looks perfectly healthy. Filter:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n jackett logs deploy/jackett -c oauth2-proxy --since=2h \
+     | grep -v "/ping" | grep -E "No valid authentication|AuthSuccess|403|401|callback"'
+```
+
+Substitute the namespace: `crafty`, `jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`.
+
+**3. The same question across every gate at once, from Loki.** Alloy collects all of these, so
+you can ask about the whole fleet and about a window that has already scrolled out of
+`kubectl logs`. Port-forward Loki and query it:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring port-forward svc/loki 13100:3100 & sleep 5; \
+   curl -s -G --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"No valid authentication in request\" [24h]))" \
+     http://127.0.0.1:13100/loki/api/v1/query | python3 -m json.tool | grep -E "namespace|value" '
+```
+
+and the denominator, which is what makes the number mean something:
+
+```
+   ... --data-urlencode "query=sum by (namespace) (count_over_time({container=\"oauth2-proxy\"} |= \"AuthSuccess\" [24h]))" ...
+```
+
+**How to read the ratio.** A healthy sign-in is roughly **one or two** `No valid authentication`
+lines per `AuthSuccess` — the first request of a session is unauthenticated by definition. A
+ratio in the tens means sessions are not surviving. Measured on prod 2026-09-30 (24h):
+
+| namespace | `No valid authentication` | `AuthSuccess` | ratio |
+|---|---|---|---|
+| crafty | 81 | 1 | 81:1 |
+| radarr | 67 | 1 | 67:1 |
+| jackett | 39 | 4 | ~10:1 |
+| pihole, qbittorrent, sonarr | 0 | 0 | (unused in the window) |
+
+All three of those are the failure, not the baseline.
+
+**The cleanest worked case of all — radarr, measured 2026-10-01.** After the cookie fix landed,
+jackett dropped to **0** `No valid authentication` lines in 13,612 (from 39), and `removing
+session` / `cookie signature not valid` went to **0 on both apps** (radarr had 8 and 8 before).
+The eviction fingerprint is gone. But radarr still logged **206** rejections — and sampling 209 of
+them, **every single one was the same request**:
+
+    GET "/signalr/messages/negotiate" ... 302    <- browser UA, over and over
+
+That is one left-open browser tab. Its cookie is gone, its SignalR socket keeps trying to
+reconnect, and every attempt is silently 302'd to Keycloak. Nobody is at the keyboard, so nobody
+is ever re-prompted; the tab just sits there looking like a working page that has stopped
+updating. **This is the entire failure class in one line**: a non-navigation request answered with
+a redirect, forever, with no error surfaced anywhere.
+
+So when you read a rejection count, **always group by request path before concluding anything**:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n radarr logs deploy/radarr -c oauth2-proxy --since=24h \
+     | grep -v "/ping" | grep "No valid authentication" -A 1 \
+     | grep -oE "\"[^\"]+\" HTTP" | sort | uniq -c | sort -rn | head'
+```
+
+One path dominating means an orphaned client, not a broken gate — annoying but harmless, and it
+clears when the tab is closed. Many different paths from one session means the session itself is
+not surviving, which is the real defect. A per-app `--api-route` is the structural fix for the
+first shape: it makes those paths answer **401** instead of 302, so the client fails loudly
+instead of looping. Note that a count of **0 `AuthSuccess`** alongside a high rejection count
+means nobody has signed in since the last restart — so `--cookie-refresh` has never run, and no
+session-lifecycle conclusion can be drawn from that window at all.
+
+**4. Did the gate admit the user, or did the app refuse them?** Look for the `[AuthSuccess]` line
+and what follows it. If `[AuthSuccess]` appears and the very next thing is the *app's own* login
+page or the app's 401, the gate did its job and layer 3 is your problem — that is the Crafty
+shape, and `runbooks/crafty-admin-credential.md` covers it.
+
+**5. Cookie names, when sessions do not survive.** Two oauth2-proxy instances emitting the
+**same** cookie name on the **same** domain hold different cookie-secrets, so each overwrites the
+other's cookie and the loser logs exactly the `No valid authentication` line above. List what
+each gate is configured to emit (arguments only — never print a Secret's data):
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'for ns in crafty jackett pihole qbittorrent radarr sonarr; do printf "%-12s " "$ns"; \
+     kubectl --context k3s-prod -n $ns get deploy $ns \
+       -o jsonpath="{range .spec.template.spec.containers[?(@.name==\"oauth2-proxy\")]}{.args}{end}" \
+     | tr "," "\n" | grep -E "cookie-name|cookie-domain" | tr "\n" " "; echo; done'
+```
+
+A gate with no `--cookie-name` is using oauth2-proxy's default `_oauth2_proxy`. If two of them
+share that default **and** a zone-wide `--cookie-domain=.home.keithjackson.dev`, they are
+evicting each other, and the symptom is intermittent and depends on which app you visited last.
+
+## Likely causes, most likely first
+
+1. **Cookie collision between two gates on one domain.** Default cookie names plus a zone-wide
+   cookie domain. Signature: high `No valid authentication`-to-`AuthSuccess` ratio, sessions
+   dying seconds after being created, symptom depends on browsing order, and *nothing* wrong at
+   Keycloak. Fix is a unique `--cookie-name` and a host-scoped `--cookie-domain` per gate — see
+   `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`, which found and fixed exactly this.
+2. **The cookie is too large to come back.** With `--pass-access-token` the session carries the
+   access token; a fat Keycloak token pushes the cookie past 4 KB and it gets split into
+   `_oauth2_proxy_0`, `_oauth2_proxy_1`… If a chunk is dropped the session is unreadable and the
+   gate logs the same line. Look for numbered cookie names in the log, and for a session that
+   never survives a single navigation.
+3. **The XHR path is not the navigation path.** The app's API is under a prefix the gate handles
+   differently, or the app issues cross-origin XHRs the cookie does not accompany. Signature:
+   `/UI/...` returns 200 and `/api/...` returns 302 in the *same second*, which is precisely the
+   jackett trace above.
+4. **Session lifetime shorter than the app's polling interval,** with no refresh. The jackett
+   session above was a 5-minute session. If `refresh_token:true` is present and sessions still
+   die early, the refresh is failing, not missing.
+5. **Group check, not authentication.** `--allowed-group` is a floor: a valid Keycloak session in
+   the wrong group is refused *after* a successful login, which reads as a loop. Look for 403 and
+   for the `groups:[...]` list on the `[AuthSuccess]` line.
+6. **The app, not the gate.** `[AuthSuccess]` followed by the app's own login page or 401.
+
+## When to escalate or stop
+
+- **Stop and hand over if the fix is a manifest change.** Everything above is read-only
+  diagnosis. Changing a `--cookie-name`, adding `--metrics-address`, or editing an alert goes
+  through a branch and a PR; `kubectl edit` on a gate is drift that the next Flux reconcile
+  reverts, and in the meantime you have an unreproducible cluster.
+- **Stop if you find yourself about to print a Secret.** The cookie-secret and client-secret live
+  in `oauth2-proxy-credentials` and the per-app equivalents. Compare *key names* and whether two
+  Secrets differ; never dump `.data` or `.stringData`. A password has been leaked into a
+  transcript here exactly this way.
+- **Escalate to the platform owner if Keycloak itself is the suspect** — check `KeycloakDown` and
+  `KeycloakDatabaseDown` first (`runbooks/keycloak-down.md`, `runbooks/keycloak-db.md`). In the
+  2026-09-30 incident Keycloak was healthy throughout and blaming it would have cost days.
+- **The honest limit of this page.** With no metrics from the sidecars there is no history beyond
+  Loki's **72h** retention (`platform/monitoring/loki.yaml`, `retention_period: 72h`). If the
+  report is older than three days, the evidence is gone and the only remaining move is to
+  reproduce it. That is the cost of the gap `OAuth2ProxyGateUnobserved` is tracking, and it is
+  the reason to close it rather than silence it.

runbooks/plex-quicksync.md

@@ -1,5 +1,10 @@
 # plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did
 
+**Status: hardware transcoding is CONFIRMED WORKING on prod as of 2026-10-02** — VAAPI for both
+decode *and* encode, zero-copy, measured in Plex's own server log (see "How to verify"). The manual
+toggle has been ticked. This page is therefore now mostly a **verification and regression**
+procedure, and its old verification section was **wrong** — see the correction at the top of
+"How to verify".
 **Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
 not an incident page. It is indexed here because the feature it completes is **inert until a human
 clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
@@ -33,7 +38,7 @@ is in git**:
 | Half | Who does it | State |
 |---|---|---|
 | The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. |
-| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
+| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **Done, 2026-10-02**, and confirmed from the **log** rather than from a preference key. `HardwareAcceleratedCodecs` is **still absent** from `Preferences.xml` with hardware transcoding working, so its absence proves nothing — see first-command #4. |
 
 **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
 grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
@@ -79,8 +84,8 @@ also the only thing that produces the cgroup allow rule. There is no graceful-de
 The one improvement is that the loss is no longer silent.
 
 What this does **not** do, so nobody reads more into a green gate than is there: it does not enable
-hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest
-renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
+hardware transcoding (that is the manual toggle below — done 2026-10-02), a green gate only proves
+the **manifest renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
 advertises, which is the scheduler's doing and not a selector in this repo.
 
 ## Running the commands in this runbook
@@ -172,17 +177,28 @@ cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with
 visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely
 causes" #2.
 
-**4. Is the toggle on?** Named key only — never the whole file:
+**4. Is the toggle on?** **You cannot answer this from `Preferences.xml` alone.** That mistake is on
+this page's record: it told readers to treat a missing `HardwareAcceleratedCodecs` as "never
+enabled", and on 2026-10-02 that key was **absent from all 27 attributes** while hardware
+transcoding was demonstrably working. Named keys only — never the whole file, it holds
+`PlexOnlineToken`:
 
 ```
 infra/devcontainer/dev.sh run bash -c \
   'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
-     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
+     grep -o -e "HardwareAcceleratedCodecs=\"[01]\"" -e "HardwareDevicePath=\"[^\"]*\"" \
+     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo NEITHER-KEY'
 ```
 
-`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
-and the remaining question is whether it *works* — go to "How to verify".
+| Reading | What it actually means |
+|---|---|
+| `HardwareDevicePath="…"` **present** | **Positive signal.** Plex writes this when hardware acceleration is configured; it was present on 2026-10-02 alongside a working VAAPI pipeline. |
+| `HardwareAcceleratedCodecs` **absent** | **Nothing. Neither on nor off.** Measured absent with the feature working (2026-10-02). Do not diagnose from it, and do not write it into the file to "fix" it. |
+| `HardwareAcceleratedCodecs="0"` | explicitly **off** — the only conclusive reading, and only in the negative. |
+| `HardwareAcceleratedCodecs="1"` | on. |
+
+**The log settles it, this file does not.** Go to "How to verify"; the `hardware transcoding:
+enabled` and `final decoder/final encoder` lines are the verdict.
 
 ## The one manual step
 
@@ -191,8 +207,11 @@ Plex web UI → **Settings → Transcoder → "Use hardware acceleration when av
 
 The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
 extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
-recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
-them and not invent them:
+recall and not from a forum post. **But do not expect to see it in `Preferences.xml` after ticking
+the box:** on 2026-10-02, with the toggle on and hardware transcoding working, the key was absent and
+`HardwareDevicePath` was what had been written. The key exists in the binary; Plex does not
+necessarily persist it. Real neighbouring keys from the same binary, so you can recognise them and
+not invent them:
 
 | Key | What it is |
 |---|---|
@@ -209,42 +228,121 @@ binary. Recorded explicitly so nobody spends an evening hunting for it or, worse
 Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
 tier, it is in no git commit, and the UI writes the key correctly. Use the UI.
 
-## How to verify — and why there is no alert
+## How to verify — the log, not the dashboard
 
-Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
-lower quality explicitly), then read Plex's own **Dashboard → Activity**:
+**Correction, 2026-10-02.** Until today this page said the Plex dashboard showing `Transcode (hw)`
+was "the *only* proof available, and nothing else will ever tell you". **That was false and it cost
+the operator real time:** hardware transcoding was fully working — VAAPI decode *and* encode, with
+zero-copy — while **Dashboard → Activity showed nothing of the sort**. A check that is silent while
+the feature works does not just fail to prove anything; it sends someone hunting a fault that does
+not exist. The authoritative check is **Plex's own server log**.
 
-- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
-- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
+### The primary check — run this one
 
-**That dashboard is the *only* proof available, and nothing else will ever tell you.**
-`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
-there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
-to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
-inert, and this repo has already been burned at scale by rules that evaluated against nothing while
-looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
-gap is ever worth closing, the honest order is a real exporter first and a rule second.
+```
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+  grep -aiE "hardware transcoding: (final|using)" \
+  "/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log" \
+  | sed -E "s/(X-Plex-Token|token)=[^&[:space:]]*/\1=REDACTED/g" | tail -5'
+```
 
-Corroborating, not a substitute — the server log on the config PVC:
+| Line | Verdict |
+|---|---|
+| `hardware transcoding: final decoder: vaapi, final encoder: vaapi` | **PASS.** Hardware in **both** directions. This is the pass condition. |
+| `hardware transcoding: using zero-copy transcoding` | Best case — frames never round-trip through the CPU. Observed on prod 2026-10-02. |
+| `hardware transcoding: final decoder: , final encoder: ` (both empty) | It **fell back to software** for that session. |
+| no matching line at all | No transcode has been attempted since the log rolled — force one (below), then look again. Absence is not a failure. |
+
+The 2026-10-02 reading on prod, in full, for the shape to recognise:
+
+```
+Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD128'
+                              (Intel CoffeeLake-S GT2 [UHD Graphics 630])
+hardware transcoding: using hardware decode accelerator vaapi
+hardware transcoding: zero-copy support present
+hardware transcoding: using zero-copy transcoding
+hardware transcoding: final decoder: vaapi, final encoder: vaapi
+```
+
+To make a line appear, start a playback that **forces a transcode**: a client that cannot
+direct-play the source, or an explicitly lower quality. Direct Play / Direct Stream never reaches
+the transcoder and logs nothing.
+
+### Three things to get right about that command
+
+1. **Always keep the `sed`.** `Plex Media Server.log` carries API URLs containing
+   **`X-Plex-Token=…`**, a live plex.tv credential. Any output that might land in a ticket, a
+   transcript or a chat window must be redacted — CLAUDE.md §6, and a credential has already been
+   leaked into a transcript in this repo exactly this way. The redacting form above is the form to
+   copy; do not "simplify" the pipe out of it.
+2. **`Plex Transcoder Statistics.log` is the WRONG FILE.** The name is irresistible and someone will
+   reach for it. It holds XML `<Variant .../>` session records — **not** ffmpeg/VAAPI output. There
+   is no decoder/encoder verdict anywhere in it. The file you want is `Plex Media Server.log`.
+3. `grep -a` is deliberate: the log carries occasional non-UTF-8 bytes, and without `-a` GNU grep
+   reports "binary file matches" and prints nothing — which reads exactly like "no hardware
+   transcoding".
+
+### Secondary and unreliable: the Plex dashboard
+
+**Dashboard → Activity** *may* label a transcoding session `Transcode (hw)`. If it does, it agrees
+with the log and you are done. **If it does not, that proves nothing at all** — measured 2026-10-02:
+hardware transcoding active and zero-copy, dashboard silent. Never conclude "software transcoding"
+from the dashboard and never open an investigation on its say-so. Kept here only because it is the
+fastest glance available; it is a convenience, **not evidence**.
+
+### Process-level, while a transcode is in flight: the transcoder's argv
+
+This shows Plex *requesting* hardware, as distinct from reporting it afterwards. Get a shell in the
+pod (if `-it` has no TTY under `dev.sh run`, wrap each line in `sh -c "…"` instead):
 
 ```
 infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
-     \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
+  'kubectl --context k3s-prod -n plex exec -it deploy/plex -- sh'
 ```
 
-Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
-Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
-of hardware. The dashboard remains the verdict.
+then, inside the pod:
+
+```
+ps -eo pid,user,comm | grep -i transcoder
+tr '\0' ' ' < /proc/<transcoder-pid>/cmdline | grep -oE '(-hwaccel[a-z_]*|vaapi|renderD[0-9]+)'
+```
+
+`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128` present ⇒ Plex is asking for hardware. Keep the
+`grep -oE`: the raw argv contains media paths and session identifiers you do not want in a
+transcript. This only works while a session is actually transcoding.
+
+### Documented false lead: scanning `/proc/*/fd` for a holder of `renderD128`
+
+**It returns nothing even when hardware transcoding is working.** Measured 2026-10-02: a
+`/proc/*/fd` sweep inside the pod found **no** process holding `/dev/dri/renderD128`, while the log
+showed `final decoder: vaapi, final encoder: vaapi` for sessions either side of the scan. Transcode
+sessions are **short** and the scan landed in a gap between two of them. An empty fd scan is a
+**timing artefact, not a finding** — do not report it as evidence of software transcoding, and do
+not escalate on it. Use the argv check above (which has the same timing constraint but tells you
+something when it does catch a session), or just read the log, which is retrospective and therefore
+does not care about timing.
+
+### Why there is no alert — this half was always right
+
+`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
+there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open
+work to "alert on software-transcode fallback": with no metric behind it such a rule would be
+permanently inert, and this repo has already been burned at scale by rules that evaluated against
+nothing while looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence
+of"). If that gap is ever worth closing, the honest order is a real exporter first and a rule
+second.
 
 ## Likely causes, most likely first
 
-When the dashboard still says plain `Transcode`:
+When the **log** says it fell back to software — `final decoder: , final encoder: ` with both
+fields empty. (Not when the dashboard is quiet: the dashboard's silence means nothing, see "How to
+verify".)
 
-1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
-   First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
-   investigating anything technical.
+1. **The toggle is off.** Historically the most likely answer and the reason this page exists. It was
+   ticked on 2026-10-02 and should stay ticked, but a `/config` restore or a stray UI save can unset
+   it. Remedy: the manual step above. **Do not diagnose this from a missing
+   `HardwareAcceleratedCodecs`** — see first-command #4; only `="0"` is conclusive, and the log line
+   `hardware transcoding: enabled` is the real signal that the toggle took.
 
 2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom:
    first-command #3 shows `renderD128` present and correctly owned, and the open returns
@@ -276,13 +374,27 @@ When the dashboard still says plain `Transcode`:
 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
    mechanism this feature leans on with no commit in this repo.
 
-5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
-   `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
-   a hardware failure.
+5. **The session was never transcoding.** Direct Play / Direct Stream never reaches the transcoder,
+   so no `hardware transcoding:` line is written for it at all. An *absent* line is this, not a
+   failure — confirm a transcode actually happened before treating anything as a hardware fault.
 
-6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
-   Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
-   sessions show `(hw)` and others do not. Not a fault.
+6. **The *source* codec may be one this iGPU cannot decode — HYPOTHESIS, not a confirmed
+   diagnosis.** The symptom that points here is fallback on **particular titles** while other
+   sessions are hardware. Measured on prod 2026-10-02, *between* two fully hardware sessions, one
+   session logged:
+
+   ```
+   hardware transcoding: enabled, but no hardware decode accelerator found
+   hardware transcoding: final decoder: , final encoder:
+   ```
+
+   The platform was demonstrably working either side of it, so the likely explanation is the **file**,
+   not the cluster: UHD 630's media engine decodes H.264, HEVC 8-bit and 10-bit, and VP9 — but **not
+   AV1**, and not 10-bit 4:2:2 HEVC. **This was NOT verified against that particular title.** It is
+   inferred from the hardware's capability set and is written here as the lead to check first, not as
+   a finding. So: fallback on *all* files is a platform problem (causes 1-4); fallback on *some*
+   files, check the source codec before suspecting the device, and expect occasional legitimate
+   per-title fallbacks to be normal rather than a fault.
 
 ## Plex will not schedule
 
@@ -360,19 +472,30 @@ infra/devcontainer/dev.sh run bash -c \
 ```
 
 `45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
-described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
-needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
-hook exists.
+described here: the hook reads the injected device's **own** ownership, which under the device plugin
+is `root:root`, so it resolves GID `0` (see "When to escalate or stop"). Gone or renamed ⇒ stop and
+re-measure. In particular the "no `supplementalGroups` needed" conclusion in `catalog/plex.yaml` and
+`schemas/renderer-contract.md` is only true while that hook exists.
 
 ## When to escalate or stop
 
 - **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
-  **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
-  `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
-  group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
-  the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
-  `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
+  **inert**, and it would mislead the next person. The image's `cont-init.d` hook stats each
+  `/dev/dri` char device and `usermod -a -G`s the `plex` user into **that device's** group, inside
+  the container; the service then execs via `s6-setuidgid plex`, which **rebuilds** the process group
+  list from `/etc/group` rather than appending to it. Anything Kubernetes supplies is therefore
   discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
+- **GID `992` is a hostPath-era number — do not expect it, and do not read the root group as a
+  defect.** Under the device plugin the injected `/dev/dri/renderD128` arrives **`root:root`**, so
+  the hook resolves GID **`0`** and adds `plex` to the **root group**. Measured 2026-10-02:
+  `Plex Media Server` runs with `Groups: 0 100 1000`, and `id plex` reports
+  `groups=1000(plex),0(root),100(users)`. Under the old hostPath the device was `root:render`
+  (GID 992 on these nodes) and the reading was `Groups: 44 100 992 1000`. **992 is now irrelevant**:
+  if any page, comment or older handoff tells you to expect it, that text is stale and the system
+  you are looking at is working. The root-group membership is an inherent property of the plugin's
+  `DeviceSpec` — it injects the host device with its host ownership — **not** a defect, and **not** a
+  reason to reintroduce `supplementalGroups`, which would still be a no-op for the reason in the
+  bullet above.
 - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
   reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement,
   because root does not bypass the cgroup device gate.
@@ -390,7 +513,8 @@ hook exists.
   watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
   lose the DB.
 - **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2
-  confirmed) *under the current mechanism*. The next step is a platform change in
+  confirmed) *under the current mechanism* — note that this open **passed** on 2026-10-02
+  (`RW-OPEN-OK` as uid 1000 via `s6-setuidgid plex`), so a failure now is a regression. The next step is a platform change in
   `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload.
 
 ## Known gaps, deliberately not fixed here
@@ -413,10 +537,25 @@ Recorded so they are findable, and so nobody assumes they were overlooked:
 
 ---
 Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
-The node measurements (all three prod nodes) and the in-container image measurements cited here were
-taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
-(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000`
-reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**.
-Hardware transcoding has **still not been observed working**: the open test has not been re-run under
-the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side
-toggle is still unset.
+
+**Hardware transcoding is CONFIRMED WORKING on prod, measured 2026-10-02.** From
+`Plex Media Server.log` in the running pod: API `vaapi` on `/dev/dri/renderD128`
+(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`,
+`zero-copy support present`, `using zero-copy transcoding`, and
+`final decoder: vaapi, final encoder: vaapi`. Also measured in the same pod: `RW-OPEN-OK` opening
+`/dev/dri/renderD128` as uid 1000 via `s6-setuidgid plex`; the pod spec carrying
+`{"limits":{"gpu.intel.com/i915":"1"},"requests":{"gpu.intel.com/i915":"1"}}`; `Groups: 0 100 1000`
+and `id plex` → `groups=1000(plex),0(root),100(users)`; and `HardwareDevicePath` present in
+`Preferences.xml` while `HardwareAcceleratedCodecs` was absent from all 27 attributes.
+
+Earlier readings, kept for provenance and **superseded** where they conflict: the node measurements
+(all three prod nodes) and the in-container image measurements taken **2026-09-30** against
+`plexinc/pms-docker` digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`); the hostPath
+`EPERM` open results and the `Groups: 44 100 992 1000` / GID-992 group reading, **2026-10-01, under
+the hostPath mechanism that no longer exists**; and the `gpu.intel.com/i915: 1` allocatable and
+`intel-gpu-plugin` DaemonSet readings, **2026-10-01**.
+
+What this page still does **not** establish: that any particular title's software fallback was caused
+by its source codec (cause 6 is an unverified hypothesis), and that Plex's dashboard *ever* labels a
+session `Transcode (hw)` on this deployment — it has not been observed doing so, which is exactly why
+the log is now the primary check.

schemas/renderer-contract.md

@@ -23,6 +23,7 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 | `dependencies` | `[]` → reaches **only DNS** (default-deny) | R-NET-01 |
 | `inClusterClients` | `[]` → no direct east-west client; only the Gateway may reach an exposed port | R-NET-01/02 |
 | `auth.mode` | `forward-auth` for any exposed UI | R-IDP-02 |
+| `auth.apiRoutes` | `[]` → the sidecar renders **no** `--api-route`; every unauthenticated request is 302'd to Keycloak | R-IDP-02 |
 | `update.channel` | `stable` | R-UPD-01 |
 | `update.autoRollbackOnFailedHealth` | `true` | R-UPD-03 |
 | `observability.metrics` | `true` | R-OBS-01 |
@@ -319,7 +320,7 @@ platform must render, and cites the requirement each mapping satisfies. Changing
   undeclared), **R-GITOPS-02** (the allowance is authored in the one reviewable object rather than
   surviving as a live-applied cluster object). Auditable via Hubble (R-SEC-03, R-OBS-04).
 
-### `auth` (`mode` / `groups[]`) — identity integration (Keycloak / oauth2-proxy)
+### `auth` (`mode` / `groups[]` / `apiRoutes[]`) — identity integration (Keycloak / oauth2-proxy)
 - `oidc` → register the app as a **Keycloak OIDC client** (declarative realm/client config, GitOps;
   client secret SOPS-encrypted). App does its own login redirect.
 - `forward-auth` → protect the service's `ingressRoutes` at the **Gateway** with **oauth2-proxy →
@@ -330,6 +331,20 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 - `groups[]` → the Keycloak groups permitted access, enforced by the OIDC client mappers /
   oauth2-proxy `allowed_groups`. **Schema requires ≥1 group for `oidc`/`forward-auth`** — access is
   "in an authorized group," not merely "logged in."
+- `apiRoutes[]` (**`forward-auth` only**) → one repeatable **`--api-route=<regex>`** per entry on
+  the per-app oauth2-proxy sidecar, making matching paths answer **401** instead of a **302** to
+  Keycloak. Default `[]` renders **no flag at all**, which is the common case. It exists because
+  oauth2-proxy's built-in AJAX guard fires only on an exact `Accept: application/json`, which
+  these apps' JavaScript does not consistently send: the XHR is 302'd, the browser follows it
+  invisibly, the live SSO session satisfies it silently, and the call resolves 200 carrying the
+  app's **HTML** where the JavaScript expected JSON — and a POST following a 302 is re-issued as
+  a GET with its **body dropped**. It is **per-service and must never become a renderer
+  constant**: the path shape belongs to the app (radarr `/signalr/`, jackett `/api/`, pihole
+  none), and a wrong constant does not merely fail to help — a path a **browser navigates to**,
+  answered 401, shows a logged-out operator a bare error page and never offers a sign-in. Each
+  entry must be **anchored at `^/`** and continue with literal path characters; `^/`, `^/.*`,
+  `.*`, an unanchored `/api/` and any alternation are **rejected** by the schema and by the
+  renderer (alternation is refused rather than supported because the flag is repeatable).
 - Keycloak, oauth2-proxy, and app→Keycloak / proxy→Keycloak / Keycloak→DB flows are ordinary
   workloads under default-deny with explicit allows (ties back to `dependencies`).
 - **Requirements:** **R-IDP-02** (protect OIDC-capable and bare UIs), **R-IDP-06** (group-based
@@ -388,6 +403,12 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 ## Cross-field invariants the renderer / L0-L1 tests rely on
 1. **kind ↔ source:** container⇒`image`, vm⇒`vmTemplate` (schema-enforced).
 2. **auth ↔ groups:** `oidc`/`forward-auth` ⇒ non-empty `groups` (schema-enforced; R-IDP-06).
+2a. **auth ↔ apiRoutes:** a non-empty `auth.apiRoutes` ⇒ `auth.mode: forward-auth`
+   (schema-enforced). It renders onto the per-app oauth2-proxy sidecar, and no sidecar exists for
+   `oidc`/`native`/`none`, so declaring it elsewhere would be a **silent no-op** — the operator
+   believes the app's XHRs now answer 401 and nothing changed. Every entry must be anchored at
+   `^/` (schema-enforced **and** enforced in `catalog/render.py`, so a breadth mistake is caught by
+   `render-golden` even on a run where the schema stage did not execute).
 3. **default-deny is unconditional:** every namespace gets the deny baseline regardless of
    `dependencies`; `dependencies` only *adds* allows (R-NET-01/02). The same holds for
    `inClusterClients`: it adds one namespace+port ingress allow and never relaxes the baseline.

schemas/servicedefinition.schema.json

@@ -2,7 +2,7 @@
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
   "title": "ServiceDefinition",
-  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 Line truncated
+  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked proLine truncated
   "type": "object",
   "additionalProperties": false,
   "required": ["name", "source"],
@@ -324,6 +324,16 @@
           "items": { "type": "string", "minLength": 1 },
           "uniqueItems": true,
           "default": []
+        },
+        "apiRoutes": {
+          "type": "array",
+          "description": "forward-auth ONLY: path regexes this app serves as a MACHINE API, which the oauth2-proxy sidecar must answer with 401 instead of a 302 to Keycloak (one --api-route per entry; the flag is repeatable and singular -- --api-routes does not exist in v7.8.1 and the proxy refuses to start with it). WHY THIS IS PER-APP AND NOT A RENDERER CONSTANT: oauth2-proxy's built-in AJAX guard fires only on an EXACT `Accept: application/json`, which these apps' JavaScript does not consistently send, so an unauthenticated XHR gets a 302 that the browser follows invisibly, the live Keycloak SSO session satisfies silently, and the call resolves 200 carrying the app's HTML where the JavaScript expected JSON -- no prompt, no error, and on a write it is worse, because a browser following a 302 re-issues a POST as a GET and DROPS THE BODY. Measured on jackett 2026-09-30: 33 x 302 against 2 x 401 on /api/ paths, with 4 of 6 POSTs answered by the proxy and never reaching the app (access-log upstream field `-`). Measured on radarr 2026-10-01, 23h after the cookie fix deployed: 232 x 302, ALL of them POST /signalr/messages/negotiate from the operator's own browser -- a left-open tab whose cookie the rename orphaned, polling forever, silently redirected every time and NEVER re-prompting. But the path shape is a property of the APP, not of the pattern: radarr's broken path is /signalr/ and jackett's is /api/, and pihole serves no /api/ at all, so a renderer-wide constant would apply one app's path shape to five. Declare only paths this app actually serves as JSON/RPC and that a browser never NAVIGATES to. BREADTH IS THE DANGER: too broad and a plain page load answers 401 instead of signing the operator in, i.e. the service looks permanently broken to a logged-out user and no sign-in is ever offered. The item pattern therefore REQUIRES each entry to be anchored at `^/` and to continue with literal path characters, which rejects `^/`, `^/.*`, `.*` and any unanchored or alternated regex; alternation is refused rather than supported because the flag is repeatable, so one regex per route costs nothing and keeps each entry readable. N/A for oidc/native/none (no sidecar is rendered).",
+          "items": {
+            "type": "string",
+            "pattern": "^\\^/[A-Za-z0-9._~/-]+(\\.\\*|\\$)?$"
+          },
+          "uniqueItems": true,
+          "default": []
         }
       }
     },
@@ -471,6 +481,26 @@
         }
       }
     },
+    {
+      "$comment": "auth.apiRoutes only has a consumer under forward-auth: it renders --api-route onto the per-app oauth2-proxy sidecar, and no sidecar is rendered for oidc/native/none. Declaring it elsewhere would be a SILENT NO-OP -- the operator believes the app's XHRs now answer 401 and nothing changed -- so it is rejected at the schema instead (R-IDP-02).",
+      "if": {
+        "required": ["auth"],
+        "properties": {
+          "auth": {
+            "required": ["apiRoutes"],
+            "properties": { "apiRoutes": { "minItems": 1 } }
+          }
+        }
+      },
+      "then": {
+        "properties": {
+          "auth": {
+            "required": ["mode"],
+            "properties": { "mode": { "const": "forward-auth" } }
+          }
+        }
+      }
+    },
     {
       "$comment": "public exposure must not be unauthenticated: an explicit auth.mode 'none' is disallowed on public services (docs/13 v1.1, docs/15, R-IDP-02). Absent auth defaults to forward-auth at render time, so it is permitted here.",
       "if": {