fix(auth): the forward-auth sidecar also died at 5 minutes and served no metrics

Widens the cookie-collision fix to the whole sidecar contract, because all three
defects live in the same function and land on the same five goldens, so they are one
atomic change or they are five conflicting ones.

Added to _oauth2_sidecar_container(), on top of the cookie-name/cookie-domain fix:

  --cookie-refresh=2m30s
      The session died FIVE MINUTES after sign-in. Keycloak's homelab realm issues
      300-second access tokens (MEASURED on a live session: expires - created =
      300.000s exactly, with refresh_token:true on the same line), and
      --cookie-refresh defaults to 0 -- which the pinned binary's own help spells
      "0 to disable". So the proxy held a refresh token it was never told to spend.
      Refresh at HALF the lifetime: two chances to renew before expiry.
      This, not the collision, is what dominates the rejection VOLUME -- ~95% of
      radarr's 155 rejections in 12h carry no cookie at all.

  --metrics-address=0.0.0.0:44180 + a `metrics` container port
      Without the flag the /metrics listener never starts, so no per-app sidecar was
      serving metrics at all (`ports` was ['http'] on all six). 44180 is the
      platform's existing convention, measured not chosen: the shared proxy already
      runs the same flag and port with 2 ready endpoints.
      PREREQUISITE ONLY -- NOT scraped. A container port is not a Service port, and
      each namespace's default-deny baseline is `ingress: []`, so this is unreachable
      until a Service port (or PodMonitor), a monitoring allow and a ServiceMonitor
      land. Do not read this as "the sidecars have metrics now".

EVERY FLAG VERIFIED AGAINST THE PINNED v7.8.1 BINARY (`--help`), not against docs or
the shared proxy -- which runs v7.15.3 and could not have vouched for v7.8.1:
  * --metrics-address exists in 7.8.1, so this cannot crashloop the pods
  * --cookie-refresh ... "0 to disable" confirms the default
  * --cookie-name ... (default "_oauth2_proxy") -- this confirms from the
    IMPLEMENTATION the constant the new L0 check is built on
  * --api-route is SINGULAR; the plural does not exist and the proxy refuses to start
  * --session-cookie-minimal exists and is mutually exclusive with --cookie-refresh,
    so dropping --pass-access-token is a least-privilege argument, never a size fix

NOT done, deliberately: --api-route=^/api/ renderer-wide. It was measured on jackett
and PiHole does not serve /api/, so a renderer constant would apply one app's path
shape to five. It needs a per-app hook the `auth` schema forbids
(additionalProperties: false) -- a schema change, separate PR.

Also records the mechanism that made this INTERMITTENT, as a renderer comment, because
nobody would reconstruct it from the flags: the loser does not just fail to read the
cookie, it calls ClearSession, which expires the cookie on its own --cookie-domain --
the whole zone. The *arr UIs poll (/api/v3/command, /api/v3/queue/details,
/signalr/negotiate), so every poll carrying an undecryptable cookie clears it again:
A LEFT-OPEN RADARR TAB DELETED JACKETT'S COOKIE EVERY FEW SECONDS, from another
namespace, with no user action. That is why signing in again never helped for longer
than the gap between two polls.

Crafty's counter-evidence resolved: the 81:1 ratio was a 24h window straddling a
13h52m-old pod. Bounded to the fixed process, crafty's ENTIRE life is 3 rejections in
6743 log lines, with ZERO `removing session` and ZERO `cookie signature not valid`
against unfixed radarr's 8 and 8. The eviction fingerprint is gone where the fix is
deployed. It logged 0 AuthSuccess too, so that is absence-of-the-failure-mode rather
than a session watched working -- stated as such in the handoff.

Renderer selftest gains six assertions, including a --cookie-refresh check that PARSES
the duration and requires 0 < refresh < 300s (so 2m30s/150s/2.5m all stay green while a
silent widening past the token lifetime does not). NOTE, and it is in the handoff as the
top follow-up: NOTHING IN ci/ RUNS `render.py --selftest`, so those assertions are
dev-time only. conformance/tests/l0-static/auditability-proofs-producible/check.sh:291
already drives other tools' selftests for exactly this reason and is the one-line fix.

Gate (toolbox, worktree shim with the spec repo mounted at /home-network-k3s): rc 3
UNDETERMINED. Green: spec-immutability, decisions-log, lint, l0, render-golden, l1,
approval-policy. Skipped: gitea-settings (no local token), conformance (declared).
The new L0 check PASSES on this branch -- all five apps fixed in one commit.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/gate.yml

@@ -131,6 +131,10 @@ jobs:
           # merging. Green here means the token resolved and the live Gitea matched the
           # declaration in infra/gitea/expected-branch-protection.json.
           REQUIRE_GITEA_SETTINGS: "true"
+          # ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
+          # PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
+          GITHUB_EVENT_NAME: ${{ github.event_name }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
         run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4

CLAUDE.md

@@ -14,7 +14,28 @@ There is no other path. Specifically:
 
 - **Do not `git push origin main`.**
 - **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
-- **Do not merge your own PR.** A human approves and clicks merge.
+- **Do not merge your own PR.** See the approval model below — and either way, the merge is
+  someone else's action, never yours.
+
+### Who approves
+
+**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
+branch requires no human approval for one. That is a deliberate transfer of authority, on the
+operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
+good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
+add a human.
+
+**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
+admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
+approving review from someone who is not its author. The reason is narrow and structural — **a
+change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
+the weakened check. Everything else merges on green.
+
+The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
+sensitive path, so the policy cannot be widened or narrowed without a human.
+
+That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
+missing credential is a lock-out, here a skip would be a bypass.
 - **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
   anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
 

apps/wave-0/jackett.yaml

@@ -100,11 +100,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_jackett
+        - --cookie-domain=jackett.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -125,6 +128,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-0/pihole.yaml

@@ -108,11 +108,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_pihole
+        - --cookie-domain=pihole.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -133,6 +136,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-0/qbittorrent.yaml

@@ -171,11 +171,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_qbittorrent
+        - --cookie-domain=qbittorrent.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -196,6 +199,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-1/radarr.yaml

@@ -153,11 +153,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_radarr
+        - --cookie-domain=radarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +181,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-1/sonarr.yaml

@@ -153,11 +153,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +181,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

catalog/render.py

@@ -133,6 +133,36 @@ OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
 OAUTH2_PROXY_PORT = 4180
 KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
 
+# --- session lifetime -----------------------------------------------------------------
+# Keycloak's `homelab` realm issues access tokens with a 300-SECOND lifetime. MEASURED on a
+# live session, not read off a config page: `expires - created = 300.000s` exactly, with
+# `refresh_token:true` on the same line — so the proxy HOLDS a refresh token it was never
+# told to spend. `--cookie-refresh` defaults to 0, which the v7.8.1 binary's own help spells
+# "0 to disable", so every one of these sidecars dropped its session five minutes after
+# sign-in and sent the user back to Keycloak. Refresh at HALF the lifetime: two chances to
+# renew before expiry, so one failed refresh is not an immediate logout.
+#
+# NOTE for anyone tempted to add `--session-cookie-minimal` to shrink the cookie: it is
+# MUTUALLY EXCLUSIVE with this (provoked: "cookie_refresh > 0 requires oauth tokens in
+# sessions"). Dropping --pass-access-token is a least-privilege argument, not a size fix.
+OAUTH2_ACCESS_TOKEN_LIFETIME_S = 300
+OAUTH2_COOKIE_REFRESH = "2m30s"
+
+# --- metrics --------------------------------------------------------------------------
+# oauth2-proxy serves /metrics only when --metrics-address is given; with it absent the
+# listener never starts, so NO per-app sidecar was scrapable at all (measured: `ports` was
+# `['http']` on all six). 44180 is the platform's existing convention, not a guess — the
+# SHARED proxy already runs `--metrics-address=0.0.0.0:44180` with a `metrics` port and 2
+# ready endpoints. The flag is confirmed present in the PINNED v7.8.1 binary (`--help`),
+# which matters because the shared proxy is on v7.15.3 and could not have vouched for it.
+#
+# THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED. A container port is not a
+# Service port: these sidecars' Services expose the app port and :4180, and each namespace's
+# default-deny baseline has `ingress: []`, so Prometheus can neither resolve nor reach this
+# port until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
+# are added. Do not read this flag as "metrics are working".
+OAUTH2_PROXY_METRICS_PORT = 44180
+
 
 # =============================================================================
 # Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
@@ -971,7 +1001,11 @@ def _oauth2_sidecar_container(s: dict) -> dict:
     auth.group (R-IDP-06: a valid session is not enough — the user must be in an authorized group,
     else /oauth2 returns 403). client-id/secret + cookie-secret come by name from the SOPS-encrypted
     `<app>-oauth2-proxy` Secret (R-IDP-04). Single provider => --skip-provider-button 302s straight
-    to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy."""
+    to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy.
+
+    --cookie-name and --cookie-domain are DERIVED per app (see the block comment on those two
+    args): every instance holds a different cookie-secret, so sharing one cookie name across the
+    zone makes the instances evict each other's sessions."""
     route = s["ingressRoutes"][0]
     host = route["host"]
     upstream_port = route["port"]
@@ -991,11 +1025,64 @@ def _oauth2_sidecar_container(s: dict) -> dict:
     args += [
         "--skip-provider-button=true",
         f"--redirect-url=https://{host}/oauth2/callback",
-        f"--cookie-domain=.{ZONE}",
+        # ── COOKIE ISOLATION (fleet fix 2026-09-30; crafty's 2026-09-29 fix, generalized) ──────
+        # EVERY oauth2-proxy on this platform — this sidecar, its four siblings, and the SHARED
+        # forward-auth proxy (platform/oauth2-proxy) — is an INDEPENDENT instance holding its OWN
+        # `cookie-secret`. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
+        # DISTINCT cookie-secret values, so no instance can decrypt any other's cookie.
+        #
+        # Left at oauth2-proxy's DEFAULTS (`_oauth2_proxy` / `_oauth2_proxy_csrf`) and scoped to
+        # `.${BASE_DOMAIN}`, the browser therefore keeps ONE cookie pair for ALL of them:
+        # whichever proxy authenticates last OVERWRITES every other's session. The loser logs
+        # "No valid authentication in request" and bounces to Keycloak, which — the SSO session
+        # still being live — redirects straight back and overwrites the cookie again. The
+        # user-visible symptom is "I signed into Keycloak but I'm still not signed in", and it
+        # sends you hunting in Keycloak, where nothing is wrong. Silent, intermittent, and
+        # misattributed: the exact shape this repo keeps getting burned by.
+        #
+        # WHY IT WAS *INTERMITTENT*, AND WHY RELOADING NEVER HELPED. The loser does not merely
+        # fail to read the cookie — it calls ClearSession, which EXPIRES the cookie on its own
+        # --cookie-domain, i.e. the whole zone. And the *arr UIs poll: measured on prod, radarr's
+        # rejections are `/api/v3/command`, `/api/v3/queue/details` and `/signalr/negotiate`,
+        # 155 of them in 12h against 1 successful auth. Every poll carrying a cookie radarr
+        # cannot decrypt calls ClearSession again — so A LEFT-OPEN RADARR TAB DELETES JACKETT'S
+        # COOKIE EVERY FEW SECONDS, from another namespace, with no user action at all. That is
+        # the intermittency, and it is why signing in again fixed nothing for longer than the
+        # gap between two polls. Host-scoping the cookie (below) is what contains the blast
+        # radius of a ClearSession to the app that issued it.
+        #
+        # DERIVED, NOT LISTED. Both flags come from the ServiceDefinition (`s['name']` and the
+        # route host), so a NEW forward-auth app is isolated the moment it is onboarded. A
+        # per-service table would leave the next app defaulting into the collision, which is how
+        # twelve ServiceMonitors acquired one identical wrong port name in a single edit.
+        # conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct asserts both properties
+        # over the WHOLE tree and asserts THIS function derives rather than lists.
+        #
+        #   --cookie-name   a unique name per instance, so two sessions cannot alias at all.
+        #                   This is the fix; the one below is defence in depth.
+        #   --cookie-domain HOST-scoped. A per-app reverse proxy has no reason to broadcast its
+        #                   session cookie to every other host in the zone — a zone-wide cookie
+        #                   is transmitted to every sibling app on `.${BASE_DOMAIN}`, so any
+        #                   FUTURE name collision would again be platform-wide instead of
+        #                   contained. The SHARED forward-auth proxy keeps the wide scope,
+        #                   because cross-host forward-auth is precisely its job.
+        #
+        # --whitelist-domain deliberately stays ZONE-WIDE: it governs permitted post-login
+        # REDIRECT targets, not cookie scope. Narrowing it here would break the sign-in bounce.
+        f"--cookie-name=_oauth2_proxy_{s['name']}",
+        f"--cookie-domain={host}",
         f"--whitelist-domain=.{ZONE}",
         "--cookie-secure=true",
+        # Spend the refresh token the proxy already holds, at half the 300s access-token
+        # lifetime. Without this the session dies 5 minutes after sign-in. See
+        # OAUTH2_ACCESS_TOKEN_LIFETIME_S above for the measurement and the
+        # --session-cookie-minimal incompatibility.
+        f"--cookie-refresh={OAUTH2_COOKIE_REFRESH}",
         "--pass-access-token=true",
         "--set-xauthrequest=true",
+        # Start the /metrics listener. Prerequisite only — not yet reachable by Prometheus;
+        # see OAUTH2_PROXY_METRICS_PORT above for exactly what is still missing.
+        f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}",
     ]
     secret = f"{s['name']}-oauth2-proxy"
     return {
@@ -1017,7 +1104,10 @@ def _oauth2_sidecar_container(s: dict) -> dict:
                 "valueFrom": {"secretKeyRef": {"name": secret, "key": "cookie-secret"}},
             },
         ],
-        "ports": [{"name": "http", "containerPort": OAUTH2_PROXY_PORT, "protocol": "TCP"}],
+        "ports": [
+            {"name": "http", "containerPort": OAUTH2_PROXY_PORT, "protocol": "TCP"},
+            {"name": "metrics", "containerPort": OAUTH2_PROXY_METRICS_PORT, "protocol": "TCP"},
+        ],
         "resources": {
             "requests": {"cpu": "25m", "memory": "64Mi"},
             "limits": {"memory": "128Mi"},
@@ -2475,6 +2565,86 @@ def _selftest() -> int:
         not any(a.startswith("--ssl-upstream-insecure-skip-verify") for a in side["args"]),
         "http upstream must NOT skip TLS verify (that flag is only for crafty's self-signed https)",
     )
+    # COOKIE ISOLATION. Every oauth2-proxy instance holds its own cookie-secret, so two instances
+    # sharing a cookie NAME on a shared DOMAIN evict each other's sessions silently. Both flags
+    # must be DERIVED from this service, not defaulted and not listed.
+    check(
+        "--cookie-name=_oauth2_proxy_fa" in side["args"],
+        "sidecar must declare a cookie name DERIVED from the service name — the oauth2-proxy "
+        "default (_oauth2_proxy) is shared with every other instance on the zone",
+    )
+    check(
+        "--cookie-domain=fa.home.keithjackson.dev" in side["args"],
+        "sidecar cookie-domain must be HOST-scoped (the app's own host), not zone-wide",
+    )
+    check(
+        f"--whitelist-domain=.{ZONE}" in side["args"],
+        "whitelist-domain must stay ZONE-wide — it governs post-login redirect targets, not "
+        "cookie scope; narrowing it breaks the sign-in bounce",
+    )
+    # SESSION LIFETIME. `--cookie-refresh` must be present AND strictly shorter than the
+    # measured 300s access-token lifetime, or the session dies mid-use. Asserted as a parsed
+    # duration rather than a string match, so "2m30s" -> "150s" -> "2.5m" stays green and a
+    # silent widening past the token lifetime does not.
+    refresh = next((a.split("=", 1)[1] for a in side["args"]
+                    if a.startswith("--cookie-refresh=")), None)
+    check(refresh is not None,
+          "sidecar must set --cookie-refresh; the v7.8.1 default is 0 = DISABLED, so the "
+          "session dies when the access token expires and the user is bounced to Keycloak")
+    if refresh is not None:
+        _m = re.fullmatch(r"(?:(\d+)m)?(?:(\d+(?:\.\d+)?)s)?", refresh)
+        _secs = (int(_m.group(1) or 0) * 60 + float(_m.group(2) or 0)) if _m else -1
+        check(0 < _secs < OAUTH2_ACCESS_TOKEN_LIFETIME_S,
+              f"--cookie-refresh={refresh} parses to {_secs}s, which is not inside "
+              f"(0, {OAUTH2_ACCESS_TOKEN_LIFETIME_S}s). A refresh at or past the access-token "
+              f"lifetime never renews in time, which is the same logout it was added to stop")
+    check(
+        not any(a.startswith("--session-cookie-minimal") for a in side["args"]),
+        "--session-cookie-minimal is MUTUALLY EXCLUSIVE with --cookie-refresh "
+        "('cookie_refresh > 0 requires oauth tokens in sessions') — the proxy refuses to start",
+    )
+    # METRICS. The listener does not exist unless the flag is given, and the port must be
+    # DECLARED or nothing can ever select it (neither a Service port nor a PodMonitor).
+    check(
+        f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}" in side["args"],
+        "sidecar must start the /metrics listener; without --metrics-address it is not merely "
+        "unscraped, it is not serving at all",
+    )
+    check(
+        [p["name"] for p in side["ports"]] == ["http", "metrics"]
+        and {p["containerPort"] for p in side["ports"]}
+        == {OAUTH2_PROXY_PORT, OAUTH2_PROXY_METRICS_PORT},
+        "sidecar must declare BOTH the http and metrics container ports — an undeclared port "
+        "cannot be named by a Service port or a PodMonitor, so the flag alone is inert",
+    )
+    # Two forward-auth apps must not be able to collide. Renders a SECOND one and compares, so a
+    # future refactor to a constant (or to a per-service table with a copy-paste slip) fails here
+    # rather than silently in a browser six weeks later.
+    fa2_side = next(
+        c
+        for c in next(
+            d[1]
+            for d in render(
+                {
+                    "name": "fa2",
+                    "source": {"image": "a:1"},
+                    "ingressRoutes": [{"host": "fa2.home.keithjackson.dev", "port": 9118}],
+                    "auth": {"mode": "forward-auth", "groups": ["media-users"]},
+                }
+            )
+            if d[1]["kind"] == "Deployment"
+        )["spec"]["template"]["spec"]["containers"]
+        if c["name"] == "oauth2-proxy"
+    )
+
+    def _cookie_name(container):
+        return next(a for a in container["args"] if a.startswith("--cookie-name="))
+
+    check(
+        _cookie_name(side) != _cookie_name(fa2_side),
+        "two forward-auth apps rendered IDENTICAL cookie names — they would overwrite each "
+        "other's session on every sign-in (the 2026-09-29 crafty defect, fleet-wide)",
+    )
     check(
         side["env"][0]["valueFrom"]["secretKeyRef"]["name"] == "fa-oauth2-proxy",
         "sidecar creds must come by name from the per-app <app>-oauth2-proxy Secret (R-IDP-04)",

ci/Makefile

@@ -35,6 +35,9 @@ decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
+approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
+	@./approval-policy.sh
+
 gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
 	@./gitea-settings.sh
 

ci/approval-policy.sh

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
+# what the pipeline verifies or the security posture it rests on.
+#
+# THE POLICY, and why it is shaped this way.
+#
+# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
+# pipeline is not good enough to be trusted for this, then it is not good enough to do its
+# intended job and needs to be improved. Manual approvers should only be for architecturally or
+# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
+#
+# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
+# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
+# A PR that weakens a check would be approved by the weakened check. The same applies to the
+# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
+# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
+# requires a human for them.
+#
+# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
+# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
+# policy cannot be widened without a human.
+#
+# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
+# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
+# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
+# BYPASS, not a gap. So: cannot determine -> BLOCK.
+#
+# EXIT CODES
+#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
+#      non-author APPROVED review is present
+#   1  sensitive paths touched without such a review, OR the policy could not be evaluated
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: approval policy (CI approves; humans approve changes to CI)"
+
+DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
+[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
+require_tool jq
+require_tool python3
+
+# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
+# by branch protection, so this is genuinely not-applicable rather than a skipped check.
+EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
+PR="${PR_NUMBER:-}"
+if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
+  ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
+  exit 0
+fi
+
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+if [ -z "$TOKEN" ]; then
+  err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
+  err "  BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
+  summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
+  exit 1
+fi
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
+PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
+REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
+for v in FILES_JSON PR_JSON REVIEWS_JSON; do
+  jq -e . >/dev/null 2>&1 <<<"${!v}" || {
+    err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
+    summary "- approval-policy: **BLOCKED — Gitea unreadable**"
+    exit 1; }
+done
+
+AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
+[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
+
+# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
+# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
+# never read. Caught before shipping, by running it.
+SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
+import json, os, fnmatch
+decl = json.load(open(os.environ["DECL"]))
+files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
+pats = decl["sensitive_paths"]
+hits = sorted({f for f in files for p in pats
+               if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
+print("\n".join(hits))
+')"
+
+if [ -z "$SENSITIVE" ]; then
+  ok "no architecturally/security-relevant paths touched — the GATE is the approver."
+  summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
+  exit 0
+fi
+
+echo "  sensitive paths touched:" >&2
+printf '    %s\n' $SENSITIVE >&2
+
+APPROVER="$(jq -r --arg a "$AUTHOR" \
+  '[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
+  <<<"$REVIEWS_JSON")"
+
+if [ -n "$APPROVER" ]; then
+  ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
+  summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
+  exit 0
+fi
+
+err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
+err "  review from anyone other than its author ('${AUTHOR}')."
+err "  A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
+err "  honestly approve for itself. Get a human review, or move the change out of:"
+printf '    %s\n' $SENSITIVE >&2
+err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
+summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
+exit 1

ci/gate.sh

@@ -113,6 +113,7 @@ run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage approval-policy "${CI_DIR}/approval-policy.sh"
 run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
@@ -190,6 +191,20 @@ case "${OUTCOME[spec-immutability]}" in
   pass) ok   "spec-immutability: green" ;;
 esac
 
+# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
+# is the approver, per operator intent. This stage is the single exception: a change to the
+# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
+# cannot honestly be approved BY the thing it changes, so those paths still need a human.
+#
+# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
+# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
+# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
+case "${OUTCOME[approval-policy]}" in
+  fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
+  skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
+  pass) ok  "approval-policy: green" ;;
+esac
+
 # gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
 # runner registration). A FAIL is real drift and always blocks -- that is the whole point.
 # A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -237,6 +237,31 @@ SCRAPE_OWNERS: list[dict] = [
                "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
                "Velero itself publishes nothing until a backup is attempted",
     },
+    {
+        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions",
+        "prefixes": ("kube_flux_",),
+        "file": KPS,
+        "require": [
+            r"customResourceState:\s*\n\s*enabled:\s*true",
+            r"metricNamePrefix:\s*kube_flux_kustomization",
+            r"metricNamePrefix:\s*kube_flux_gitrepository",
+            # The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io
+            # above: without read access to these two API groups KSM publishes the
+            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would
+            # leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured.
+            r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]',
+            r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]',
+            # A config-only edit does not restart KSM (the chart sets no checksum annotation over
+            # the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop
+            # it and a future change to the flux collector set ships green and never takes effect.
+            r"homelab\.keithjackson\.dev/crs-config-rollout",
+        ],
+        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
+        "why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no "
+               "revision label anywhere (measured 2026-09-29) and the Flux event stream emits "
+               "ReconciliationSucceeded on every interval tick whether or not anything changed, so "
+               "without these metrics nothing can tell a deploy starting from an idle cluster",
+    },
     {
         "owner": "kube-prometheus-stack: bundled kube-state-metrics",
         "prefixes": ("kube_",),

conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/chainsaw-test.yaml

@@ -0,0 +1,152 @@
+# =============================================================================
+# L0 STATIC — no two oauth2-proxy instances can share a browser cookie
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. Every oauth2-proxy on this platform is an INDEPENDENT instance with
+# its OWN `cookie-secret`: five per-app reverse-proxy sidecars (jackett, pihole, qbittorrent,
+# radarr, sonarr), crafty's, and the SHARED forward-auth proxy. Measured on prod 2026-09-30:
+# seven `*-oauth2-proxy` Secrets, SEVEN DISTINCT cookie-secret values — so no instance can
+# decrypt any other's cookie.
+#
+# They were all emitting oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and
+# `_oauth2_proxy_csrf`), scoped to `.${BASE_DOMAIN}`. A cookie is keyed by (name, domain, path),
+# so the browser kept ONE pair for ALL of them. Whichever proxy authenticated last OVERWROTE
+# every other's session; the loser logged `No valid authentication in request`, bounced to
+# Keycloak, and — the SSO session still being live — was redirected straight back and overwrote
+# the cookie again. Six live instances, one cookie, mutual eviction.
+#
+# WHY IT IS THE WORST KIND OF BUG. The symptom is "I signed into Keycloak but I'm still not
+# signed in", which points the investigator at Keycloak, where nothing is wrong. It is
+# INTERMITTENT — it only appears once a browser has visited a SECOND gated app, so the obvious
+# single-app reproduction succeeds — and it is SILENT: every Deployment stays Ready, every
+# Kustomization stays Ready, the proxy's own log line names an authentication failure rather
+# than a collision. It went unnoticed across five services for the whole life of the pattern,
+# and was found only as a side effect of investigating something else (2026-09-29, crafty).
+#
+# WHY A COMMIT IS NOT ENOUGH, AND THIS IS A GATE STAGE. The collision is a property of the
+# PATTERN, not of any one app: the next forward-auth service onboarded inherits the defect by
+# doing nothing at all, because a missing flag IS the bug. Nothing about its absence is loud.
+# This is the same shape as the twelve inert ServiceMonitors in this tree — one omission,
+# copied, invisible for weeks — and the same remedy: make the omission fail the merge.
+#
+# GIVEN every oauth2-proxy instance this repo authors — a container in any manifest (the
+#       rendered sidecars in apps/, crafty's hand-authored one in platform/) and the shared
+#       proxy's HelmRelease values,
+# WHEN  each instance's EFFECTIVE cookie name is resolved (its declared `--cookie-name` /
+#       `OAUTH2_PROXY_COOKIE_NAME`, or oauth2-proxy's built-in default `_oauth2_proxy` when it
+#       declares none — an omission is a NAME, not an absence, and that is the whole bug),
+# THEN  (1) no two instances share an effective cookie name;
+#       (2) no instance's effective name is another's name + `_csrf` — oauth2-proxy derives the
+#           CSRF cookie that way, so `_oauth2_proxy` and `_oauth2_proxy_csrf` as two SESSION
+#           names would collide on the second cookie while looking distinct on the first;
+#       (3) the DEFAULT name has exactly ONE owner and it is the instance declared in
+#           SHARED_DEFAULT_COOKIE_OWNER. This is the deliberate ASYMMETRY. The shared
+#           forward-auth proxy MUST keep `_oauth2_proxy`: it is the single session every
+#           OTHER forward-auth-gated surface on the platform rides on, so renaming it signs
+#           every one of them out at once. The rule is therefore NOT "every instance must
+#           declare a name" — it is "the default is a NAME LIKE ANY OTHER, held by exactly one
+#           REGISTERED owner". That formulation needs no exemption from (1), and it closes the
+#           hole an exemption would leave: if the shared proxy is ever removed, an app silently
+#           inheriting the default would still FAIL here instead of quietly becoming its heir;
+#       (4) every instance that is NOT that owner is HOST-SCOPED: its `--cookie-domain` equals
+#           the host of its own `--redirect-url`. See the argument below;
+#       (5) catalog/render.py DERIVES both flags from the ServiceDefinition (the service name
+#           and the route host) and does not carry a per-service table. A list is how twelve
+#           ServiceMonitors acquired one identical wrong port name in a single edit, and a list
+#           leaves the NEXT app defaulting into the collision;
+#       (6) the SHIPPED renderer is proven END-TO-END on synthetic ServiceDefinitions, run as a
+#           subprocess: two forward-auth fixtures must yield two different, non-default cookie
+#           names, each scoped to its own host. Without this, once the tree is compliant (1)-(4)
+#           are satisfied by a tree with nothing left to catch, and a guard whose only evidence
+#           is "nothing failed" cannot be told apart from one that stopped working;
+#   AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope:
+#           whole-repo`, so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into
+#           EVERY gate run, and ci/gate.sh blocks the merge when the l0 stage fails. It
+#           deliberately adds NO stage to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired
+#           fails the gate if anyone adds one.
+#
+# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh maps `apps/*` to R-IDP-02/R-IDP-06
+# today, but the instances live in THREE unrelated subtrees (apps/, platform/crafty/,
+# platform/oauth2-proxy/) and the file most likely to reintroduce the defect is
+# catalog/render.py, which maps to R-GITOPS-02 and to no identity requirement at all. A
+# requirement-scoped selection would filter this check out of exactly the change that breaks it.
+#
+# ON ASSERTION (4), HOST-SCOPED COOKIE-DOMAIN — the argument, and the decision.
+#   AGAINST asserting it (overreach): distinct NAMES already close the eviction bug completely.
+#   Two cookies with different names do not overwrite each other however wide their domains are,
+#   so (4) fixes nothing that (1) has not already fixed. `--cookie-domain` is also a legitimate
+#   knob — a future proxy fronting two hostnames of one app would need a wider value — and a
+#   check that fails correct configurations is a check someone eventually disables.
+#   FOR asserting it (and this wins): a zone-wide cookie is TRANSMITTED BY THE BROWSER TO EVERY
+#   HOST IN THE ZONE. qBittorrent's session token is therefore sent to pihole, to radarr, to
+#   sonarr, and to every other name under `.${BASE_DOMAIN}` — including hosts behind no proxy at
+#   all. That is a session-credential disclosure across trust boundaries that no one needs and
+#   nobody asked for; the sidecar's session is only ever meaningful to the one app it fronts.
+#   And it is what makes a FUTURE name collision catastrophic rather than contained: with
+#   host-scoped cookies, two instances that accidentally share a name cannot even see each
+#   other's cookie, so the failure this whole test exists to prevent degrades from platform-wide
+#   to nothing. Defence in depth on the exact axis that already failed once.
+#   DECISION: assert it, for every instance except the registered shared proxy — whose whole
+#   job IS cross-host forward-auth, and which is exempted by name and with a reason, not by
+#   loosening the rule. `--whitelist-domain` is deliberately NOT touched: it governs permitted
+#   post-login REDIRECT targets, not cookie scope, and narrowing it breaks the sign-in bounce.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py
+# and the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it
+# directly, and this Chainsaw Test wraps the SAME script.
+#
+# NOT proven (honest gaps; this is STATIC, with no cluster and no browser):
+#   - That anyone can actually sign in. No browser round-trip is performed, and none can be
+#     from L0. The 2026-09-29 crafty entry recorded the same gap and it is still open: the
+#     symptom has never been REPRODUCED in a browser, so this is a latent-bug fix reasoned from
+#     measured configuration, not a fix watched to work.
+#   - That the LIVE pods carry what the tree says. git is not the cluster (CLAUDE.md §2b). The
+#     live flags were measured separately, once, by hand; nothing here re-measures them, and a
+#     hand-applied drift would not be visible to this check.
+#   - That the cookie-secrets really differ. That is a Secret comparison and belongs in-cluster;
+#     it was measured out of band (seven distinct values, compared without printing any of
+#     them). If they were ever made IDENTICAL the collision would become harmless — but sharing
+#     one session-encryption key across trust zones is a worse idea than the bug it would mask,
+#     so this check does not offer it as an alternative and does not look for it.
+#   - Anything about the CONTENT of a session, group authorization, or the CSRF cookie's own
+#     behaviour. Names and domains only.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: oauth2-proxy-cookie-names-distinct
+  labels:
+    conformance.home.k3s/R-IDP-02: "true"
+    conformance.home.k3s/R-IDP-06: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-02, R-IDP-06"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — the instances span apps/wave-0, apps/wave-1, platform/crafty and
+    # platform/oauth2-proxy, and the generator that emits five of them is catalog/render.py.
+    # No single requirement path-maps to that set, and the renderer maps to no identity
+    # requirement at all, so a requirement-scoped selection would skip this check on precisely
+    # the change that can reintroduce the defect fleet-wide in one edit.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      No two oauth2-proxy instances can share a browser cookie: every instance's effective
+      cookie name is unique, the built-in default belongs to exactly one registered owner (the
+      shared forward-auth proxy, which cannot be renamed without signing every gated app out),
+      every other instance is host-scoped, and catalog/render.py derives both flags from the
+      ServiceDefinition rather than from a per-service list — proven end-to-end on synthetic
+      fixtures so the guard stays falsifiable once the tree is clean.
+spec:
+  steps:
+    - name: assert-no-two-oauth2-proxy-instances-share-a-cookie
+      description: >-
+        Inventory every oauth2-proxy container and HelmRelease in the tree, resolve each one's
+        effective cookie name (declared, or oauth2-proxy's default when omitted), assert
+        pairwise distinctness including the session/CSRF derivation, ratchet the single
+        registered owner of the default name, assert host-scoped cookie-domain on every other
+        instance, assert catalog/render.py derives rather than lists, and prove the shipped
+        renderer on two synthetic forward-auth ServiceDefinitions.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/check.sh

@@ -0,0 +1,498 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — no two oauth2-proxy instances can share a browser cookie.
+# See chainsaw-test.yaml's header for WHY this exists, what it measured, and the argument
+# for assertion (4). In one paragraph:
+#
+# Every oauth2-proxy here holds its OWN cookie-secret (seven distinct values on prod,
+# 2026-09-30), so two instances sharing a cookie NAME on a shared DOMAIN cannot read each
+# other's session — they OVERWRITE it. The loser bounces to Keycloak, is redirected straight
+# back by the live SSO session, and overwrites the winner in turn. The symptom is "I signed
+# into Keycloak but I'm still not signed in", it only shows up after a second gated app is
+# visited, and nothing in the cluster reports an error. A missing --cookie-name IS the bug,
+# so the next forward-auth app inherits it by doing nothing.
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner
+# turns into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope)
+# only if the tree contains no oauth2-proxy instance at all — genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
+# Matches the IMAGE every instance runs and the CHART the shared one installs; if NEITHER
+# appears anywhere, the forward-auth pattern has left this repo and the check has no subject.
+#
+# ANCHORED ON `image:` DELIBERATELY. A bare `oauth2-proxy/oauth2-proxy:` also matches
+# platform/oauth2-proxy/forward-auth-cec.yaml's Envoy cluster reference
+# (`serviceName: oauth2-proxy/oauth2-proxy:80` — namespace/service:port, not a registry path).
+# That file configures the ext_authz listener that CALLS the proxy; it runs no proxy and holds
+# no cookie flags, so counting it made the grep/parse cross-check below report a phantom
+# unaudited instance.
+if ! grep -rlE 'image:[[:space:]]*[^[:space:]]*oauth2-proxy|chart:[[:space:]]*oauth2-proxy' \
+       "${ROOT}/apps" "${ROOT}/platform" "${ROOT}/catalog" >/dev/null 2>&1; then
+  echo "check SKIPPED: no oauth2-proxy image or chart found under apps/, platform/ or catalog/ — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: no two oauth2-proxy instances can share a browser cookie.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather
+than line-based parsing: a container's args live in a nested structure and a HelmRelease's
+flags live in a values map, and a line-based reader of either is precisely the approximate
+parser that silently stops matching — the failure class this check exists to prevent.
+"""
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# oauth2-proxy's OWN built-in defaults (pkg/apis/options/cookie.go): the session cookie is
+# `_oauth2_proxy` and the CSRF cookie is that name with `_csrf` appended. An instance that
+# declares nothing is NOT unconfigured — it has claimed this exact name, which is why the
+# inventory below treats the default as a name like any other rather than as an absence.
+DEFAULT_COOKIE_NAME = "_oauth2_proxy"
+CSRF_SUFFIX = "_csrf"
+
+# =============================================================================================
+# REGISTER — SHARED_DEFAULT_COOKIE_OWNER. Exactly one entry, and it must match a real instance.
+#
+# The one instance permitted to hold DEFAULT_COOKIE_NAME, and the one permitted a zone-wide
+# cookie-domain. Both exemptions belong to the SAME object for the SAME reason, so they are one
+# register rather than two: the shared forward-auth proxy is not a per-app sidecar, it is the
+# single session that every forward-auth-gated surface on the platform rides on. Renaming its
+# cookie signs all of them out simultaneously, and host-scoping it would break the cross-host
+# forward-auth that is its entire job.
+#
+# Shrink-only, and exact in BOTH directions: an entry that matches no instance FAILS as stale
+# (the proxy was renamed or removed and the exemption is now a hole), and an instance holding
+# the default WITHOUT an entry FAILS as an unregistered heir. Adding an entry is a review event
+# on a sensitive path.
+#
+# Keyed by "<relpath>:<instance id>".
+# =============================================================================================
+SHARED_DEFAULT_COOKIE_OWNER: dict[str, str] = {
+    "platform/oauth2-proxy/helmrelease.yaml:oauth2-proxy":
+        "The SHARED forward-auth proxy (ADR-0014, platform/oauth2-proxy/README.md). It is the "
+        "single authenticator behind every forward-auth-gated surface that is NOT fronted by "
+        "its own sidecar, so its cookie is the platform session: renaming it signs every one "
+        "of them out at once, and that is why the 2026-09-29 crafty fix changed only the "
+        "sidecar side. Its cookie-domain is zone-wide for the same structural reason — "
+        "cross-host forward-auth is what it is for.",
+}
+
+
+def is_test_fixture(p: Path) -> bool:
+    """conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests.
+
+    Auditing a golden would be auditing a copy of something already audited through apps/.
+    Same exclusion the sibling l0-static checks make, and the same "reference, not
+    declaration" distinction ci/touched-requirements.sh draws for fixture paths.
+    """
+    parts = p.parts
+    return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
+
+
+def iter_containers(node):
+    """Yield every container mapping under any `containers`/`initContainers` list, at any depth.
+
+    Depth-agnostic on purpose: a sidecar sits under spec.template.spec.containers in a
+    Deployment, but the same container shape appears under a CronJob's jobTemplate, a
+    DaemonSet, or a Pod, and a reader that hardcoded one path would silently stop seeing an
+    instance the day one moved.
+    """
+    if isinstance(node, dict):
+        for key, val in node.items():
+            if key in ("containers", "initContainers") and isinstance(val, list):
+                for c in val:
+                    if isinstance(c, dict):
+                        yield c
+            else:
+                yield from iter_containers(val)
+    elif isinstance(node, list):
+        for item in node:
+            yield from iter_containers(item)
+
+
+def flag_from_args(args: list, flag: str):
+    """Value of `--flag=value` or `--flag value` in an argv list, else None."""
+    pref = "--" + flag
+    for i, a in enumerate(args):
+        if not isinstance(a, str):
+            continue
+        if a.startswith(pref + "="):
+            return a.split("=", 1)[1]
+        if a == pref and i + 1 < len(args) and isinstance(args[i + 1], str):
+            return args[i + 1]
+    return None
+
+
+def host_of(url):
+    """Host part of an https://host/path URL, else None."""
+    if not isinstance(url, str):
+        return None
+    m = re.match(r"^https?://([^/]+)", url)
+    return m.group(1) if m else None
+
+
+# ============================================================ (0) INVENTORY
+# instances: list of dicts {rel, ident, kind, declared_name, cookie_domain, redirect_host}
+instances: list[dict] = []
+unparseable: list[str] = []
+# Files the grep gate says contain an instance, so the parse can be cross-checked against it.
+grep_files: set[str] = set()
+
+for p in sorted(ROOT.rglob("*.yaml")):
+    relparts = p.relative_to(ROOT).parent.parts
+    # NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES
+    # such as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent
+    # runs, so a bare rglob audited up to 18 duplicates of every file — which both invents
+    # FAILs and, worse, can satisfy a "something declares X" check from a copy rather than from
+    # real source. Same fix as egress-paths-selected-by-rules. .claude/ is gitignored.
+    if any(part.startswith(".") for part in relparts) or is_test_fixture(p) or ".sops." in p.name:
+        continue
+    txt = read(p)
+    if not txt:
+        continue
+    rel = str(p.relative_to(ROOT))
+    # Same anchoring as check.sh's skip gate: `image:` (or a chart name), never a bare path —
+    # see the comment there for the Envoy `serviceName: oauth2-proxy/oauth2-proxy:80` near-miss.
+    if re.search(r"image:\s*\S*oauth2-proxy|chart:\s*oauth2-proxy", txt):
+        grep_files.add(rel)
+    elif "oauth2-proxy" not in txt:
+        continue
+    try:
+        docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
+    except yaml.YAMLError as exc:
+        unparseable.append(f"{rel} ({exc})")
+        continue
+    for d in docs:
+        # --- shape 1: a container running the oauth2-proxy image -------------------------
+        for c in iter_containers(d):
+            image = str(c.get("image") or "")
+            name = str(c.get("name") or "")
+            if "oauth2-proxy" not in image and name != "oauth2-proxy":
+                continue
+            args = [a for a in (list(c.get("args") or []) + list(c.get("command") or []))]
+            env = {e.get("name"): e.get("value")
+                   for e in (c.get("env") or []) if isinstance(e, dict) and "value" in e}
+            owner = (d.get("metadata") or {}).get("name") or "?"
+            instances.append({
+                "rel": rel,
+                "ident": f"{owner}/{name or 'oauth2-proxy'}",
+                "kind": "container",
+                "declared_name": flag_from_args(args, "cookie-name")
+                                 or env.get("OAUTH2_PROXY_COOKIE_NAME"),
+                "cookie_domain": flag_from_args(args, "cookie-domain")
+                                 or env.get("OAUTH2_PROXY_COOKIE_DOMAIN"),
+                "redirect_host": host_of(flag_from_args(args, "redirect-url")
+                                         or env.get("OAUTH2_PROXY_REDIRECT_URL")),
+            })
+        # --- shape 2: the shared proxy's HelmRelease values ------------------------------
+        if d.get("kind") != "HelmRelease":
+            continue
+        spec = d.get("spec") or {}
+        chart = (((spec.get("chart") or {}).get("spec") or {}).get("chart") or "")
+        if "oauth2-proxy" not in str(chart):
+            continue
+        values = spec.get("values") or {}
+        extra = values.get("extraArgs") or {}
+        if isinstance(extra, list):          # the chart also accepts a bare argv list
+            extra = {a.lstrip("-").split("=", 1)[0]: a.split("=", 1)[1]
+                     for a in extra if isinstance(a, str) and "=" in a}
+        cfg = values.get("config") or {}
+        get = lambda k: extra.get(k, cfg.get(k))     # noqa: E731 - two-source lookup, used 3x
+        instances.append({
+            "rel": rel,
+            "ident": str((d.get("metadata") or {}).get("name") or "oauth2-proxy"),
+            "kind": "helmrelease",
+            "declared_name": get("cookie-name"),
+            "cookie_domain": get("cookie-domain"),
+            "redirect_host": host_of(get("redirect-url")),
+        })
+
+for u in unparseable:
+    check(False, f"{u} is not parseable YAML — every oauth2-proxy instance in it went unaudited")
+
+for ins in instances:
+    ins["key"] = f"{ins['rel']}:{ins['ident']}"
+    # THE CORE MODELLING DECISION: an omitted --cookie-name is not "unset", it is a CLAIM on
+    # oauth2-proxy's built-in default. Resolving it here is what lets assertion (1) below be a
+    # plain distinctness test with no special case, and it is exactly the step whose absence
+    # made the defect invisible to every reviewer who read these manifests.
+    ins["effective_name"] = ins["declared_name"] or DEFAULT_COOKIE_NAME
+
+# Anti-vacuous. The floor is deliberately low (a real removal should not turn this check red),
+# so the load-bearing guard is the grep/parse CROSS-CHECK beneath it: every file the cheap
+# text scan says holds an instance must yield at least one PARSED instance. A parser that
+# quietly stopped matching would keep the floor and fail this.
+check(len(instances) >= 5,
+      f"only {len(instances)} oauth2-proxy instance(s) parsed out of the tree. This repo "
+      f"authors more than that (five rendered sidecars, crafty's, and the shared proxy), so "
+      f"the inventory has stopped seeing instances and every assertion below would pass "
+      f"vacuously while a collision went unaudited")
+
+parsed_files = {ins["rel"] for ins in instances}
+for rel in sorted(grep_files - parsed_files):
+    check(False,
+          f"{rel} contains an oauth2-proxy image or chart reference but NO instance was parsed "
+          f"out of it. Either it uses a shape iter_containers()/the HelmRelease reader does not "
+          f"recognise, or the file stopped parsing. Its cookie configuration is UNAUDITED, which "
+          f"is the same state the whole fleet was in before this check existed")
+
+# ============================================================ (1) DISTINCT EFFECTIVE NAMES
+by_name: dict[str, list[dict]] = {}
+for ins in instances:
+    by_name.setdefault(ins["effective_name"], []).append(ins)
+
+for nm, group in sorted(by_name.items()):
+    if len(group) < 2:
+        continue
+    who = ", ".join(sorted(g["key"] for g in group))
+    default_note = (
+        " Every one of them OMITS --cookie-name, so they are all claiming oauth2-proxy's "
+        "built-in default. Omitting the flag is not neutral: it IS this collision."
+        if nm == DEFAULT_COOKIE_NAME else "")
+    check(False,
+          f"{len(group)} oauth2-proxy instances share the cookie name '{nm}': {who}.{default_note}\n"
+          f"       WHY THIS IS A BUG AND NOT A STYLE RULE: these are INDEPENDENT instances with "
+          f"DIFFERENT cookie-secrets (measured on prod 2026-09-30: seven distinct values), so "
+          f"neither can DECRYPT the other's cookie — it can only OVERWRITE it. A browser keys a "
+          f"cookie by (name, domain, path), so signing into one app destroys the session of every "
+          f"other app sharing that key. The loser logs 'No valid authentication in request', "
+          f"bounces to Keycloak, is redirected straight back because the SSO session is still "
+          f"live, and overwrites the winner in turn. The user sees 'I signed into Keycloak but "
+          f"I'm still not signed in' and goes looking in Keycloak, where nothing is wrong.\n"
+          f"       FIX: give each per-app instance its own --cookie-name. For a service rendered "
+          f"from catalog/, do NOT hand-edit the golden — catalog/render.py derives the flag from "
+          f"the service name; re-render instead (ci/render-golden.sh's header has the canonical "
+          f"command). Exactly one instance may hold '{DEFAULT_COOKIE_NAME}', and it must be the "
+          f"one registered in SHARED_DEFAULT_COOKIE_OWNER.")
+
+# ============================================================ (2) SESSION vs CSRF DERIVATION
+# oauth2-proxy names the CSRF cookie <cookie-name>_csrf. So a SESSION name that happens to equal
+# another instance's name + "_csrf" collides on the CSRF cookie while assertion (1) sees two
+# perfectly distinct strings. This is the near-miss the obvious fix invites: naming the shared
+# proxy's neighbour `_oauth2_proxy_csrf` reads as "clearly different" and is not.
+effective = {ins["effective_name"] for ins in instances}
+for ins in sorted(instances, key=lambda i: i["key"]):
+    base = ins["effective_name"]
+    if not base.endswith(CSRF_SUFFIX):
+        continue
+    victim = base[: -len(CSRF_SUFFIX)]
+    check(victim not in effective,
+          f"{ins['key']} uses the SESSION cookie name '{base}', which is exactly the CSRF cookie "
+          f"oauth2-proxy derives for the instance named '{victim}' (it appends '{CSRF_SUFFIX}'). "
+          f"The two session names look distinct and the CSRF cookies collide, so sign-in breaks "
+          f"with a state-mismatch instead of a session loss — the same bug wearing a different "
+          f"error message. Pick a name that is not another instance's name + '{CSRF_SUFFIX}'")
+
+# ============================================================ (3) THE DEFAULT HAS ONE OWNER
+default_holders = sorted(ins["key"] for ins in instances
+                         if ins["effective_name"] == DEFAULT_COOKIE_NAME)
+registered = set(SHARED_DEFAULT_COOKIE_OWNER)
+
+for key in sorted(registered - set(ins["key"] for ins in instances)):
+    check(False,
+          f"SHARED_DEFAULT_COOKIE_OWNER registers '{key}', but no such oauth2-proxy instance "
+          f"exists in the tree. The register is SHRINK-ONLY and exact: a stale entry is a "
+          f"standing exemption for an object that is gone, so the next instance to inherit the "
+          f"default name would be silently excused. Remove the entry, or restore the instance")
+
+for key in default_holders:
+    check(key in registered,
+          f"{key} holds oauth2-proxy's DEFAULT cookie name '{DEFAULT_COOKIE_NAME}' but is NOT "
+          f"registered in SHARED_DEFAULT_COOKIE_OWNER. Exactly ONE instance may hold the default, "
+          f"and it is the SHARED forward-auth proxy — whose cookie is the platform-wide session "
+          f"every forward-auth-gated surface rides on, so renaming IT signs them all out at once. "
+          f"Every other instance must declare its own --cookie-name. If this instance really is "
+          f"the new shared proxy, register it here with the reason; if it is a per-app sidecar, "
+          f"give it a unique name")
+
+check(len(default_holders) <= 1,
+      f"{len(default_holders)} instances hold the default cookie name: {', '.join(default_holders)}. "
+      f"The default is a NAME, not an absence — it can have exactly one owner")
+
+# ============================================================ (4) HOST-SCOPED COOKIE DOMAIN
+# See the chainsaw-test header for the argument (distinct names alone already fix the eviction;
+# host-scoping is asserted anyway because a zone-wide cookie is TRANSMITTED to every host in the
+# zone, and because it makes a future name collision contained instead of platform-wide).
+for ins in sorted(instances, key=lambda i: i["key"]):
+    if ins["key"] in registered:
+        continue                          # the shared proxy: zone-wide by design, registered
+    dom, rhost = ins["cookie_domain"], ins["redirect_host"]
+    check(dom is not None,
+          f"{ins['key']} declares no --cookie-domain. oauth2-proxy then scopes the cookie to the "
+          f"request host, which happens to be correct — but it is correct BY ACCIDENT and says "
+          f"nothing to the next reader. Declare it explicitly as this app's own host")
+    if dom is None:
+        continue
+    check(not dom.startswith("."),
+          f"{ins['key']} sets --cookie-domain={dom}. A leading dot makes the cookie ZONE-WIDE: "
+          f"the browser sends this app's session token to every other host under that domain, "
+          f"including hosts behind no proxy at all, and it is what turns any future cookie-name "
+          f"collision into a platform-wide outage instead of a contained one. Scope it to this "
+          f"app's own host. Only the registered shared forward-auth proxy may be zone-wide, "
+          f"because cross-host forward-auth is its job")
+    if rhost:
+        check(dom == rhost,
+              f"{ins['key']} sets --cookie-domain={dom} but its --redirect-url host is {rhost}. "
+              f"The cookie is then set for a domain the sign-in flow never returns to (or for a "
+              f"wider one than this app owns), so the session either does not stick or leaks "
+              f"beyond the app. Both flags describe the SAME app: make them agree")
+
+# ============================================================ (5) THE RENDERER DERIVES
+# Five of these instances are GENERATED. Auditing only the goldens would let someone "fix" the
+# tree by hand and leave the generator emitting the defect for the next service onboarded — and
+# the next re-render would then silently revert the hand fix.
+render_py = ROOT / "catalog" / "render.py"
+check(render_py.is_file(),
+      "catalog/render.py is missing — the generator of five of these instances cannot be audited")
+if render_py.is_file():
+    src = read(render_py)
+    m = re.search(r"def _oauth2_sidecar_container\(.*?\n(?=\n*def |\n*# =)", src, re.S)
+    fn = m.group(0) if m else ""
+    check(bool(fn),
+          "could not locate _oauth2_sidecar_container() in catalog/render.py. It was renamed or "
+          "removed; assertions (5) below cannot run, and the generator is unaudited")
+    if fn:
+        cookie_name_lines = [ln for ln in fn.splitlines() if "--cookie-name" in ln]
+        check(bool(cookie_name_lines),
+              "catalog/render.py's _oauth2_sidecar_container() emits NO --cookie-name at all, so "
+              "every service it renders claims oauth2-proxy's default and collides with every "
+              "other. This is the defect itself, back in the one place that reintroduces it "
+              "fleet-wide in a single edit")
+        check(any("s['name']" in ln or 's["name"]' in ln for ln in cookie_name_lines),
+              "catalog/render.py emits --cookie-name without DERIVING it from the "
+              "ServiceDefinition's own name. A constant collides with every sibling by "
+              "construction; a per-service table leaves the NEXT app onboarded defaulting into "
+              "the collision, which is how twelve ServiceMonitors acquired one identical wrong "
+              "port name in a single edit. Derive it from s['name']")
+        cookie_dom_lines = [ln for ln in fn.splitlines() if "--cookie-domain" in ln]
+        check(any("{host}" in ln for ln in cookie_dom_lines),
+              "catalog/render.py's sidecar --cookie-domain is not derived from the route host. A "
+              "zone-wide literal (`.{ZONE}`) broadcasts every app's session cookie across the "
+              "whole zone — see assertion (4)")
+        check(not any(re.search(r"--cookie-name=_oauth2_proxy_[a-z0-9]", ln) for ln in cookie_name_lines),
+              "catalog/render.py hardcodes a literal service name into --cookie-name. Whatever "
+              "service that is, every OTHER service now renders the same constant and they all "
+              "collide. Derive the name instead")
+
+# ============================================================ (6) PROVE THE SHIPPED RENDERER
+# Once the tree is compliant, (1)-(4) have nothing left to catch and their passing is evidence
+# of nothing. This runs the SHIPPED renderer as a SUBPROCESS on two synthetic ServiceDefinitions
+# and asserts the property directly, so the guard stays falsifiable.
+def _sd(name: str, port: int) -> str:
+    return yaml.safe_dump({
+        "name": name,
+        "kind": "container",
+        "source": {"image": "example/app:1"},
+        "exposure": "internal",
+        "ingressRoutes": [{"host": f"{name}.home.keithjackson.dev", "port": port}],
+        "auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
+    }, sort_keys=False)
+
+
+def _sidecar_args(rendered: str) -> list:
+    for doc in yaml.safe_load_all(rendered):
+        if not isinstance(doc, dict) or doc.get("kind") != "Deployment":
+            continue
+        for c in iter_containers(doc):
+            if c.get("name") == "oauth2-proxy":
+                return list(c.get("args") or [])
+    return []
+
+
+if render_py.is_file():
+    fixtures = {"cookiefixa": 8111, "cookiefixb": 8112}
+    rendered_args: dict[str, list] = {}
+    with tempfile.TemporaryDirectory() as td:
+        for svc, port in fixtures.items():
+            f = Path(td) / f"{svc}.yaml"
+            f.write_text(_sd(svc, port))
+            proc = subprocess.run([sys.executable, str(render_py), str(f)],
+                                  capture_output=True, text=True)
+            check(proc.returncode == 0,
+                  f"the shipped catalog/render.py failed on a synthetic forward-auth fixture "
+                  f"({svc}): rc={proc.returncode} {proc.stderr.strip()[:200]}")
+            if proc.returncode == 0:
+                rendered_args[svc] = _sidecar_args(proc.stdout)
+
+    names, domains = {}, {}
+    for svc, args in rendered_args.items():
+        names[svc] = flag_from_args(args, "cookie-name")
+        domains[svc] = flag_from_args(args, "cookie-domain")
+        check(bool(args),
+              f"the shipped renderer produced no oauth2-proxy sidecar for synthetic forward-auth "
+              f"fixture {svc} — the sidecar is how a forward-auth app is gated at all (R-IDP-02)")
+        check(names[svc] not in (None, DEFAULT_COOKIE_NAME),
+              f"the shipped renderer gave synthetic fixture {svc} the cookie name "
+              f"{names[svc]!r}. A newly-onboarded forward-auth app would therefore default "
+              f"straight into the fleet collision — which is exactly how five services acquired "
+              f"this defect without anyone making a decision")
+        check(domains[svc] == f"{svc}.home.keithjackson.dev",
+              f"the shipped renderer scoped synthetic fixture {svc}'s cookie to "
+              f"{domains[svc]!r} rather than to its own host — a newly-onboarded app would "
+              f"broadcast its session cookie across the whole zone")
+        check(any(isinstance(a, str) and a.startswith("--whitelist-domain=.") for a in args),
+              f"synthetic fixture {svc} lost its ZONE-WIDE --whitelist-domain. That flag governs "
+              f"permitted post-login REDIRECT targets, not cookie scope; narrowing it with the "
+              f"cookie domain breaks the sign-in bounce")
+
+    if len(rendered_args) == 2:
+        a, b = sorted(rendered_args)
+        check(names[a] != names[b],
+              f"the shipped renderer gave two DIFFERENT synthetic services the SAME cookie name "
+              f"({names[a]!r}). Whatever the committed goldens currently look like, the generator "
+              f"will collide every forward-auth app it renders from now on")
+        check(domains[a] != domains[b],
+              f"the shipped renderer gave two different synthetic services the same cookie domain "
+              f"({domains[a]!r}) — the cookies would share a scope even with distinct names")
+
+# ============================================================ VERDICT
+inv = " | ".join(f"{i['key']} -> {i['effective_name']} @ {i['cookie_domain'] or '<host>'}"
+                 for i in sorted(instances, key=lambda x: x["key"]))
+print(f"oauth2-proxy instances audited ({len(instances)}): {inv}")
+if FAILURES:
+    print(f"\nCHECK FAIL (oauth2-proxy-cookie-names-distinct): {len(FAILURES)} failed, "
+          f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  FAIL {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: oauth2-proxy-cookie-names-distinct — {CHECKS} assertions (R-IDP-02, R-IDP-06)")
+PY

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -76,11 +76,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_paas-demo
+        - --cookie-domain=paas-demo.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -101,6 +104,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -99,11 +99,14 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +127,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -99,11 +99,14 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +127,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

decisions/2026-09-29-flux-deploy-notifications.md

@@ -0,0 +1,144 @@
+---
+date: 2026-09-29
+branch: feat/flux-deploy-notifications
+title: Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes
+---
+
+## What changed
+
+The operator was getting **no notification when Flux began applying a git change, and none when it
+finished**. Root cause, measured rather than inferred: `notification-controller` v1.4.0 is running
+(`1/1` ready) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3`, but
+there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get
+providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*`
+rules in `platform/monitoring/alerts.yaml` only cover *failures* (`FluxReconciliationFailure`,
+`FluxResourceSuspended`) — nothing covered the healthy path.
+
+**The obvious fix does not work, and this is the substance of the change.** Neither Flux event can be
+filtered into a deploy signal:
+
+- `ReconciliationSucceeded` is a **heartbeat, not a change signal**. 16 Kustomizations, 15 on
+  `interval: 10m` → **101 such events in one hour**, with the message `Reconciliation finished in
+  324.161192ms, next run in 10m0s` — byte-identical in shape whether the apply changed anything or
+  not. An `Alert` at `eventSeverity: info` would have mailed ~100 times an hour.
+- `Progressing` is worse. Five Kustomizations re-apply the same object on **every** tick — event
+  counts **2505 / 2504 / 2501 / 2495 / 2490** (`apps-vms` → `NetworkPolicy/unifi/default-deny`,
+  `platform` → `CustomResourceDefinition/cdis.cdi.kubevirt.io`, `apps-wave-0/1/2` →
+  `NetworkPolicy/{gitea,nextcloud,seerr}/default-deny`). Anything keyed on `Progressing` would mail
+  five times every ten minutes, forever.
+- `gotk_*` carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series,
+  `gotk_suspend_status` 40, not one with a revision.
+
+The applied **revision** is the only signal that moves when and only when git moves, and it lives in
+object status alone. So:
+
+1. **`platform/monitoring/kube-prometheus-stack.yaml`** — two resources added to the *existing*
+   `kube-state-metrics.customResourceState` config (the Velero-Schedule path, already enabled):
+   `kube_flux_kustomization_info{revision}` + `kube_flux_kustomization_suspended` (from
+   `status.lastAppliedRevision` / `spec.suspend`) and `kube_flux_gitrepository_info{revision}` (from
+   `status.artifact.revision`). Plus the two mandatory read-only `rbac.extraRules` entries, and a
+   `podAnnotations` roll-counter — the chart sets **no checksum annotation** over that ConfigMap, so
+   a config-only edit would have left KSM running the old collector set with the new metrics absent
+   and the new alerts silently inert. Named `kube_flux_*`, never `gotk_*`, for the same reason
+   `kube_velero_schedule_*` is not `velero_*`: these are KSM series about Kubernetes objects.
+   Cost: 33 series.
+2. **`platform/monitoring/alerts.yaml`**, group `platform-health.flux` — `FluxRolloutInProgress`
+   (`severity: info`, **no `for:`**) fires the moment any non-suspended Kustomization's applied
+   revision is not the revision the GitRepository fetched from `main`, and **clears when all have
+   caught up**. Fired mail = "a deploy started"; **resolved mail = "it completed"** — the completion
+   half is delivered by `send_resolved`, not by a second rule. `FluxRolloutStuck` (`severity:
+   warning`, `for: 30m`) is the same expression as the escalation for a rollout that started and
+   never converged.
+3. **Alertmanager** — a dedicated `severity = "info"` route and `email-deploy` receiver, subject
+   `[FLUX] applying|applied — …`. Without it a routine deploy would have arrived subject-lined
+   `[WARNING]` via the catch-all receiver and become indistinguishable from a real fault.
+   `group_by: [alertname]` collapses all 16 Kustomizations into one mail per rollout;
+   `group_interval: 2m` so completion is not held 4m behind the event.
+4. **`platform/monitoring/tests/flux-rollout.test.yaml`** — 6 promtool scenarios, both directions.
+5. **`runbooks/flux-rollout.md`** (+ its `configMapGenerator` entry) and a `kube_flux_` entry in
+   `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` whose `require` patterns
+   pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so
+   deleting any of them fails the gate instead of silently re-inerting the alerts.
+
+Two guards in the expression, each with a test whose only job is to fail if that clause is removed:
+`and on () (count(kube_flux_gitrepository_info…) > 0)` (without it, a missing exporter makes
+`unless on (revision) <empty>` remove nothing and **all 16 Kustomizations alert forever** — the
+permanently-firing mode this repo shipped once already), and `unless on (namespace, name)
+(kube_flux_kustomization_suspended == 1)` (a suspended Kustomization pins an old revision by design
+and would hold the rollout open, so the "applied" mail would never arrive).
+
+## What I verified
+
+Measured against live prod (`--context k3s-prod`) **before** writing anything, and quoted here as
+measured:
+
+- `notification-controller` `1/1`, image `ghcr.io/fluxcd/notification-controller:v1.4.0`;
+  `alerts`/`providers`/`receivers.notification.toolkit.fluxcd.io` all serve `v1beta3` (storage);
+  `get providers,alerts -A` → **`No resources found`**. This is the defect.
+- Event counts above, from `kubectl get events -A -o json` filtered to `toolkit.fluxcd.io`
+  involvedObjects: 101 `Kustomization/ReconciliationSucceeded`, 10 `Progressing`, 15
+  `DependencyNotReady`, 4 `GitRepository/GitOperationSucceeded`, 1 `NewArtifact`.
+- The `gotk_*` family **is** in the TSDB now (`flux-podmonitor` works): 6 targets up in job
+  `flux-system/flux-system`, `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40
+  (`{kind="Kustomization"}` = 16) — and none carries a revision.
+- `customResourceState` already works end to end: `kube_velero_schedule_{created,paused,
+  protects_volume_data,info}` = 2 series each.
+- Both CRDs serve **`v1` as the storage version**; `GitRepository/flux-system`
+  `.status.artifact.revision` = `main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de`, `spec.interval`
+  `1m`; all 16 Kustomizations report that same `lastAppliedRevision`; **none** sets `spec.suspend`
+  (which is why `nilIsZero: true` is mandatory).
+- Reachability for the follow-up below: `flux-system/allow-egress` is `egress: [{}]` (allow-all) and
+  the `monitoring` namespace has **no NetworkPolicy at all**, so nothing blocks flux-system →
+  `kube-prometheus-stack-alertmanager` (`10.43.60.242:9093`).
+- `promtool test rules` on the new suite: **SUCCESS**, 53 rules parsed, promtool 3.14.0 from the
+  pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that
+  proves the alert empties itself (the completion mail).
+- Full gate in the pinned toolbox: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`,
+  `spec-immutability` all **green** — `l0` includes `alert-metrics-have-scrape-targets`,
+  `alert-rules-unit-tested` and `alert-runbooks-resolvable`, i.e. the new owner entry, the new tests
+  and the new runbook all cleared their own checks. Coverage 51/51.
+
+## What I did NOT verify
+
+- **No mail has been sent.** Nothing here has run on the cluster: the metrics do not exist yet, the
+  rules are not loaded, and the `email-deploy` receiver has never delivered. The end-to-end claim
+  ("a merge produces an `[FLUX] applying` mail and then an `[FLUX] applied` mail") is **UNVERIFIED**
+  and can only be proven by merging this and watching the *next* commit. Agents cannot apply, and
+  CI has no cluster (`vars.CLUSTER_AVAILABLE` unset → L2/L3 did not run).
+- **The KSM custom-resource-state paths are asserted, not observed.** I verified the CRD storage
+  versions and that each status field is populated on the live objects, but I have not seen KSM emit
+  `kube_flux_*`. If a path is wrong KSM emits the metric with the label **omitted**, `revision!=""`
+  then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the
+  first thing to check after merge:
+  `count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names.
+- **KSM memory was not re-measured.** Limit is 384Mi against a working set of ~111–123Mi; this adds
+  two small informers (16 + 1 objects). 91 OOMKills followed the *last* CRS addition at a 128Mi
+  limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])`
+  after this settles rather than assuming the headroom holds.
+- **Alertmanager's `.Status` subject template is untested.** `[FLUX] applying|applied` relies on
+  `.Status` being `firing`/`resolved` in the notification template data. Wrong, and the mail still
+  arrives with a poorer subject — cosmetic, but unproven.
+- **The `notification-controller` `Provider`/`Alert` half is designed but NOT shipped.** It was cut
+  deliberately: its unique value over the rules above is the human-readable **commit subject**
+  (`GitRepository`/`NewArtifact`, count 1 per commit, message `stored artifact for commit '<subject>'`,
+  revision in `source.toolkit.fluxcd.io/revision` metadata), and notification-controller has **no
+  SMTP provider**, so it must post into Alertmanager via `type: alertmanager` at
+  `http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts`. I could not
+  establish **what label set that provider actually emits** (whether it sets `alertname`, and whether
+  `severity` is `info`) without applying it, and the `email-deploy` route matches on `severity`. So
+  the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off
+  Alertmanager, then wire the route — and put it in its **own `wait: false` Flux Kustomization**
+  (never `platform` or `platform-config`, both `wait: true`), because a notification mechanism that
+  cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped
+  cold-start deadlock this repo already documents.
+- **The 2500-count drift loop is NOT fixed, and is not mine.** Five Kustomizations have been
+  re-applying the same object every 10 minutes for ~25 days: something outside git is mutating
+  `NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and
+  `CustomResourceDefinition/cdis.cdi.kubevirt.io` after Flux applies them. Cause **UNVERIFIED** —
+  I only measured the event counts. It is a real defect (constant write load on the API server and
+  etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The
+  alerts here are immune to it by construction: they read revisions, not events.
+- **Branch protection / runner state re-measured? No.** The `gitea-settings` stage **SKIPPED**
+  (`SETTINGS_API_TOKEN` unset), so whether `main` is still protected and the gate still required is
+  UNVERIFIED by this run. The gate's overall verdict is therefore **rc 3, not rc 0** — every stage
+  that ran is green, and one stage did not run.

decisions/2026-09-30-flux-deploy-notification-inhibited.md

@@ -0,0 +1,95 @@
+---
+date: 2026-09-30
+branch: fix/flux-deploy-notification-inhibited
+title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack
+---
+
+## What changed
+
+`FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real
+deploy and **its mail was suppressed before Alertmanager sent it**. One label changed:
+
+- `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and
+  **no `severity` label at all**, where it previously carried `severity: info`.
+- `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches
+  `notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged.
+- `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels`
+  blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected.
+
+**The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules`
+ships:
+
+    InfoInhibitor:
+      group by (namespace) (ALERTS{severity="info"} == 1)
+        unless on (namespace) group by (namespace)
+          (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
+
+which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in
+it**. The chart's third `inhibit_rule` then drops exactly that alert:
+
+    source_matchers: [alertname="InfoInhibitor"]
+    target_matchers: [severity="info"]
+    equal:           [namespace]
+
+For a deploy notification this is the worst possible interaction, because the suppressed case is the
+*only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning
+or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been
+delivered only when something else in `flux-system` was already broken.
+
+Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a
+real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a
+label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping
+`severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working
+around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names
+the severity label as a third thing that must not regress — "do not tidy this by adding a severity"
+— because the next reader's instinct will be to add one back.
+
+## What I verified
+
+Against live prod, after PR #25 merged as `64c72fd` and Flux converged:
+
+- **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16,
+  `kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS
+  populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` /
+  `status.artifact.revision` paths are correct — previously UNVERIFIED.
+- **The `podAnnotations` roll mechanism works.** The KSM pod carries
+  `homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod
+  (`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was
+  genuinely closed rather than merely documented.
+- **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had
+  converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and
+  `revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert
+  **cleared**. Firing and self-clearing are both confirmed on real data, which is the completion
+  notification's whole mechanism.
+- **The routing works.** `amtool config routes` shows
+  `{notification="deploy"} → receiver: email-deploy` under the default route, and
+  `amtool config routes test notification=deploy` returns `email-deploy`.
+- **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment
+  the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in
+  Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}`
+  is 0 for every reason, so nothing failed to send; it was never dispatched.
+- promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox.
+
+## What I did NOT verify
+
+- **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The
+  suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven
+  (`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be
+  proven by merging this and watching the next commit produce `[FLUX] applying` and then
+  `[FLUX] applied`. Same structural gap as PR #25, one layer further along.
+- **Whether the single email during the window was ours.**
+  `alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts
+  (`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver
+  label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the
+  attribution is genuinely unavailable from this metric.
+- **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`;
+  inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not
+  apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs.
+- **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be
+  `firing`/`resolved` in the notification template data.
+- **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection
+  reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the
+  model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report
+  this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only.
+- **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every
+  10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.

decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md

@@ -0,0 +1,378 @@
+---
+date: 2026-09-30
+branch: fix/oauth2-proxy-cookie-collision-fleet
+title: The cookie collision was never Crafty's — it was the renderer's, and it shipped to five apps
+---
+
+## What changed
+
+Follow-up #3 of `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`: *"Any further per-app
+reverse-proxy sidecar needs a unique `--cookie-name` too. The collision is a property of the
+pattern, not of Crafty."*
+
+It is a property of the pattern, and this branch found where the pattern actually lives.
+
+**Read the scope claim carefully before the detail.** This branch fixes THREE defects in the
+forward-auth sidecar contract, all in one function, all landing on all five apps at once:
+
+| Defect | Fix | Measured evidence |
+|---|---|---|
+| Six independent instances, seven different cookie-secrets, **one** cookie name on **one** zone-wide domain — so they overwrite and `ClearSession`-delete each other's sessions | `--cookie-name=_oauth2_proxy_<app>` + host-scoped `--cookie-domain` | 7/7 distinct secrets; 6 instances on the default name; radarr logs the eviction fingerprint 8× in 12h |
+| The session dies **5 minutes** after sign-in — `--cookie-refresh` defaults to 0 = *disabled*, so the refresh token the proxy already holds is never spent | `--cookie-refresh=2m30s` (half the lifetime) | access-token lifetime = **300.000s exactly**, `refresh_token:true`, flag absent |
+| **No** per-app sidecar serves `/metrics` at all — without `--metrics-address` the listener never starts | `--metrics-address=0.0.0.0:44180` + a `metrics` container port | `ports` was `['http']` on all six |
+
+Rejection *volume* is dominated by the second defect, not the first: ~95% of radarr's rejections
+carry no cookie at all. **Do not merge this expecting one number to drop** — three mechanisms were
+tangled together, and separating them is most of the value here.
+
+**Boundary note.** My brief originally reserved `apps/wave-0/jackett.yaml` for a concurrent branch.
+That split is not achievable and has been revoked: the file is generated, the fix is one renderer
+edit, and `render-golden` requires all five goldens regenerated in the *same commit*. I own all
+five. The proof I ran before touching jackett is under "What I verified".
+
+**The defect was not in the four manifests it was reported in.** `apps/wave-0/pihole.yaml`,
+`apps/wave-0/qbittorrent.yaml`, `apps/wave-1/radarr.yaml`, `apps/wave-1/sonarr.yaml` — and
+`apps/wave-0/jackett.yaml` — are **generated goldens**. One function emits all five sidecars:
+`catalog/render.py::_oauth2_sidecar_container()`, which hardcoded `--cookie-domain=.{ZONE}` and
+emitted no `--cookie-name` at all. Hand-editing a golden is not a fix twice over: `ci/render-golden.sh`
+FAILS on the drift (provoked, not assumed — see below), and the next re-render silently reverts it.
+
+So the fix is in the generator, and both flags are now **derived** from the ServiceDefinition:
+
+    --cookie-name=_oauth2_proxy_{s['name']}     unique per service, by construction
+    --cookie-domain={host}                      the app's own route host, not the zone
+
+`--whitelist-domain` deliberately stays zone-wide — it governs permitted post-login *redirect*
+targets, not cookie scope. `catalog/render.py --selftest` gained four assertions, including one
+that renders a **second** synthetic forward-auth service and requires the two cookie names to
+differ, so a future refactor to a constant fails in the renderer's own test.
+
+Re-rendering with the canonical command regenerated **five** goldens, not four. Jackett is the
+fifth, and this is the one thing in this branch the PM brief did not anticipate — see
+"the jackett collision" below. The three `conformance/tests/l1-rendering/*/golden/` fixtures are
+expected-output copies of the same renderer, so they are regenerated too; that is not collateral
+damage, it is the L1 stage doing its job (it FAILED on the first gate run and named the drift).
+
+**The new gate check** is `conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/`,
+declaring `conformance.home.k3s/scope: whole-repo` so `ci/l0.sh`'s `--include-scope` union runs it
+on **every** gate invocation. No bespoke `ci/l0.sh` stage was added — `whole-repo-invariants-wired`
+fails the gate if anyone adds one, and the scope annotation is the supported mechanism.
+
+It inventories every oauth2-proxy instance the repo authors (containers at any depth, plus the
+shared proxy's HelmRelease values) and resolves each one's **effective** cookie name — its declared
+name, **or oauth2-proxy's built-in `_oauth2_proxy` when it declares none**. That single modelling
+decision is the whole check: an omitted flag is not an absence, it is a *claim on a name*, and
+treating it as an absence is precisely why five reviewers read these manifests and saw nothing.
+Then: (1) no two effective names collide; (2) no name is another's name + `_csrf`, because that is
+how oauth2-proxy derives the CSRF cookie and two such session names look distinct while their CSRF
+cookies collide; (3) the default name has exactly one owner, and it is the one registered in
+`SHARED_DEFAULT_COOKIE_OWNER`; (4) every non-registered instance is host-scoped; (5)
+`catalog/render.py` derives rather than lists; (6) the **shipped** renderer is proven end-to-end on
+two synthetic ServiceDefinitions run as a subprocess.
+
+**On the shared proxy — "distinct" or "explicitly declared"?** The brief asked me to decide, and
+the answer is *neither, quite*. The shared forward-auth proxy at `platform/oauth2-proxy/` **must
+keep** `_oauth2_proxy`: it is the single session every forward-auth-gated surface without its own
+sidecar rides on, so renaming it signs all of them out at once. Requiring *every* instance to
+declare a name would therefore need an exemption, and an exemption is a hole: delete the shared
+proxy and whichever app next omitted the flag would silently inherit the default with the rule
+still satisfied. So the assertion is plain **distinctness over effective names**, with the default
+treated as a name like any other — held by **exactly one owner, named in a shrink-only register
+with its reason**. The shared proxy needs no exemption from the rule; it is simply the registered
+holder of that one name. A stale register entry FAILS (the exemption would outlive its object) and
+an unregistered holder FAILS (no silent heirs). Both were provoked and are quoted below.
+
+**On host-scoped `--cookie-domain` — overreach, or not?** Asserted, and here is the argument both
+ways. *Against:* distinct names alone close the eviction bug completely — two differently-named
+cookies do not overwrite each other however wide their domains are — so this assertion fixes
+nothing (1) has not already fixed, and a check that fails a legitimate configuration is a check
+someone eventually disables. *For, and this wins:* a zone-wide cookie is **transmitted by the
+browser to every host in the zone**. qBittorrent's session token was being sent to pihole, radarr,
+sonarr and every other name under the zone, including hosts behind no proxy at all — a
+session-credential disclosure across trust boundaries that nothing needs. And it is what makes a
+*future* name collision catastrophic rather than contained: with host-scoped cookies, two
+instances that accidentally share a name cannot see each other's cookie at all. Defence in depth
+on the exact axis that already failed once. The escape hatch is the same register, by name and
+with a reason, rather than a loosened rule.
+
+### The jackett boundary, and why it could not hold
+
+`apps/wave-0/jackett.yaml` carries the identical defect and was initially assigned to a concurrent
+branch, with an instruction to me not to touch it. **That separation is not achievable** — the
+ownership split was drawn across a generated file — and I did not discover this by reasoning about
+it, I provoked it. With the renderer fixed and jackett's golden left at its `main` content,
+`./ci/render-golden.sh` exits **1**:
+
+    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render:
+    -        - --cookie-domain=.${BASE_DOMAIN}
+    +        - --cookie-name=_oauth2_proxy_jackett
+    +        - --cookie-domain=jackett.${BASE_DOMAIN}
+    [fail] render==golden check FAILED
+
+Jackett's golden is an *output* of the file that contains the defect. There are exactly three
+options and two of them are worse: hand-edit four goldens and leave the generator broken (fails
+`render-golden`, and reverts on the next render); add a per-service carve-out to the renderer so
+jackett alone keeps defaulting (ships the exact "list, don't derive" anti-pattern this check
+exists to forbid, and leaves jackett broken *by design*); or fix the generator and regenerate all
+five. I took the third. Jackett's golden here is **machine output of a one-line generator change**,
+not an independent edit of Mira's file, and it is byte-identical to what her branch must produce.
+
+**Consequence:** the jackett branch has no cookie work left in it — this commit fixes jackett
+identically and by the same mechanism, and both branches independently arrived at the *same two
+values* (`_oauth2_proxy_jackett`, `jackett.${BASE_DOMAIN}`), which is a good sign for the
+diagnosis. Its hand-edit of the rendered file will fail `render-golden` on its own terms, so it
+should be closed rather than rebased. **What must not be lost with it** is its evidence: the
+interleaved radarr/jackett log showing `removing session` deleting a clean session 26s after login,
+and the `expires − created = 300.000s` measurement that produced `--cookie-refresh` here. Both are
+carried into this entry and into the renderer's comments; the branch itself is now redundant, the
+work behind it is not.
+
+**The general lesson, which is worth more than this bug:** an ownership split across
+`apps/wave-*/*.yaml` cannot hold, because those files are *outputs*. Two agents assigned two
+rendered apps are assigned the same function and will collide in `render-golden` no matter how
+carefully they stay in their own files. Ownership of a rendered app is ownership of
+`catalog/render.py`, and briefs should say so.
+
+### What this branch deliberately does NOT do
+
+- **`--api-route=^/api/` is NOT added**, and not because of scope discipline alone: it would be
+  **wrong** renderer-wide. It was measured on jackett, and **PiHole does not serve `/api/`** — so
+  a renderer constant applies one app's path shape to five. Doing it properly means declaring API
+  path prefixes per service, and the `auth` block is `additionalProperties: false`, so that is a
+  *schema change*. Separate PR. (Note the flag is **`--api-route`, singular** — verified in the
+  pinned binary's own `--help`. The plural `--api-routes` does not exist in v7.8.1 and the proxy
+  refuses to start with it. That error was propagating through briefs; it stops here.)
+- **`--session-cookie-minimal` is NOT added** and must not be, as a cookie-size fix: it is
+  **mutually exclusive** with `--cookie-refresh` (*"cookie_refresh > 0 requires oauth tokens in
+  sessions"*). Dropping `--pass-access-token` is a least-privilege argument, never a size one.
+- **Metrics are NOT scraped, only served.** The flag and container port are the prerequisite;
+  a container port is not a Service port. These Services expose the app port and `:4180`, and each
+  namespace's default-deny baseline is `ingress: []`, so Prometheus can neither resolve nor reach
+  `:44180` until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
+  exist. That wiring is someone else's follow-up and I have deliberately not faked its start.
+  **Nobody should read this commit as "the sidecars have metrics now."**
+- **The new check was NOT widened to cover cookie-refresh or metrics.** Its subject is cookie
+  *identity*; those two share neither its inputs nor its failure mode. Bundling them would make
+  the check's name a lie and couple unrelated ratchets, so a future metrics decision could only
+  be expressed by weakening a *cookie* check. The cost of that discipline is named honestly in
+  the follow-ups: those two flags are currently **ungated**.
+
+## What I verified
+
+- **The premise, in-cluster, rather than inherited from the handoff** (CLAUDE.md §2b). Read-only,
+  `kubectl --context k3s-prod`, through the pinned toolbox.
+  - **Seven** `*-oauth2-proxy` Secrets carry a `cookie-secret`: crafty, jackett, pihole,
+    qbittorrent, radarr, sonarr, and the shared `oauth2-proxy-credentials`. **All seven values are
+    DISTINCT.** Compared without printing anything: each value was hashed **in-cluster-side script,
+    never echoed**, and only the first 8 hex characters of each `sha256` were emitted — enough to
+    say same-or-different, recovering nothing. Secret *key names* were listed; no `.data` or
+    `.stringData` was printed (CLAUDE.md §6). **So no instance can decrypt another's cookie — it can
+    only overwrite it. The premise holds.**
+  - **The default cookie names really were in use.** Of 8 live oauth2-proxy containers, **6 declare
+    no `--cookie-name` at all** — jackett, pihole, qbittorrent, radarr, sonarr, and both replicas of
+    the shared proxy — so all six emit `_oauth2_proxy` / `_oauth2_proxy_csrf`. All six also carry
+    `--cookie-domain=.home.keithjackson.dev`. **One cookie pair, six independent instances.**
+  - Crafty's live pod (`crafty-647c7d8cc5-5spv9`) already carries
+    `--cookie-name=_oauth2_proxy_crafty` and `--cookie-domain=crafty.home.keithjackson.dev`, so
+    PR #22 is merged and reconciled. It is the only compliant per-app instance on the cluster.
+- **THE COLLISION IS REAL. IT IS NOT ESTABLISHED AS THE CAUSE OF THE REPORTED FAILURES.** These
+  are two different claims and only the first is supported. I was handed counter-evidence —
+  crafty, the app already fixed, showed the *worst* rejection ratio of the set over a 24h Loki
+  window (`crafty 81:1`, `radarr 67:1`, `jackett 39:4`) — and it is right to refuse the causal
+  claim. What I could do was bound the window. Read from **each sidecar's own log** rather than
+  Loki, because `kubectl logs` on a pod that has not restarted cannot reach a pod older than
+  itself, so a window inside the pod's age is inside the fixed spec's lifetime *by construction*:
+
+      namespace      age(h) rst  reject success    ratio  cleared  badsig
+      crafty           13.9   0       3       0      n/a        0       0   <- FIXED spec, whole life
+      radarr          422.5   0     155       1  155.0:1        8       8
+      jackett         422.6   0      39       4    9.8:1        1       1
+      pihole          395.7   0       0       0      0:0        0       0
+      qbittorrent     404.1   0       0       0      0:0        0       0
+      sonarr          422.5   0       0       0      0:0        0       0
+      oauth2-proxy    422.6   0       0       0      0:0        0       0   (both replicas)
+
+  `reject` = `No valid authentication in request`; `success` = `AuthSuccess`;
+  `cleared` = `removing session` (oauth2-proxy's `ClearSession`); `badsig` = `cookie signature
+  not valid`. Window `--since=24h`, which exceeds every pod's age only for crafty — whose pod is
+  13.9h old with **0 restarts**, so these are its complete post-fix figures.
+
+  Then I read crafty's four auth-relevant lines rather than counting them, because at n=3 the
+  content is the evidence:
+
+      03:57:20  Cookie settings: name:_oauth2_proxy_crafty secure(https):true httponly:true
+                expiry:168h0m0s domains:crafty.home.keithjackson.dev path:/
+      04:02:55  No valid authentication in request. Initiating login.
+      15:22:39  No valid authentication in request. Initiating login.
+      15:22:44  No valid authentication in request. Initiating login.
+
+  6743 log lines; those are all four. **Zero** `removing session`, **zero** `cookie signature
+  not valid`, zero anything else.
+
+  Four conclusions, kept separate:
+  1. **`crafty 81:1` is a window artefact. It is not evidence against the fix and must not be
+     cited as such.** The fixed pod emitted **3** rejections in 13h 52m; the 81 came from a 24h
+     window straddling it, i.e. from the pre-fix process.
+  2. **The eviction FINGERPRINT is gone where the fix is deployed.** `cookie signature not
+     valid` + `removing session` is what a clobbered instance logs — a cookie *is* present,
+     written with a key this instance does not hold, so it cannot decrypt it and then
+     `ClearSession` expires it on its own `--cookie-domain`. Fixed crafty: **0 and 0** across
+     its whole life. Unfixed radarr: **8 and 8** in 12h. Unfixed jackett: 1 and 1. That is a
+     real, directional signal and it is the strongest evidence available that the fix works.
+  3. **It is still not a demonstrated successful sign-in.** Crafty logged **0 `AuthSuccess`** —
+     so what is shown is *absence of the failure mode*, not a session observed working
+     end-to-end. Those three rejections are consistent with benign first-hits (one client, two
+     of them 5s apart, i.e. a page plus a `/ws` reconnect). I am comfortable saying the fix
+     works; I am not willing to say it has been watched working, and the distinction is the
+     whole reason this section exists.
+  4. **The collision is a MINORITY of the rejection volume.** radarr: 8 of 155 (5%) carry the
+     fingerprint; ~95% carry **no cookie at all**. Those are the 5-minute session expiry and
+     `302`-on-XHR — the first of which this branch now also fixes, the second of which it
+     deliberately does not.
+
+  Two cautions about the instrument. `reject` ranks **chattiness, not breakage**: radarr's are
+  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/negotiate` polls, so a busy *arr
+  inflates the count with nobody inconvenienced. And that same polling is the *amplifier* that
+  made this intermittent — every poll carrying an undecryptable cookie calls `ClearSession`
+  again on the zone-wide domain, so **a left-open Radarr tab deleted Jackett's cookie every few
+  seconds, from another namespace, with no user action at all.** That is why signing in again
+  never helped for longer than the gap between two polls. It is now a comment in the renderer,
+  because it is the part of this bug that no one would reconstruct from the flags alone.
+- **The four target files are generated, not authored.** Rendering `catalog/` with the canonical
+  command into a temp dir and diffing against `apps/` reported all 10 onboarded goldens
+  byte-identical before any change. Hand-editing one is drift by definition.
+- **`render-golden` FAILS if jackett is excluded** — provoked (output quoted above), rc **1**.
+  Restoring the regenerated golden returns it to `all 10 onboarded service golden(s) reproduce`,
+  rc **0**.
+- **Every flag verified against the PINNED v7.8.1 binary**, not against docs, memory, or the
+  shared proxy (which runs **v7.15.3** and therefore could not vouch for v7.8.1):
+  `podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help`.
+  - `--cookie-name string ... (default "_oauth2_proxy")` — **this confirms the constant my whole
+    check is built on, from the binary itself.** The check's central claim, that an omitted flag
+    is a *claim on a name* rather than an absence, now rests on the implementation's own help
+    text rather than on my reading of its source.
+  - `--cookie-refresh duration  refresh the cookie after this duration; 0 to disable` — confirms
+    the default really is disabled.
+  - `--metrics-address string` — present in v7.8.1, so the new flag cannot crashloop the pods.
+  - `--api-route strings` — **singular**, confirming the brief's plural was wrong.
+  - `--session-cookie-minimal` — exists, and is the flag that conflicts with `--cookie-refresh`.
+- **44180 is the platform's existing convention, measured, not chosen.** The shared proxy already
+  runs `--metrics-address=0.0.0.0:44180` with a `metrics` container port, a `metrics:44180`
+  Service port, a ServiceMonitor whose endpoint is `metrics`, and **2 ready endpoint addresses** —
+  i.e. it demonstrably works on this platform.
+- **`catalog/render.py --selftest`: PASS**, now including the two-service distinctness assertion,
+  a `--cookie-refresh` assertion that PARSES the duration and requires `0 < refresh < 300s` (so
+  `2m30s` -> `150s` -> `2.5m` all stay green but a silent widening past the token lifetime does
+  not), a `--session-cookie-minimal` mutual-exclusion guard, and both metrics assertions.
+- **The check is selected by the whole-repo union.** `conformance/run.sh --layer l0 --requirements
+  r-nosuchreq-99 --include-scope whole-repo --print-selection` lists
+  `tests/l0-static/oauth2-proxy-cookie-names-distinct` among 20 tests — i.e. it runs even when the
+  touched-requirement filter matches nothing, which is the blind spot it was scoped for.
+- **The check was tested in BOTH directions**, against throwaway trees outside the repo, never by
+  mutating the branch:
+  - **Compliant tree (this branch): rc 0, 39 assertions, 7 instances audited.**
+  - **Pre-fix tree** (`git archive HEAD` + this test dropped in): **rc 1, 26 failures**, naming all
+    five violators, the 6-way `_oauth2_proxy` collision, the zone-wide domains, the renderer
+    emitting no `--cookie-name`, and both synthetic fixtures rendering `None`.
+  - **Targeted mutations on a compliant copy**, one assertion class each, all fired and nothing
+    else did: duplicate name (sonarr←radarr) → 1 failure; CSRF derivation
+    (`_oauth2_proxy_sonarr_csrf` as a session name) → fired; zone-wide `--cookie-domain` on pihole
+    → 2 failures; crafty's `--cookie-name` deleted → unregistered-default-heir + 2-way collision;
+    shared proxy renamed → **stale register entry** failure.
+  - A **false positive was found and fixed** by this testing: `platform/oauth2-proxy/forward-auth-cec.yaml`
+    contains `serviceName: oauth2-proxy/oauth2-proxy:80` — an Envoy `namespace/service:port`
+    reference, not an image — which the first inventory scan counted as an unaudited instance. The
+    scan is now anchored on `image:`, and the near-miss is documented in the check.
+- **Gate:** toolbox `./ci/gate.sh` — exit code and stage list in the next section, because it is
+  not a pass and that belongs there.
+
+## What I did NOT verify
+
+- **Local gate: rc 3 (UNDETERMINED), not 0.** Green: `spec-immutability`, `decisions-log`, `lint`,
+  `l0`, `render-golden`, `l1`, `approval-policy`. **Skipped:** `gitea-settings` (no
+  `SETTINGS_API_TOKEN` locally) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset). Nothing
+  FAILED; nothing runtime was certified either. **rc 3 is UNDETERMINED, never green.** Touched
+  subset `R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-PLAT-02 R-TEST-01
+  R-TEST-02 R-TEST-03 R-TEST-04` — **UNVERIFIED** at L2/L3, all twelve.
+- **`spec-immutability` was skipping because of a BUG IN MY TOOLING, not the environment.** The
+  worktree ENGINE shim in circulation mounts only the main checkout's `.git`, and `dev.sh`
+  computes the spec path from its own `REPO_ROOT` — which in a worktree is
+  `.claude/worktrees/agent-*/`, so `../home-network-k3s` does not resolve and it adds no spec
+  mount at all. The gate then reported the skip, and the skip read as "the one tolerated
+  environmental skip" while actually being a path bug being waved through. Adding
+  `-v /var/home/keith/source/home-network-k3s:/home-network-k3s:ro -e SPEC_REPO=/home-network-k3s`
+  makes **R-TEST-06 actually run and pass**. Worth putting in every worktree agent brief; the
+  two-mount shim in CLAUDE.md §3 should say so.
+- **CI: `gate / gate (pull_request)` reports FAILURE on `75a4029`, for exactly ONE reason, and it
+  is not this change.** Read from the job log rather than the tick:
+
+      [fail] approval-policy: a sensitive change lacks a non-author approval -> blocks merge
+      [fail] MERGE GATE: BLOCKED
+
+  Every substantive stage is **green in CI** — `gitea-settings: pass` (CI has the token my local
+  run does not), `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. Skipped: `spec-immutability`
+  (the workflow does not check the spec repo out — the documented tolerated skip) and
+  `conformance` (declared). So CI is a **stricter, more complete** run than my local one on every
+  axis except the spec repo, and it agrees.
+  `infra/gitea/approval-policy.json` lists `conformance/**` as sensitive — *"the tests and the
+  harness; changing them changes what green MEANS"* — so this PR is **fail-closed pending a
+  non-author approving review**, which is CLAUDE.md §1 working, not a defect to route around.
+  **The red tick is the policy, and it clears when a human reviews it.** Note also that
+  `gate / conformance (pull_request)` reports `success` with description **"Has been skipped"** —
+  the precise trap CLAUDE.md §2b names; it certifies nothing.
+- **A SUCCESSFUL SIGN-IN, ANYWHERE.** The strongest post-fix evidence is the *absence* of the
+  eviction fingerprint on crafty (0 `removing session`, 0 `cookie signature not valid`, across
+  13h 52m) against radarr's 8-and-8. That is directional and I believe it. But crafty logged
+  **0 `AuthSuccess`** in the same window, so no session has been observed working end to end on a
+  fixed instance. Whoever closes this out should re-measure crafty **after real sign-in traffic**.
+- **No browser round-trip by me at all**, for any of the three defects. Reproducing the collision
+  needs two gated apps visited in one session. The concurrent jackett branch does carry an observed
+  log-level reproduction (interleaved radarr/jackett `removing session` deleting a clean session
+  26s after login) — real evidence for the MECHANISM, and it is theirs, not mine.
+- **`--cookie-refresh=2m30s` has never been exercised.** The value is derived from a measured
+  300.000s access-token lifetime and is asserted to sit inside `(0, 300s)`, but no session has been
+  held past the five-minute cliff to watch a refresh succeed. If Keycloak's refresh-token or SSO
+  idle timeouts are shorter than expected, the symptom moves rather than disappears. **This is the
+  fix most likely to need a second pass**, and the first thing to watch after merge.
+- **The `metrics` port has never been fetched.** The listener is asserted to start by flag
+  presence only; nobody has curled `:44180` on a rendered pod. And it is unreachable by design
+  until the Service/policy/ServiceMonitor wiring lands — see above.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. Cluster access was
+  read-only throughout; the one container I ran was the oauth2-proxy image locally under podman to
+  read its `--help`. The running pods are still the pre-fix spec. **Merging is what deploys**, and
+  it will roll five Deployments; every user's existing session on those five apps is invalidated by
+  the cookie rename and they will re-authenticate once. Radarr/sonarr/qbittorrent use
+  `strategy: Recreate` on RWO volumes, so there is a brief gap per app. Pick the moment — and
+  note this now changes **three** flags plus a port on all five, so a bad `--cookie-refresh` value
+  would surface as five apps misbehaving, not one.
+- **The live flags are not re-measured by the check.** git is not the cluster. I measured the live
+  pods by hand, once; nothing in the gate re-reads them, so hand-applied drift would be invisible
+  to it. A live-state assertion is L2 and is not written.
+- **`--cookie-refresh` and `--metrics-address` are UNGATED, and I found out why.** Nothing in
+  `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the twenty-odd
+  assertions in that selftest, including the six I just added, are dev-time only: a future edit
+  that drops either flag *and* re-renders the goldens would pass the entire merge gate.
+  `render-golden` catches accidental drift (golden vs renderer) but not a deliberate removal of
+  both together, and my L0 check deliberately does not cover these two flags. **The repo already
+  has the precedent for the fix and the one-line shape of it:**
+  `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` drives
+  `conformance/lib/<tool> --selftest` as a subprocess for exactly this reason — its own header
+  says it does so "because nothing [else] drives them". Adding `catalog/render.py` to that list,
+  or giving the renderer its own scope-declaring L0 test, closes it. I did not do it here because
+  it widens an already review-blocked PR into a third subject; it is the highest-value follow-up
+  in this entry.
+- **Log-derived counts are a weak instrument and I did not strengthen them.** `kubectl logs
+  --since=` returns what the kubelet still retains, so an old, busy container's window may be
+  truncated; and `No valid authentication in request` counts unauthenticated first hits and XHR
+  polls indiscriminately. The `cookie signature not valid` / `removing session` pair is the only
+  collision-specific signal in there, and it is the one I leaned on. None of this is gated —
+  there is no alert on it and no L2 check reads it.
+- **The cookie-secrets' distinctness is not gated either.** It was measured out of band and is
+  asserted nowhere. If they were ever made identical the collision would become harmless — but
+  sharing one session-encryption key across trust zones is a worse idea than the bug it would mask,
+  so the check does not offer it as an alternative and does not look for it.
+- **The shared proxy's own configuration is unchanged and untested by me.** I did not touch
+  `platform/oauth2-proxy/**` (owned by another agent this cycle); the check only reads it.
+- **Nothing about group authorization, CSRF behaviour, or session contents.** Names and domains only.
+- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so the PR's green tick — if it goes green —
+  proves the L0/L1 offline subset and nothing about runtime.

infra/gitea/approval-policy.json

@@ -0,0 +1,28 @@
+{
+  "_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
+  "_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
+  "_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
+  "sensitive_paths": [
+    "ci/**",
+    ".gitea/**",
+    "conformance/**",
+    "clusters/**",
+    "bootstrap/**",
+    "infra/gitea/**",
+    "platform/kyverno/**",
+    "platform-config/kustomization.yaml",
+    ".sops.yaml",
+    "**/*.sops.yaml",
+    "**/rbac.yaml",
+    "CLAUDE.md"
+  ],
+  "_why_each": {
+    "ci/**, .gitea/**": "the pipeline itself -- the thing doing the approving",
+    "conformance/**": "the tests and the harness; changing them changes what green MEANS",
+    "clusters/**, bootstrap/**": "Flux wiring -- what reconciles, in what order, with what substitution",
+    "infra/gitea/**, platform-config/kustomization.yaml": "branch protection, the drift detector, and the list that decides which admission policies are actually applied",
+    "platform/kyverno/**": "the admission policy engine",
+    ".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
+    "CLAUDE.md": "the delivery contract agents follow"
+  }
+}

platform/monitoring/alerts.yaml

@@ -799,6 +799,123 @@ spec:
             summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
             description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
+        # ===================================================================================
+        # DEPLOY NOTIFICATIONS (added 2026-09-29). The only rules in this repo whose PURPOSE is to
+        # send mail on the HEALTHY path: the operator asked to be told when Flux starts applying a
+        # git change and when it finishes. Read the two together — they share one expression and
+        # differ only in `for` and severity.
+        #
+        # WHY A RULE AND NOT A FLUX Alert/Provider. Neither Flux event can be filtered into a
+        # deploy signal. Measured on prod 2026-09-29: a Kustomization emits ReconciliationSucceeded
+        # on EVERY interval tick (101 events in one hour across 16 Kustomizations) with the message
+        # "Reconciliation finished in 324ms, next run in 10m0s" — byte-identical whether the apply
+        # changed anything or not; and `Progressing` is worse, because five Kustomizations re-apply
+        # the same object on every tick (event counts 2505/2504/2501/2495/2490), so anything keyed
+        # on it mails five times every ten minutes forever. The applied REVISION is the only signal
+        # that moves when and only when git moves. See the kube_flux_* block in
+        # kube-prometheus-stack.yaml for how it reaches the TSDB.
+        #
+        # THE EXPRESSION. `unless on (revision)` keeps a Kustomization only while its applied
+        # revision is absent from the set of revisions the GitRepository has fetched — i.e. while it
+        # is behind main. It empties itself as each Kustomization catches up, which is what makes
+        # the RESOLVED notification mean "the rollout completed" rather than "the alert timed out".
+        # Alertmanager's email-deploy receiver carries send_resolved: true for exactly that reason,
+        # and group_by: [alertname] collapses all 16 Kustomizations into one mail per rollout.
+        #
+        # TWO GUARDS, BOTH LOAD-BEARING, BOTH UNIT-TESTED (tests/flux-rollout.test.yaml):
+        #   * `and on () (count(...) > 0)` — if kube_flux_gitrepository_info is ABSENT (KSM rolled
+        #     without the ConfigMap, the flux RBAC lost, customResourceState reverted) then
+        #     `unless on (revision) <empty>` removes NOTHING and all 16 Kustomizations alert
+        #     forever, for no reason. That is the permanently-firing failure mode this repo already
+        #     shipped once (CNPG, 2026-09-12). The guard makes the missing-exporter case SILENT,
+        #     which is the correct direction for a notification — a broken exporter is what
+        #     FluxReconciliationFailure and the platform-health.monitoring-self rules are for.
+        #   * `unless on (namespace, name) (kube_flux_kustomization_suspended == 1)` — a suspended
+        #     Kustomization pins an old revision BY DESIGN, and would otherwise hold a rollout open
+        #     for as long as it stays suspended, so the resolved mail (the "it finished" half of the
+        #     feature) would never arrive. FluxResourceSuspended above is what reports that state.
+        #
+        # `revision!=""` on both sides: KSM omits a label whose path is absent, so a Kustomization
+        # that has never applied anything — or a GitRepository with no artifact yet — publishes the
+        # series with no `revision` label. Without the matcher those group together under the empty
+        # string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
+        #
+        # A THIRD THING THAT MUST NOT REGRESS — THE SEVERITY LABEL. These alerts are delivered
+        # because of the labels they do NOT carry as much as the ones they do. Do not "tidy" this
+        # pair by giving FluxRolloutInProgress a `severity`, and never `severity: info`: the chart's
+        # InfoInhibitor rule plus its `severity="info"` inhibit_rule suppress an info alert whenever
+        # it is the only thing firing in its namespace, which is every healthy deploy. Measured on
+        # prod 2026-09-30 — the alert fired correctly for 12 Kustomizations, resolved correctly, and
+        # was inhibited in between. The full reasoning is on the `notification: deploy` label below.
+        # ===================================================================================
+        - alert: FluxRolloutInProgress
+          expr: |
+            (
+              (
+                kube_flux_kustomization_info{revision!=""}
+                unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
+              )
+              unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
+            )
+            and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
+          # NO `for:`, ON PURPOSE. Every other rule in this file waits, because every other rule
+          # reports a fault and a fault that clears in 30s is noise. This one reports an EVENT the
+          # operator asked to be told about, so the failure mode to avoid is the opposite: a `for:`
+          # of even 1m would silently swallow any rollout that converged faster than that (a manual
+          # `flux reconcile`, a change touching one Kustomization), and a deploy notification that
+          # only sometimes arrives is worse than no notification at all.
+          labels:
+            # ROUTED BY `notification`, AND DELIBERATELY CARRYING NO `severity` (fixed 2026-09-30).
+            # This alert originally used `severity: info`. That was WRONG, and wrong in the silent
+            # direction — measured on prod within minutes of the first real deploy.
+            # kube-prometheus-stack ships `general.rules` -> InfoInhibitor:
+            #     group by (namespace) (ALERTS{severity="info"} == 1)
+            #       unless on (namespace) group by (namespace)
+            #         (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
+            # which fires for any namespace that has a firing info alert AND NOTHING ELSE WRONG in
+            # it; paired with the chart's inhibit_rule
+            #     source_matchers: [alertname="InfoInhibitor"]
+            #     target_matchers: [severity="info"]
+            #     equal: [namespace]
+            # the effect is that an info alert is suppressed EXACTLY WHEN it is the only thing
+            # happening in its namespace. For a deploy notification that is the only case that
+            # matters: on a healthy rollout of a healthy cluster, flux-system has no warning or
+            # critical alert, so InfoInhibitor fired and this alert never reached the receiver.
+            # Verified live: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment this
+            # alert began firing for 12 Kustomizations.
+            # Upstream's intent is explicit — `severity: info` means "only worth showing next to a
+            # real problem". A deploy notice is not that, so it must not claim that severity.
+            # `notification` is a label no inhibit_rule in the chart targets (all three match on
+            # severity values), and omitting `severity` also stops InfoInhibitor firing for
+            # flux-system at all. Route: platform/monitoring/kube-prometheus-stack.yaml matches
+            # `notification = "deploy"` -> the email-deploy receiver, subject "[FLUX] applying|applied".
+            notification: deploy
+          annotations:
+            summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
+            description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+        # The escalation: a rollout that STARTED and never finished. Same expression, so it cannot
+        # disagree with the notification above about what "behind main" means; 30m is comfortably
+        # past the slowest legitimate convergence (the longest Kustomization interval is 1h for
+        # cilium, but its dependents are 10m and a wave chain converges well inside 30m).
+        # severity: warning, so this one takes the normal email-warning path and reads as a fault.
+        - alert: FluxRolloutStuck
+          expr: |
+            (
+              (
+                kube_flux_kustomization_info{revision!=""}
+                unless on (namespace, name) (kube_flux_kustomization_suspended == 1)
+              )
+              unless on (revision) kube_flux_gitrepository_info{name="flux-system", revision!=""}
+            )
+            and on () (count(kube_flux_gitrepository_info{name="flux-system", revision!=""}) > 0)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Flux rollout has not completed: {{ $labels.name }} stuck on an old revision"
+            description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has been behind the flux-system GitRepository revision for 30m (applied {{ $labels.revision }}) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
 
 # =====================================================================================
 # Certificates — cert-manager

platform/monitoring/kube-prometheus-stack.yaml

@@ -131,6 +131,44 @@ spec:
               # page-worthy — notify fast, remind often
               group_wait: 10s
               repeat_interval: 1h
+            # DEPLOY NOTICES, not health alerts (added 2026-09-29). FluxRolloutInProgress fires
+            # the moment the applied revision diverges from main and RESOLVES when every
+            # Kustomization has caught up — so the FIRED mail is "a deploy started" and the
+            # RESOLVED mail is "it finished". Both halves are the product, which is why
+            # send_resolved must stay true on the receiver below.
+            #
+            # A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
+            # routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
+            # real platform fault, which is how alert mail stops being read at all.
+            #
+            # MATCHES `notification`, NOT `severity` (corrected 2026-09-30). This route first read
+            # `severity = "info"`, which routed correctly and then had the mail SUPPRESSED before it
+            # was sent: the chart's own InfoInhibitor rule fires for any namespace holding a firing
+            # info alert with no warning/critical beside it, and its inhibit_rule
+            # (source alertname="InfoInhibitor" -> target severity="info", equal [namespace]) drops
+            # exactly that alert. A healthy deploy in a healthy flux-system is precisely that case,
+            # so the notification was inhibited on every occasion it was wanted. Measured live on
+            # prod 2026-09-30: InfoInhibitor went active at 04:58:48Z as the alert began firing.
+            # Routing on a non-severity label sidesteps all three of the chart's inhibit_rules,
+            # which match on severity values only. See the label's own comment in alerts.yaml.
+            - matchers: ['notification = "deploy"']
+              receiver: email-deploy
+              # One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
+              # Kustomization (16 possible), and grouping on alertname alone collapses them into a
+              # single mail that lists all of them. The default group_by would have split them by
+              # namespace as well — harmless today (all 16 are in flux-system) and wrong the moment
+              # a Kustomization lives elsewhere.
+              group_by: ["alertname"]
+              group_wait: 30s
+              # 2m, not the 5m default: group_interval also bounds how long a group's RESOLVED
+              # notification waits, and holding "the deploy finished" four minutes behind the
+              # deploy finishing makes the completion half of this feature feel broken.
+              group_interval: 2m
+              # A deploy notice is a statement about one revision, so re-sending it on the 12h
+              # default would just repeat old news. 24h is long enough never to fire twice for one
+              # rollout, while still re-nagging if a rollout is genuinely stuck for a whole day
+              # (FluxRolloutStuck, severity warning, is the real escalation for that case).
+              repeat_interval: 24h
         receivers:
           - name: email-warning
             email_configs:
@@ -144,6 +182,19 @@ spec:
                 send_resolved: true
                 headers:
                   subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
+          # Flux deploy notices (the `severity = "info"` route above).
+          - name: email-deploy
+            email_configs:
+              - to: "[email protected]"    # alert inbox (operator: change if a different inbox is wanted)
+                # BOTH transitions are the deliverable here: firing = "Flux started applying a new
+                # revision", resolved = "every Kustomization reached it". Turning this off would
+                # delete the completion notification, not merely reduce noise.
+                send_resolved: true
+                headers:
+                  # `[FLUX] applying` / `[FLUX] applied` — readable in a notification preview
+                  # without opening the mail, and never the word WARNING for a healthy deploy.
+                  # .Status is "firing" or "resolved" in Alertmanager's notification template data.
+                  subject: '[FLUX] {{ if eq .Status "firing" }}applying{{ else }}applied{{ end }} — {{ .CommonLabels.alertname }}'
 
     # --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO ---
     grafana:
@@ -272,11 +323,33 @@ spec:
       #     `config` alone does not restart the pod: after editing it, `kubectl -n monitoring rollout
       #     restart deploy/kube-prometheus-stack-kube-state-metrics`. Enabling it the first time
       #     changes the pod spec (new arg + volume), so the initial rollout is automatic.
+      # FORCE A POD ROLL WHEN `customResourceState.config` BELOW CHANGES. The chart renders that
+      # config into a ConfigMap and mounts it, but the KSM Deployment carries NO checksum annotation
+      # over it — documented in the block above, and the reason the Velero metrics needed a manual
+      # `rollout restart` when they were first tuned. So a PR that edits ONLY `config` changes the
+      # ConfigMap, leaves the pod spec byte-identical, and KSM keeps running the OLD collector set:
+      # the new metrics never appear and every alert reading them is inert, while the PR is merged
+      # and green. That is the exact shape of defect this repo keeps getting bitten by, so it is
+      # closed declaratively rather than by remembering an operator step. BUMP THIS VALUE IN THE
+      # SAME COMMIT as any `customResourceState.config` edit and Flux rolls the pod for you.
+      # (Same mechanism as grafana.podAnnotations/oidc-rollout above.)
+      podAnnotations:
+        homelab.keithjackson.dev/crs-config-rollout: "1"
       rbac:
         extraRules:
           - apiGroups: ["velero.io"]
             resources: ["schedules"]
             verbs: ["get", "list", "watch"]
+          # Flux applied-revision metrics (the kube_flux_* block at the end of `config` below).
+          # MANDATORY for the same measured reason as velero.io above: without read access KSM
+          # publishes the custom-resource-state metrics as an EMPTY SET and logs a forbidden
+          # error, which would leave FluxRolloutInProgress/FluxRolloutStuck inert — silently.
+          - apiGroups: ["kustomize.toolkit.fluxcd.io"]
+            resources: ["kustomizations"]
+            verbs: ["get", "list", "watch"]
+          - apiGroups: ["source.toolkit.fluxcd.io"]
+            resources: ["gitrepositories"]
+            verbs: ["get", "list", "watch"]
       customResourceState:
         enabled: true
         config:
@@ -328,6 +401,89 @@ spec:
                         labelsFromPath:
                           cron: [spec, schedule]
                           phase: [status, phase]
+              # --- FLUX APPLIED REVISION AS METRICS (added 2026-09-29) --------------------------
+              # WHY. Nothing in the TSDB could answer "is the cluster running the commit that is on
+              # main?", so "Flux started applying a git change" and "Flux is idle" were
+              # indistinguishable and a deploy could start, stall halfway and never finish with
+              # nothing to say so. Measured on prod 2026-09-29, all three candidate signals fail:
+              #   * gotk_* (scraped by flux-podmonitor.yaml) carries NO revision label anywhere —
+              #     gotk_reconcile_condition 120 series, gotk_suspend_status 40, not one with a
+              #     revision. It can say a Kustomization is unhealthy, never which commit it holds.
+              #   * the EVENT stream cannot be filtered into a deploy signal: a Kustomization emits
+              #     ReconciliationSucceeded on EVERY interval tick — 101 events in one hour across
+              #     16 Kustomizations — with the message "Reconciliation finished in 324ms, next run
+              #     in 10m0s", byte-identical whether the apply changed anything or not.
+              #   * `Progressing` is worse, not better: five Kustomizations re-apply the same object
+              #     on every single tick (2505/2504/2501/2495/2490 on one event object each), so
+              #     anything keyed on it fires five times every ten minutes, forever.
+              # The applied REVISION is the one signal that moves only when git moves, and it lives
+              # in these two objects' status and nowhere else:
+              #   GitRepository/flux-system  .status.artifact.revision   = what main is (poll 1m)
+              #   Kustomization/*            .status.lastAppliedRevision = what is applied
+              # Exporting both makes "applied != desired" a PromQL question, which is what
+              # FluxRolloutInProgress / FluxRolloutStuck ask (platform/monitoring/alerts.yaml).
+              #
+              # NAMING: kube_flux_*, NOT gotk_* — the same rule as kube_velero_schedule_* above.
+              # These are kube-state-metrics series ABOUT Kubernetes objects and must never be
+              # mistaken for the gotk_* family the Flux controllers themselves export.
+              #
+              # VERIFIED ON PROD before writing this (2026-09-29): both CRDs serve `v1` as the
+              # STORAGE version (kustomizations.kustomize.toolkit.fluxcd.io,
+              # gitrepositories.source.toolkit.fluxcd.io) and both status paths below are populated
+              # — artifact.revision=main@sha1:b93594b7…, and all 16 Kustomizations reporting that
+              # same revision as lastAppliedRevision.
+              #
+              # Cost: 2 series per Kustomization (16 today) + 1 for the GitRepository = 33.
+              - groupVersionKind:
+                  group: kustomize.toolkit.fluxcd.io
+                  version: v1
+                  kind: Kustomization
+                metricNamePrefix: kube_flux_kustomization
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision a Flux Kustomization has applied (status.lastAppliedRevision)."
+                    each:
+                      type: Info
+                      info:
+                        # KSM OMITS a label whose path is absent (the same behaviour noted for the
+                        # Velero `phase` label above), so a Kustomization that has never applied
+                        # anything publishes this series with NO `revision` label at all rather than
+                        # an empty one. Every rule that reads it therefore matches revision!="" —
+                        # otherwise those would collapse together under the empty string and a
+                        # never-applied Kustomization would compare equal to a converged one.
+                        labelsFromPath:
+                          revision: [status, lastAppliedRevision]
+                  - name: suspended
+                    help: "1 if the Flux Kustomization is suspended (spec.suspend)."
+                    each:
+                      type: Gauge
+                      gauge:
+                        # nilIsZero is MANDATORY, exactly as for the Velero `paused` gauge above:
+                        # measured 2026-09-29, NONE of the 16 Kustomizations sets spec.suspend at
+                        # all. Without it this metric would not exist, and the `unless` that stops a
+                        # suspended Kustomization holding a rollout notification open forever would
+                        # silently match nothing.
+                        path: [spec, suspend]
+                        nilIsZero: true
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: v1
+                  kind: GitRepository
+                metricNamePrefix: kube_flux_gitrepository
+                labelsFromPath:
+                  name: [metadata, name]
+                  namespace: [metadata, namespace]
+                metrics:
+                  - name: info
+                    help: "Revision of the artifact a Flux GitRepository has fetched (what main is)."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          revision: [status, artifact, revision]
       # MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
       # with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
       # roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert

platform/monitoring/tests/flux-rollout.test.yaml

@@ -0,0 +1,268 @@
+# promtool rule-unit tests for FluxRolloutInProgress / FluxRolloutStuck — the deploy-notification
+# pair in the platform-health.flux group (platform/monitoring/alerts.yaml).
+# Run with: platform/monitoring/tests/run.sh   (the merge gate runs the same harness via
+#           conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THESE TESTS CARRY MORE WEIGHT THAN USUAL. This pair is a NOTIFICATION, so both failure
+# directions are silent in production and neither is visible on a healthy cluster:
+#   * fire when it should not -> a permanently-firing "deploy in progress" that mails on every
+#     repeat_interval and trains the operator to ignore Flux mail. Shipped once already in this
+#     repo (CNPG, 2026-09-12).
+#   * fail to fire -> no deploy notification at all, which is the exact complaint this pair was
+#     written to answer. Indistinguishable from "no deploys happened".
+# So every clause is pinned in both directions here, and the two GUARDS in the expression (the
+# missing-exporter guard and the suspended-Kustomization exclusion) each get a test whose ONLY
+# job is to fail if that clause is deleted.
+#
+# MODELLING A REVISION CHANGE. `revision` is a LABEL, so a Kustomization moving from revision A to
+# revision B is modelled as series A ending and series B beginning — not as one series changing
+# value. Prometheus' 5m lookback then means series A is still resolvable for 5m after its last
+# sample, so every eval_time below is placed at least 5m clear of a handoff. That is deliberate:
+# it keeps these tests about the RULE and not about staleness semantics.
+#
+# Sample interval 1m; the longest `for` under test is 30m.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. CONVERGED — the steady state this cluster is in almost all of the time.
+  #    Both rules MUST be silent. This is the "not permanently firing" half of the proof, and it
+  #    is the test that fails if `unless on (revision)` is mis-spelled: the left-hand side is
+  #    non-empty here, and only the revision match suppresses it.
+  # =============================================================================================
+  - interval: 1m
+    name: converged (every Kustomization on the GitRepository revision) - silent
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      - eval_time: 35m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 35m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. A COMMIT LANDS — the notification this whole change exists to produce.
+  #    GitRepository moves to revision bbbbbbb at t=11m. apps-wave-0 catches up at t=14m;
+  #    `platform` never does.
+  #      t=25m -> FluxRolloutInProgress fires for `platform` ONLY (apps-wave-0 has converged),
+  #               and FluxRolloutStuck is still quiet because its 30m has not elapsed.
+  #      t=50m -> both fire. ("started and never finished" is a real, separate condition.)
+  # =============================================================================================
+  - interval: 1m
+    name: new revision on main - InProgress fires for the lagging Kustomization only
+    input_series:
+      # The old artifact revision ends at t=10m, the new one starts at t=11m.
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x10'
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '_x10 1x70'
+      # `platform` stays on the old revision for the whole test.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x80'
+      # apps-wave-0 catches up at t=14m.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x8'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '_x13 1x67'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+    alert_rule_test:
+      # The deploy notification, ~14m after the commit was fetched. Exactly ONE alert: a rule that
+      # ignored the revision match would also list apps-wave-0, which has already converged.
+      - eval_time: 25m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — platform is behind main"
+              description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+      # `for: 30m` on the escalation is real — not yet firing 14m in.
+      - eval_time: 25m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+      # ...and it does fire once the rollout has genuinely hung.
+      - eval_time: 50m
+        alertname: FluxRolloutStuck
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux rollout has not completed: platform stuck on an old revision"
+              description: "Kustomization flux-system/platform has been behind the flux-system GitRepository revision for 30m (applied main@sha1:aaaaaaa) — a git change started rolling out and has not converged. Check the Kustomization's Ready condition and its dependencies."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+
+  # =============================================================================================
+  # 3. THE ROLLOUT COMPLETES -> the alert clears, which is what sends the "applied" mail.
+  #    Without this test the completion half of the feature is unproven: an expression that fired
+  #    correctly but never emptied itself would pass test 2 and still never deliver a resolved
+  #    notification.
+  # =============================================================================================
+  - interval: 1m
+    name: rollout completes - InProgress clears (this is the resolved/completion notification)
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:aaaaaaa",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x10'
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '_x10 1x70'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x20'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:bbbbbbb",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '_x23 1x57'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x80'
+    alert_rule_test:
+      # Mid-rollout: firing.
+      - eval_time: 20m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: platform
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — platform is behind main"
+              description: "Kustomization flux-system/platform has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+      # Converged (5m clear of the old series' last sample at t=20m): silent, and it never had
+      # 30m of continuous firing, so the escalation must not have fired either.
+      - eval_time: 32m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 32m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. GUARD 1 — kube_flux_gitrepository_info ABSENT (KSM rolled without the ConfigMap, the flux
+  #    RBAC was lost, customResourceState reverted). `unless on (revision) <empty set>` removes
+  #    NOTHING, so without the `and on () (count(...) > 0)` guard EVERY Kustomization alerts
+  #    forever. This test exists to go red the moment that guard is deleted.
+  #    Correct behaviour is SILENCE: a broken exporter is FluxReconciliationFailure's job and the
+  #    platform-health.monitoring-self rules', not a deploy notification's.
+  # =============================================================================================
+  - interval: 1m
+    name: GUARD - GitRepository metric absent - silent, NOT 16 alerts forever
+    input_series:
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x60'
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x60'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x60'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="apps-wave-0",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x60'
+    promql_expr_test:
+      # Stated explicitly so the reason for the silence is unambiguous in the test output: the
+      # guard's own count is empty, not merely false.
+      - expr: 'count(kube_flux_gitrepository_info{name="flux-system", revision!=""})'
+        eval_time: 45m
+        exp_samples: []
+    alert_rule_test:
+      - eval_time: 45m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 45m
+        alertname: FluxRolloutStuck
+        exp_alerts: []
+
+  # =============================================================================================
+  # 5. GUARD 2 — a SUSPENDED Kustomization pins an old revision by design. It must not hold the
+  #    rollout notification open (or the "applied" mail never arrives), while a non-suspended
+  #    Kustomization on the same old revision MUST still alert.
+  #    Both halves are asserted in one test on purpose: an `unless` written with the wrong labels
+  #    would suppress EVERYTHING and would pass a suppression-only test.
+  # =============================================================================================
+  - interval: 1m
+    name: GUARD - suspended Kustomization excluded, non-suspended one still alerts
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      # Suspended and behind -> excluded.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="platform",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="platform",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      # Not suspended and behind -> must alert.
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="conformance",revision="main@sha1:aaaaaaa",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="conformance",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      - eval_time: 20m
+        alertname: FluxRolloutInProgress
+        exp_alerts:
+          - exp_labels:
+              notification: deploy
+              job: kube-state-metrics
+              namespace: flux-system
+              name: conformance
+              revision: "main@sha1:aaaaaaa"
+              customresource_group: kustomize.toolkit.fluxcd.io
+              customresource_kind: Kustomization
+              customresource_version: v1
+            exp_annotations:
+              summary: "Flux is applying a new revision — conformance is behind main"
+              description: "Kustomization flux-system/conformance has applied main@sha1:aaaaaaa, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-rollout"
+
+  # =============================================================================================
+  # 6. A Kustomization that has NEVER applied anything publishes no `revision` label at all (KSM
+  #    omits absent paths). `revision!=""` must drop it rather than let it group under the empty
+  #    string and compare equal to the GitRepository revision.
+  # =============================================================================================
+  - interval: 1m
+    name: never-applied Kustomization (no revision label) is not counted as converged or lagging
+    input_series:
+      - series: 'kube_flux_gitrepository_info{job="kube-state-metrics",namespace="flux-system",name="flux-system",revision="main@sha1:bbbbbbb",customresource_group="source.toolkit.fluxcd.io",customresource_kind="GitRepository",customresource_version="v1"}'
+        values: '1x40'
+      # No `revision` label at all — this is what KSM publishes for status.lastAppliedRevision="".
+      - series: 'kube_flux_kustomization_info{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '1x40'
+      - series: 'kube_flux_kustomization_suspended{job="kube-state-metrics",namespace="flux-system",name="brand-new",customresource_group="kustomize.toolkit.fluxcd.io",customresource_kind="Kustomization",customresource_version="v1"}'
+        values: '0x40'
+    alert_rule_test:
+      # Silent: FluxReconciliationFailure (Ready=False for 15m) is what covers a Kustomization
+      # that cannot apply at all. This pair only ever speaks about revisions it can compare.
+      - eval_time: 30m
+        alertname: FluxRolloutInProgress
+        exp_alerts: []
+      - eval_time: 30m
+        alertname: FluxRolloutStuck
+        exp_alerts: []

runbooks/flux-rollout.md

@@ -0,0 +1,118 @@
+# flux-rollout — a git change is rolling out, or has stopped halfway
+
+**Alerts:** `FluxRolloutInProgress` (info, no `for:`) and `FluxRolloutStuck` (warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`, group `platform-health.flux`
+**Metrics:** `kube_flux_kustomization_info` / `kube_flux_kustomization_suspended` /
+`kube_flux_gitrepository_info` — kube-state-metrics custom-resource-state, configured in
+`platform/monitoring/kube-prometheus-stack.yaml`
+
+## What this means here
+
+These two are the **deploy notification**, not a fault report. On this cluster merging a PR into
+`main` IS the deploy, and until this pair existed nothing told you a deploy had begun: the Flux event
+stream emits `ReconciliationSucceeded` on every 10-minute tick whether or not anything changed (101
+such events in one hour, measured 2026-09-29), so "applying a commit" and "idle" looked identical.
+
+The pair reads the one signal that moves only when git moves — the **applied revision**:
+
+- `GitRepository/flux-system` `.status.artifact.revision` — what `main` is (polled every 1m)
+- `Kustomization/*` `.status.lastAppliedRevision` — what each tier has actually applied
+
+**`FluxRolloutInProgress` fires** the moment any non-suspended Kustomization's applied revision is
+not the revision fetched from `main`, and **clears when every one of them has caught up**. So:
+
+| Mail | Subject | Means |
+|---|---|---|
+| firing | `[FLUX] applying — FluxRolloutInProgress` | a commit started rolling out |
+| resolved | `[FLUX] applied — FluxRolloutInProgress` | every Kustomization reached it; rollout complete |
+
+Both are expected, routine, and not a problem. The `severity: info` label routes them to the
+dedicated `email-deploy` Alertmanager receiver so they never arrive subject-lined `[WARNING]`.
+
+**`FluxRolloutStuck` is the one that matters.** Same expression, `for: 30m`, `severity: warning`. It
+means a deploy started and did not converge — the "applying" mail arrived and the "applied" mail
+never did. The 14-tier chain is dependency-ordered and revision-gated, so a single stalled tier holds
+every tier below it on "dependency not ready".
+
+## First commands
+
+`kubectl` does not work on the workstation — the toolchain lives only in the pinned container, and
+the context must always be named explicitly.
+
+Which tiers are behind, and on what:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n flux-system get gitrepository flux-system \
+    -o jsonpath="WANTED: {.status.artifact.revision}{\"\n\"}"
+  kubectl --context k3s-prod get kustomizations -A \
+    -o custom-columns="NAME:.metadata.name,APPLIED:.status.lastAppliedRevision,READY:.status.conditions[?(@.type==\"Ready\")].status"'
+```
+
+Any row whose `APPLIED` differs from `WANTED` is a tier this alert is naming. Then ask **why** that
+tier is not moving — its Ready condition carries the reason:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n flux-system get kustomization <name> \
+    -o jsonpath="{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{\"\n\"}{end}"'
+```
+
+Confirm the alert is reading live data rather than a stale exporter:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+  POD=$(kubectl --context k3s-prod -n monitoring get pod -l app.kubernetes.io/name=prometheus -o name | head -1)
+  kubectl --context k3s-prod -n monitoring exec "$POD" -c prometheus -- \
+    /bin/promtool query instant http://localhost:9090 "count by (__name__) ({__name__=~\"kube_flux.*\"})"'
+```
+
+Three names must come back (`kube_flux_kustomization_info`, `kube_flux_kustomization_suspended`,
+`kube_flux_gitrepository_info`). If that query is **empty**, the alerts are inert — go to the last
+cause below.
+
+## Likely causes, most likely first
+
+1. **Nothing is wrong — the rollout is simply in flight.** Most Kustomizations reconcile on a 10m
+   interval and the chain is sequential, so several minutes between the "applying" and "applied"
+   mails is normal. `cilium` is on a 1h interval and can legitimately be the last to converge.
+2. **A gating tier is unhealthy, so everything below it is parked.** Look for `DependencyNotReady`
+   in the stuck tier's message and walk *up* the chain to the first tier that is not Ready; that one
+   is the real fault. Its own alert (`FluxReconciliationFailure`) and runbook `flux-reconcile` own it.
+3. **A Kustomization is suspended.** A suspended tier is excluded from this alert by design (so it
+   cannot hold the "applied" mail open forever), but anything that `dependsOn` it still stalls and
+   *will* show here. `FluxResourceSuspended` / runbook `flux-suspended` covers it.
+4. **A health check is timing out.** A `wait: true` tier only records `lastAppliedRevision` once its
+   objects pass health checks, so a workload that never goes Ready keeps the tier on the old
+   revision even though the manifests applied fine.
+5. **kube-state-metrics is not publishing the `kube_flux_*` family.** Then the rules are inert and
+   you would get *no* mail at all rather than a stuck one — so this is the cause when the complaint
+   is silence. The metrics come from `customResourceState` plus a read-only RBAC rule for
+   `kustomize.toolkit.fluxcd.io` / `source.toolkit.fluxcd.io`; without that RBAC, KSM emits the set
+   **empty** and logs a forbidden error. Check its logs:
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c '
+     kubectl --context k3s-prod -n monitoring logs deploy/kube-prometheus-stack-kube-state-metrics \
+       --tail=50 | grep -i "forbidden\|custom.resource" || echo "no RBAC complaints"'
+   ```
+   Note the chart puts **no checksum annotation** over that ConfigMap, so a change to the config
+   alone does not restart the pod — that is why `podAnnotations` carries a
+   `homelab.keithjackson.dev/crs-config-rollout` counter that must be bumped in the same commit.
+
+## When to escalate or stop
+
+- **`FluxRolloutInProgress` alone, cleared within ~15 minutes: do nothing.** That is the feature
+  working. Do not suspend, reconcile or apply anything in response to it.
+- **`FluxRolloutStuck` firing: fix the upstream fault, not this symptom.** This alert names the
+  tier that is behind; it never names the cause. Walk up to the first non-Ready tier and use
+  `flux-reconcile`.
+- **Never `kubectl apply`, `edit` or `patch` the change through by hand.** Flux reconciles `main`;
+  a hand-applied object is drift that the next reconcile reverts or, worse, adopts — and it makes
+  the applied-revision signal these alerts read a lie. `flux reconcile` (an operator action, never
+  an agent's) is the only nudge that is safe, and it only *hurries* the same pull.
+- **Escalate to a revert if a stuck rollout is degrading service.** Reverting the offending commit
+  on `main` through the normal PR flow is the rollback path; see runbook `update-rollback`.
+- **If both alerts are silent during a deploy you know happened,** the notification itself is broken,
+  not the deploy. That is cause 5 above, and it is worth fixing immediately — silence from these
+  rules is indistinguishable from "no deploys happened", which is the exact defect they were
+  written to close.

runbooks/kustomization.yaml

@@ -65,6 +65,7 @@ configMapGenerator:
       - crafty-admin-credential=crafty-admin-credential.md
       - denied-flows=denied-flows.md
       - flux-reconcile=flux-reconcile.md
+      - flux-rollout=flux-rollout.md
       - flux-suspended=flux-suspended.md
       - image-automation=image-automation.md
       - image-policy-stalled=image-policy-stalled.md