docs(decisions-log): addendum — detector armed, and why an agent cannot merge here

Records two findings from the unattended session.

The drift detector was INERT: set-ci-token.sh stored the secret under a name Gitea
reserves (GITEA_ prefix, HTTP 400 "invalid secret name"), sent the PUT to /dev/null,
and printed success anyway. A token existed; no secret did. Enabling
REQUIRE_GITEA_SETTINGS on top of that would have blocked every PR including its own
fix. PR #15 fixes it and its CI is green WITH require active, which is the only
proof that counts.

And: merging as gitea-admin returns HTTP 405 "Does not have enough approvals".
required_approvals:1 + block_admin_merge_override:true + an author who cannot
self-approve means "merge on green CI" is not executable by an agent. Recorded as
the control working rather than a fault, and deliberately not circumvented --
self-approval would hollow out the boundary just built. The choice of what to do
about it is the operator s, and whatever is chosen must also change the declaration
in expected-branch-protection.json or the detector will correctly fail on the drift.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/README.md

@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
 *collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
 (`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
 
-## Honest SKIP — never fake green (Phase 0)
+## Honest SKIP — never fake green
 
-**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
+**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
+exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
+skip logic below is still correct and still honest, but read it as describing a **live
+coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
+phrased as a phase is part of why the gap went unexamined.
+
+Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
 they do not pass**:
 
 - The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
 ## Triggers & branch model (CONTRIBUTING.md)
 
 - `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
-- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
+- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
+  so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
+  **No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
+  protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
 
-## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
+## Transition: COMPLETE (2026-09-29)
 
-Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
-[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
+This section used to describe a migration as pending. It has happened. Gitea is live at
+`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
+`gate.yml` runs on every PR.
 
-1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
-   but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
-   local feature-branch → `main`, the gate enforced by convention + the local runner.
-2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
-3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
-   enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
-   `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
-   **platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
+**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
+anyone checking the platform. In reality `main` had **no branch protection** and **zero
+runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
+between rested on an assertion nobody verified. PR #1 — the first pull request this
+repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
+
+So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
+Query the API. `infra/gitea/` holds the scripts and the measured state.
+
+Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
+`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
+CI. That is a real coverage gap, not a phase.
 
 ## Configuration (set in Gitea; referenced by name only)
 

.gitea/workflows/gate.yml

@@ -44,16 +44,31 @@
 ---
 name: gate
 
+# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
+#
+# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
+# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
+#
+#   * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
+#     SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
+#     clock on the one result that actually gates the merge. That cost real time during the
+#     2026-09-29 plex claim, where the loop is racing a short-lived token.
+#   * The list silently did not match this repo's own convention. Branches here are `feat/**`,
+#     which was absent, so the "fast feedback" never fired for the most common prefix anyway —
+#     a mismatch nobody noticed for a month because its only symptom was a MISSING run.
+#     Any hand-maintained prefix list has that failure mode; deleting it removes the class.
+#   * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
+#     you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
+#     same checks buys nothing.
+#
+# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
+# which no PR context exists.
 on:
   pull_request:
     branches: [main]
   push:
     branches:
       - main
-      - 'feature/**'
-      - 'fix/**'
-      - 'docs/**'
-      - 'chore/**'
 
 concurrency:
   group: gate-${{ github.ref }}
@@ -95,9 +110,22 @@ jobs:
       # UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
       # tolerated exit 3 are explained in the file header; the tolerance covers exactly one
       # thing, the unverifiable spec repo, and nothing else can produce a 3 here.
+      # GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
+      # OWN database (branch protection + runner registration) — the blind spot that let `main`
+      # sit unprotected with zero runners for a month while this repo recorded both as working.
+      # Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
+      #
+      # WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
+      # (spec-immutability). It now also covers gitea-settings skipping when that secret is
+      # absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
+      # every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
+      # exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
+      # tolerance narrows back to spec-immutability alone.
       - name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
         env:
           GATE_STRICT_SKIPS: "true"
+          GITEA_SETTINGS_TOKEN: ${{ secrets.GITEA_SETTINGS_TOKEN }}
+          # REQUIRE_GITEA_SETTINGS: "true"   # <- uncomment AFTER seeing gitea-settings green
         run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4

.gitea/workflows/merge-gate.yaml

@@ -11,11 +11,14 @@
 #                                          v
 #                                     merge-gate decision  [R-TEST-03: touched subset green]
 #
-# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
-# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
-# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
-# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
-# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
+# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
+# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
+# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
+# as a plan and never revisited once the plan completed.
+#
+# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
+# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
+# statement about the live platform; query the API instead (see infra/gitea/).
 #
 # CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
 # scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a

CLAUDE.md

@@ -0,0 +1,147 @@
+# CLAUDE.md — how work lands in this repo
+
+Read this before doing anything durable. It is the delivery contract for AI coding
+sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
+takes precedence on *what* to work on; this file governs *how* it lands.
+
+## 1. Never push to `main`. Ever.
+
+Every durable change reaches `main` exactly one way:
+
+    branch  ->  push the branch  ->  open a PR  ->  CI gate green  ->  human approval  ->  merge
+
+There is no other path. Specifically:
+
+- **Do not `git push origin main`.**
+- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
+- **Do not merge your own PR.** A human approves and clicks merge.
+- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
+  anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
+
+> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
+> That wording is **stale** — it predates the PR flow. It describes what Flux does
+> (main is what the cluster follows), not what you are permitted to do. Honour this
+> file, not that line. Fixing those runbooks is tracked work.
+
+## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
+
+Verified against the live Gitea on 2026-09-29:
+
+| Expected | Actual |
+|---|---|
+| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
+| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
+| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
+| Approval required before merge | ✅ `required_approvals: 1` |
+
+All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
+true. They are true now because each was measured, and the push rejection was provoked rather
+than assumed. **Re-measure before relying on any of them** — see §2b.
+
+Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
+CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
+
+**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
+the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
+strength of a check you have not seen report — and note that a **skipped** Gitea job
+reports as `success`, so a green tick is not by itself evidence that anything ran.
+
+## 2b. Never infer platform state from this repository
+
+**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
+Both describe what someone believed at the time of writing, and both have been
+confidently wrong for a month at a stretch.
+
+On 2026-09-29 the log stated the merge gate was a required status check with a runner
+online. The API said `branch_protections: []` and **zero** runners. Everything downstream
+of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
+was wrong, and nothing in the tree could have revealed it, because the tree was the source
+of the error.
+
+**So, before asserting that any gate, protection, runner, secret, or workload is in a
+given state: query the thing itself.**
+
+| Claim about... | Check |
+|---|---|
+| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
+| a runner exists | `GET /api/v1/admin/actions/runners` |
+| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
+| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
+| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
+
+Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
+"unverified" is a legitimate report, and a far better one than a confident guess that
+matches the docs.
+
+**A green tick is not proof anything ran.** Read what actually executed.
+
+## 3. Gate before you push
+
+    ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+
+Exit codes are a contract — do not flatten them:
+
+| rc | Meaning |
+|---|---|
+| 0 | PASS |
+| 1 | BLOCKED — something failed |
+| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
+
+`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
+beside this one; that single skip is expected and tolerated. Any *other* skip is a
+real gap in your run.
+
+**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
+so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
+
+### Running the gate from a git worktree
+
+Two unrelated failures, both with known workarounds — put both in any agent brief:
+
+1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
+   and exists only in the main checkout. Do **not** copy it (it is a credential).
+   Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
+   normally.
+2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
+   gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
+   the `ENGINE` hook — a shim that intercepts `run` and re-execs
+   `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
+   `tracked_files()` is this bug, not a broken tree.
+
+## 4. Branch naming
+
+Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
+trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
+The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
+
+## 5. Cluster access
+
+`kubectl` does not work on the operator workstation; the toolchain lives only in the
+pinned container. Always name the context explicitly:
+
+    ./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+`dev.sh` merges both clusters and picks a current-context **per invocation**; that
+default has already changed once. Never rely on it. Do not run two `dev.sh`
+invocations concurrently.
+
+**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
+
+## 6. Secrets
+
+- **Never `sops -d` a file into a terminal, a log, or a transcript.**
+- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
+  list key names or object names only. A password has been leaked into a transcript
+  this way before.
+- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
+  the single creation-rules file; do not add a second one in a subtree.
+- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
+  flow. Do not design around committing one — read the expiry from the issuing page and
+  treat a burnt token as free to retry.
+
+## 7. Before you finish
+
+Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
+verified, and what you did **not** verify. State gaps plainly — an honest
+"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
+burned by the difference.

DECISIONS-LOG.md

@@ -1192,6 +1192,13 @@ fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` an
   Platform-admins require TOTP MFA.
 - **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
   required status check; runner `incluster-dind` online (ns gitea-actions).
+  > **[CORRECTION, 2026-09-29 — THIS BULLET WAS FALSE WHEN WRITTEN AND STAYED FALSE FOR A MONTH.]**
+  > Measured against the live Gitea API on 2026-09-29: `branch_protections` returned `[]` (`main`
+  > `protected:false`, `enable_status_check:false`, `user_can_push:true`) and **zero** Actions runners
+  > were registered at repo, org or site level. The gate was neither enforced nor able to run. PR #1
+  > — the first pull request this repo ever had — was opened on 2026-09-29 and sat `pending` for
+  > 30.6 minutes with nothing to execute it. Both halves of this bullet were asserted, not verified.
+  > See the 2026-09-29 entries for the root cause and the fix.
 
 ### What changed this session (durable)
 1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
@@ -2424,6 +2431,148 @@ container dies with `PermissionError` on every run; my older base had `0400` and
 - The apex flip itself is `migration/dns-apex-inversion`'s job. This change only guarantees the realm
   follows git once the knob moves; it does not move it.
 
+## 2026-09-12 — The silent-gateway outage now has an alert, and the direct signal turned out not to exist (branch `feat/egress-gateway-staleness-alert`)
+
+Hand-off item 4 from the section above: *"A silent gateway outage still has no alert… nothing
+watches the invariant `egress-uplink` label == `cilium-dbg bpf egress list`'s Egress Ifindex."*
+Closed, as far as Prometheus can close it — and the part that cannot be closed is recorded here
+rather than papered over with a plausible-looking metric name.
+
+### The direct signal is not available, verified rather than assumed
+
+The invariant wants the BPF egress map's contents. They are not in Prometheus, for two independent
+reasons, both checked read-only against live prod:
+
+| check | result |
+|---|---|
+| `{__name__=~"cilium.*egress.*"}`, `cilium_drop.*`, `hubble_drop.*` | **empty vector** — no `cilium_*`/`hubble_*` series exist at all |
+| container ports on the cilium DaemonSet | `health=9879`, `peer-service=4244` — **no metrics port**; only the operator has `operator-prometheus-serve-addr :9963` |
+| `kubectl get servicemonitors -A \| grep -iE 'cilium\|hubble'` | none — Hubble/Cilium metrics export is still the open TODO in `platform/cilium/helmrelease.yaml` |
+
+So `HubbleDeniedFlowSpike` cannot fire today either; that is a separate, pre-existing gap, noted
+and left alone. And even with agent metrics turned on, **Cilium OSS publishes no series carrying
+egress-map contents** — the map is readable only through `cilium-dbg bpf egress list`, a CLI over
+the BPF map. Nothing was invented to fill the hole.
+
+What node-exporter *does* export is the live tun0 **ifindex as a metric value**:
+`node_network_iface_id{device="tun0"}` = `129` on `k3s-prod-3`. Read-only on the live cluster, all
+three agree right now:
+
+```
+node label    homelab.keithjackson.dev/egress-uplink = 10.150.192.86-129
+node-exporter node_network_iface_id{device="tun0"}   = 129
+bpf map       Egress IP 10.150.192.86 / Egress Ifindex 129   (all 16 rows, 4 vpn-class pods)
+```
+
+(The map on the two NON-gateway agents reads `Egress IP 0.0.0.0 / Ifindex 0` with
+`Gateway IP 192.168.2.12` — only the gateway node resolves the interface. Worth knowing before
+debugging the wrong agent.)
+
+### PromQL cannot express the equality test — so the invariant is counted instead
+
+The label's value is `<addr>-<ifindex>`; the live ifindex is a metric VALUE. PromQL has no
+label-to-number conversion, so `label == value` is **not expressible**, and there is no
+level-triggered form of this comparison. What is expressible is the same invariant counted over a
+window: **the published label must change at least as often as tun0 is re-created.**
+
+```promql
+(
+    changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+  >
+    (count(count_over_time(max by (label_homelab_keithjackson_dev_egress_uplink) (
+       kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})[1h:1m])) - 1)
+)
+and on()
+(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+```
+
+`count(count_over_time(max by (<label>) (…)[1h:1m]))` is the number of DISTINCT label values in the
+window. The `max by (<label>)` wrapper is load-bearing: it aggregates away `pod`/`instance` so a
+kube-state-metrics restart is not read as a label change. Both sides reduce to a single unlabelled
+series, which is why `and on()` joins — the same reason `VPNTunnelDown` needs `on()`.
+
+### Proof it evaluates — executed against prod's Prometheus, port-forward + `/api/v1/query`
+
+There is no promtool in this repo, so query execution IS the verification. The **rendered** expr was
+extracted from `kubectl kustomize platform-config` output and run verbatim:
+
+| state | query | result |
+|---|---|---|
+| healthy, now | rendered `VPNEgressSteeringStale` | `[]` — not firing |
+| **no gateway node** (`@ 1789245000`, before prod's stand-up: no tun0 series, no labels) | same expr | `[]` — cannot fire |
+| **stale**, constructed from real data | same expr with the already-exported `egress-gateway` label substituted for `egress-uplink`, window 3h | **`1` — FIRES** |
+| same expr, window 1h | | `[]` |
+
+The stale case is honest and cost prod nothing: prod's tun0 really did go **128 → 129** ~2.5h
+earlier when it moved to PIA `.135`, and the `egress-gateway` label really did *not* follow it
+(`changes(...[3h:1m])` = 1, distinct label values = 1 → `1 > 0`). That is bit-for-bit the shape of
+a label that missed a flap. The healthy counterfactual is the one thing left as reasoning rather
+than a query: over that same 3h the REAL `egress-uplink` label held two values
+(`10.72.192.7-128`, then `10.150.192.86-129`), giving `1 > 1` = false. It could not be queried
+because the label was not exported yet — which is the next point.
+
+Supporting queries, also live: the distinct-value idiom returns **1** for a label that held still
+over the window and **2** for one whose value changed inside it (node 3's `egress-gateway` label
+over 12h, which went `""` → `"true"` at stand-up); the gateway guard returns 1; subquery +
+`changes()` returns 0/1/1/1 over 15m/3h/6h/12h as the flap history requires.
+
+### Behaviour at the edges
+
+- **No gateway node:** three independent reasons it cannot fire — no tun0 series, no uplink label,
+  and the `kube_node_labels` guard `VPNTunnelDown` already established. Verified at a past
+  timestamp when prod genuinely had no gateway. This matters: `VPNTunnelDown` fired permanently on
+  prod for exactly this reason and had to be retrofitted.
+- **Convergence:** the up-script rewrites the label within seconds of a connect, worst case ~5m
+  (its 30 × 10s kube-API retry budget on a cold boot, when the API is not up yet). The condition is
+  true during that gap, so `for: 15m` — comfortably past the 2-minute timer and the 5-minute retry
+  budget — is what keeps a normal reconnect quiet.
+- **Bounded, deliberately:** a missed flap alerts from +15m to +1h, then RESOLVES even if egress is
+  still dead. An unbounded critical on `email-critical` is the worse failure for this repo, and 15
+  minutes still beats 80. A level-triggered version needs the map's ifindex as a real metric — the
+  node-exporter textfile collector route, still open.
+
+### The second rule exists because the first one fails OPEN
+
+`VPNEgressSteeringStale` needs `egress-uplink` in kube-state-metrics. Without it the comparison
+degrades to an empty vector and alerts nothing — so `VPNEgressUplinkLabelMissing` (warning,
+`for: 30m`) watches the precondition: a node labelled `egress-gateway=true` with no `egress-uplink`
+label visible. Live right now it returns **1**, correctly: the KSM allowlist extension is in this
+same branch and not yet deployed. It self-clears on the first reconcile, and `for: 30m` covers the
+kube-state-metrics roll. The rule and the allowlist are in the same Flux kustomization on purpose.
+
+`kube-state-metrics.metricLabelsAllowlist` now reads
+`nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]` — a label
+DIMENSION on the same 3 `kube_node_labels` series, not new series, on exactly one node.
+
+### Rejected, with reasons
+
+- **A black-box egress probe** is the most honest signal — it is the only one that answers "does a
+  vpn-class pod's traffic actually leave via the tunnel?" and the only one that catches "the label
+  was written and Cilium ignored it anyway". Rejected **for now**: there is no blackbox-exporter and
+  no probe CronJob in this repo, so it means a new always-on component (itself vpn-class, itself an
+  alerting surface, itself dependent on a third-party endpoint) — and it could not have been
+  verified on prod today. It remains the right next step.
+- **A node-exporter textfile metric from the re-steer script** (hand-off item 4's own suggestion) is
+  the only route to a true level-triggered label-vs-MAP comparison. It needs a textfile directory
+  mounted into the node-exporter DaemonSet plus an Ansible change applied on a prod node — neither
+  of which this branch could execute or verify. Left open, now with the reason recorded.
+- **`absent(node_network_receive_bytes_total{device="tun0"})`-style presence checks** — that IS
+  `VPNTunnelDown`, and it is exactly the blind spot: green while steering points at a dead ifindex.
+
+### Left for the operator / not settled here
+
+1. **The rules are unproven in the firing sense.** The expressions evaluate against live prod data
+   and the stale branch was exercised with a real flap, but no alert has actually reached
+   Alertmanager or the inbox. `conformance/tests/l3-fault-injection/vpn-killswitch` stops and
+   restarts the tunnel and is the natural place to assert `VPNEgressSteeringStale` stays quiet
+   through a healthy reconnect; the alerting L3 test with the SMTP sink is where a fired-and-emailed
+   proof would come from. Neither was fired at prod unasked.
+2. **`HubbleDeniedFlowSpike` cannot fire** — no cilium/hubble metrics are scraped on either cluster.
+   Pre-existing, orthogonal, and the same TODO in `platform/cilium/helmrelease.yaml` blocks it.
+3. **Window vs. stickiness** is a judgement call, not a measurement: 1h chosen so a missed flap
+   alerts for ~45 minutes. If the operator would rather have a sticky alert that needs manual
+   acknowledgement, that is the textfile-collector design, not a tweak to this one.
+
 ## 2026-09-13 — **PROD IS THE APEX.** Dev decommissioned; and a GitOps deadlock found the hard way
 
 Prod serves `home.keithjackson.dev`. Dev's three VMs (Proxmox 101/102/105) are stopped with
@@ -4183,3 +4332,780 @@ restarting, since a restart may silently restore DHCP and destroy the evidence.
 The README's dev-era addressing was corrected throughout (VIP `.1.54`→`192.168.2.21`, pool and Gateway,
 device subnet, node name) along with several stale factual claims, since it doubles as the rebuild
 runbook.
+
+---
+
+## 2026-09-18 — VERIFIED: internal services no longer advertise themselves publicly
+
+The R-DNS-03 disclosure is closed for the `internal` tier. Confirmed at the authority
+(`betty.ns.cloudflare.com`, not a cache):
+
+| name | public DNS | via Pi-hole `.23` |
+|---|---|---|
+| sonarr, radarr, qbittorrent, jackett, semaphore, pihole | **GONE** | 192.168.2.20 |
+| plex, gitea, seerr, keycloak, nextcloud | 192.168.2.20 (untouched, as designed) | — |
+
+So split-horizon now genuinely exists: the names resolve internally and are absent externally, which
+is what `docs/09-dns-and-ingress.md` specified all along. The internal subnet, the Gateway VIP and six
+service names are no longer publicly enumerable, and the DNS-rebind primitive is gone for those six.
+
+**A correction to my own sequencing.** I had insisted this could not land until the operator repointed
+DHCP, on the grounds that removing public records would break LAN resolution. That was wrong, and
+measuring it took one command: the SOHO router at 192.168.2.1 returns **nothing** for all six names, so
+the public records were serving nobody on the LAN. The dependency I asserted did not exist. Landing it
+early closed a security finding a day sooner than my own plan allowed.
+
+**Operator clarifications recorded** (theirs, 2026-09-18):
+1. Pi-hole is **DNS only** — the SOHO router remains the DHCP server. Nothing in the Pi-hole change
+   touches DHCP; it exposes udp/53 and tcp/53 and no DHCP feature is enabled. The remaining action is
+   on the router: hand out `192.168.2.23` as the DNS server, ideally with a public resolver secondary
+   so a Pi-hole outage degrades to "internal names stop resolving" rather than "the LAN loses DNS".
+2. **Reverse proxy by hostname is the accepted design; a separate IP is overkill.** So no second
+   Gateway and no second LoadBalancer address — the existing Gateway already distinguishes by Host/SNI.
+   Per-service public DNS targeting, if wanted, therefore goes through `DNSEndpoint` objects plus the
+   `crd` source rather than a second Gateway carrying a target annotation.
+3. Minecraft stays open with no whitelist — accepted.
+
+**Still open, and it is a fact about the WAN, not a repo decision.** The five `public` services have no
+working public path at all: `136.51.115.252` returns **HTTP 403** for `gitea.home.keithjackson.dev` and
+presents a placeholder certificate (`CN=%D`), so nothing reverse-proxies these names from outside today.
+Either the WAN forwards 80/443 to the Gateway at `192.168.2.20` — in which case their records should
+point at the WAN address — or those five should be re-declared `internal`, because `exposure: public`
+currently asserts something untrue.
+
+**Also fixed:** `apps/vms/unifi/unifi-uos-installer.sops.yaml` had been committed **unencrypted** with
+both fields as literal `REPLACE-ME`, unlike every other `*.sops.yaml` here. `sops <file>` would have
+failed with "sops metadata not found", and Flux would have created the Secret containing the
+placeholder strings. Encrypted in place, round-trip verified. Both fields need setting, not just the
+password — I had omitted `installer-url` from the operator instructions.
+
+---
+
+## 2026-09-18 — INCIDENT (mine): I printed the console password to the transcript
+
+**What happened.** Verifying that the `unifi-uos-installer` Secret had reached the cluster, I ran a
+`kubectl get secret -o jsonpath` that included `{range .data}{@}{end}`. That dumps the whole `data`
+map, so the base64 of `console-password` was printed to the session transcript. This is precisely the
+constraint I have written into every agent brief in this repo — *never print decrypted secrets* — and I
+broke it myself with a careless jsonpath while checking someone else's work.
+
+**Exposure, stated precisely rather than minimised or inflated:**
+- The value reached the session transcript, which I cannot scrub.
+- It did **not** reach git: `git log --all -S<value>` returns **0 commits**.
+- The scratch file that captured it (`tasks/*.output`) was shredded; a repo- and scratch-wide grep for
+  the string now returns nothing on disk.
+
+**Why it is not yet a working credential — the one thing that limits this.** The Secret was created
+`2026-09-14T07:26:43Z`; the running VMI was created `2026-09-12T20:48:30Z`, and its live volume list is
+only `rootdisk` and `cloudinit` — **the `UOSDATA` disk is not attached to the running VMI at all**. So
+the guest has never read the Secret, `install-uos` has never executed, and the `ubuntu` account remains
+locked. The leaked string authenticates nothing today. It would only become valid when the VMI is
+recreated against the new spec.
+
+**Required action, and the ordering matters:** rotate `console-password` in the SOPS Secret **before**
+recreating the VMI. Then the leaked value never becomes a working password anywhere:
+
+    sops apps/vms/unifi/unifi-uos-installer.sops.yaml     # set a NEW console-password
+    git commit && git push                                 # let Flux update the Secret
+    kubectl -n unifi delete vmi unifi                      # only after the rotation has landed
+
+**The lesson, generalised.** Reading a Secret to check *existence* never requires reading its
+*values*. `-o jsonpath='{.metadata.name}'` or listing `.data`'s KEYS answers "did it land?"; dumping
+`.data` answers a question nobody asked. Every previous secret check in this session used a
+keys-only or boolean form — this one regressed because I appended a convenience field to an
+otherwise-safe command without re-reading what it expanded to.
+
+## 2026-09-18 — Two fail-open holes in the gate itself: nested worktrees, and L1 never running
+
+Three pieces of work, one theme: the gate was reporting green over things it had not checked.
+
+### 1. `runbooks/node-memory-swap.md` — the procedure the operator asked for
+
+Keith has a matched 32 GB pair to install and asked which node needs it. **`k3s-prod-1`**: `DIMM1` is a
+Micron `8ATF2G64HZ-3G2E1` (16 GB, DDR4-3200, 1Rx8), `DIMM3` a `16ATF2G64HZ-2G6E1` (16 GB, DDR4-2667,
+2Rx8). Capacity is correct; the pair runs at the slower 2667. Nothing is faulty. `k3s-prod-3` is a
+matched Crucial pair.
+
+The runbook front-loads the go/no-go checks because this is the awkward node to lose: 24 Longhorn
+replicas, one of three etcd members (drain leaves quorum at 2 of 3, no further margin), and its
+rebuilds are the load that starved etcd once. A BLOCKED drain is treated as the safety property
+working, never something to force past.
+
+Three things I had written and then removed, because they were not true:
+
+- **`EtcdIsVoter` does not exist in this cluster.** I had an assertion built on it; it appears nowhere
+  in the repo. The house check is `curl http://127.0.0.1:2381/metrics | grep etcd_server_has_leader`
+  (`runbooks/node-down.md:93`).
+- **`node-drain-policy` is not pinned** in `platform/longhorn/helmrelease.yaml`. The runbook has the
+  operator READ the live Setting and set it before draining rather than assuming the chart default —
+  that Setting is the entire safety story for 24 replicas.
+- I had written "rebuild concurrency is deliberately 1". That is the purge/restore limits, not replica
+  rebuild — the same 5-vs-1 confusion already corrected once in `longhorn-degraded.md`.
+
+Verification hinges on **`Configured Memory Speed`**, not `Speed`: a matched 3200 pair being clocked
+down still reports `Speed: 3200 MT/s`, so matched part numbers alone do not prove the swap worked.
+
+Also recorded: the `node-health` artifacts de-duplicate DIMM part numbers, so they can prove "two
+different modules" but cannot distinguish one module from two identical ones — verification collects
+`dmidecode` live. **Node 2's inventory remains stale and self-contradictory** (documented as the only
+single-DIMM node at 16 GB, yet reporting ~31.1 GiB) and needs re-inventory before anyone claims
+three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was an EFI memattr kernel bug.
+
+### 2. The whole-repo L0 checks were auditing 18 copies of the repo
+
+Six discovery sites walked `ROOT` without pruning dot-directories, so they descended into
+`.claude/worktrees/agent-*/` — full repo copies from isolated agent runs, gitignored at
+`.gitignore:43`.
+
+The visible half was false FAILs: four ServiceMonitors "selecting no Service", nine PrometheusRules
+"under test but not in the repo", every path a worktree path. The whole-repo gate was red for reasons
+with nothing to do with authored surface.
+
+The dangerous half is quieter, and is why all six were fixed rather than only the failing ones: a check
+asking *"does anything in the repo declare X"* can be satisfied by a worktree copy instead of real
+source. The same bug that invents failures can manufacture a pass.
+
+`egress-paths-selected-by-rules` already had the fix and documents hitting it (it once saw 5
+`CiliumEgressGatewayPolicy` objects instead of the 1 authored). Its dot-directory predicate is now
+applied everywhere, plus both `find` calls in `platform/monitoring/tests/run.sh`, whose `--list`
+reported each manifest once per worktree — 21 real pairs where it had been returning them ~19x over.
+
+### 3. L1 rendering never ran on a requirement-scoped gate — closed, with the incident reproduced
+
+`ci/l1.sh` has passed `--include-scope whole-repo` since it was written, with a comment saying no L1
+test declared the scope yet and it was wired ahead of time. Nothing ever declared it, so **the entire
+L1 rendering layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
+R-NET-05/06/07.**
+
+All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`. The justification is one
+shared code path — `catalog/render.py` against `schemas/servicedefinition.schema.json` — which
+requirement-scoped selection cannot see.
+
+The 2026-09-04 external-dns merge is the proof, reproduced rather than asserted.
+`ci/touched-requirements.sh` attributes `platform/externaldns/helmrelease.yaml` to R-ENV-02,
+R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — **none** of the rendering requirements — yet that
+change added `external-dns.alpha.kubernetes.io/controller: none` to the rendered output of all three
+goldens. Measured with `--print-selection`:
+
+| selection for that change | L1 tests selected |
+|---|---|
+| requirement-scoped only (the 2026-09-04 behaviour) | **0** |
+| with `--include-scope whole-repo` (how `ci/l1.sh` invokes it) | **3** |
+
+### 4. …which exposed two defects in the meta-check that guards all of this
+
+Declaring the scope at L1 made `whole-repo-invariants-wired` (R-TEST-03) fail, and it was right to
+change behaviour — but for the wrong reason, and it could not say so:
+
+- **It was layer-agnostic where the design is layer-bounded.** It compared *every* scope-declaring dir
+  against an `--layer l0` selection, assuming only L0 tests would ever declare the scope — while
+  `ci/l1.sh` was written from the opposite premise, that a whole-repo invariant could appear "at any
+  layer". So it blamed the union for correctly excluding another layer. Now partitioned by layer, with
+  the L1 selection and L1 leakage assertions added — which is what makes the L1 declaration
+  load-bearing instead of decorative.
+- **Every assertion in it failed silently.** The idiom was `<test>` on one line, `chk $?` on the next,
+  under `set -euo pipefail` — so a false test aborted the script *before* `chk` could print. The real
+  failure I hit emitted 26 `ok` lines, rc=1, and no message. All 12 sites now capture the status
+  (`chk $arc`), proven by forcing `MIN_DECLARED=999`: the failure prints its full diagnostic and the
+  run continues to the remaining assertions.
+
+A meta-check that cannot report why it failed is worse than no meta-check, because the silence reads
+as a tooling glitch rather than a finding.
+
+### Gate
+
+Offline `l0,l1,meta --include-scope whole-repo` in the pinned toolbox: **25 tests, 0 failures, 0
+errors, exit 0**. L0 whole-repo went 4 failures → 1 → 0 across the two fixes;
+`whole-repo-invariants-wired` now runs 35 assertions. Host runs report one skip because `promtool` is
+not on `$PATH` there — the toolbox run is the real result, and that difference is worth remembering
+before trusting a host-only gate run.
+
+### PM HANDOFF
+
+Open, in the order I would take them:
+
+1. **The memory swap is a free R-STATE-02 proof.** `node-loss` is Phase 7's last exit item and is still
+   proven only on the decommissioned dev cluster. The swap is a planned, supervised single-node loss on
+   prod — exactly the event R-STATE-02 describes. Capturing evidence during it costs nothing extra;
+   manufacturing the event later costs a second maintenance window.
+2. **Node 2 DIMM re-inventory** — blocked on the vault become password for `node_health`. Ansible only;
+   never `ansible-inventory --list` (it prints decrypted vault values).
+3. **Workstation DNS, one sudo command.** `/etc/dnsmasq.d/10-home.keithjackson.dev.conf` contains
+   `local=/home.keithjackson.dev/` plus `address=/.home.keithjackson.dev/192.168.1.53` — the whole zone
+   hardcoded to the decommissioned dev cluster, with forwarding forbidden so no upstream could correct
+   it. That is one root cause for both the sonarr failure and `gitea` not resolving. Pi-hole answers
+   correctly (`gitea` → `192.168.2.20`); only this file is wrong. Correction to an earlier claim of
+   mine: there is no separate `20-prod.*` file — it is a replace of this one file. Router DHCP option
+   6 → `.23` is the fix that makes it unnecessary per-machine.
+4. **UniFi guest has no IP** — needs console access; `cat /etc/netplan/*.yaml` is the confirming command.
+5. **Five `public` services have no working public path** — waiting on WAN forwarding, which Keith has
+   said is deliberately not ready.
+6. **18 stale agent worktrees** on already-merged branches remain on disk. No longer load-bearing for
+   the gate; cleaning them up is cosmetic and should check each for uncommitted work first.
+7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, 103 unmapped
+   paths, and two rendered manifests nothing applies (the keycloak one would stand up a conflicting
+   instance).
+
+## 2026-09-19 — Overnight: the backup alert was crying wolf, and 49 app manifests were gated by nothing
+
+Worked autonomously while the operator slept. One theme again: things that reported green, or red, over
+something other than what they claimed to measure.
+
+### 1. VeleroNoRecentBackup was firing CRITICAL over backups that protected everything
+
+`velero-daily-20260919030041`: **PartiallyFailed, 3079 of 3079 items backed up, 2 errors** — both
+`PodVolumeBackups` for the emptyDir `plugins` + `scratch` volumes of a
+`homepage-default-kopia-maintain-job-…` pod. Velero runs transient kopia repository-maintenance Jobs in
+its own namespace, one per BackupRepository; under `defaultVolumesToFsBackup: true` it tries to
+fs-backup their volumes, the Job finishes, the pod disappears mid-backup, and the PVB fails. Four such
+failures had accumulated.
+
+Nothing real was ever missed. But `PartiallyFailed` does not advance
+`velero_backup_last_successful_timestamp`, so the alert guarding R-STATE-03 fired nightly over Velero
+cleaning up after itself. **An alert that cries wolf every night is how a real backup failure gets
+ignored** — which is precisely the failure mode that alert exists to prevent.
+
+**First fix was right about the bug and wrong about the layer.** I excluded ns `velero`'s volume data
+(keeping its resources in schedule 2) and explicitly deferred the general fix — skip emptyDir by type —
+because Velero's `volumeTypes` policy condition was unproven here and an unparseable `policies.yaml`
+breaks EVERY backup. I wrote "revisit if the shape shows up elsewhere."
+
+It showed up elsewhere inside the hour. The verification backup came back PartiallyFailed with 4 errors,
+all in ns `runbooks`: the nginx Deployment's emptyDir `tmp` + `cache`, failing with
+`error to expose PVB: error getting client pod …: pods "…" not found`. **Cause worth stating plainly: I
+caused it** — pushing a runbook edit changed the configMap hash, rolled the Deployment, and the backup
+lost the pods mid-snapshot. Self-inflicted, not a standing defect. But the mechanism is not
+self-inflicted: a PVB exposes a volume *through its pod*, so any rollout landing in the backup window
+fails one, and one failed PVB makes the whole Backup PartiallyFailed. On a GitOps cluster that is
+routine, not an edge case.
+
+So the property is what matters, not the namespace: an emptyDir is created empty with its pod and
+destroyed with it, so backing one up cannot protect anything while attempting it can break the backup
+that protects everything else.
+
+**The `volumeTypes` risk was retired by testing, not by hoping.** New policy into a THROWAWAY
+ConfigMap first, live one untouched, proven on prod (velero 1.18.1) with a control:
+
+| ns `runbooks` | PodVolumeBackups |
+|---|---|
+| control, no policy | **4** (2× `tmp`, 2× `cache`), Completed |
+| with the policy | **0**, Backup Completed, 0 errors |
+
+The control is the point — "zero PVBs" alone is equally consistent with nothing having been attempted,
+which is exactly the empty green I shipped earlier in this session with a 24h baseline built from ten
+minutes of data.
+
+Then reverted the namespace exclusion as redundant (`plugins`/`scratch` are emptyDir, verified on a live
+maintain pod) rather than leaving two overlapping fixes.
+
+**Proof, full cluster, policy live:** `pre-memswap-202609190525` — **Completed, 3926/3926 items, 0
+errors, 21/21 PodVolumeBackups Completed, 0 emptyDir volumes backed up.** `VeleroNoRecentBackup`
+cleared. `VeleroBackupPartialFailure` remains at `increase = 1.008` from the pre-fix failure and
+self-clears about an hour after it, by design — the rule is `increase(...[1h]) > 0`, which is correct.
+
+That backup also serves as the pre-maintenance backup the memory-swap runbook requires.
+
+### 2. The runbook claimed the CNPG primaries sit on node 1. They do not.
+
+Verified live: all three CNPG clusters run 3 instances spread one per node by required anti-affinity,
+and **k3s-prod-1 holds a REPLICA of each** (`keycloak-db-2`, `nextcloud-db-2`, `semaphore-db-2`). No
+primary is on node 1, so draining it forces **no failover at all** — the best state this maintenance
+could ask for. The original sentence was stale session state. The runbook now prints the command to
+re-derive it, because primaries move after any restart.
+
+That also settles `CNPGPrimariesCoLocated`, firing for k3s-prod-2 (both `keycloak-db` and
+`semaphore-db` primaries). **Do not tidy it before the swap.** Rebalancing means promoting a replica,
+the only node with spare primary capacity is node 1, so "fixing" the alert would put a primary on the
+node about to be powered off and convert a clean drain into a two-database failover. Rebalance after
+node 1 is back and caught up. Warning is now in the runbook.
+
+### 3. 49 app manifests attributed no requirement, so the merge gate ran no behaviour test for them
+
+`ci/touched-requirements.sh`'s coarse path map had no arm for `apps/*`. A change adding or editing any
+workload selected **zero** requirement-scoped tests — so none of the L2 behaviour tests that enforce
+default-deny, egress class or auth mode ran on exactly the changes that can violate them. Kyverno
+admission and the L0 egress-path check are real backstops, but admission is post-merge and the gate
+exists to catch it before.
+
+Also unmapped: `platform/homepage` and `platform/headlamp` — **the C2 UI**. This was already documented
+as a known gap in `changed_diff()`'s rule-3 comment ("the ONLY route to R-C2-01/02 was rule 3", i.e.
+editing the C2 test itself), and had sat there since.
+
+Twelve arms added: `apps/vms/*` (before `apps/*`, since `case` takes the first match), `apps/*`,
+`platform/homepage|headlamp`, `platform/kubevirt`, `platform/gateway`, `platform/coredns`,
+`platform/crafty`, `platform/external-snapshotter|backup`, `runbooks/*`, `platform-config/*`,
+`platform/kustomization.yaml` and `.sops.yaml`.
+
+`platform/kustomization.yaml` deserves its own note: it is the assembly list, so **one deleted line
+disables an entire platform component** while every file it points at stays valid on disk. That is the
+most consequential single-line edit in the repo and it attributed nothing.
+
+Measured effect for an `apps/` change: **0 → 13 requirement-scoped tests** (3 L1, 8 L2, 2 L3). Coverage
+stays 51/51, so the gate's "attributed IDs must be covered" assertion does not false-block.
+
+Ten new assertions in `ci/tests/touched-requirements.test.sh` (26 total, all passing), including the
+ORDER property — `apps/vms/*` must win over `apps/*` — since that is a real failure mode of `case` and
+not merely a matter of both arms existing.
+
+**A correction on my own numbers.** I first reported "192 unmapped files", then "182". Both were
+inflated by my audit script, not by the repo: the regex used `[^)]*`, which matched across a newline, so
+the first arm parsed as `case "$1" in\n    catalog/*` and never matched anything — and I separately
+mis-parsed alternation arms like `.gitea/*|ci/*`, which made `ci/` look unmapped when it is mapped.
+True remainder after this change is **18 files, all prose or captured artifacts** (`docs/plans/`,
+root READMEs, `.standup-dns/*.txt`).
+
+### 4. Finished `feat/egress-gateway-staleness-alert` — it had been backlogged because it could not pass the gate
+
+Merged the two alerts for the silent egress-gateway failure (`VPNEgressSteeringStale` critical,
+`VPNEgressUplinkLabelMissing` warning) plus the KSM allowlist entry that makes them non-inert.
+
+Confirmed first that they are **live on prod, not decorative**: `k3s-prod-3` carries
+`egress-gateway=true` and `egress-uplink=10.150.192.86-139`, `vpn-egress-steering` has been in place 14
+days, and the label was exported on **0 series** before this change — so the allowlist entry is the
+switch that turns the alerts on.
+
+Three whole-repo checks failed the branch, each a real gap: the runbook the alerts link to did not
+exist (a firing alert would hand the operator a 404); `alerts.yaml` is an already-covered file, so new
+rules in it must be unit-tested; and the scrape-target guard broke. Wrote
+`runbooks/vpn-egress-stale.md` (wired into the generator, four required sections) and
+`platform/monitoring/tests/vpn-egress.test.yaml` (four promtool fixtures — tests 1 and 2 are the SAME
+reconnect with and without the label following it, which isolates the mechanism: a label value change
+retires one series and starts another rather than changing a sample).
+
+**Two of the three failures were defects in the checks, not the branch:**
+
+- `alert-metrics-have-scrape-targets` asserted the allowlist matched `nodes=[...egress-gateway]` with a
+  literal closing bracket — that the list held *exactly* that one label. Correctly adding a second label
+  therefore failed the check while the guarded property stayed satisfied. A guard that breaks when you
+  correctly extend the thing it guards teaches people to edit the guard, which is how it stops guarding.
+- `UNTESTED_ALERTS` flagged `VPNTunnelDown` as stale because "asserted on" is a **bare word match over
+  the test file** — my header *comment* mentioning it counted as coverage. I did not delete the register
+  entry to make that go away; instead the fixtures now assert `VPNTunnelDown` **quiet** in all four
+  cases. That required making them faithful: they had no `node_network_receive_bytes_total{device="tun0"}`
+  at all, so `VPNTunnelDown` would have been firing throughout unnoticed, because promtool only checks
+  the alertnames you name. A healthy tun0 passes traffic — and that silence IS the claim these rules
+  exist for.
+
+**Known, deliberately not fixed:** that word match means any alert named in prose counts as tested.
+Tightening it to `alertname:` positions could expose other prose-only "coverage", and the register is
+shrink-only, so the cascade wants doing deliberately — not at 01:00 unattended.
+
+### 5. Stale worktrees: audited, deliberately not touched
+
+18 agent worktrees remain under `.claude/worktrees/`. **14 clean, 4 dirty** — the dirty ones being
+`fix/gate-does-not-pass-on-skipped-stages`, `fix/postbuild-substitution-catches-dollar-mangling`,
+`fix/scrape-targets-for-inert-alerts` and `fix/qbittorrent-durable-webui-credential`. Every branch is
+already merged into main, so the uncommitted state is leftover agent scratch rather than unmerged work.
+
+Left in place. They no longer affect the gate (the scanner now skips dot-directories), so removal is
+disk hygiene, and deleting someone's uncommitted changes unasked is not a call to make while they are
+asleep. `git worktree remove` preserves branches and their commits, so only that scratch is at risk.
+
+### PM HANDOFF
+
+1. **R-STATE-02 is still the Phase 7 exit item, and it is blocked on permission, not on readiness.**
+   `./conformance/run.sh --layer l3 --requirements R-STATE-02 --record`. The permission classifier
+   refuses it and I did not route around it — hand-rolling the cordon would defeat the guard. I read the
+   test first: it cordons ONE node and evicts only its OWN writer pod, never `kubectl drain`; kubelet
+   and etcd stay up so quorum is untouched; PVC and pods live in the per-test namespace; and `finally`
+   uncordons via a unique label even on failure or panic. Preconditions verified: 3 Ready nodes,
+   `longhorn` default SC, nothing pre-cordoned, 24/24 volumes healthy. Either run it, or add a Bash
+   permission rule.
+2. **The memory swap is still the cheapest R-STATE-02 evidence** — a supervised single-node loss on prod
+   is the event that requirement describes. Node 1 currently holds no DB primary, which is the ideal
+   window.
+3. **Node 2 DIMM re-inventory** — blocked on the vault become password. Never `ansible-inventory --list`.
+4. **Pi-hole needs A records** for `unifi.prod` → `.21` and `crafty-mc.prod` → `.22` before router DHCP
+   option 6 → `.23` is safe. It currently answers `.20` for both, and neither service speaks HTTP
+   through that Gateway VIP, so the LAN-wide change would break UniFi adoption and Minecraft. Until
+   then the per-machine dnsmasq override is load-bearing.
+5. **UniFi guest has no IP** — needs console; `cat /etc/netplan/*.yaml` is the confirming command.
+6. **Five `public` services have no working public path** — waiting on WAN forwarding, deliberately.
+7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, two rendered
+   manifests nothing applies (the keycloak one would stand up a conflicting instance).
+
+## 2026-09-29 — The CI/CD merge gate does not exist: no runner, no branch protection. **PM HANDOFF.**
+
+Went to land the plex claim wiring through the documented PR flow and discovered the flow has
+never run. **PR #1 is the first pull request this repository has ever had.** It is still queued.
+
+### What is actually true of the live Gitea (verified via API, 2026-09-29)
+
+| Claimed (DECISIONS-LOG 2026-08-31) | Measured |
+|---|---|
+| "required status check" on `main` | `branch_protections` returns `[]`. `protected: false`, `enable_status_check: false`, `status_check_contexts: []`, `required_approvals: 0`, `user_can_push: true` |
+| "runner `incluster-dind` online (ns gitea-actions)" | **Zero runners registered** at repo, org and site level |
+
+Both halves of that entry are false. Nothing enforces the gate, and the gate cannot run.
+PR #1's two contexts (`gate / gate`, `gate / conformance`) sat `pending` for 14+ minutes with
+nothing to execute them, and will sit there indefinitely.
+
+This is a **fail-open governance hole**, the same class as the two found on 2026-09-18 — and it
+was invisible precisely because the log asserted the opposite. Every "the gate is enforcing"
+statement since 2026-08-31 rested on an unverified claim.
+
+### Ordering trap — read before fixing
+
+**Do not enable required status checks until a runner is live.** Requiring
+`gate / gate (pull_request)` with no runner deadlocks every PR permanently, including the PR
+that would fix it. Sequence: runner first, verify the gate actually runs and passes on a real
+PR, *then* turn on enforcement with the context name CI actually reported.
+
+Related: `gate.yml`'s `conformance` job is `if: false` by design (honest skip, never fake green),
+but it still publishes a `gate / conformance (pull_request)` context. If that context is ever
+listed as required, it deadlocks every PR for the same reason. Wire it or exclude it.
+
+### Delivery contract written down
+
+Added `CLAUDE.md` at the repo root — the first agent-facing delivery doc here. It states the
+branch -> PR -> gate -> approval -> merge rule, records that enforcement is currently weaker than
+the rule (so agents follow the rule rather than what they can get away with), and carries the
+toolbox-gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and the secret
+handling rules.
+
+It also flags that **37 of 39 runbooks still instruct `git push origin main`**. Those were
+accurate before the PR flow and are stale now. Correcting them is tracked, not done.
+
+### Also found, not fixed
+
+- `gate.yml`'s push trigger lists `feature/**`; this repo's branches use `feat/**`, so branch
+  pushes get no fast-feedback run. PR trigger is unaffected.
+- The gitea-actions runner is GitOps-managed (`apps/wave-1/gitea-actions.yaml` +
+  `gitea-actions-runner-secret.sops.yaml`), so it is deployed but not registering — most likely a
+  stale registration token in that SOPS secret. Needs in-cluster diagnosis (operator).
+- **Plex is still unclaimed and it was never checked** whether it is already claimed. Do that
+  before spending a token — `runbooks/plex-claim.md` §First commands.
+- `runbooks/plex-claim.md` asserts a "~4-minute TTL" for a plex.tv claim token in four places,
+  **uncited**, and the whole stopwatch procedure is built on it. Upstream `pms-docker`'s README
+  says nothing about expiry. The number is plausible community folklore, not a measured fact.
+  The authoritative reading is the countdown rendered on plex.tv/claim at issue time.
+
+### Next steps
+
+1. **Operator:** diagnose the gitea-actions runner; get one registered.
+2. **Operator:** enable branch protection on `main` — push blocked, approvals required — and only
+   then add the gate as a required status check.
+3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
+4. Claim plex, after checking it is not already claimed.
+5. Correct the 37 runbooks' `git push origin main` wording.
+
+## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
+
+Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
+rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
+present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
+library scan is running over the 8.33T NFS share.
+
+### The finding: both halves of the 2026-08-31 CI claim were false
+
+That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
+`incluster-dind` online". Measured against the live API on 2026-09-29:
+
+| Claimed | Measured |
+|---|---|
+| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
+| Runner online | **zero** runners at repo, org AND site level |
+
+**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
+for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
+statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
+and the reason it hid so long is that the log itself was the evidence.
+
+### Root cause of the dead runner
+
+`act_runner` registers **once** and persists its identity in `/data/.runner` on the
+`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
+**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
+runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
+genuine re-registration (`.runner is missing` → `Runner registered successfully`).
+
+It then did not come back for several minutes, because Flux could not recreate the PVC:
+`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
+`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
+`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
+**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
+
+### Corrections to things this log and the runbooks asserted
+
+- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
+  `success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
+  Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
+  than a deadlock, because a deadlock is loud. Never require it; require
+  `gate / gate (pull_request)` alone. Corrected in #3.
+- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
+  With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
+  which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
+  `infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
+- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
+  documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
+  issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
+  `catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
+  comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
+
+### Measured, once the pipeline actually worked
+
+`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
+inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
+That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
+
+### State now
+
+- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
+- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
+- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
+  instruction had become actively blocked, not merely stale.
+- `infra/plex/claim-ship.sh` drives the claim loop end to end.
+
+### Next steps
+
+1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
+   prerequisite is met: the gate has been observed green on real PRs.
+2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
+   org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
+   fixed would leave nobody able to approve.)
+3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
+   bypass. `user_can_push:false` is the API's opinion, not proof.
+4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
+5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
+   FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
+   the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
+6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
+
+## 2026-09-29 (later) — the merge gate is now real, enforced, and proven. **PM HANDOFF.**
+
+Closes the gap opened by the earlier entry today. Eight PRs, all gated; every claim below was
+measured against the live API or cluster, not inferred from this repository.
+
+### Final state
+
+| | Verified |
+|---|---|
+| `main` protected | `protected:true`, `user_can_push:false` — and a direct push as site-admin `gitea-admin` was **provoked and rejected** (`pre-receive hook declined`) |
+| Gate required | `enable_status_check:true`, contexts `["gate / gate (pull_request)"]` |
+| Approvals | `required_approvals:1`; `jakman85` holds `owner` |
+| Runner | `incluster-dind` online, now a **native sidecar** (`initContainers`, `restartPolicy: Always`, TLS-verified `docker info` startupProbe) |
+| Gate time | ~1m35s |
+| Plex | claimed 2026-09-29; `PlexOnlineToken` present on the `plex-config` PVC |
+
+### What was corrected in the documentation
+
+`CLAUDE.md` §2b — *never infer platform state from this repository* — is the durable outcome.
+The 2026-08-31 entry asserting an enforced gate is now annotated **in place**. The "Phase 0 /
+no cluster exists" framing across `ci/`, `clusters/` and `conformance/` READMEs is banner-marked
+as describing a live gap rather than a stage. All 37 runbooks now describe the PR flow. Two
+`.sops.yaml` headers and `catalog/plex.yaml` were corrected (comment edits verified not to break
+the SOPS MAC, and not to move the rendered golden).
+
+### A claim of mine that shipped wrong, and is now corrected in place
+
+PR #7 stated the sidecar conversion would remove the one-per-start dockerd line
+`http: TLS handshake error from 127.0.0.1: EOF`. **It does not.** Exactly one still appears,
+immediately after `API listen on [::]:2376` — it is now the startupProbe's first attempt,
+landing microseconds after dockerd begins listening and cut off by `timeoutSeconds: 1`; the
+next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one
+benign startup line would slow every subsequent poll. The manifest comment now says so.
+
+### The root cause nobody has fixed yet
+
+Gitea's repo settings live in its own database — invisible to Flux and unreadable by the
+conformance gate. **No check exists that could have caught "protection off, runner gone."**
+`infra/gitea/` declares and can re-assert the desired state, but it is not a detector. A
+conformance check that reads the live Gitea API and fails on divergence is the highest-value
+remaining work in this repo, and it is the difference between this having been fixed and this
+being unable to recur.
+
+### Next steps
+
+1. **Write that Gitea-settings conformance check.** Everything else here is a symptom.
+2. Wire `vars.CLUSTER_AVAILABLE` + `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` so **L2/L3 runs in
+   CI**. A green PR currently proves the L0/L1 offline subset only — runtime behaviour is
+   unverified on every merge.
+3. Re-run `infra/gitea/verify-then-enforce.sh` after any future act-runner change.
+4. `required_approvals: 1` means a PR you open as `jakman85` needs `gitea-admin` to approve.
+   If that friction bites, either open agent work as `gitea-admin` and review as `jakman85`, or
+   drop back to relying on the status check alone.
+
+## 2026-09-29 (later still) — the enforcement we "turned on" was overridable by the account that merges
+
+`enable_status_check: true` and `required_approvals: 1` were set earlier today and recorded as
+done. **They did not bite.** `block_admin_merge_override` was `false`, and Gitea lets repository
+admins override merge requirements without it. `jakman85` holds `owner`, so both requirements
+were bypassable by exactly the account that performs every merge.
+
+**The evidence was already on the record and nobody read it:** PRs #5, #6, #7, #8 and #9 all
+merged while `required_approvals: 1` was in force, with **zero approvals recorded on any of
+them**. A one-line check (`/pulls/{n}/reviews`) would have shown it at any point.
+
+Two process failures, both mine, both the same shape as the original defect:
+
+1. **I verified with the wrong view.** `/branches/main` reports `protected`,
+   `enable_status_check` and `required_approvals` but NOT `block_admin_merge_override`. It
+   showed everything green while the rule was overridable. The authoritative object is
+   `/branch_protections/main` — read it and enumerate its keys.
+2. **I probed for a field name I assumed existed.** This Gitea (1.24.3) has no
+   `apply_to_admins`. Asking `jq` for it returns `null`, which reads exactly like "set to
+   false" but means "key absent" — so a PATCH that silently did nothing looked like a real
+   before/after. **`jq` returning null is not a value; it may be an absence.** Enumerate with
+   `to_entries`.
+
+Now `block_admin_merge_override: true`, applied and read back. `infra/gitea/require-gate.sh`
+sets it, and `infra/gitea/block-admin-override.sh` fixes it in isolation.
+
+**This is still only a setting.** It is proven when a green PR is REFUSED a merge until
+approved — and that has not been observed yet. This PR is the test: if it merges without an
+approval, the fix did not work and the finding stands.
+
+### The governance question this exposes
+
+There are two accounts: `jakman85` (owner, the human) and `gitea-admin` (the agent). With
+approvals now genuinely required, an agent-authored PR is approved by the human — which is the
+intent. But a PR authored by `jakman85` needs `gitea-admin` to approve it, i.e. the automation
+account rubber-stamping the human. That is ceremony, not review. Options: keep it and accept
+the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
+that actually verifies something; or add a second human account. **This is a decision for the
+operator, not a default to be chosen silently.**
+
+## 2026-09-29 (end of session) — **PM HANDOFF.** Six PRs open; read this before touching anything.
+
+The session that began "set a SOPS secret" ended having rebuilt the delivery pipeline. Plex is
+claimed and scanning. `main` is genuinely protected and the gate genuinely enforces. What is
+open is listed below with the reason each is not merged.
+
+### MERGE THESE — both gated green
+
+| PR | What | Risk |
+|---|---|---|
+| #12 | `CLUSTER_AVAILABLE=true` would have FAKED A GREEN over 54 skipped tests — now blocks | low; makes a wrong state loud |
+| #13 | Admission boundary constraining the conformance CI trigger (R-SEC-02) + a fail-open fix in `ci/l0.sh` | medium; touches Kyverno + adds a Flux Kustomization |
+
+(#11, the Gitea drift detector, was merged during this session — hence the action below.)
+
+Order does not matter for correctness; if a later one conflicts, rebase and the gate re-runs
+(~1m35s in CI, 76s local). **Each was gated in the pinned toolbox, not on a host.**
+
+### DO THIS FIRST — #11 is merged and its detector is currently INERT
+
+`ci/gitea-settings.sh` SKIPS until its token exists, and **a skipped detector detects nothing**:
+
+    ./infra/gitea/set-ci-token.sh          # mints + stores GITEA_SETTINGS_TOKEN, never prints it
+
+Then open any PR and confirm the gate prints `gitea-settings: green`. ONLY THEN uncomment
+`REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml`, so a future missing token blocks
+instead of skipping. Doing that before seeing it green locks the repo.
+
+Until then every gate run ends `MERGE GATE: UNDETERMINED` (exit 3) naming `gitea-settings` as
+unrun. That is BY DESIGN and `gate.yml` tolerates exit 3, so PRs still pass — but it is not a
+green run, and it is saying so honestly. The token is what turns it green.
+
+### A DIFFERENT COMPUTER NEEDS THREE THINGS THAT ARE NOT IN GIT
+
+This is the part that will silently waste an hour otherwise. None of these are in the repo, by
+design, and without them the toolbox and the cluster are unreachable:
+
+1. **`~/.config/sops/age/keys.txt`** — the age PRIVATE key. Without it NO `*.sops.yaml` can be
+   read or edited, and `sops` will fail in a way that looks like a broken file. Copy it from this
+   workstation; the public recipient in `.sops.yaml` is only half the pair.
+2. **`infra/ansible/.cache/prod.kubeconfig`** — untracked, so it exists only in the main checkout
+   on this machine. Without it `dev.sh` refuses to start. In a git WORKTREE it is absent even
+   here: use `DEV_KUBECONFIG=/dev/null` for anything that needs no cluster (see `CLAUDE.md` §3).
+3. **A Gitea credential** for `gitea.home.keithjackson.dev` (this session used the stored
+   `~/.git-credentials`). `git push` and every `infra/gitea/*.sh` script needs it.
+
+### STATE, MEASURED — do not re-derive it from this log, re-measure it
+
+`CLAUDE.md` §2b exists because this log was confidently wrong for a month. Verify before relying:
+
+    protected:true  user_can_push:false  enable_status_check:true
+    status_check_contexts:["gate / gate (pull_request)"]  required_approvals:1
+    block_admin_merge_override:true      runner incluster-dind: online      gate ~1m35s
+
+### NEXT WORK, in priority order
+
+1. **Finish the L2/L3 wiring** now that the credential model is decided and #13 lands the
+   boundary. Still missing (`docs/plans/l2-l3-conformance-ci-wiring.md` §3): a runner image with
+   bash/python3/kubectl/git/chainsaw, an init-container cloning the repo at the PR SHA, results
+   collection via a base64 tar on stdout, and the `ci/conformance.sh` rewrite to drive the Job.
+   **Tighten the admission policy's image pattern to the literal digest when that image exists.**
+2. **Do NOT set `vars.CLUSTER_AVAILABLE=true`** until step 1 is done. It does not enable L2/L3; it
+   only changes which way the stage fails (see #12).
+3. The `gitea-settings` detector covers branch protection + runners. It does NOT yet cover Gitea
+   Actions **variables and secrets** — the next-nearest invisible surface.
+4. `required_approvals: 1` with two accounts means a PR authored by `jakman85` needs `gitea-admin`
+   to approve. The operator holds ultimate authority and does not require that review; revisit if
+   it becomes friction.
+
+### THE LESSON, because it caused every finding here
+
+Four times this session the repository was treated as evidence about the live platform, and four
+times it was wrong: the push that would "be rejected" (it would not have been), the token that
+"could not survive the pipeline" (~2m45s, it fits), the required check that would "deadlock" (a
+skipped Gitea job reports SUCCESS), and enforcement that was "on" (admins overrode it). Two
+fail-opens in the gate itself were found the same way — `kyverno test` certifying policies that do
+not load, and `CLUSTER_AVAILABLE` certifying 54 skipped tests as passed.
+
+**Measure, then assert. `CLAUDE.md` §2b is the rule; this entry is the evidence for it.**
+
+## 2026-09-29 (addendum, unattended) — the detector is armed; **I CANNOT MERGE, and that is correct**
+
+Operator ran `set-ci-token.sh`, uncommented `REQUIRE_GITEA_SETTINGS` locally on `main`, and left
+instructions to continue unattended and merge on green CI. Two things came out of that.
+
+### 1. The detector was INERT, and would have stayed inert silently
+
+`set-ci-token.sh` PUT the secret as `GITEA_SETTINGS_TOKEN`. **Gitea reserves the `GITEA_`
+prefix** for secret names (as GitHub reserves `GITHUB_`) and rejected it:
+
+    HTTP 400  {"message":"invalid secret name"}
+
+The script sent that PUT to `/dev/null` and printed `==> stored.` unconditionally; `curl -sS`
+exits 0 on an HTTP error. **A token existed; no secret did.** So `ci/gitea-settings.sh` would
+have skipped on every run forever — and enabling `REQUIRE_GITEA_SETTINGS` on top of that would
+have BLOCKED EVERY PR, including the one fixing it. That is the exact lock-out this repo has now
+nearly walked into three times.
+
+Fixed in **PR #15**: the secret is `SETTINGS_API_TOKEN`, and `set-ci-token.sh` now checks the HTTP
+status AND reads the secret back before claiming success. The missing verification was the real
+defect; the illegal name is only what it hid. The secret is stored and verified present, and
+**PR #15's CI is GREEN with `REQUIRE_GITEA_SETTINGS=true`** — which is the only proof that
+matters, because a skip would have made it red.
+
+The operator's local edit could not be pushed to `main` (protected). It is carried in PR #15.
+
+### 2. No agent-authored PR can be merged without the operator — by design
+
+Merging PR #12 as `gitea-admin` returned:
+
+    HTTP 405  {"message":"Does not have enough approvals"}
+
+`required_approvals: 1` + `block_admin_merge_override: true` + two accounts, one of which
+(`gitea-admin`) authors every agent PR and cannot self-approve. **So "merge on green CI" is not
+something an agent can execute here.** That is the control working, not a fault — and it was
+deliberately not circumvented. Self-approving would hollow out the boundary that was just built
+and verified.
+
+**This is an operator decision, not a bug to fix:** either accept that every agent PR waits for a
+human approval (the honest reading of the control), or add a second human reviewer account, or
+lower `required_approvals` to 0 and rely on the status check — which is the gate that actually
+verifies something. Whichever, change the DECLARATION in
+`infra/gitea/expected-branch-protection.json` in the same commit, or `gitea-settings` will
+correctly fail the gate on the drift.
+
+### Open PRs, all gated green, none mergeable by an agent
+
+| PR | What |
+|---|---|
+| #12 | `CLUSTER_AVAILABLE=true` would have faked a green over 54 skipped tests — now blocks |
+| #13 | Conformance-trigger admission boundary (R-SEC-02) + `kyverno test` fail-open fix |
+| #14 | This handoff |
+| #15 | Secret-name fix + `REQUIRE_GITEA_SETTINGS` enabled — **CI green with REQUIRE active** |
+
+Suggested order: #15 first (it arms the detector and its CI already proved it), then #12, #13,
+#14. If a later one conflicts, rebase; the gate re-runs in ~1m36s.
+
+### Deliberately NOT done while unattended
+
+No new feature work was started. Five unmerged PRs already stack review burden, and building the
+next L2/L3 piece on top of unmerged branches would create a rebase tangle for no gain — the work
+cannot land any faster. The next piece is specified and ready to start once these are merged:
+`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
+boundary landed in #13.

apps/vms/unifi/30-virtualmachine.yaml

@@ -95,8 +95,22 @@ spec:
               disk:
                 bus: virtio
           interfaces:
+            # MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
+            # KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
+            # /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
+            # that file while the NoCloud instance-id is unchanged — and the instance-id here is
+            # domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
+            # handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
+            # administratively DOWN with no address and no DHCP attempt.
+            # That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
+            # device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
+            # (because the per-instance modules were skipped), yet UOS is installed and
+            # qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
+            # Pinning the MAC to the value the current VMI already holds makes the guest's netplan
+            # match permanently, so this cannot recur on the next recreation.
             - name: default
               masquerade: {}
+              macAddress: a6:d7:c0:b3:72:fa
               # Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
               # these and (b) surfaces them as containerPorts on the virt-launcher pod so the
               # LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.

apps/vms/unifi/unifi-uos-installer.sops.yaml

@@ -5,8 +5,8 @@ metadata:
     namespace: unifi
 type: Opaque
 stringData:
-    installer-url: ENC[AES256_GCM,data:CGCWLMiHlF57/w==,iv:/tilBVekwbHkAjSBi5UKUljfaqPo/hXhNqMYS4LY/kA=,tag:b7SeEM30ua/HvutGHYEdXg==,type:str]
-    console-password: ENC[AES256_GCM,data:gkPbmZ4g7he2WA==,iv:T9PUgAmOsrt4LUB+5vXBr9ctOPsXP5z/vMWr2ls+3ko=,tag:83HDr4kCrUF7u83fpIkm4A==,type:str]
+    installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
+    console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -16,14 +16,14 @@ sops:
         - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIV3hvNUVRTWxINmtaQkp5
-            a0VkYitYNHFWUGRhYncxVXl4MFdET1VmQTJZCnVXaDhaRHpQVkdHWG4xUElqRFNG
-            VUhPU3dpa0VMWGxQdUxGeDZuNFZWVm8KLS0tIEFMY2RNVDBvMG51SkhqVzlXUjRR
-            TG9kcHdSTU5QOGpnLzkyVW9hUWJLem8KW536eEsviMq5/uyg3/RT/PrEgr4PBL3E
-            yNemOTU76DdU81/neHb/6NVWT44ffLoCltgSrCQ0ieBwDAorjQ3bEg==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
+            WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
+            TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
+            cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
+            kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-09-19T01:51:00Z"
-    mac: ENC[AES256_GCM,data:yKGOdLKnIUUeSyJuzyp4CiJFhTyIJv6S8PTEdnioCpeSoRpptWYQIKR8BUrYx1/4NnghvOEyS5o0nRT56dq7vf2G+CoRf03/alDW/YHUjkAfIRjorPCO638mjdIPDytfsTBOyBXnjzjMlhHwoDjHskk+J66Qm42xFd476w9cYig=,iv:M1SwKW2ZUehklgUP+n3fefb2XoDDPBs5HnTklVh3KwM=,tag:q6eoA+pdHnwWKxQODCN7jQ==,type:str]
+    lastmodified: "2026-09-19T02:47:13Z"
+    mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

apps/wave-0/kustomization.yaml

@@ -17,6 +17,7 @@ resources:
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
+- plex-claim-secret.sops.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml

apps/wave-0/plex-claim-secret.sops.yaml

@@ -0,0 +1,77 @@
+# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
+# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
+#
+# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
+# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
+# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
+# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
+# not prose, so the upstream conditions below are described in words.
+#
+# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
+# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
+# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
+# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
+# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
+#
+# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
+# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
+# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
+# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
+#
+# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
+# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
+# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
+#
+# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
+# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
+# the same file extracted from the running `plexinc/pms-docker:latest` image):
+#   - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
+#     PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
+#     skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
+#     README: "If server is already logged in, this parameter is ignored."
+#   - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
+#     empty body. curl's exit status is never checked, the sed for the authentication-token element
+#     yields the empty string, and the success branch at line 120 simply does not fire. There is no
+#     else branch and no non-zero exit; the script has no `set -e` and ends by touching
+#     /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
+#     It does NOT crashloop.
+#   - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
+#     if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
+# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
+# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
+# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
+# by hand at claim time.
+#
+# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
+#   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+# Never decrypt this file into a shell, a log, or a shared terminal.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: plex-claim
+    namespace: plex
+    labels:
+        app.kubernetes.io/name: plex
+type: Opaque
+stringData:
+    PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
+            cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
+            THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
+            QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
+            RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-29T06:31:16Z"
+    mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/plex.yaml

@@ -132,6 +132,11 @@ spec:
           value: Etc/UTC
         - name: ALLOWED_NETWORKS
           value: 10.0.0.0/8,192.168.0.0/16
+        - name: PLEX_CLAIM
+          valueFrom:
+            secretKeyRef:
+              name: plex-claim
+              key: PLEX_CLAIM
         volumeMounts:
         - name: config
           mountPath: /config

apps/wave-1/gitea-actions-runner-secret.sops.yaml

@@ -4,7 +4,13 @@
 # against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
 # decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
 # in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
-# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
+# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
+# Gitea API and writes it here via `sops set`, without the value ever being printed.
+# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
+# it still decrypts, redirect: `sops -d <file> > /dev/null`.
+# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
+# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
+# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
 apiVersion: v1
 kind: Secret
 metadata:
@@ -14,7 +20,7 @@ metadata:
         app.kubernetes.io/name: act-runner
 type: Opaque
 stringData:
-    token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
+    token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -30,8 +36,8 @@ sops:
             MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
             UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-08-26T04:48:14Z"
-    mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
+    lastmodified: "2026-09-29T05:48:35Z"
+    mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

apps/wave-1/gitea-actions.yaml

@@ -99,70 +99,79 @@ spec:
       # CI holds no cluster credentials — do not mount a SA token into the runner or
       # (via the shared netns) the job containers (R-SEC-02, docs/14).
       automountServiceAccountToken: false
+      # ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
+      #
+      # NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
+      # `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
+      # running alongside them, and is terminated after them. kubelet holds `runner` until
+      # this container's startupProbe passes, so the ordering is enforced by the kubelet
+      # instead of hand-rolled shell in the runner.
+      #
+      # This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
+      # Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
+      # postBuild substitution over this ENTIRE manifest, comments included, and those loops
+      # depended on bare dollar-names surviving it untouched. There is now no shell here at
+      # all, so there is nothing to escape or mis-escape.
+      #
+      # WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
+      # re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
+      # and only THEN begins listening, so the port can accept a connection while the certs
+      # are absent or incomplete -- and act_runner hard-exits on a missing
+      # /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
+      # are usable and that the daemon answers.
+      #
+      # CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
+      # introduced this claimed the conversion would REMOVE the one-per-start dockerd line
+      # `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
+      # appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
+      # attempt, landing microseconds after dockerd begins listening and cut off by
+      # timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
+      # was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
+      # startup line would slow every subsequent poll. Left documented rather than silenced.
+      #
+      # periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
+      initContainers:
+        - name: dind
+          restartPolicy: Always   # <- this field is what makes it a SIDECAR
+          image: docker:27-dind
+          securityContext:
+            privileged: true
+          env:
+            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
+            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
+            - name: DOCKER_TLS_CERTDIR
+              value: /certs
+          volumeMounts:
+            - name: certs
+              mountPath: /certs
+            - name: dind-storage
+              mountPath: /var/lib/docker
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 2Gi
+          startupProbe:
+            exec:
+              command:
+                - docker
+                - --host=tcp://127.0.0.1:2376
+                - --tlsverify
+                - --tlscacert=/certs/client/ca.pem
+                - --tlscert=/certs/client/cert.pem
+                - --tlskey=/certs/client/key.pem
+                - info
+            periodSeconds: 1
+            failureThreshold: 60
       containers:
         # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
         - name: runner
           image: gitea/act_runner:0.2.11
-          # TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
-          # entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
-          # boot and runs `act_runner daemon`.
-          #
-          # STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
-          #   and hard-exits if it is missing (a startup race against the sidecar).
-          # STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
-          #   necessary but NOT sufficient. dockerd writes the certs and only THEN begins
-          #   listening on :2376, so a runner that proceeds on the file alone still hard-exits
-          #   with "cannot ping the docker daemon at tcp://localhost:2376" — observed
-          #   crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
-          # Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
-          # surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
-          #
-          # ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
-          #   braced form or the double-dollar form; both are wrong here, for different reasons.
-          #   clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
-          #   Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
-          #   Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
-          #   and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
-          #     - A bare  $name  is NOT a reference to envsubst: it passes through untouched, and
-          #       that holds even when the name IS set in cluster-vars. So $i and $j reach the
-          #       cluster verbatim and both 60s bounds below are real, enforced bounds.
-          #     - $((i + 1))  is likewise untouched: "(" cannot begin a variable name.
-          #     - A BRACED dollar-name IS a reference and gets replaced — with the empty string
-          #       when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
-          #       which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
-          #     - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
-          #       form — still wrong for us, because on any path where envsubst does NOT run
-          #       (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
-          #       reads that leading double-dollar as its own PID, so a bound would compare a
-          #       string like 1234{i} against 60. The bare form is the only spelling that is
-          #       correct on BOTH paths.
-          #     - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
-          #       name") that fails this whole Kustomization — and this Kustomization gates every
-          #       merge in the repo. So never write one anywhere in this file, INCLUDING in a
-          #       comment. That is why this note names the three forms in words rather than
-          #       showing them: an earlier draft of it spelled them out literally and broke
-          #       `flux envsubst` on the file. (Same trap the indexed-expansion warning in
-          #       clusters/base/apps-vms.yaml describes.)
-          command:
-            - /bin/sh
-            - -c
-            - |
-              # Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
-              i=0
-              until [ -f /certs/client/ca.pem ]; do
-                i=$((i + 1))
-                [ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
-                echo "waiting for dind TLS certs ($i)"; sleep 1
-              done
-              # Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
-              # Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
-              j=0
-              until nc -z -w1 127.0.0.1 2376; do
-                j=$((j + 1))
-                [ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
-                echo "waiting for dind daemon on :2376 ($j)"; sleep 1
-              done
-              exec /sbin/tini -- /opt/act/run.sh
+          # No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
+          # registers from env on first boot and runs `act_runner daemon`. The two wait-loops
+          # that used to live here are gone: kubelet now holds this container until the dind
+          # sidecar's startupProbe passes (see initContainers above).
           env:
             - name: GITEA_INSTANCE_URL
               value: https://gitea.${BASE_DOMAIN}
@@ -198,52 +207,6 @@ spec:
               memory: 128Mi
             limits:
               memory: 512Mi
-        # ---- Docker-in-Docker: the job executor daemon (privileged) ----
-        #
-        # FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
-        #   Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
-        #   k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
-        #   ordering that the loops above hand-roll — which also removes the envsubst hazard
-        #   from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
-        #   Shape to build:
-        #     1. Move this whole container into `initContainers:` as the FIRST entry.
-        #     2. Add `restartPolicy: Always` to it. That single field is what makes an
-        #        initContainer a sidecar: it starts before the regular containers, keeps
-        #        running alongside them, and is terminated after them.
-        #     3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
-        #        connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
-        #        --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
-        #        --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
-        #        it would re-open the same "port accepting but TLS/certs not ready" gap that
-        #        Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
-        #        preserves today's 60s cap.
-        #     4. kubelet then holds the `runner` container until that startupProbe passes, so
-        #        `runner` can drop its `command:` override completely and run the image's
-        #        default entrypoint.
-        #   Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
-        #   entire repository. If the conversion is wrong, nothing can merge — so it must land
-        #   alone, where the failure is unambiguous and not entangled with anything else, and
-        #   only after a real workflow run has been observed green on the converted pod.
-        - name: dind
-          image: docker:27-dind
-          securityContext:
-            privileged: true
-          env:
-            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
-            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
-            - name: DOCKER_TLS_CERTDIR
-              value: /certs
-          volumeMounts:
-            - name: certs
-              mountPath: /certs
-            - name: dind-storage
-              mountPath: /var/lib/docker
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              memory: 2Gi
       volumes:
         - name: data
           persistentVolumeClaim:

catalog/plex.yaml

@@ -32,11 +32,46 @@ env:
   # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
   - name: ALLOWED_NETWORKS
     value: 10.0.0.0/8,192.168.0.0/16
-  # PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
-  # shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
-  # is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
-  # single API POST) against the running server; the claim then persists on the /config PVC and
-  # survives pod rolls (including the roll this env change causes), so it never belongs in git.
+  # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
+  # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
+  # workload). Plex cannot create libraries over the mounted bulk share while the server is
+  # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
+  # half of that reasoning survived contact with the upstream source.
+  #
+  # A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
+  # expires in MINUTES — read the countdown the claim page renders; do not budget against a number
+  # written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
+  # UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
+  # for the whole procedure design — including a since-disproven claim that the delivery pipeline was
+  # too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
+  #
+  # The live token is injected by the operator inside that window, with an explicit `flux reconcile`
+  # to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
+  # the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
+  # committed value is the spent token from that claim rather than the original placeholder.
+  #
+  # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
+  # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+  # the running `:latest` image), NOT assumed:
+  #   - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
+  #     PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
+  #     no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
+  #     logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
+  #   - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
+  #     status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
+  #     there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
+  #     SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
+  #     default 0, which continues startup even when a cont-init script fails.
+  # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
+  # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
+  # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
+  # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
+  # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
+  - name: PLEX_CLAIM
+    valueFrom:
+      secretKeyRef:
+        name: plex-claim
+        key: PLEX_CLAIM
   # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
   # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
   # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
@@ -56,10 +91,12 @@ storage:
                               # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
                               # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
                               # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
-                              # makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
-                              # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
-                              # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
-                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
+                              # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
+                              # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
+                              # exchange is then skipped on every later start. (The older note here claimed a
+                              # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
+                              # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
+                              # failure, and that is why one is committed.)
   # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
   # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
   # downloads/ movies/ photos/ tv/).

catalog/render.py

@@ -1844,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
     convention:
       - `<svc>.yaml`                        -> wave(svc)
       - `<svc>-bootstrap-secret.sops.yaml`  -> wave(svc)   (consumed by svc's env, same apply)
+      - `<svc>-claim-secret.sops.yaml`      -> wave(svc)   (a one-shot upstream-service claim token
+        consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
+        bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
+        in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
       - `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc)  (forward-auth sidecar OIDC creds)
       - `<svc>-db.yaml`                     -> wave(svc-db) (a strictly-lower wave; CNPG DB)
       - `<svc>-db-app.sops.yaml`            -> wave(svc-db) (CNPG initdb owner secret)
@@ -1870,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
         w = waves[name]
         add(w, f"{name}.yaml")
         add(w, f"{name}-bootstrap-secret.sops.yaml")
+        # A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
+        # app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
+        # so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
+        # Placed only when the file exists, so services without one are unaffected.
+        add(w, f"{name}-claim-secret.sops.yaml")
         # forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
         # Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
         # so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
@@ -2548,7 +2557,8 @@ def _selftest() -> int:
     # 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
     #     EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
     present = {
-        "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
+        "a.yaml", "a-claim-secret.sops.yaml",
+        "svc.yaml", "svc-bootstrap-secret.sops.yaml",
         "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
         "svc-db-backup-credentials-prod.sops.yaml",
         "gitea.yaml",
@@ -2559,6 +2569,10 @@ def _selftest() -> int:
     plan = plan_apps_waves(plan_sds, present)
     flat = {f: wv for wv, fs in plan.items() for f in fs}
     check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
+    check(flat.get("a-claim-secret.sops.yaml") == 0,
+          "a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
+          "as its app — an env secretKeyRef has no `optional`, so a later wave means "
+          "CreateContainerConfigError")
     check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
           and flat.get("svc-db-backup-credentials.sops.yaml") == 0
           and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,

ci/Makefile

@@ -32,6 +32,9 @@ render-golden: ## render==golden: canonical render reproduces committed apps/ (n
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
+gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
+	@./gitea-settings.sh
+
 conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
 	@"$(CI_DIR)conformance.sh"
 

ci/README.md

@@ -1,5 +1,19 @@
 # `ci/` — portable merge gate (runnable today, without Gitea)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 **Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
 
 The same merge gate the Gitea Actions workflow runs

ci/gate.sh

@@ -112,6 +112,7 @@ run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
 echo >&2
@@ -188,6 +189,31 @@ case "${OUTCOME[spec-immutability]}" in
   pass) ok   "spec-immutability: green" ;;
 esac
 
+# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
+# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
+# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
+# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
+# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
+# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
+# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
+# token is wired, and the skip becomes a block as it should.
+case "${OUTCOME[gitea-settings]}" in
+  fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
+  skip)
+    if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
+      err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
+      summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
+      BLOCK=1
+    else
+      warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
+      warn "  registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
+      warn "  (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
+      summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
+      NOT_ESTABLISHED+=("gitea-settings")
+    fi ;;
+  pass) ok "gitea-settings: green" ;;
+esac
+
 # L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
 # unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
 # degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a

ci/gitea-settings.sh

@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
+# match what git declares.
+#
+# WHY THIS STAGE EXISTS AT ALL
+# ----------------------------
+# Every other check in this repo reads the repository, the rendered manifests, or the
+# cluster. NONE of them can see Gitea's own database — and that is where branch
+# protection and runner registration live. On 2026-09-29 that blind spot was measured:
+#
+#   * `main` had NO branch protection, and
+#   * ZERO Actions runners were registered,
+#
+# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
+# and human in the loop repeated that claim because the repository said so. PR #1 — the
+# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
+# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
+# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
+# merged with zero approvals recorded against `required_approvals: 1`.
+#
+# Three separate failures, one shape: a setting that READ as enforced while not being, with
+# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
+# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
+#
+# WHAT IT ASSERTS
+#   1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
+#      LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
+#   2. at least `.runners_online_min` Actions runners report status "online".
+#
+# (2) is not decoration. A required status check with no runner is the deadlock this repo
+# nearly walked into: every PR pends forever, including the one that would fix it.
+#
+# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
+# `protected`, `enable_status_check` and `required_approvals` but NOT
+# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
+# only account that merges. This stage queries /branch_protections/ deliberately.
+#
+# EXIT CODES (the gate contract)
+#   0  live Gitea matches the declaration
+#   1  DIVERGENCE — something drifted, or was never applied. Blocks the merge.
+#   3  SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
+#      skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
+#      that an unwired credential cannot lock the repository.
+#
+# TO WIRE IT: create a Gitea repo secret GITEA_SETTINGS_TOKEN (a token that can read repo
+# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
+# the drift it exists to catch is still undetected.
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: Gitea platform settings match the declaration"
+
+DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+TOKEN="${GITEA_SETTINGS_TOKEN:-}"
+
+[ -f "$DECL" ] || die "declaration missing: ${DECL}"
+require_tool jq
+require_tool curl
+
+if [ -z "$TOKEN" ]; then
+  skip "GITEA_SETTINGS_TOKEN is not set — cannot read the live Gitea rule."
+  skip "  Branch protection and runner registration are therefore UNVERIFIED. They were"
+  skip "  wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
+  summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
+  exit 3
+fi
+
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+BRANCH="$(jq -r '.branch' "$DECL")"
+LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
+if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
+  skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
+  skip "  lacks permission). Not treating this as a pass."
+  summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
+  exit 3
+fi
+
+RC=0
+
+# --- 1. every declared protection key must match the live rule, exactly ---------------
+while IFS= read -r key; do
+  want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
+  got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
+  if [ "$want" = "$got" ]; then
+    ok "  ${key} = ${got}"
+  else
+    err "  ${key}: DECLARED ${want} but LIVE is ${got}"
+    RC=1
+  fi
+done < <(jq -r '.protection | keys[]' "$DECL")
+
+# --- 2. a runner must actually be online ----------------------------------------------
+MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
+RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
+N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
+if [ "${N:-0}" -ge "$MIN" ]; then
+  ok "  runners online: ${N} (>= ${MIN})"
+else
+  err "  runners online: ${N}, DECLARED minimum ${MIN}."
+  err "  A required status check with no runner leaves every PR pending forever —"
+  err "  including the PR that would fix it. This is the 2026-09-29 outage condition."
+  RC=1
+fi
+
+if [ "$RC" -eq 0 ]; then
+  ok "Gitea settings match the declaration."
+  summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
+else
+  err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
+  err "  Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
+  err "  state is what you now want, change the DECLARATION in the same commit — deliberately."
+  summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
+fi
+exit "$RC"

ci/l1.sh

@@ -30,11 +30,13 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
 # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
 # touched subset, then COLLECT its artifacts into the CI artifact dirs.
 # --include-scope: union the touched subset with every L1 test DECLARING
-# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
-# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
-# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
-# "someone must remember" is the failure mode this whole change removes; see
-# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
+# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
+# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
+# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
+# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
+# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
+# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
+# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
 : "${WHOLE_REPO_SCOPE:=whole-repo}"
 l1_rc=0
 "$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \

ci/tests/touched-requirements.test.sh

@@ -136,6 +136,65 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
 assert_present "R-ENV-02"     "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
 assert_present "R-GITOPS-01"  "$OUT4" "clusters/* attributes R-GITOPS-01"
 
+# ---------------------------------------------------------------------------------------
+# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
+#
+# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
+# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
+# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
+# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
+# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
+# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
+#
+# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
+# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
+#
+# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
+# from the PATH and not from the content scan.
+
+# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
+# under test here, not just the presence of both arms.
+git -C "$TMP" checkout -q -b feature-apps-vms main
+mkdir -p "$TMP/apps/vms/demo"
+printf 'kind: VirtualMachine\nmetadata:\n  name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
+OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
+assert_present "R-PLAT-03"  "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
+assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
+assert_absent  "R-PLAT-02"  "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
+
+# A plain ServiceDefinition under apps/ gets the container-workload set.
+git -C "$TMP" checkout -q -b feature-apps main
+mkdir -p "$TMP/apps/wave-1"
+printf 'kind: ServiceDefinition\nmetadata:\n  name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
+OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
+assert_present "R-PLAT-02"  "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
+assert_present "R-NET-01"   "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
+assert_present "R-NET-06"   "$OUT_APPS" "apps/* attributes the egress-class requirements"
+assert_present "R-IDP-02"   "$OUT_APPS" "apps/* attributes the auth-mode requirements"
+
+# homepage / headlamp ARE the command-centre surface.
+git -C "$TMP" checkout -q -b feature-c2 main
+mkdir -p "$TMP/platform/headlamp"
+printf 'kind: Deployment\nmetadata:\n  name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
+OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
+assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
+assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
+
+# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
+git -C "$TMP" checkout -q -b feature-assembly main
+mkdir -p "$TMP/platform"
+printf 'resources:\n  - cilium\n  - longhorn\n' > "$TMP/platform/kustomization.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
+OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
+assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
+
 # ---------------------------------------------------------------------------------------
 # THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
 #

ci/touched-requirements.sh

@@ -177,6 +177,23 @@ path_requirements() {
   case "$1" in
     catalog/*)                 echo "R-GITOPS-02" ;;
     schemas/*)                 echo "R-GITOPS-02" ;;
+    # apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
+    # manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
+    # adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
+    # checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
+    # class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
+    # violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
+    # catch it at apply time" is post-merge, and the gate exists to catch it before.
+    #
+    # ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
+    # apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
+    # subject to default-deny (R-NET-01).
+    apps/vms/*)                echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
+    # Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
+    # workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
+    # (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
+    # default-deny (R-NET-01).
+    apps/*)                    echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
     platform/kyverno/*)        echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
     platform/cilium/*)         echo "R-NET-01 R-NET-02 R-NET-05" ;;
     platform/longhorn/*)       echo "R-STATE-01 R-STATE-02" ;;
@@ -208,6 +225,39 @@ path_requirements() {
     platform/keycloak/*)       echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
     platform/oauth2-proxy/*)   echo "R-IDP-02 R-IDP-06" ;;
     platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
+    # The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
+    # rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
+    # platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
+    # itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
+    # docs/12-command-center-ui.md), so editing them must select the C2 tests.
+    platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
+    # KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
+    platform/kubevirt/*)       echo "R-PLAT-03 R-OBS-01" ;;
+    # The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
+    # TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
+    platform/gateway/*)        echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
+    # CoreDNS is cluster DNS resolution (R-DNS-01).
+    platform/coredns/*)        echo "R-DNS-01" ;;
+    # crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
+    # its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
+    platform/crafty/*)         echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
+    # Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
+    platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
+    # runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
+    # resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
+    # is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
+    # runbook change is an observability change.
+    runbooks/*)                echo "R-OBS-04" ;;
+    # platform-config/ is the env-specific configuration Flux substitutes into and applies.
+    platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
+    # platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
+    # Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
+    # file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
+    # change. That is the most consequential single-line edit in the repo and it attributed nothing.
+    platform/kustomization.yaml) echo "R-GITOPS-01" ;;
+    # .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
+    # here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
+    .sops.yaml)                echo "R-SEC-01 R-GITOPS-01" ;;
     conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
     # infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
     # nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but

clusters/README.md

@@ -1,5 +1,19 @@
 # clusters/
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
 
 Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),

conformance/README.md

@@ -1,5 +1,19 @@
 # conformance/ — the home-k3s conformance harness (WS-TEST)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
 on a demo, a screenshot, or "it reconciled": every requirement is proven by an
 **automated, reproducible, behavior-based test** tagged with the requirement ID(s) it

conformance/incluster/README.md

@@ -1,5 +1,19 @@
 # In-cluster conformance Job (L2/L3)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
 cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
 on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster

conformance/runs/20260919T051633Z-6eaaaae/run.log

@@ -0,0 +1,6 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
+Install the pinned binary or run via the in-cluster Job (incluster/).

conformance/runs/20260919T052225Z-9339d5a/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="115.855688" />
+  </testsuite>
+</testsuites>

conformance/runs/20260919T052225Z-9339d5a/run.log

@@ -0,0 +1,232 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 05:22:26 | node-loss | @chainsaw                               | CREATE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | TRY       | BEGIN |
+    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1789795346371286043
+        victim node (currently hosts the attached volume): k3s-prod-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-prod-2 labeled
+        node/k3s-prod-2 cordoned
+        cordoned + labelled victim node: k3s-prod-2
+        pod "writer" deleted from chainsaw-advanced-boxer namespace
+        evicted writer off the victim node; volume detached from k3s-prod-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
+        recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | TRY       | END   |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | FINALLY   | BEGIN |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        node/k3s-prod-2 uncordoned
+        node/k3s-prod-2 unlabeled
+        restored node to service: k3s-prod-2
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-advanced-boxer namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | FINALLY   | END   |
+    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | CLEANUP   | BEGIN |
+    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | DELETE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:24:21 | node-loss | @chainsaw                               | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (115.86s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/ledger.jsonl

@@ -5,3 +5,4 @@
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -256,7 +256,13 @@ SCRAPE_OWNERS: list[dict] = [
         "file": KPS,
         "require": [
             r"metricLabelsAllowlist:",
-            r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
+              # The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
+              # end in a literal `]`, asserting the list held exactly this one label — so adding a
+              # second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
+              # while the property it guards stayed perfectly satisfied. A guard that breaks when you
+              # correctly extend the thing it guards teaches people to edit the guard, which is how it
+              # stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
+              r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
         ],
         "forbid": [],
         "why": "KSM exposes NO custom labels by default. Without the allowlist entry "
@@ -448,9 +454,14 @@ def is_golden_fixture(p: Path) -> bool:
     return "conformance" in parts and "golden" in parts
 
 
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 rule_files = sorted(
     p for p in ROOT.rglob("*.yaml")
-    if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
+    if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
 )
 check(len(rule_files) > 0,
       "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
@@ -619,7 +630,7 @@ def yaml_docs(path: Path) -> list[dict]:
 # job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
 same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
 for p in sorted(ROOT.rglob("*.yaml")):
-    if ".git" in p.parts:
+    if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts):
         continue
     txt = read(p)
     if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:

conformance/tests/l0-static/alert-rules-unit-tested/check.sh

@@ -179,6 +179,10 @@ UNTESTED_FILES = {
 # and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
 # that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
 # detectors are no longer unasserted.
+# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
+# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
+# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
+# had to become an assertion rather than a sentence in a comment.
 UNTESTED_ALERTS = {
     ("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
     ("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
@@ -195,7 +199,6 @@ UNTESTED_ALERTS = {
     ("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
     ("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
     ("platform/monitoring/alerts.yaml", "CertificateNotReady"),
-    ("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
     ("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
     ("platform/monitoring/alerts.yaml", "KeycloakDown"),
     ("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
@@ -280,11 +283,16 @@ FETCH_RE = re.compile(
 #     versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
 #     version, the network and the machine are decided silently at test time, and a missing
 #     toolchain becomes a green run instead of an exit 3.
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
 INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
 for p in sorted(ROOT.rglob("*.sh")):
     rel = str(p.relative_to(ROOT))
-    if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
+    if rel.startswith((".git/", ".claude/", "conformance/runs/", SELF_DIR)):
         continue
     try:
         text = p.read_text()
@@ -319,7 +327,7 @@ def tracked_yaml() -> list[Path]:
         return [ROOT / p for p in out.stdout.split("\0") if p]
     # No usable index (the gate container mounts a worktree whose .git points outside the
     # mount). Fall back to a filesystem walk rather than auditing nothing.
-    return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
+    return sorted(p for p in ROOT.rglob("*.y*ml") if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts))
 
 
 KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
@@ -473,7 +481,34 @@ for rel in sorted(alert_files):
           f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
           f"shrink-only: delete the line so the next missing test is caught")
     text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
-    covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
+    # "COVERED" MUST MEAN ASSERTED, NOT MENTIONED (tightened 2026-09-19).
+    # This used to be a bare word match over the whole test file, so an alert named anywhere —
+    # including in a PROSE COMMENT — counted as tested. Hit for real while adding
+    # platform/monitoring/tests/vpn-egress.test.yaml: its header comment explains that VPNTunnelDown
+    # stays quiet during an egress-steering failure, and that sentence alone was enough to make this
+    # check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Deleting
+    # the register entry would then have recorded coverage that did not exist — the exact "looks fixed"
+    # outcome this check exists to prevent, produced BY the check.
+    # Now an alert counts only when a test file names it in an `alertname:` position, with comment
+    # lines stripped first so a commented-out assertion does not count either.
+    # MEASURED BLAST RADIUS BEFORE TIGHTENING: of 61 alerts across the PrometheusRule manifests, 32 are
+    # asserted in one of the two idioms below, 29 are not mentioned at all (already declared in the
+    # registers), and ZERO are prose-only — so this change failed nothing on the day it landed. That
+    # mattered because the registers are shrink-only: had any alert been prose-only, tightening would
+    # have needed a register entry it is not allowed to add. (The first measurement said 31/30/0 because
+    # it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companions.)
+    text_no_comments = "\n".join(
+        l for l in text.split("\n") if not l.lstrip().startswith("#")
+    )
+    # BOTH promtool idioms count, because both are real assertions:
+    #   alert_rule_test:   - alertname: CNPGPrimariesCoLocated
+    #   promql_expr_test:  - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
+    # Matching only the first produced a FALSE NEGATIVE on CNPGPrimariesCoLocated, which
+    # platform/cnpg/alerts.promtool-test.yaml pins across FOUR scenarios entirely through the second
+    # form. Had that been believed, the "fix" would have been a redundant duplicate test for an alert
+    # already better covered than most.
+    covered = [n for n in names
+               if re.search(rf'(?:alertname:\s*|alertname="){re.escape(n)}\b', text_no_comments)]
     covered_total += len(covered)
     for n in names:
         if n in covered:

conformance/tests/l0-static/alert-runbooks-resolvable/check.sh

@@ -126,9 +126,14 @@ def read(p: Path) -> str:
 SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
 RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
 
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 rule_files = sorted(
     p for p in ROOT.rglob("*.yaml")
-    if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
+    if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and "kind: PrometheusRule" in read(p)
 )
 check(len(rule_files) > 0,
       "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "

conformance/tests/l0-static/placeholder-guard-coupling/check.sh

@@ -238,7 +238,7 @@ def bail(msg):
 
 DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
 SELF_DIR = ROOT / "conformance" / "tests" / "l0-static" / "placeholder-guard-coupling"
-SKIP_DIRS = {".git", "_artifacts", "node_modules", ".venv"}
+SKIP_DIRS = {".git", ".claude", "_artifacts", "node_modules", ".venv"}
 
 # ------------------------------------------------------------------ discover the guard FILE
 # Hardcoding a path is what let a file that never landed disable this whole check.

conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh

@@ -154,8 +154,13 @@ services: list[tuple] = []
 monitors: list[tuple] = []
 unparseable: list[str] = []
 
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 for p in sorted(ROOT.rglob("*.yaml")):
-    if ".git" in p.parts or is_test_fixture(p) or ".sops." in p.name:
+    if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) or is_test_fixture(p) or ".sops." in p.name:
         continue
     txt = read(p)
     if not txt:

conformance/tests/l0-static/whole-repo-invariants-wired/check.sh

@@ -39,20 +39,20 @@ if [ "$FAILED" -ne 0 ]; then
 fi
 
 grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \
-  && grep -q 'conf_dirs_for_scope' "$COMMON_SH"
-chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
+  && grep -q 'conf_dirs_for_scope' "$COMMON_SH" && arc=0 || arc=$?
+chk $arc "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
        "conformance/lib/common.sh no longer resolves the scope annotation — no test can be
        selected BY declaring a scope, so every whole-repo invariant is back to being
        selectable away by the requirement filter"
 
-grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH"
-chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
+grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH" && arc=0 || arc=$?
+chk $arc "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
        "conformance/run.sh does not union scope-declared tests (--include-scope /
        conf_dirs_for_scopes missing) — the annotation would be inert decoration"
 
 grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \
-  || grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH"
-chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
+  || grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH" && arc=0 || arc=$?
+chk $arc "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
        "ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a
        purely requirement-scoped run, and every whole-repo invariant is filtered out of
        exactly the change that breaks it"
@@ -63,16 +63,16 @@ while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <(
   CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE"
 )
 MIN_DECLARED="${MIN_DECLARED:-5}"
-[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ]
-chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
+[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ] && arc=0 || arc=$?
+chk $arc "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
        "only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the
        annotation was stripped from the whole-repo checks, or the resolver stopped matching
        it. Every assertion below would then pass while checking nothing"
 
 for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
   rel="${d#"$ROOT"/}"
-  [ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ]
-  chk $? "${rel}: declares the scope AND carries an executable offline check" \
+  [ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ] && arc=0 || arc=$?
+  chk $arc "${rel}: declares the scope AND carries an executable offline check" \
          "${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an
          unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a
          permanent not-green, not a guard"
@@ -91,25 +91,58 @@ BOGUS_REQ="r-nosuchreq-99"
 sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
 sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)"
 
-[ -z "$sel_without" ]
-chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
+[ -z "$sel_without" ] && arc=0 || arc=$?
+chk $arc "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
        "the control case is broken: --requirements ${BOGUS_REQ} still selected tests
        ($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing
        about --include-scope"
 
+# PARTITIONED BY LAYER, because the union is layer-BOUNDED by design: `--layer l0` must select the
+# scope-declaring L0 tests and NOT the L1 ones (asserted separately below). This loop compared EVERY
+# scope-declaring dir against an `--layer l0` selection, which silently assumed only L0 tests would
+# ever declare the scope — and ci/l1.sh was written from the opposite premise, that a whole-repo
+# invariant could appear "at any layer". The moment the three L1 rendering tests declared it
+# (2026-09-18) this check failed, blaming the union for correctly excluding another layer.
 missing=""
 for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
+  case "$d" in */tests/l0-static/*) ;; *) continue ;; esac
   printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}"
 done
-[ -z "$missing" ]
-chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
+[ -z "$missing" ] && arc=0 || arc=$?
+chk $arc "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
        "the scope union does not actually select:${missing}. This is the blind spot itself: a
        change that attributes no requirement would not run these checks"
 
+# The SAME property one layer up. This is what makes the L1 declaration load-bearing rather than
+# decorative: ci/l1.sh has always passed --include-scope, but until an L1 test declared the scope
+# nothing proved the union reached L1 at all. The 2026-09-04 external-dns merge is the case in
+# point — it attributed no rendering requirement, selected ZERO L1 tests, gated GREEN, and left all
+# three rendering goldens stale.
+sel_l1="$(bash "$RUN_SH" --layer l1 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" \
+          --print-selection 2>/dev/null || true)"
+missing_l1=""
+declared_l1=0
+for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
+  case "$d" in */tests/l1-rendering/*) ;; *) continue ;; esac
+  declared_l1=$((declared_l1+1))
+  printf '%s\n' "$sel_l1" | grep -qxF "$d" || missing_l1="${missing_l1} ${d#"$ROOT"/}"
+done
+[ "$declared_l1" -gt 0 ] && [ -z "$missing_l1" ] && arc=0 || arc=$?
+chk $arc "every scope-declared L1 test is selected under --layer l1 DESPITE a requirement filter matching none" \
+       "scope-declared L1 tests: ${declared_l1}; not selected by --layer l1 --include-scope ${SCOPE}:
+       ${missing_l1:-<none>}. If the count is 0 the rendering goldens are back to being checked only
+       when a touched path happens to attribute R-PLAT-02/R-NET-01/R-NET-05/06/07"
+
+# And the L1 union must not leak the other way either.
+strays_l1="$(printf '%s\n' "$sel_l1" | grep -v '/tests/l1-rendering/' || true)"
+[ -z "$strays_l1" ] && arc=0 || arc=$?
+chk $arc "the scope union stays inside --layer l1 (no cross-layer leakage)" \
+       "--layer l1 --include-scope ${SCOPE} selected non-L1 test(s): $(printf '%s' "$strays_l1" | tr '\n' ' ')"
+
 # The union must not smuggle in another layer's tests when --layer bounds the run.
 strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)"
-[ -z "$strays" ]
-chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \
+[ -z "$strays" ] && arc=0 || arc=$?
+chk $arc "the scope union stays inside --layer l0 (no cross-layer leakage)" \
        "--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')"
 
 # ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN -------------------------
@@ -143,23 +176,23 @@ run_offline() {  # run_offline <dir...> -> echoes rc
 rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")"
 rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")"
 
-[ "$rc_declared" = 3 ]
-chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
+[ "$rc_declared" = 3 ] && arc=0 || arc=$?
+chk $arc "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
        "a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A
        whole-repo invariant reporting 'nothing to audit' means the tree it audits has
        VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's
        rc=3. Exit 0 here turns that back into a silent green"
 
-[ "$rc_undeclared" = 0 ]
-chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
+[ "$rc_undeclared" = 0 ] && arc=0 || arc=$?
+chk $arc "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
        "a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict
        rule leaked onto ordinary scaffolds, which would make every unfinished test block"
 
 # ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass.
 l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")"
 printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \
-  && printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3'
-chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
+  && printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3' && arc=0 || arc=$?
+chk $arc "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
        "ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared
        check that SKIPPED would be reported as a PASS — the exact false green the deleted
        stages were written to prevent"
@@ -193,8 +226,8 @@ done
 
 # ---- (5) the gate still BLOCKS on an l0 failure -----------------------------------------
 # Carried over from the three deleted stages' own checks — asserted once, here.
-grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH"
-chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \
+grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH" && arc=0 || arc=$?
+chk $arc "ci/gate.sh sets the merge block when the l0 stage fails" \
        "ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above
        would hold while a red L0 stage merged anyway"
 

conformance/tests/l1-rendering/container-workload-render/chainsaw-test.yaml

@@ -23,6 +23,17 @@ metadata:
     conformance.home.k3s/requirements: "R-PLAT-02"
     conformance.home.k3s/layer: "L1"
     conformance.home.k3s/safety-critical: "false"
+    # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
+    # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
+    # rendered output breaks these goldens regardless of which requirement the touched path maps to.
+    # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
+    # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
+    # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
+    # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
+    # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
+    # GREEN, and left every golden stale; the staleness surfaced only on a later full run.
+    # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
+    conformance.home.k3s/scope: "whole-repo"
     conformance.home.k3s/intent: >-
       A kind: container ServiceDefinition renders a Deployment + Service (the PaaS path).
 spec:

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -196,6 +196,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

conformance/tests/l1-rendering/default-deny-baseline-render/chainsaw-test.yaml

@@ -27,6 +27,17 @@ metadata:
     conformance.home.k3s/requirements: "R-NET-01"
     conformance.home.k3s/layer: "L1"
     conformance.home.k3s/safety-critical: "false"
+    # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
+    # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
+    # rendered output breaks these goldens regardless of which requirement the touched path maps to.
+    # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
+    # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
+    # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
+    # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
+    # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
+    # GREEN, and left every golden stale; the staleness surfaced only on a later full run.
+    # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
+    conformance.home.k3s/scope: "whole-repo"
     conformance.home.k3s/intent: >-
       Rendering any service's namespace yields a default-deny baseline policy.
 spec:

conformance/tests/l1-rendering/egress-class-render/chainsaw-test.yaml

@@ -32,6 +32,17 @@ metadata:
     conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07"
     conformance.home.k3s/layer: "L1"
     conformance.home.k3s/safety-critical: "false"
+    # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
+    # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
+    # rendered output breaks these goldens regardless of which requirement the touched path maps to.
+    # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
+    # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
+    # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
+    # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
+    # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
+    # GREEN, and left every golden stale; the staleness surfaced only on a later full run.
+    # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
+    conformance.home.k3s/scope: "whole-repo"
     conformance.home.k3s/intent: >-
       egress: vpn renders the egress-gateway + kill-switch path; internet does not.
 spec:

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -223,6 +223,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -223,6 +223,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

infra/gitea/README.md

@@ -0,0 +1,168 @@
+# infra/gitea — platform settings that are NOT GitOps-managed
+
+Gitea's own repo settings (branch protection, required status checks, approvals) live
+in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
+and invisible to the gate** — exactly how the state below went unnoticed.
+
+## The gap this closes
+
+Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
+(2026-08-31) asserting the opposite:
+
+| Claimed | Measured |
+|---|---|
+| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
+| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
+
+So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
+rule; this directory is how the rule gets enforced by the machine.
+
+## `branch-protection.sh`
+
+Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
+do not trust the API response**, since Gitea may let site admins bypass protection.
+
+    ./infra/gitea/branch-protection.sh
+
+It authenticates with the credential `git` already has stored for this host, and never
+prints the secret.
+
+## `runner-token-rotate.sh`
+
+Mints a fresh instance-level registration token from the Gitea API and writes it into
+`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
+never printed.**
+
+    ./infra/gitea/runner-token-rotate.sh
+
+A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
+and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
+longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
+and polls with an identity Gitea does not recognise — so it never appears in the runner
+list. That is precisely the observed symptom: zero runners, pod running, no error anyone
+looks at. Clearing the PVC is what forces a real re-registration; the script prints the
+commands.
+
+## Sequence — DONE 2026-09-29
+
+All four steps are complete and each was verified, not assumed:
+
+| Step | State |
+|---|---|
+| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
+| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
+| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
+| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
+
+Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
+`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
+account agents authenticate as.
+
+### 5. Block the admin override — the step the first pass MISSED
+
+`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
+bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
+merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
+like the gate were overridable by precisely the account that merges.
+
+**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
+`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
+*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
+(`/branch_protections/main`) and enumerate its keys.
+
+> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
+> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
+> means "key absent". Enumerate with `to_entries` and act on what is really there.
+
+Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
+proven when a green PR is REFUSED a merge until approved, and not before.
+
+### `require-gate.sh` / `verify-then-enforce.sh`
+
+`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
+merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
+the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
+enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
+
+Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
+
+### `expected-branch-protection.json` + `ci/gitea-settings.sh` — the detector
+
+`expected-branch-protection.json` **declares** the desired rule. `ci/gitea-settings.sh` runs as
+a merge-gate stage, reads the LIVE rule, and **fails the gate on any divergence** — plus asserts
+at least one runner is online. It is the only thing in this repository that can see Gitea's own
+database.
+
+It reads `/branch_protections/main`, not `/branches/main`: the branch view does **not** expose
+`block_admin_merge_override`, and it showed all-green for hours while the rule was overridable.
+
+Behaviour, proven by test rather than assumed:
+
+| Situation | Exit | Effect |
+|---|---|---|
+| live matches the declaration | 0 | green |
+| any key drifted, or no runner online | 1 | **blocks the merge** |
+| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
+
+The skip is deliberately *not* a block: an unwired credential blocking every PR is the
+lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
+the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS=true` in
+`gate.yml` so a future missing token blocks instead of skipping.
+
+**Changing the declaration is how you change policy.** If the live state is what you now want,
+edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
+
+### Why these scripts exist at all, and what is still missing
+
+Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
+conformance gate can read. Nothing detects drift from what this directory declares. That is
+the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
+gone for a month while the log recorded both as working, and **no check existed that could
+have caught it.**
+
+These scripts encode the desired state and can re-assert it; `ci/gitea-settings.sh` above is the
+**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
+root cause of the 2026-09-29 session.
+
+It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
+detector detects nothing. That is the remaining step.
+
+## Historical sequence (kept for context)
+
+### Original ordering notes
+
+1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
+   the runner actually re-registers, then confirm it shows `online` in the API. Do not
+   skip the confirmation: the whole reason this went unnoticed is that a runner was
+   *recorded* as online without anyone checking.
+2. **Watch the gate actually report on a real PR**, and note the context name it produced.
+3. **Only then set `enable_status_check: true`** with that exact context.
+4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
+
+> **Do not do step 3 before step 1.** A required status check with no runner leaves every
+> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
+> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
+> reported.
+
+### Measured, once the runner came up (2026-09-29)
+
+The first CI jobs this repo ever ran, on PR #1 and PR #2:
+
+| Context | Result | Time |
+|---|---|---|
+| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
+| `gate / gate (push)` | success | 1m32s (PR #2 only) |
+| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
+
+**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
+but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
+does not hang a status check: Gitea reports it as **`success`** with the description
+"Has been skipped". So requiring it would not deadlock anything — it would pass
+*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
+a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
+exactly the fail-open shape this repo keeps finding.
+
+The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
+(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
+is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
+`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.

infra/gitea/block-admin-override.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Stop repository admins from overriding the merge requirements on `main`.
+#
+# WHY. Gitea lets repo admins override branch-protection merge requirements unless
+# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
+# required_approvals and the required status check were both overridable by the one account
+# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
+# `required_approvals: 1`.
+#
+# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
+# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
+# real keys (`to_entries`) rather than probing names you assume exist.
+#
+# Recoverable: an owner can PATCH this back to false if it gets in the way.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
+
+echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
+api -X PATCH -H 'Content-Type: application/json' \
+  -d '{"block_admin_merge_override": true}' "$B" >/dev/null
+AFTER=$(api "$B" | jq -r .block_admin_merge_override)
+echo "==> after:  block_admin_merge_override = $AFTER"
+[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
+
+api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
+cat <<'NOTE'
+
+NOW PROVE IT -- the setting is not the evidence. On the next PR:
+  * let the gate go green, then try to merge WITHOUT approving it.
+  * Gitea must REFUSE. If it merges, this still is not doing its job.
+Do not record approvals as enforced until you have seen a merge refused.
+NOTE

infra/gitea/branch-protection.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# Enable branch protection on main for home-network-k3s-impl.
+#
+# Sets enable_push:false  -> no direct pushes to main; changes land only by merging a PR.
+#
+# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner
+# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the
+# gate context would deadlock EVERY PR permanently -- including the PR that fixes it.
+# Turn it on only after a runner is live and you have seen the gate actually report on
+# a real PR, using the context name CI itself produced.
+#
+# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT
+# a repo collaborator, so requiring an approval now could leave nobody able to approve.
+# Add the collaborator first, then raise this to 1.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev
+OWNER=Home-Network
+REPO=home-network-k3s-impl
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"
+P="$(sed -n 's/^password=//p' <<<"$creds")"
+[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
+echo "authenticating as '${U}' (secret not shown)"
+
+echo "==> creating branch protection on 'main'"
+jq -n '{
+  branch_name: "main",
+  enable_push: false,
+  enable_status_check: false,
+  status_check_contexts: [],
+  required_approvals: 0,
+  block_on_rejected_reviews: true,
+  block_on_outdated_branch: false,
+  dismiss_stale_approvals: true
+}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
+  -X POST -H 'Content-Type: application/json' --data @- \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \
+| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end'
+
+echo "==> verifying (this is the part that matters)"
+curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
+| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}'
+
+cat <<'NOTE'
+
+Now prove it actually blocks you -- do not take the API's word for it. From a clean
+checkout on main, with nothing to lose:
+
+    git commit --allow-empty -m 'protection probe'
+    git push origin main          # EXPECT: rejected by the remote
+    git reset --hard HEAD~1       # tidy up either way
+
+If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not
+yet doing what you want.
+NOTE

infra/gitea/expected-branch-protection.json

@@ -0,0 +1,20 @@
+{
+  "_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
+  "_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
+  "branch": "main",
+  "runners_online_min": 1,
+  "protection": {
+    "enable_push": false,
+    "enable_push_whitelist": false,
+    "enable_merge_whitelist": false,
+    "enable_status_check": true,
+    "status_check_contexts": [
+      "gate / gate (pull_request)"
+    ],
+    "required_approvals": 1,
+    "enable_approvals_whitelist": false,
+    "block_admin_merge_override": true,
+    "block_on_rejected_reviews": true,
+    "dismiss_stale_approvals": true
+  }
+}

infra/gitea/require-gate.sh

@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# Items 1 + 2: make the gate a REQUIRED status check and require one approval.
+#
+# RUN THIS LAST. Prerequisites, in order:
+#   a) PR #7 (dind native sidecar) is MERGED, and
+#   b) you have SEEN a real workflow run go green on the converted pod.
+# The runner Deployment is the merge gate for the whole repo. Turning on a required check
+# while the runner is broken means no PR can merge -- including the revert. (Recoverable:
+# an owner can set enable_status_check back to false. But do not rely on that.)
+#
+# Requires exactly ONE context: `gate / gate (pull_request)`.
+#   * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
+#     SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
+#     ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
+# block_admin_merge_override: true is LOAD-BEARING and was missing on the first pass. Gitea
+# lets repo admins override merge requirements without it, so required_approvals and the
+# required status check are both bypassable by exactly the account that does the merging.
+# Measured: PRs #5-#9 merged with ZERO approvals recorded against required_approvals: 1.
+#
+# FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins`. Asking jq for it returns null,
+# which reads like "false" but means "key absent" -- enumerate real keys with `to_entries`
+# instead of probing names you assume exist. That mistake cost a wasted PATCH.
+#
+#   * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
+#     PR would never produce that context.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+echo "==> updating branch protection on main"
+jq -n '{
+  enable_push: false,
+  enable_status_check: true,
+  status_check_contexts: ["gate / gate (pull_request)"],
+  required_approvals: 1,
+  block_on_rejected_reviews: true,
+  dismiss_stale_approvals: true,
+  block_on_outdated_branch: false,
+  block_admin_merge_override: true
+}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
+
+echo "==> verifying"
+api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
+ | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
+
+cat <<'NOTE'
+
+NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
+  * Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
+  * PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
+    agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
+    and rely on the status check alone.
+
+Then prove it, rather than trusting the API:
+  git commit --allow-empty -m probe && git push origin main   # EXPECT: rejected
+  git reset --hard HEAD~1
+NOTE

infra/gitea/runner-token-rotate.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+# Mint a fresh Gitea Actions registration token and write it into the SOPS secret.
+#
+# The token value is NEVER printed. It goes from the Gitea API into `sops set` and
+# nowhere else. (It is briefly visible in this host's process list as a sops argument;
+# acceptable on a single-operator workstation, and far better than echoing it.)
+#
+# This ALONE may not bring the runner back -- see "IMPORTANT" at the end of the output.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev
+SECRET=apps/wave-1/gitea-actions-runner-secret.sops.yaml
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+cd "$REPO_ROOT"
+[ -f "$SECRET" ] || { echo "not in the repo root, or $SECRET is missing" >&2; exit 1; }
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"
+P="$(sed -n 's/^password=//p' <<<"$creds")"
+[ -n "$U" ] && [ -n "$P" ] || { echo "no stored git credential for $HOST" >&2; exit 1; }
+echo "==> minting an instance-level registration token as '${U}'"
+
+TOKEN="$(curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
+  "https://${HOST}/api/v1/admin/runners/registration-token" | jq -r '.token // empty')"
+[ -n "$TOKEN" ] || { echo "Gitea did not return a token" >&2; exit 1; }
+echo "==> got a ${#TOKEN}-character token (value not shown)"
+
+echo "==> writing it into ${SECRET} via sops"
+DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run \
+  sops set "$SECRET" '["stringData"]["token"]' "\"${TOKEN}\""
+unset TOKEN
+
+echo "==> done. Verify the file is still encrypted (no plaintext token):"
+grep -q 'ENC\[' "$SECRET" && echo "    OK - stringData is still SOPS-encrypted" \
+  || { echo "    REFUSING TO CONTINUE: $SECRET is not encrypted" >&2; exit 1; }
+
+cat <<'NOTE'
+
+Next: commit this on a branch, open a PR, and merge it so Flux applies the new Secret.
+
+IMPORTANT -- a new token is usually NOT sufficient on its own.
+act_runner registers exactly ONCE and then persists its identity in /data/.runner on the
+act-runner-data PVC. If Gitea no longer has a matching runner row (DB restore, the row was
+deleted, the instance was rebuilt), the pod starts, sees .runner, SKIPS registration, and
+polls with an identity Gitea does not recognise -- so it never appears in the runner list,
+which is exactly the observed symptom (zero runners, pod running).
+
+So after the Secret lands, force a genuine re-registration by clearing that state:
+
+    kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=0
+    kubectl --context k3s-prod -n gitea-actions delete pvc act-runner-data
+    kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=1
+
+(The PVC holds only runner registration state -- no build artifacts, nothing durable.
+Deleting it is the supported way to re-register. Flux recreates it from the manifest.)
+
+Then confirm it actually registered -- do not assume:
+
+    curl -s -u '<user>' https://gitea.home.keithjackson.dev/api/v1/admin/actions/runners | jq
+
+A runner must appear with status "online" and the labels ubuntu-latest / homelab.
+NOTE

infra/gitea/set-ci-token.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# Mint a Gitea API token and store it as the repo Actions secret GITEA_SETTINGS_TOKEN,
+# so ci/gitea-settings.sh can read the live branch-protection rule during the gate.
+#
+# The token value is NEVER printed. It goes from the token API into the secrets API and
+# nowhere else -- not to stdout, not to a file, not into an agent transcript.
+#
+# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
+# runner list). Gitea token scopes are coarse; grant the least that works on your version and
+# confirm the check goes green rather than assuming the scope name.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+NAME="ci-gitea-settings-$(date +%Y%m%d%H%M%S)"
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+echo "==> minting token '${NAME}' as ${U}"
+TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
+  -d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
+  "https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
+[ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
+echo "==> got a ${#TOKEN}-character token (value not shown)"
+
+echo "==> storing it as repo Actions secret GITEA_SETTINGS_TOKEN"
+basic -X PUT -H 'Content-Type: application/json' \
+  -d "$(jq -n --arg d "$TOKEN" '{data:$d}')" \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/GITEA_SETTINGS_TOKEN" >/dev/null
+unset TOKEN
+echo "==> stored."
+
+cat <<'NOTE'
+
+Next:
+  1. .gitea/workflows/gate.yml already passes this secret to the gate step.
+  2. Open any PR and confirm the gate prints "gitea-settings: green" -- it SKIPS silently
+     without the token, so a green run is the only proof the wiring works.
+  3. THEN set REQUIRE_GITEA_SETTINGS=true in gate.yml, so a future missing token BLOCKS
+     instead of skipping. Do that only after you have seen it green, or you lock the repo.
+NOTE

infra/gitea/verify-then-enforce.sh

@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order.
+#
+# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and
+# merges). This script does the mechanical half: prove the converted runner actually works,
+# and only then turn on enforcement.
+#
+# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a
+# required status check while the runner is broken means NO PR can merge -- including the
+# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and
+# only then does enforcement go on.
+#
+# Aborts at the first checkpoint that does not pass. Safe to re-run.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+REPO_ROOT=/var/home/keith/source/home-network-k3s-impl
+D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "$REPO_ROOT"
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
+say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
+die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; }
+
+say "PRECHECK — all four PRs must be merged first"
+for n in 8 5 6 7; do
+  m=$(api "$A/pulls/$n" | jq -r .merged)
+  printf '  PR #%s merged=%s\n' "$n" "$m"
+  [ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run."
+done
+
+say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval"
+./infra/devcontainer/dev.sh run bash -c \
+  'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source'
+
+say "STEP 2 — CHECKPOINT: the converted pod must come up healthy"
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s
+  echo "--- sidecar (initContainers) ---"
+  kubectl --context k3s-prod -n gitea-actions get deploy act-runner \
+    -o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}"
+  kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \
+  || die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else"
+
+say "STEP 3 — CHECKPOINT: a runner must be ONLINE again"
+n=0
+for i in $(seq 1 40); do
+  n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length')
+  printf '\r  online runners: %s  (%ss)   ' "$n" "$((i*5))"
+  [ "${n:-0}" -ge 1 ] && { echo; break; }
+  sleep 5
+done
+[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7"
+
+say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD"
+MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id)
+echo "  watching the post-merge push run on main (${MAIN_SHA:0:8})..."
+st=""
+for i in $(seq 1 60); do
+  st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state)
+  printf '\r  main gate: %-8s (%ss)   ' "$st" "$((i*10))"
+  case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac
+  sleep 10
+done
+[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement"
+echo "  the converted sidecar ran a real job green. Enforcement is now safe."
+
+say "STEP 5 — require the gate + one approval"
+"$D/require-gate.sh"
+
+say "STEP 6 — prove a direct push is rejected (do not trust the API's word)"
+git fetch -q origin && git checkout -q main && git reset -q --hard origin/main
+git commit -q --allow-empty -m 'protection probe'
+if git push origin main 2>&1 | grep -q 'protected branch'; then
+  echo "  CONFIRMED: push to main rejected by the pre-receive hook."
+else
+  echo "  WARNING: the push was NOT rejected -- protection is not doing what it claims."
+fi
+git reset -q --hard origin/main
+
+say "FINAL STATE"
+api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'

infra/plex/claim-ship.sh

@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
+# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
+# stopwatch so you can see the budget being spent.
+#
+#   branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
+#
+# It never sees, prints, or handles the token: the file is already encrypted by sops.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev
+OWNER=Home-Network
+REPO=home-network-k3s-impl
+SECRET=apps/wave-0/plex-claim-secret.sops.yaml
+T0=$(date +%s)
+el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
+
+cd "$(git rev-parse --show-toplevel)"
+
+# --- preflight: refuse to ship a file that is not what we think it is -------------
+git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
+  echo "  infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
+grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
+[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered
+# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a
+# matching prefix fired a redundant run that serialised ahead of the PR run. Those
+# per-branch push triggers were deleted, so no prefix produces a duplicate run any more.
+BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
+
+el "committing on $BRANCH"
+git checkout -q -b "$BRANCH"
+git add "$SECRET"
+git commit -q -m 'chore(plex): fresh plex.tv claim token
+
+Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
+SHA=$(git rev-parse HEAD)
+
+el "pushing"
+git push -q -u origin "$BRANCH"
+
+el "opening PR"
+PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
+       '{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
+     | api -X POST -H 'Content-Type: application/json' --data @- \
+         "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
+[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
+echo "  PR #$PR"
+
+el "waiting for the gate (expect ~1m40s)"
+for i in $(seq 1 60); do
+  st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
+  printf '\r  gate=%s  [T+%ss]   ' "$st" "$(( $(date +%s)-T0 ))"
+  case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
+  sleep 5
+done
+[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
+
+el "merging PR #$PR"
+api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
+git checkout -q main && git pull -q --ff-only origin main
+
+el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
+./infra/devcontainer/dev.sh run bash -c \
+  'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
+
+# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
+# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
+# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
+el "rolling the pod (mandatory — a changed Secret restarts nothing)"
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
+   kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
+
+el "verdict"
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
+cat <<'NOTE'
+
+  "Attempting to obtain server token..." + "Token obtained successfully"  -> CLAIMED
+  "Attempting..." with no success line                                    -> token expired; retry, nothing is broken
+  neither line                                                            -> already claimed, or env did not reach the container
+
+Confirm it persisted (prints the attribute NAME only, never its value):
+
+  infra/devcontainer/dev.sh run bash -c \
+    'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
+       "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
+NOTE

platform/monitoring/alerts.yaml

@@ -904,6 +904,132 @@ spec:
             summary: "Egress VPN tunnel down"
             description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
+        # -----------------------------------------------------------------------------------------
+        # THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
+        #
+        # `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
+        # having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
+        # exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
+        # re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
+        # at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
+        # 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
+        # carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
+        # exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
+        #
+        # The datapath now self-heals: `roles/vpn_egress_gateway` publishes
+        # `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
+        # OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
+        # and that node write is what makes Cilium re-resolve. These two rules watch whether it
+        # ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
+        #
+        # WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
+        # `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
+        # observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
+        #   * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
+        #     `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
+        #     there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
+        #     metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
+        #     `cilium_*` and `hubble_*` query returns an empty vector today (which also means
+        #     `HubbleDeniedFlowSpike` above cannot fire yet).
+        #   * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
+        #     CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
+        #     map, not an exporter. No metric name is invented here to paper over that.
+        # So the observable is the label: the input to the re-steer, compared against the live tun0
+        # identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
+        # ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
+        # `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
+        #
+        # WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
+        # firing, the label write failing or never being deployed, the convergence timer being dead —
+        # i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
+        # "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
+        # until either the re-steer script exports the map's ifindex through a node-exporter textfile
+        # collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
+        # A probe was considered and rejected for now: there is no blackbox-exporter and no probe
+        # CronJob in this repo, so it means a NEW always-on component whose own failures become a new
+        # alerting surface — worth doing, but not as the thing that closes a hole today.
+        #
+        # WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
+        # VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
+        # label-to-number conversion, so the level-triggered equality test is not expressible. What IS
+        # expressible is the same invariant counted over a window: the published label must change at
+        # least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
+        # is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
+        # that held still, 2 for one that changed value inside the window); the `max by (<label>)`
+        # wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
+        # restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
+        # which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
+        #
+        # Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
+        # +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
+        # trade against the repo's alert-fatigue history (an unbounded critical on the one channel
+        # that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
+        # email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
+        - alert: VPNEgressSteeringStale
+          # FIRES when the live tun0 was re-created more times in the last hour than the gateway
+          # node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
+          # tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
+          # and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
+          # seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
+          # cold boot), so `for: 15m` cannot fire on a healthy reconnect.
+          expr: |
+            (
+                changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+              >
+                (
+                  count(
+                    count_over_time(
+                      max by (label_homelab_keithjackson_dev_egress_uplink) (
+                        kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
+                      )[1h:1m]
+                    )
+                  ) - 1
+                )
+            )
+            and on()
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "VPN egress steering is stale — vpn-class pods have no egress"
+            description: >-
+              The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
+              Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
+              being dropped, silently, while tun0 still looks healthy. Compare
+              `kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
+              `cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
+              (or setting any node label) forces a fresh resolution.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        - alert: VPNEgressUplinkLabelMissing
+          # The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
+          # in kube-state-metrics, and if that label is missing its comparison degrades to an empty
+          # vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
+          # node with no published uplink means the re-steer mechanism is not in place at all
+          # (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
+          # kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
+          # the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
+          # rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
+          # that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
+          # on the way in, and so an Ansible run in progress does not either.
+          expr: |
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+            and on()
+            absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Egress gateway node publishes no egress-uplink label"
+            description: >-
+              A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
+              `homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
+              nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
+              `VPNEgressSteeringStale` is inert. Check the label on the node, then the
+              `vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
+              metricLabelsAllowlist.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        # -----------------------------------------------------------------------------------------
         # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
         # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
         # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`

platform/monitoring/kube-prometheus-stack.yaml

@@ -231,7 +231,7 @@ spec:
       # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
       # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
       metricLabelsAllowlist:
-        - "nodes=[homelab.keithjackson.dev/egress-gateway]"
+        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
       # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
       # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
       # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that

platform/monitoring/tests/run.sh

@@ -116,10 +116,14 @@ fi
 # `.git` file points outside the mount, so git cannot read the index there — and with
 # `pipefail` that failure silently emptied the discovery, i.e. a green run over ZERO tests
 # (observed 2026-09-12). `find` needs no repo metadata, so discovery cannot be disabled by
-# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up.
+# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up,
+# and .claude/ with it: .claude/worktrees/agent-*/ holds full repo copies for isolated agent
+# runs, so an unpruned find reported the SAME manifest under test once per worktree (up to 18
+# times), and the L0 check that compares this listing against the tree failed on paths that
+# are not authored surface at all (.claude/ is gitignored, .gitignore:43).
 discover() {
   # Companion convention, anywhere in the tree.
-  find "${ROOT}" -path "${ROOT}/.git" -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
+  find "${ROOT}" \( -path "${ROOT}/.git" -o -path "${ROOT}/.claude" \) -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
   # Suite convention, ANYWHERE in the tree — not just this directory.
   #
   # This used to be `find "${HERE}" -maxdepth 1`, which made the suite convention a
@@ -129,7 +133,7 @@ discover() {
   # as the rules this harness exists to test. Found 2026-09-13 adding apps/wave-*/tests/. Scoped by
   # NAME only; manifests_for() then refuses (loudly) any *.test.yaml that is not inside a tests/
   # directory, so a misplaced file fails rather than disappearing.
-  find "${ROOT}" -path "${ROOT}/.git" -prune -o -name '*.test.yaml' -print 2>/dev/null
+  find "${ROOT}" \( -path "${ROOT}/.git" -o -path "${ROOT}/.claude" \) -prune -o -name '*.test.yaml' -print 2>/dev/null
 }
 
 TESTS=()

platform/monitoring/tests/vpn-egress.test.yaml

@@ -0,0 +1,188 @@
+# promtool rule-unit tests for VPNEgressSteeringStale + VPNEgressUplinkLabelMissing
+# (platform/monitoring/alerts.yaml)
+#
+# Run with: platform/monitoring/tests/run.sh
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
+# `egress-class=vpn` pods leave through one gateway node's tun0. Cilium resolves that interface to an
+# IFINDEX and caches it; an OpenVPN reconnect destroys and re-creates tun0 with a NEW ifindex, and the
+# gateway node's up-script publishes `homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex>` so that
+# node write forces Cilium to re-resolve. If the label does not follow the tunnel, the cache points at a
+# destroyed interface and every vpn-class pod is dropped — while tun0 itself stays up, so VPNTunnelDown
+# says nothing. That shape ran ~80 minutes undetected on the dev gateway.
+#
+# VPNEgressSteeringStale detects it by comparing HOW MANY TIMES tun0's ifindex changed in the last hour
+# against HOW MANY DISTINCT `egress-uplink` label values appeared in the same hour. The subtlety worth
+# testing is that a label VALUE change is not a sample change: it RETIRES one series and STARTS another,
+# so "the label followed the tunnel" shows up as two overlapping series in the window, not as a
+# changing value. Tests 1 and 2 are the same tunnel reconnect with and without that second series, so
+# they isolate exactly that mechanism — anything that broke the series-counting half would make test 2
+# fire, and anything that broke the tun0 half would make test 1 quiet.
+#
+# VPNEgressUplinkLabelMissing is the WATCHDOG for the rule above: with no uplink label anywhere, the
+# staleness comparison has nothing to compare and is structurally INERT — present, evaluating, and
+# incapable of firing. Tests 3 and 4 pin that, because an inert critical alert is the failure mode this
+# whole group exists to prevent.
+#
+# Fixture values are prod's real shape: gateway k3s-prod-3, tun0 ifindex 139 with address
+# 10.150.192.86 (live label value `10.150.192.86-139`), reconnecting to ifindex 140.
+
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. THE BUG — tun0 was re-created, the label did NOT follow. Steering is stale: FIRES.
+  # =============================================================================================
+  - interval: 1m
+    name: tunnel reconnect with no label update - VPNEgressSteeringStale fires
+    input_series:
+      # tun0 re-created at t=41m: ifindex 139 -> 140. One change inside the trailing hour.
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '139+0x40 140+0x40'
+      # tun0 is UP and passing traffic the whole time — this is what makes the failure deceptive, and
+      # it is asserted below: VPNTunnelDown stays QUIET while egress is actually broken.
+      - series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '0+120000x80'
+      # The node label NEVER changes, so only ONE distinct uplink value exists in the window.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
+        values: '1+0x80'
+    alert_rule_test:
+      # 1 tunnel change > (1 distinct label value - 1) = 0, held for 15m.
+      - eval_time: 60m
+        alertname: VPNEgressSteeringStale
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+            exp_annotations:
+              summary: "VPN egress steering is stale — vpn-class pods have no egress"
+              description: >-
+                The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
+                Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
+                being dropped, silently, while tun0 still looks healthy. Compare `kubectl get nodes -L
+                homelab.keithjackson.dev/egress-uplink` against `cilium-dbg bpf egress list` on the
+                gateway node; re-applying the CiliumEgressGatewayPolicy (or setting any node label)
+                forces a fresh resolution.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+      # Not yet: at 45m the condition has been true only ~4m, under the 15m `for`.
+      - eval_time: 45m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      # No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
+      # no VPN gateway must not be told its tunnel is down forever.
+      - eval_time: 45m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+      # The watchdog must stay quiet — the label IS present here.
+      - eval_time: 60m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      # THE HEADLINE CLAIM, asserted rather than asserted-in-prose: the tunnel alert says NOTHING
+      # while vpn-class egress is completely broken. If VPNTunnelDown covered this failure, the
+      # staleness rule would not need to exist.
+      - eval_time: 60m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. THE HEALTHY RECONNECT — same tunnel change, but the label followed it. STAYS QUIET.
+  #    This is the half that proves the rule is not simply "any reconnect alerts".
+  # =============================================================================================
+  - interval: 1m
+    name: tunnel reconnect WITH the label following - stays quiet
+    input_series:
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '139+0x40 140+0x40'
+      - series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '0+120000x80'
+      # A label VALUE change retires one series and starts another. Both are inside the trailing
+      # hour, so the window holds TWO distinct uplink values: 2 - 1 = 1, which is NOT < 1 change.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
+        values: '1+0x40 _x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-140"}'
+        values: '_x41 1+0x39'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      - eval_time: 75m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+
+  # =============================================================================================
+  # 3. THE INERT-ALERT CASE — a gateway node publishing NO uplink label at all.
+  #    VPNEgressSteeringStale cannot fire in this state, which is exactly why this warning exists.
+  # =============================================================================================
+  - interval: 1m
+    name: gateway node with no egress-uplink label - VPNEgressUplinkLabelMissing fires
+    input_series:
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true"}'
+        values: '1+0x60'
+      # tun0 is reconnecting the whole time. The staleness rule STILL cannot fire, because it has no
+      # label series to compare against — that is the inertness being alerted on.
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '139+0x30 140+0x30'
+      - series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '0+120000x60'
+    alert_rule_test:
+      - eval_time: 35m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+            exp_annotations:
+              summary: "Egress gateway node publishes no egress-uplink label"
+              description: >-
+                A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
+                `homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without
+                it nothing can tell whether Cilium's egress steering followed the last tunnel reconnect,
+                and `VPNEgressSteeringStale` is inert. Check the label on the node, then the
+                `vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
+                metricLabelsAllowlist.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+      # Under the 30m `for`.
+      - eval_time: 20m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      # And the proof of inertness: the critical rule is silent despite a real reconnect.
+      - eval_time: 55m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      - eval_time: 55m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. NO GATEWAY NODE AT ALL — neither rule may fire. Both are guarded by an `and on()` on the
+  #    egress-gateway label so they stay silent on a cluster with no VPN gateway, rather than
+  #    alerting forever about an absent feature.
+  # =============================================================================================
+  - interval: 1m
+    name: no egress-gateway node - both rules silent
+    input_series:
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
+        values: '1+0x60'
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.10:9100",device="eth0"}'
+        values: '2+0x60'
+    alert_rule_test:
+      - eval_time: 45m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      - eval_time: 45m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      # No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
+      # no VPN gateway must not be told its tunnel is down forever.
+      - eval_time: 45m
+        alertname: VPNTunnelDown
+        exp_alerts: []

platform/velero/volume-policies.yaml

@@ -24,6 +24,36 @@
 # pod, and nothing tells them. A volume policy tests the PROPERTY that makes a volume bulk-tier —
 # its source is the out-of-cluster NFS export — so it covers volumes that do not exist yet.
 #
+# RULE 2 — emptyDir (added 2026-09-19). Every emptyDir volume is skipped, because an emptyDir holds
+# no durable state BY DEFINITION: created empty with its pod, destroyed with it. Backing one up
+# cannot protect anything, and ATTEMPTING it actively breaks the backup.
+#
+# HOW IT BREAKS THINGS. A PodVolumeBackup exposes the volume through its pod; if that pod is replaced
+# mid-flight the PVB fails with `error to expose PVB: error getting client pod <pod>: pods "<pod>"
+# not found`, and ONE failed PVB makes the whole Backup PartiallyFailed. PartiallyFailed does not
+# advance velero_backup_last_successful_timestamp, so VeleroNoRecentBackup fires CRITICAL over a
+# backup that protected every real volume. Any Deployment rollout landing in the backup window is
+# enough — on a GitOps cluster that is routine, not an edge case. An alert that cries wolf nightly is
+# how a real backup failure gets ignored, and this one guards R-STATE-03.
+#
+# OBSERVED TWICE IN ONE HOUR in unrelated namespaces, which is what justified a general rule over a
+# targeted exclusion:
+#   * ns velero   — Velero's OWN transient kopia repository-maintenance Jobs
+#                   (`<ns>-default-kopia-maintain-job-<ts>`), emptyDir `plugins` + `scratch`. Four
+#                   failed PVBs had accumulated; velero-daily-20260919030041 was PartiallyFailed,
+#                   2 errors, 3079 of 3079 items backed up.
+#   * ns runbooks — the nginx Deployment's emptyDir `tmp` + `cache` (a read-only root filesystem needs
+#                   writable scratch). Four failed PVBs when a runbook edit rolled the Deployment
+#                   while the backup was running.
+#
+# PROVEN ON PROD, WITH A CONTROL (2026-09-19, velero 1.18.1). `volumeTypes` support was the open
+# question, and a policies.yaml Velero cannot parse breaks EVERY backup — so it was tested against a
+# THROWAWAY ConfigMap first, leaving this one untouched until it passed:
+#     control, ns runbooks, NO policy  -> 4 PodVolumeBackups (2x `tmp`, 2x `cache`), Completed
+#     same namespace, WITH this policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
+# The control is the point: "zero PVBs" on its own is equally consistent with nothing being attempted.
+#
+
 # `nfs: {}` is the documented "match any NFS volume source" condition (server/path may be given to
 # narrow it; deliberately not, so a second export is covered too). First matching rule wins.
 #
@@ -44,6 +74,12 @@ data:
   policies.yaml: |
     version: v1
     volumePolicies:
+      # emptyDir first: cheapest condition, and the most common match.
+      - conditions:
+          volumeTypes:
+            - emptyDir
+        action:
+          type: skip
       - conditions:
           nfs: {}
         action:

platform/vpn-egress/README.md

@@ -157,6 +157,31 @@ kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
 If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
 gateway node is the next stop.
 
+### And now something watches it (2026-09-12)
+
+The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
+silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
+
+- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
+  more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
+  about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
+  steering pointed at a destroyed ifindex.
+- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
+  `egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
+  in which the rule above would be inert. The watchdog for the watchdog.
+
+Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
+so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
+`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
+the label to exist as a metric at all — the two changes must ship together.
+
+**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
+exposes no metric carrying the egress map's contents (the map is readable only through
+`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
+place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
+map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
+probe from a vpn-class pod. Neither exists yet.
+
 ## Resilience — there is NO gateway failover (spec correction, escalated)
 
 **ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)

runbooks/README.md

@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form
 
     runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
 
-46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
+46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
 live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
 help. These pages are what makes the link resolve.
 
@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
 `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
 `volumes.longhorn.io`, `nodes.longhorn.io`.
 
-`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
+Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
+direct push is rejected — durable changes go through a branch, a PR, and a green gate.
 
 ## Alerts you should not trust the silence of (measured 2026-09-13)
 
@@ -84,8 +85,10 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`.
 
 ### Networking
-- **`vpn-down`** — the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert
-  cannot see, and what the kill-switch actually does.
+- **`vpn-down`** — the egress VPN tunnel. Explains what the kill-switch actually does, and the silent
+  egress-gateway staleness this alert cannot see — which now has its own alerts and page below.
+- **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached
+  ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
 - **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
 
 ### Backups and DR
@@ -106,6 +109,13 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
   every green signal missed.
 
+### Operator procedures (no alert links here)
+- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
+  applies after a `/config` wipe. The claim token is short-lived — read the countdown on
+  plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
+  token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
+  Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
 - **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.

runbooks/backup-failure.md

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/backup-missed.md

@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/backup-partial.md

@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/cdi-import-stuck.md

@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/cert-expiry.md

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/cert-issuance.md

@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/cnpg-archiving.md

@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/cnpg-backup-missed.md

@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/cnpg-exporter.md

@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/cnpg-primary-spread.md

@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/cnpg-wal-growth.md

@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/denied-flows.md

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/flux-reconcile.md

@@ -7,7 +7,7 @@
 ## What this means here
 
 Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
-it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
+it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
 Kustomization means git and the cluster have diverged and nothing will close the gap.
 
 The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/flux-suspended.md

@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/image-automation.md

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/image-policy-stalled.md

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/keycloak-client-secrets.md

@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/keycloak-db.md

@@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/keycloak-down.md

@@ -59,8 +59,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/kubevirt-control-plane.md

@@ -57,8 +57,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/kubevirt-crashloop.md

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/kubevirt-launch-failed.md

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/kustomization.yaml

@@ -80,7 +80,14 @@ configMapGenerator:
       - node-down=node-down.md
       - node-io=node-io.md
       - node-memory=node-memory.md
+      - node-memory-swap=node-memory-swap.md
+      # Operator procedure, not an alert page (no alert links to it — the
+      # alert-runbooks-resolvable check treats that as advisory). It is served
+      # because the plex.tv claim window is short and is not a thing to go looking
+      # for in git: /runbooks/plex-claim has to resolve the moment it is needed.
+      - plex-claim=plex-claim.md
       - pvc-capacity=pvc-capacity.md
       - targets-down=targets-down.md
       - update-rollback=update-rollback.md
       - vpn-down=vpn-down.md
+      - vpn-egress-stale=vpn-egress-stale.md

runbooks/longhorn-capacity.md

@@ -83,8 +83,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/longhorn-degraded.md

@@ -74,8 +74,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/longhorn-faulted.md

@@ -75,8 +75,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/monitoring-capacity.md

@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/node-cpu.md

@@ -49,8 +49,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/node-disk.md

@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/node-down.md

@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/node-io.md

@@ -227,8 +227,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/node-memory-swap.md

@@ -0,0 +1,201 @@
+# node-memory-swap — replacing a node's DIMM pair (planned maintenance)
+
+**Not alert-driven.** This is a planned hardware procedure. If a node is *reporting* a memory problem,
+start at `/runbooks/node-memory` instead.
+
+Written for the one known mismatch: **`k3s-prod-1` (192.168.2.10)**.
+
+## Why node 1, and what "mismatched" means here
+
+| slot | part number | size | rated speed | organisation |
+|---|---|---|---|---|
+| `DIMM1` | Micron **8**ATF2G64HZ-**3G2**E1 | 16 GB | DDR4-**3200** | 1Rx8 |
+| `DIMM3` | Micron **16**ATF2G64HZ-**2G6**E1 | 16 GB | DDR4-**2667** | 2Rx8 |
+
+Micron's part numbering carries the proof independently of any inventory doc: the `8ATF` / `16ATF`
+prefix encodes rank organisation, the `-3G2` / `-2G6` suffix encodes 3200 vs 2666 MT/s. Capacity is
+correct — 32 GB, ~31.1 GiB usable — so **nothing is faulty**. The pair simply runs at the slower 2667
+and the ranks differ. `k3s-prod-3` is a matched Crucial pair (`CT16G4SFD8266.C16FE`).
+
+Source: `infra/ansible/_artifacts/node-health/k3s-prod-1.json` records both part numbers; node 3's
+records one. **Caveat on the artifacts:** the extraction de-duplicates part numbers, so it can prove
+"two different modules" but cannot distinguish one module from two identical ones. Only `dmidecode`
+per-slot output settles that, and no raw per-slot block is recorded — which is why verification below
+collects it live.
+
+**Node 2's inventory is stale and contradicts itself** (`infra/PROD-PROVISIONING.md` calls it the only
+single-DIMM node with one 16 GB Samsung module, yet it reports the same ~31.1 GiB as the others). That
+does not change this procedure, but re-inventory node 2 before claiming three-way symmetry.
+
+**DO NOT confuse this with §0b.** The provisioning doc says, correctly, *"do not buy RAM"* — that is
+about node 2's old 2.6-GiB-of-16-GB symptom, which was an **EFI memattr kernel bug, not hardware**.
+Making node 1's pair matched is a different and legitimate change.
+
+## Before you touch anything
+
+*If `kubectl` hangs from the workstation rather than failing, the OIDC context is the reason — prefix
+these with `infra/devcontainer/dev.sh run`.*
+
+**1. Decide what capacity you are installing, because it is a symmetry decision.**
+
+- **2 × 16 GB** keeps all three nodes at 32 GB, which is what `infra/PROD-PROVISIONING.md` §2 assumes
+  (symmetric nodes; replica placement and the scheduler both lean on it).
+- **2 × 32 GB** gives node 1 **64 GB** while nodes 2 and 3 stay at 32 GB — a *new* asymmetry, on the
+  node that is already the asymmetric one on disk. Workloads will schedule there preferentially.
+  Allowed, but make it a decision rather than a side effect.
+
+**2. Confirm the cluster can lose this node.** All three nodes are control-plane **and** etcd members,
+so losing one leaves quorum at 2 of 3 — survivable, but with **zero further margin**. Do not start if
+another node is unhealthy.
+
+    kubectl --context k3s-prod get nodes -o wide
+    # etcd's own view, per node (the house check — embedded etcd metrics on 2381):
+    for n in 1 2 3; do echo "== k3s-prod-$n"; ssh k3s-prod-$n \
+      'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'; done
+
+Every node must report `etcd_server_has_leader 1`. Note which one reports `etcd_server_is_leader 1`:
+if it is node 1, powering it off forces an election, which is normal — you just want to confirm a new
+leader afterwards rather than be surprised by it.
+
+**3. Confirm Longhorn is fully healthy first.** Node 1 held **24 replicas** when last measured. Every
+volume must be healthy *before* you drain, or you are draining into an already-degraded state.
+
+    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state
+
+Expect every attached volume `healthy` (a `detached` volume reporting `unknown` is normal for an unused
+PVC). **If anything reads `degraded`, stop and let it rebuild.**
+
+**4. Confirm the drain policy is what protects you**, rather than assuming it — it is not pinned in
+`platform/longhorn/helmrelease.yaml`, so it is whatever the chart defaults to on this version:
+
+    kubectl --context k3s-prod -n longhorn-system get setting node-drain-policy -o jsonpath='{.value}{"\n"}'
+
+You want `block-if-contains-last-replica`. If it reads something more permissive, **set it before
+draining** — this is the setting that stops the drain from stranding a volume whose only healthy
+replica is on node 1.
+
+**5. Take a fresh backup.** There is player-authored state here now (the Minecraft world on
+`crafty-data`). Find the schedule's real resource name rather than guessing it:
+
+    kubectl --context k3s-prod -n velero get schedules
+    kubectl --context k3s-prod -n velero create backup pre-memswap-$(date -u +%Y%m%d%H%M) \
+      --from-schedule <the daily schedule from above>
+
+Wait for `Completed` with `errors: 0` before continuing.
+
+## Drain
+
+    kubectl --context k3s-prod cordon k3s-prod-1
+    kubectl --context k3s-prod drain k3s-prod-1 --ignore-daemonsets --delete-emptydir-data --timeout=30m
+
+**If the drain blocks, that is the policy doing its job.** Do not reach for `--force` or
+`--disable-eviction`. Watch replicas rebuild onto the other two nodes and let the drain proceed:
+
+    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy
+
+Expect real time to pass — this is 24 replicas of actual data movement, and the rebuild path is
+deliberately throttled (see `/runbooks/node-io` for which limits are set and which do not exist on
+this Longhorn version). Watch the pressure it generates while it runs:
+
+    rate(node_pressure_io_stalled_seconds_total{instance="192.168.2.10:9100"}[5m])
+
+The warning threshold is **0.05** and critical is **0.10**; node 1 sat at 0.10 during the incident that
+starved etcd. A rebuild storm pushing this up is expected, sustained time above 0.10 is not.
+
+Note what moves, and one thing that happily does not. Verified live 2026-09-19:
+
+    kubectl --context k3s-prod get pods -A -l cnpg.io/podRole \
+      -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,ROLE:.metadata.labels.role
+
+**No PostgreSQL primary is on node 1.** All three CNPG clusters run 3 instances spread one per node by
+required anti-affinity, and node 1 holds a *replica* of each (`keycloak-db-2`, `nextcloud-db-2`,
+`semaphore-db-2`). Draining it evicts three replicas and forces **no failover at all** — the best state
+this maintenance could ask for. Re-run the command above before you start rather than trusting this
+paragraph: primaries move on their own after any restart or failover.
+
+`CNPGPrimariesCoLocated` is currently firing for `k3s-prod-2`, which holds both the `keycloak-db` and
+`semaphore-db` primaries. **Do not "fix" that before the swap.** Rebalancing means promoting a replica,
+and the only node with spare primary capacity is node 1 — so tidying the alert would put a primary on
+the very node you are about to power off and convert a no-failover drain into a two-database failover.
+Rebalance *after* node 1 is back and its replicas have caught up.
+
+`sonarr`, `qbittorrent` and `jackett` do sit on node 1 and will reschedule. The UniFi VM
+(`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice today, so expect it to be
+recreated, which also re-runs its cloud-init.
+
+## Power off and swap
+
+    sudo systemctl poweroff   # from the node itself, or via the console
+
+HP EliteDesk 800 G5 Desktop Mini, 2 SODIMM slots. **HP's supported population requires `DIMM1` to be
+populated** — fill `DIMM1` first, then `DIMM3`. Seat both fully; a partially seated SODIMM in these
+machines will POST with one module silently missing.
+
+## Post-swap verification
+
+In order. **Do not uncordon until 1–3 pass.**
+
+**1. The pair is actually matched** — the whole point of the exercise:
+
+    ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s-prod-1   # read-only
+
+or directly on the node:
+
+    sudo dmidecode -t memory | grep -E 'Locator|Size|Part Number|Speed|Rank'
+
+Both slots must report the **same Part Number**, the **same Rank**, and — the part that proves the
+mismatch is gone — the same `Speed` *and* the same `Configured Memory Speed`. **`Configured Memory
+Speed` is the field that matters:** a pair rated 3200 that is being clocked down still reports
+`Speed: 3200 MT/s` while `Configured Memory Speed` sits lower. Matched part numbers with a
+below-rated configured speed means the job is not done.
+
+**2. Capacity is what you installed.**
+
+    kubectl --context k3s-prod get node k3s-prod-1 -o jsonpath='{.status.capacity.memory}{"\n"}'
+
+Roughly 31.1 GiB reported for 2 × 16 GB, roughly double for 2 × 32 GB. If it reads about half what you
+installed, a module is not seated or not detected — power off and reseat rather than debugging it in
+software, because the scheduler will believe whatever capacity is reported.
+
+**3. No new memory errors.** These machines are non-ECC, so this is a sanity check, not a guarantee:
+
+    sudo journalctl -k -b | grep -iE 'mce|memory error|ECC|Hardware Error'
+
+**4. Rejoin the cluster.**
+
+    kubectl --context k3s-prod uncordon k3s-prod-1
+    kubectl --context k3s-prod get nodes
+    ssh k3s-prod-1 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'
+
+`Ready` on the node and `etcd_server_has_leader 1` from node 1's own etcd before you call this done.
+
+**5. Longhorn returns to fully healthy.** Rebuilds run back onto node 1, and this is the phase that
+historically produced I/O pressure on this node — watch it rather than walking away:
+
+    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -vc healthy
+
+with the same PSI query from the drain section. `/runbooks/node-io` records what the post-SSD-swap
+18-replica rebuild measured, for comparison.
+
+**6. Alerts quiet.** The only thing that should be firing is `Watchdog`:
+
+    kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
+      -c prometheus -- /bin/promtool query instant http://localhost:9090 'ALERTS{alertstate="firing"}'
+
+## If it goes wrong
+
+- **Node does not POST**: reseat both modules; failing that, put the original pair back. The old DIMMs
+  are known-good — the mismatch was a consistency/performance issue, not a fault.
+- **Node POSTs with half the memory**: one module not detected. Power off and reseat.
+- **Node rejoins but volumes stay degraded**: `/runbooks/longhorn-degraded`. Do **not** drain a second
+  node while rebuilds are outstanding — that loses quorum and replicas at the same time.
+- **Abort mid-drain**: `kubectl --context k3s-prod uncordon k3s-prod-1` is safe any time before poweroff.
+
+## Update the record when you are done
+
+`infra/PROD-PROVISIONING.md` carries the DIMM inventory table and the "not a matched pair" paragraph.
+Both become wrong the moment this succeeds. Update them in the same change that records the swap —
+stale hardware documentation is exactly what made node 2's inventory untrustworthy.

runbooks/node-memory.md

@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/plex-claim.md

@@ -0,0 +1,189 @@
+# plex-claim — claiming the Plex Media Server inside the token window
+
+**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
+it is the page you need at the moment you need it, with a stopwatch running.
+**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml`
+(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden).
+
+## What this means here
+
+Plex will not let you **create libraries** over the mounted bulk media share while the server is
+**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the
+server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable
+`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on
+the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
+`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
+
+**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
+claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
+merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
+which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
+waited for. Measured end to end on 2026-09-29: **~2m45s**.
+
+**An expired or wrong token does not break anything.** Verified against the image's own first-run
+script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
+body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
+no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
+serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
+ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no
+`optional`, so the Secret must always exist even when its value is useless).
+
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that
+hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and that default has already changed once (k3s-dev until dev
+was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once.
+
+Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator
+work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change
+is still a git change.
+
+## First commands
+
+**Before spending a token, find out whether the server is already claimed.** This prints only the
+attribute NAME, never its value — do not widen it to print the token:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
+     "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
+```
+
+`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn.
+Empty output ⇒ unclaimed, continue.
+
+Is the workload healthy enough to claim at all, and is the wave reconciling?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
+```
+
+## The procedure
+
+> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
+> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
+> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
+> this page again after a `/config` wipe or a rebuild onto an empty PVC.
+
+**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
+rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
+TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
+is the only authority, and it is on screen at T+0.)
+
+**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
+never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
+`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.
+
+### One command
+
+1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
+   <https://plex.tv/claim>, copy the `claim-…` string.
+
+2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:
+
+   ```
+   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+   ```
+
+3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:
+
+   ```
+   ./infra/plex/claim-ship.sh
+   ```
+
+   branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
+   → verdict. It refuses to run if the file is unchanged or not encrypted.
+
+**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
+container start ~30s.
+
+### Why it is a script and not six pasted commands
+
+`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
+the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
+pressure, and two are silent when skipped:
+
+- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
+  revision, not your commit.
+- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
+  template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
+  container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
+  old value forever.
+- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
+  trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
+  second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.
+
+### Reading the verdict
+
+* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
+  ⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
+  create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
+* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
+  back to step 1 with a new one.
+* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
+  claimed — the expected case now), or the env did not reach the container. Check by NAME only:
+
+  ```
+  infra/devcontainer/dev.sh run bash -c \
+    'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+       -o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
+  ```
+
+## Likely causes, most likely first
+
+When the claim does not take, in order of how often it is the answer here:
+
+1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
+   merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
+   Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
+2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
+   whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
+   container still holds the placeholder.
+3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4
+   finished, so the new pod read the OLD Secret. Check
+   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'`
+   against your commit, then roll again.
+4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a
+   server claimed earlier ignores every later token silently. This is the benign case.
+5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled
+   connection can leave the container sitting in cont-init with **nothing listening on 32400** — the
+   one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after
+   "Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly.
+6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at
+   step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".
+
+## When to escalate or stop
+
+- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt
+  is how people start editing live Secrets by hand, which leaves the cluster holding a value that is
+  in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from.
+- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never
+  `kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is
+  reverted (or adopted with the wrong value) on the next reconcile.
+- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the
+  `PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential.
+- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means
+  removing the server from that account on plex.tv and clearing `PlexOnlineToken` from
+  `Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git
+  change.
+- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata
+  and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a
+  two-line pref, not a reason to lose the DB.
+- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is
+  skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no
+  `optional` — deleting it would put the pod into `CreateContainerConfigError`.
+
+---
+Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
+Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
+`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
+the running `plexinc/pms-docker:latest` image.

runbooks/pvc-capacity.md

@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/targets-down.md

@@ -60,8 +60,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/update-rollback.md

@@ -54,8 +54,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/vpn-down.md

@@ -66,8 +66,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
 
 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.
 
 ## First commands
 

runbooks/vpn-egress-stale.md

@@ -0,0 +1,123 @@
+# vpn-egress-stale — VPN egress steering did not follow the tunnel
+
+**Alerts:** `VPNEgressSteeringStale` (critical, 15m) and `VPNEgressUplinkLabelMissing` (warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`
+**Related:** `/runbooks/vpn-down` is for the tunnel being DOWN. This page is for the tunnel being **up**
+while the pods that must use it have no egress at all.
+
+## What this means here
+
+`egress-class=vpn` pods leave the cluster through one gateway node's `tun0`, selected by the
+`vpn-egress-steering` CiliumEgressGatewayPolicy. Cilium resolves that interface to an **ifindex** and
+caches it. When OpenVPN reconnects it destroys and re-creates `tun0`, which gets a **new ifindex** — and
+Cilium does not re-resolve on its own. The gateway node publishes
+`homelab.keithjackson.dev/egress-uplink=<tun0 address>-<ifindex>` from the OpenVPN up-script on every
+connect, and that node write is what makes Cilium re-resolve.
+
+If the label write does not happen, the cache keeps pointing at a destroyed interface. **`tun0` looks
+perfectly healthy the whole time** — it is up, it has an address, `VPNTunnelDown` stays quiet — while
+every vpn-class pod is silently dropped. That exact shape ran for roughly **80 minutes** undetected on
+the dev gateway, which is why these two alerts exist.
+
+`VPNEgressSteeringStale` compares how many times `tun0`'s ifindex changed in the last hour against how
+many distinct `egress-uplink` label values appeared in the same hour. More tunnel changes than label
+changes means a reconnect happened that the label did not follow.
+
+**`VPNEgressUplinkLabelMissing` is the watchdog for that comparison.** If the label is absent entirely,
+the staleness rule has nothing to compare against and is **inert** — present, evaluating, and incapable
+of firing. So the warning matters more than its severity suggests: it means your critical alert is
+switched off.
+
+Today's gateway is **`k3s-prod-3`** (`homelab.keithjackson.dev/egress-gateway=true`), and the policy has
+been in place since early September. It is a **singleton** — there is no second gateway to fail over to.
+
+## First commands
+
+Run cluster commands through the pinned toolbox, which supplies the kube context:
+
+```
+./infra/devcontainer/dev.sh run kubectl --context k3s-prod get nodes \
+  -L homelab.keithjackson.dev/egress-gateway -L homelab.keithjackson.dev/egress-uplink
+```
+
+You want exactly one node carrying `egress-gateway=true`, and that same node carrying a non-empty
+`egress-uplink` of the form `<address>-<ifindex>`.
+
+```
+# 2. what Cilium currently believes, on the gateway node itself
+ssh k3s-prod-3 'sudo cilium-dbg bpf egress list'
+ssh k3s-prod-3 'ip -o link show tun0'          # the REAL ifindex, first field
+```
+
+Compare the ifindex in the label and in `bpf egress list` against `ip link`. **A mismatch is the whole
+bug.**
+
+```
+# 3. is the re-steer mechanism itself alive?
+ssh k3s-prod-3 'systemctl status vpn-egress-resteer.service vpn-egress-resteer.timer --no-pager'
+ssh k3s-prod-3 'journalctl -u vpn-egress-resteer --since -2h --no-pager | tail -30'
+```
+
+```
+# 4. are vpn-class pods actually broken, or only the metric?
+./infra/devcontainer/dev.sh run kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io \
+  vpn-egress-steering -o yaml | head -40
+```
+
+Prove egress from a pod the policy selects rather than inferring it — the observable symptom is
+"outbound traffic silently disappears", so test outbound traffic:
+
+```
+./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n qbittorrent exec deploy/qbittorrent -- \
+  sh -c 'curl -s --max-time 10 https://ifconfig.me || echo EGRESS-FAILED'
+```
+
+That should return the **VPN provider's** address. The gateway VIP or a timeout both mean egress is
+broken; your own ISP address means the kill-switch is not holding and traffic is leaking, which is worse
+than a drop — see `/runbooks/vpn-down`.
+
+## Likely causes, most likely first
+
+1. **A tunnel reconnect that the label did not follow.** The common case, and what the alert is built to
+   catch. The OpenVPN up-script either did not run or could not write the node label (RBAC, expired
+   kubeconfig on the node, API unreachable at that moment). Check `journalctl -u vpn-egress-resteer` and
+   the up-script's own log around the reconnect time.
+2. **Cilium cached a dead ifindex despite the label changing.** The label write happened but the
+   re-resolution did not take. Forcing any node-label write makes Cilium re-resolve — that is the same
+   lever the up-script pulls, so it is a safe nudge, not a workaround:
+
+       ./infra/devcontainer/dev.sh run kubectl --context k3s-prod annotate node k3s-prod-3 \
+         egress-resteer-nudge="$(date -u +%s)" --overwrite
+
+   Re-applying the CiliumEgressGatewayPolicy has the same effect. Re-check `bpf egress list` after.
+3. **`metricLabelsAllowlist` lost the label** (this fires `VPNEgressUplinkLabelMissing`, not the stale
+   alert). kube-state-metrics exposes **no** custom node labels by default; the allowlist entry in
+   `platform/monitoring/kube-prometheus-stack.yaml` is what publishes them. If that entry is edited or
+   dropped, `kube_node_labels` carries no `label_homelab_keithjackson_dev_egress_uplink` and the critical
+   alert goes quiet without anything looking broken. Confirm what is actually exported:
+
+       ./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
+         prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
+         /bin/promtool query instant http://localhost:9090 'kube_node_labels'
+
+   Note that KSM does **not** restart on a ConfigMap-only change; the deployment carries no checksum
+   annotation over it, so a `rollout restart -n monitoring deploy/kube-prometheus-stack-kube-state-metrics`
+   is required after editing the allowlist.
+4. **The gateway node was replaced or relabelled.** If `egress-gateway=true` moved, the up-script must be
+   installed on the new node (`infra/ansible/roles/vpn_egress_gateway`). A gateway label with no
+   up-script behind it produces exactly this alert on the next reconnect.
+
+## When to escalate or stop
+
+- **Stop and do not "fix" it by deleting the policy.** Removing `vpn-egress-steering` does not restore
+  egress; it sends vpn-class traffic out of your **own** ISP connection, which is the leak the class
+  exists to prevent. A broken tunnel that drops traffic is the intended failure mode.
+- **Escalate to a maintenance window** if the ifindex mismatch returns after every reconnect. That is the
+  up-script or its RBAC, not a transient, and chasing it live means repeatedly breaking egress for the
+  vpn-class workloads.
+- **This is a singleton gateway.** There is no failover. If `k3s-prod-3` needs to be drained while this is
+  unresolved, accept that vpn-class egress is down for the duration rather than improvising a second
+  gateway under time pressure.
+- If `VPNEgressUplinkLabelMissing` is firing and you cannot restore the label quickly, say so explicitly
+  in the incident notes: for as long as it is missing, **`VPNEgressSteeringStale` is not watching
+  anything**, so the absence of that critical alert proves nothing.