docs(runbooks): two more uncited "4-minute" claims the first pass missed

A systematic grep for the TTL assertion found it still in runbooks/README.md and in runbooks/kustomization.yaml, outside the page being rewritten. Same correction: the countdown on plex.tv/claim is the authority, not a number in our tree. README.md also now records that the server is CLAIMED and points at claim-ship.sh with the measured ~2m45s loop, rather than describing a push-based sequence that no longer exists. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
docs(runbooks): the PR flow, not git push origin main -- that instruction is now blocked
2026-09-29 01:57:47 -05:00 · 2026-09-29 01:42:35 -05:00 · 2026-09-29 06:33:04 +00:00 · 2026-09-29 01:31:24 -05:00 · 2026-09-29 06:18:14 +00:00 · 2026-09-29 06:12:30 +00:00
63 changed files with 2214 additions and 92 deletions
@@ -0,0 +1,117 @@
+# CLAUDE.md — how work lands in this repo
+
+Read this before doing anything durable. It is the delivery contract for AI coding
+sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
+takes precedence on *what* to work on; this file governs *how* it lands.
+
+## 1. Never push to `main`. Ever.
+
+Every durable change reaches `main` exactly one way:
+
+    branch  ->  push the branch  ->  open a PR  ->  CI gate green  ->  human approval  ->  merge
+
+There is no other path. Specifically:
+
+- **Do not `git push origin main`.**
+- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
+- **Do not merge your own PR.** A human approves and clicks merge.
+- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
+  anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
+
+> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
+> That wording is **stale** — it predates the PR flow. It describes what Flux does
+> (main is what the cluster follows), not what you are permitted to do. Honour this
+> file, not that line. Fixing those runbooks is tracked work.
+
+## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
+
+Verified against the live Gitea on 2026-09-29:
+
+| Expected | Actual |
+|---|---|
+| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
+| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
+| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
+| Approval required before merge | **`required_approvals: 0`** |
+
+So nothing mechanically stops you from pushing to `main` right now. **That is a gap
+being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
+"required status check" with a runner online; neither was true, which is how this went
+unnoticed for a month. The runner half is now genuinely fixed and observed green. The
+protection half is not — see `infra/gitea/`.
+
+**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
+the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
+strength of a check you have not seen report — and note that a **skipped** Gitea job
+reports as `success`, so a green tick is not by itself evidence that anything ran.
+
+## 3. Gate before you push
+
+    ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+
+Exit codes are a contract — do not flatten them:
+
+| rc | Meaning |
+|---|---|
+| 0 | PASS |
+| 1 | BLOCKED — something failed |
+| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
+
+`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
+beside this one; that single skip is expected and tolerated. Any *other* skip is a
+real gap in your run.
+
+**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
+so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
+
+### Running the gate from a git worktree
+
+Two unrelated failures, both with known workarounds — put both in any agent brief:
+
+1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
+   and exists only in the main checkout. Do **not** copy it (it is a credential).
+   Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
+   normally.
+2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
+   gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
+   the `ENGINE` hook — a shim that intercepts `run` and re-execs
+   `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
+   `tracked_files()` is this bug, not a broken tree.
+
+## 4. Branch naming
+
+Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
+trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
+The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
+
+## 5. Cluster access
+
+`kubectl` does not work on the operator workstation; the toolchain lives only in the
+pinned container. Always name the context explicitly:
+
+    ./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+`dev.sh` merges both clusters and picks a current-context **per invocation**; that
+default has already changed once. Never rely on it. Do not run two `dev.sh`
+invocations concurrently.
+
+**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
+
+## 6. Secrets
+
+- **Never `sops -d` a file into a terminal, a log, or a transcript.**
+- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
+  list key names or object names only. A password has been leaked into a transcript
+  this way before.
+- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
+  the single creation-rules file; do not add a second one in a subtree.
+- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
+  flow. Do not design around committing one — read the expiry from the issuing page and
+  treat a burnt token as free to retry.
+
+## 7. Before you finish
+
+Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
+verified, and what you did **not** verify. State gaps plainly — an honest
+"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
+burned by the difference.
@@ -2424,6 +2424,148 @@ container dies with `PermissionError` on every run; my older base had `0400` and
 - The apex flip itself is `migration/dns-apex-inversion`'s job. This change only guarantees the realm
  follows git once the knob moves; it does not move it.

+## 2026-09-12 — The silent-gateway outage now has an alert, and the direct signal turned out not to exist (branch `feat/egress-gateway-staleness-alert`)
+
+Hand-off item 4 from the section above: *"A silent gateway outage still has no alert… nothing
+watches the invariant `egress-uplink` label == `cilium-dbg bpf egress list`'s Egress Ifindex."*
+Closed, as far as Prometheus can close it — and the part that cannot be closed is recorded here
+rather than papered over with a plausible-looking metric name.
+
+### The direct signal is not available, verified rather than assumed
+
+The invariant wants the BPF egress map's contents. They are not in Prometheus, for two independent
+reasons, both checked read-only against live prod:
+
+| check | result |
+|---|---|
+| `{__name__=~"cilium.*egress.*"}`, `cilium_drop.*`, `hubble_drop.*` | **empty vector** — no `cilium_*`/`hubble_*` series exist at all |
+| container ports on the cilium DaemonSet | `health=9879`, `peer-service=4244` — **no metrics port**; only the operator has `operator-prometheus-serve-addr :9963` |
+| `kubectl get servicemonitors -A \| grep -iE 'cilium\|hubble'` | none — Hubble/Cilium metrics export is still the open TODO in `platform/cilium/helmrelease.yaml` |
+
+So `HubbleDeniedFlowSpike` cannot fire today either; that is a separate, pre-existing gap, noted
+and left alone. And even with agent metrics turned on, **Cilium OSS publishes no series carrying
+egress-map contents** — the map is readable only through `cilium-dbg bpf egress list`, a CLI over
+the BPF map. Nothing was invented to fill the hole.
+
+What node-exporter *does* export is the live tun0 **ifindex as a metric value**:
+`node_network_iface_id{device="tun0"}` = `129` on `k3s-prod-3`. Read-only on the live cluster, all
+three agree right now:
+
+```
+node label    homelab.keithjackson.dev/egress-uplink = 10.150.192.86-129
+node-exporter node_network_iface_id{device="tun0"}   = 129
+bpf map       Egress IP 10.150.192.86 / Egress Ifindex 129   (all 16 rows, 4 vpn-class pods)
+```
+
+(The map on the two NON-gateway agents reads `Egress IP 0.0.0.0 / Ifindex 0` with
+`Gateway IP 192.168.2.12` — only the gateway node resolves the interface. Worth knowing before
+debugging the wrong agent.)
+
+### PromQL cannot express the equality test — so the invariant is counted instead
+
+The label's value is `<addr>-<ifindex>`; the live ifindex is a metric VALUE. PromQL has no
+label-to-number conversion, so `label == value` is **not expressible**, and there is no
+level-triggered form of this comparison. What is expressible is the same invariant counted over a
+window: **the published label must change at least as often as tun0 is re-created.**
+
+```promql
+(
+    changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+  >
+    (count(count_over_time(max by (label_homelab_keithjackson_dev_egress_uplink) (
+       kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})[1h:1m])) - 1)
+)
+and on()
+(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+```
+
+`count(count_over_time(max by (<label>) (…)[1h:1m]))` is the number of DISTINCT label values in the
+window. The `max by (<label>)` wrapper is load-bearing: it aggregates away `pod`/`instance` so a
+kube-state-metrics restart is not read as a label change. Both sides reduce to a single unlabelled
+series, which is why `and on()` joins — the same reason `VPNTunnelDown` needs `on()`.
+
+### Proof it evaluates — executed against prod's Prometheus, port-forward + `/api/v1/query`
+
+There is no promtool in this repo, so query execution IS the verification. The **rendered** expr was
+extracted from `kubectl kustomize platform-config` output and run verbatim:
+
+| state | query | result |
+|---|---|---|
+| healthy, now | rendered `VPNEgressSteeringStale` | `[]` — not firing |
+| **no gateway node** (`@ 1789245000`, before prod's stand-up: no tun0 series, no labels) | same expr | `[]` — cannot fire |
+| **stale**, constructed from real data | same expr with the already-exported `egress-gateway` label substituted for `egress-uplink`, window 3h | **`1` — FIRES** |
+| same expr, window 1h | | `[]` |
+
+The stale case is honest and cost prod nothing: prod's tun0 really did go **128 → 129** ~2.5h
+earlier when it moved to PIA `.135`, and the `egress-gateway` label really did *not* follow it
+(`changes(...[3h:1m])` = 1, distinct label values = 1 → `1 > 0`). That is bit-for-bit the shape of
+a label that missed a flap. The healthy counterfactual is the one thing left as reasoning rather
+than a query: over that same 3h the REAL `egress-uplink` label held two values
+(`10.72.192.7-128`, then `10.150.192.86-129`), giving `1 > 1` = false. It could not be queried
+because the label was not exported yet — which is the next point.
+
+Supporting queries, also live: the distinct-value idiom returns **1** for a label that held still
+over the window and **2** for one whose value changed inside it (node 3's `egress-gateway` label
+over 12h, which went `""` → `"true"` at stand-up); the gateway guard returns 1; subquery +
+`changes()` returns 0/1/1/1 over 15m/3h/6h/12h as the flap history requires.
+
+### Behaviour at the edges
+
+- **No gateway node:** three independent reasons it cannot fire — no tun0 series, no uplink label,
+  and the `kube_node_labels` guard `VPNTunnelDown` already established. Verified at a past
+  timestamp when prod genuinely had no gateway. This matters: `VPNTunnelDown` fired permanently on
+  prod for exactly this reason and had to be retrofitted.
+- **Convergence:** the up-script rewrites the label within seconds of a connect, worst case ~5m
+  (its 30 × 10s kube-API retry budget on a cold boot, when the API is not up yet). The condition is
+  true during that gap, so `for: 15m` — comfortably past the 2-minute timer and the 5-minute retry
+  budget — is what keeps a normal reconnect quiet.
+- **Bounded, deliberately:** a missed flap alerts from +15m to +1h, then RESOLVES even if egress is
+  still dead. An unbounded critical on `email-critical` is the worse failure for this repo, and 15
+  minutes still beats 80. A level-triggered version needs the map's ifindex as a real metric — the
+  node-exporter textfile collector route, still open.
+
+### The second rule exists because the first one fails OPEN
+
+`VPNEgressSteeringStale` needs `egress-uplink` in kube-state-metrics. Without it the comparison
+degrades to an empty vector and alerts nothing — so `VPNEgressUplinkLabelMissing` (warning,
+`for: 30m`) watches the precondition: a node labelled `egress-gateway=true` with no `egress-uplink`
+label visible. Live right now it returns **1**, correctly: the KSM allowlist extension is in this
+same branch and not yet deployed. It self-clears on the first reconcile, and `for: 30m` covers the
+kube-state-metrics roll. The rule and the allowlist are in the same Flux kustomization on purpose.
+
+`kube-state-metrics.metricLabelsAllowlist` now reads
+`nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]` — a label
+DIMENSION on the same 3 `kube_node_labels` series, not new series, on exactly one node.
+
+### Rejected, with reasons
+
+- **A black-box egress probe** is the most honest signal — it is the only one that answers "does a
+  vpn-class pod's traffic actually leave via the tunnel?" and the only one that catches "the label
+  was written and Cilium ignored it anyway". Rejected **for now**: there is no blackbox-exporter and
+  no probe CronJob in this repo, so it means a new always-on component (itself vpn-class, itself an
+  alerting surface, itself dependent on a third-party endpoint) — and it could not have been
+  verified on prod today. It remains the right next step.
+- **A node-exporter textfile metric from the re-steer script** (hand-off item 4's own suggestion) is
+  the only route to a true level-triggered label-vs-MAP comparison. It needs a textfile directory
+  mounted into the node-exporter DaemonSet plus an Ansible change applied on a prod node — neither
+  of which this branch could execute or verify. Left open, now with the reason recorded.
+- **`absent(node_network_receive_bytes_total{device="tun0"})`-style presence checks** — that IS
+  `VPNTunnelDown`, and it is exactly the blind spot: green while steering points at a dead ifindex.
+
+### Left for the operator / not settled here
+
+1. **The rules are unproven in the firing sense.** The expressions evaluate against live prod data
+   and the stale branch was exercised with a real flap, but no alert has actually reached
+   Alertmanager or the inbox. `conformance/tests/l3-fault-injection/vpn-killswitch` stops and
+   restarts the tunnel and is the natural place to assert `VPNEgressSteeringStale` stays quiet
+   through a healthy reconnect; the alerting L3 test with the SMTP sink is where a fired-and-emailed
+   proof would come from. Neither was fired at prod unasked.
+2. **`HubbleDeniedFlowSpike` cannot fire** — no cilium/hubble metrics are scraped on either cluster.
+   Pre-existing, orthogonal, and the same TODO in `platform/cilium/helmrelease.yaml` blocks it.
+3. **Window vs. stickiness** is a judgement call, not a measurement: 1h chosen so a missed flap
+   alerts for ~45 minutes. If the operator would rather have a sticky alert that needs manual
+   acknowledgement, that is the textfile-collector design, not a tweak to this one.
+
 ## 2026-09-13 — **PROD IS THE APEX.** Dev decommissioned; and a GitOps deadlock found the hard way

 Prod serves `home.keithjackson.dev`. Dev's three VMs (Proxmox 101/102/105) are stopped with
@@ -4396,3 +4538,328 @@ Open, in the order I would take them:
 7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, 103 unmapped
   paths, and two rendered manifests nothing applies (the keycloak one would stand up a conflicting
   instance).
+
+## 2026-09-19 — Overnight: the backup alert was crying wolf, and 49 app manifests were gated by nothing
+
+Worked autonomously while the operator slept. One theme again: things that reported green, or red, over
+something other than what they claimed to measure.
+
+### 1. VeleroNoRecentBackup was firing CRITICAL over backups that protected everything
+
+`velero-daily-20260919030041`: **PartiallyFailed, 3079 of 3079 items backed up, 2 errors** — both
+`PodVolumeBackups` for the emptyDir `plugins` + `scratch` volumes of a
+`homepage-default-kopia-maintain-job-…` pod. Velero runs transient kopia repository-maintenance Jobs in
+its own namespace, one per BackupRepository; under `defaultVolumesToFsBackup: true` it tries to
+fs-backup their volumes, the Job finishes, the pod disappears mid-backup, and the PVB fails. Four such
+failures had accumulated.
+
+Nothing real was ever missed. But `PartiallyFailed` does not advance
+`velero_backup_last_successful_timestamp`, so the alert guarding R-STATE-03 fired nightly over Velero
+cleaning up after itself. **An alert that cries wolf every night is how a real backup failure gets
+ignored** — which is precisely the failure mode that alert exists to prevent.
+
+**First fix was right about the bug and wrong about the layer.** I excluded ns `velero`'s volume data
+(keeping its resources in schedule 2) and explicitly deferred the general fix — skip emptyDir by type —
+because Velero's `volumeTypes` policy condition was unproven here and an unparseable `policies.yaml`
+breaks EVERY backup. I wrote "revisit if the shape shows up elsewhere."
+
+It showed up elsewhere inside the hour. The verification backup came back PartiallyFailed with 4 errors,
+all in ns `runbooks`: the nginx Deployment's emptyDir `tmp` + `cache`, failing with
+`error to expose PVB: error getting client pod …: pods "…" not found`. **Cause worth stating plainly: I
+caused it** — pushing a runbook edit changed the configMap hash, rolled the Deployment, and the backup
+lost the pods mid-snapshot. Self-inflicted, not a standing defect. But the mechanism is not
+self-inflicted: a PVB exposes a volume *through its pod*, so any rollout landing in the backup window
+fails one, and one failed PVB makes the whole Backup PartiallyFailed. On a GitOps cluster that is
+routine, not an edge case.
+
+So the property is what matters, not the namespace: an emptyDir is created empty with its pod and
+destroyed with it, so backing one up cannot protect anything while attempting it can break the backup
+that protects everything else.
+
+**The `volumeTypes` risk was retired by testing, not by hoping.** New policy into a THROWAWAY
+ConfigMap first, live one untouched, proven on prod (velero 1.18.1) with a control:
+
+| ns `runbooks` | PodVolumeBackups |
+|---|---|
+| control, no policy | **4** (2× `tmp`, 2× `cache`), Completed |
+| with the policy | **0**, Backup Completed, 0 errors |
+
+The control is the point — "zero PVBs" alone is equally consistent with nothing having been attempted,
+which is exactly the empty green I shipped earlier in this session with a 24h baseline built from ten
+minutes of data.
+
+Then reverted the namespace exclusion as redundant (`plugins`/`scratch` are emptyDir, verified on a live
+maintain pod) rather than leaving two overlapping fixes.
+
+**Proof, full cluster, policy live:** `pre-memswap-202609190525` — **Completed, 3926/3926 items, 0
+errors, 21/21 PodVolumeBackups Completed, 0 emptyDir volumes backed up.** `VeleroNoRecentBackup`
+cleared. `VeleroBackupPartialFailure` remains at `increase = 1.008` from the pre-fix failure and
+self-clears about an hour after it, by design — the rule is `increase(...[1h]) > 0`, which is correct.
+
+That backup also serves as the pre-maintenance backup the memory-swap runbook requires.
+
+### 2. The runbook claimed the CNPG primaries sit on node 1. They do not.
+
+Verified live: all three CNPG clusters run 3 instances spread one per node by required anti-affinity,
+and **k3s-prod-1 holds a REPLICA of each** (`keycloak-db-2`, `nextcloud-db-2`, `semaphore-db-2`). No
+primary is on node 1, so draining it forces **no failover at all** — the best state this maintenance
+could ask for. The original sentence was stale session state. The runbook now prints the command to
+re-derive it, because primaries move after any restart.
+
+That also settles `CNPGPrimariesCoLocated`, firing for k3s-prod-2 (both `keycloak-db` and
+`semaphore-db` primaries). **Do not tidy it before the swap.** Rebalancing means promoting a replica,
+the only node with spare primary capacity is node 1, so "fixing" the alert would put a primary on the
+node about to be powered off and convert a clean drain into a two-database failover. Rebalance after
+node 1 is back and caught up. Warning is now in the runbook.
+
+### 3. 49 app manifests attributed no requirement, so the merge gate ran no behaviour test for them
+
+`ci/touched-requirements.sh`'s coarse path map had no arm for `apps/*`. A change adding or editing any
+workload selected **zero** requirement-scoped tests — so none of the L2 behaviour tests that enforce
+default-deny, egress class or auth mode ran on exactly the changes that can violate them. Kyverno
+admission and the L0 egress-path check are real backstops, but admission is post-merge and the gate
+exists to catch it before.
+
+Also unmapped: `platform/homepage` and `platform/headlamp` — **the C2 UI**. This was already documented
+as a known gap in `changed_diff()`'s rule-3 comment ("the ONLY route to R-C2-01/02 was rule 3", i.e.
+editing the C2 test itself), and had sat there since.
+
+Twelve arms added: `apps/vms/*` (before `apps/*`, since `case` takes the first match), `apps/*`,
+`platform/homepage|headlamp`, `platform/kubevirt`, `platform/gateway`, `platform/coredns`,
+`platform/crafty`, `platform/external-snapshotter|backup`, `runbooks/*`, `platform-config/*`,
+`platform/kustomization.yaml` and `.sops.yaml`.
+
+`platform/kustomization.yaml` deserves its own note: it is the assembly list, so **one deleted line
+disables an entire platform component** while every file it points at stays valid on disk. That is the
+most consequential single-line edit in the repo and it attributed nothing.
+
+Measured effect for an `apps/` change: **0 → 13 requirement-scoped tests** (3 L1, 8 L2, 2 L3). Coverage
+stays 51/51, so the gate's "attributed IDs must be covered" assertion does not false-block.
+
+Ten new assertions in `ci/tests/touched-requirements.test.sh` (26 total, all passing), including the
+ORDER property — `apps/vms/*` must win over `apps/*` — since that is a real failure mode of `case` and
+not merely a matter of both arms existing.
+
+**A correction on my own numbers.** I first reported "192 unmapped files", then "182". Both were
+inflated by my audit script, not by the repo: the regex used `[^)]*`, which matched across a newline, so
+the first arm parsed as `case "$1" in\n    catalog/*` and never matched anything — and I separately
+mis-parsed alternation arms like `.gitea/*|ci/*`, which made `ci/` look unmapped when it is mapped.
+True remainder after this change is **18 files, all prose or captured artifacts** (`docs/plans/`,
+root READMEs, `.standup-dns/*.txt`).
+
+### 4. Finished `feat/egress-gateway-staleness-alert` — it had been backlogged because it could not pass the gate
+
+Merged the two alerts for the silent egress-gateway failure (`VPNEgressSteeringStale` critical,
+`VPNEgressUplinkLabelMissing` warning) plus the KSM allowlist entry that makes them non-inert.
+
+Confirmed first that they are **live on prod, not decorative**: `k3s-prod-3` carries
+`egress-gateway=true` and `egress-uplink=10.150.192.86-139`, `vpn-egress-steering` has been in place 14
+days, and the label was exported on **0 series** before this change — so the allowlist entry is the
+switch that turns the alerts on.
+
+Three whole-repo checks failed the branch, each a real gap: the runbook the alerts link to did not
+exist (a firing alert would hand the operator a 404); `alerts.yaml` is an already-covered file, so new
+rules in it must be unit-tested; and the scrape-target guard broke. Wrote
+`runbooks/vpn-egress-stale.md` (wired into the generator, four required sections) and
+`platform/monitoring/tests/vpn-egress.test.yaml` (four promtool fixtures — tests 1 and 2 are the SAME
+reconnect with and without the label following it, which isolates the mechanism: a label value change
+retires one series and starts another rather than changing a sample).
+
+**Two of the three failures were defects in the checks, not the branch:**
+
+- `alert-metrics-have-scrape-targets` asserted the allowlist matched `nodes=[...egress-gateway]` with a
+  literal closing bracket — that the list held *exactly* that one label. Correctly adding a second label
+  therefore failed the check while the guarded property stayed satisfied. A guard that breaks when you
+  correctly extend the thing it guards teaches people to edit the guard, which is how it stops guarding.
+- `UNTESTED_ALERTS` flagged `VPNTunnelDown` as stale because "asserted on" is a **bare word match over
+  the test file** — my header *comment* mentioning it counted as coverage. I did not delete the register
+  entry to make that go away; instead the fixtures now assert `VPNTunnelDown` **quiet** in all four
+  cases. That required making them faithful: they had no `node_network_receive_bytes_total{device="tun0"}`
+  at all, so `VPNTunnelDown` would have been firing throughout unnoticed, because promtool only checks
+  the alertnames you name. A healthy tun0 passes traffic — and that silence IS the claim these rules
+  exist for.
+
+**Known, deliberately not fixed:** that word match means any alert named in prose counts as tested.
+Tightening it to `alertname:` positions could expose other prose-only "coverage", and the register is
+shrink-only, so the cascade wants doing deliberately — not at 01:00 unattended.
+
+### 5. Stale worktrees: audited, deliberately not touched
+
+18 agent worktrees remain under `.claude/worktrees/`. **14 clean, 4 dirty** — the dirty ones being
+`fix/gate-does-not-pass-on-skipped-stages`, `fix/postbuild-substitution-catches-dollar-mangling`,
+`fix/scrape-targets-for-inert-alerts` and `fix/qbittorrent-durable-webui-credential`. Every branch is
+already merged into main, so the uncommitted state is leftover agent scratch rather than unmerged work.
+
+Left in place. They no longer affect the gate (the scanner now skips dot-directories), so removal is
+disk hygiene, and deleting someone's uncommitted changes unasked is not a call to make while they are
+asleep. `git worktree remove` preserves branches and their commits, so only that scratch is at risk.
+
+### PM HANDOFF
+
+1. **R-STATE-02 is still the Phase 7 exit item, and it is blocked on permission, not on readiness.**
+   `./conformance/run.sh --layer l3 --requirements R-STATE-02 --record`. The permission classifier
+   refuses it and I did not route around it — hand-rolling the cordon would defeat the guard. I read the
+   test first: it cordons ONE node and evicts only its OWN writer pod, never `kubectl drain`; kubelet
+   and etcd stay up so quorum is untouched; PVC and pods live in the per-test namespace; and `finally`
+   uncordons via a unique label even on failure or panic. Preconditions verified: 3 Ready nodes,
+   `longhorn` default SC, nothing pre-cordoned, 24/24 volumes healthy. Either run it, or add a Bash
+   permission rule.
+2. **The memory swap is still the cheapest R-STATE-02 evidence** — a supervised single-node loss on prod
+   is the event that requirement describes. Node 1 currently holds no DB primary, which is the ideal
+   window.
+3. **Node 2 DIMM re-inventory** — blocked on the vault become password. Never `ansible-inventory --list`.
+4. **Pi-hole needs A records** for `unifi.prod` → `.21` and `crafty-mc.prod` → `.22` before router DHCP
+   option 6 → `.23` is safe. It currently answers `.20` for both, and neither service speaks HTTP
+   through that Gateway VIP, so the LAN-wide change would break UniFi adoption and Minecraft. Until
+   then the per-machine dnsmasq override is load-bearing.
+5. **UniFi guest has no IP** — needs console; `cat /etc/netplan/*.yaml` is the confirming command.
+6. **Five `public` services have no working public path** — waiting on WAN forwarding, deliberately.
+7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, two rendered
+   manifests nothing applies (the keycloak one would stand up a conflicting instance).
+
+## 2026-09-29 — The CI/CD merge gate does not exist: no runner, no branch protection. **PM HANDOFF.**
+
+Went to land the plex claim wiring through the documented PR flow and discovered the flow has
+never run. **PR #1 is the first pull request this repository has ever had.** It is still queued.
+
+### What is actually true of the live Gitea (verified via API, 2026-09-29)
+
+| Claimed (DECISIONS-LOG 2026-08-31) | Measured |
+|---|---|
+| "required status check" on `main` | `branch_protections` returns `[]`. `protected: false`, `enable_status_check: false`, `status_check_contexts: []`, `required_approvals: 0`, `user_can_push: true` |
+| "runner `incluster-dind` online (ns gitea-actions)" | **Zero runners registered** at repo, org and site level |
+
+Both halves of that entry are false. Nothing enforces the gate, and the gate cannot run.
+PR #1's two contexts (`gate / gate`, `gate / conformance`) sat `pending` for 14+ minutes with
+nothing to execute them, and will sit there indefinitely.
+
+This is a **fail-open governance hole**, the same class as the two found on 2026-09-18 — and it
+was invisible precisely because the log asserted the opposite. Every "the gate is enforcing"
+statement since 2026-08-31 rested on an unverified claim.
+
+### Ordering trap — read before fixing
+
+**Do not enable required status checks until a runner is live.** Requiring
+`gate / gate (pull_request)` with no runner deadlocks every PR permanently, including the PR
+that would fix it. Sequence: runner first, verify the gate actually runs and passes on a real
+PR, *then* turn on enforcement with the context name CI actually reported.
+
+Related: `gate.yml`'s `conformance` job is `if: false` by design (honest skip, never fake green),
+but it still publishes a `gate / conformance (pull_request)` context. If that context is ever
+listed as required, it deadlocks every PR for the same reason. Wire it or exclude it.
+
+### Delivery contract written down
+
+Added `CLAUDE.md` at the repo root — the first agent-facing delivery doc here. It states the
+branch -> PR -> gate -> approval -> merge rule, records that enforcement is currently weaker than
+the rule (so agents follow the rule rather than what they can get away with), and carries the
+toolbox-gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and the secret
+handling rules.
+
+It also flags that **37 of 39 runbooks still instruct `git push origin main`**. Those were
+accurate before the PR flow and are stale now. Correcting them is tracked, not done.
+
+### Also found, not fixed
+
+- `gate.yml`'s push trigger lists `feature/**`; this repo's branches use `feat/**`, so branch
+  pushes get no fast-feedback run. PR trigger is unaffected.
+- The gitea-actions runner is GitOps-managed (`apps/wave-1/gitea-actions.yaml` +
+  `gitea-actions-runner-secret.sops.yaml`), so it is deployed but not registering — most likely a
+  stale registration token in that SOPS secret. Needs in-cluster diagnosis (operator).
+- **Plex is still unclaimed and it was never checked** whether it is already claimed. Do that
+  before spending a token — `runbooks/plex-claim.md` §First commands.
+- `runbooks/plex-claim.md` asserts a "~4-minute TTL" for a plex.tv claim token in four places,
+  **uncited**, and the whole stopwatch procedure is built on it. Upstream `pms-docker`'s README
+  says nothing about expiry. The number is plausible community folklore, not a measured fact.
+  The authoritative reading is the countdown rendered on plex.tv/claim at issue time.
+
+### Next steps
+
+1. **Operator:** diagnose the gitea-actions runner; get one registered.
+2. **Operator:** enable branch protection on `main` — push blocked, approvals required — and only
+   then add the gate as a required status check.
+3. Land `feat/plex-claim-sops-secret` (PR #1) once the gate can actually run.
+4. Claim plex, after checking it is not already claimed.
+5. Correct the 37 runbooks' `git push origin main` wording.
+
+## 2026-09-29 — Plex is claimed; the CI/CD gate that "enforced" it never existed. **PM HANDOFF.**
+
+Began as "inject a plex.tv claim token". Ended by discovering the delivery pipeline was fiction,
+rebuilding it, and shipping the token through it. Plex is **claimed** — `PlexOnlineToken` is
+present in `/config` on the `plex-config` PVC, so the exchange is now a permanent no-op and the
+library scan is running over the 8.33T NFS share.
+
+### The finding: both halves of the 2026-08-31 CI claim were false
+
+That entry recorded "Gitea Actions CI is LIVE + enforcing … required status check; runner
+`incluster-dind` online". Measured against the live API on 2026-09-29:
+
+| Claimed | Measured |
+|---|---|
+| `main` protected, gate required | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
+| Runner online | **zero** runners at repo, org AND site level |
+
+**PR #1 was the first pull request this repository has ever had.** Its contexts sat `pending`
+for 30.6 minutes across 122 polls and never reported. For a month, every "the gate enforces this"
+statement rested on an assertion nobody had checked — the same fail-open class as 2026-09-18,
+and the reason it hid so long is that the log itself was the evidence.
+
+### Root cause of the dead runner
+
+`act_runner` registers **once** and persists its identity in `/data/.runner` on the
+`act-runner-data` PVC. Gitea no longer held a matching row, so the pod booted, saw `.runner`,
+**skipped registration**, and polled with an identity Gitea did not recognise. Symptom: zero
+runners, pod Running, nothing crashlooping, no error anyone reads. Deleting the PVC forced a
+genuine re-registration (`.runner is missing` → `Runner registered successfully`).
+
+It then did not come back for several minutes, because Flux could not recreate the PVC:
+`apps-wave-1` was blocked four levels deep on a **stale** `DependencyNotReady` for
+`platform-config`, which itself read `Ready=True`. Each wave re-evaluates only on its own
+`interval: 10m`, so a resource deleted at the bottom of the chain can take ~40 min to return.
+**Force it** with `flux reconcile kustomization apps-wave-1 --with-source` rather than waiting.
+
+### Corrections to things this log and the runbooks asserted
+
+- **A skipped Gitea job reports `success`, not pending.** Measured: `gate / conformance` returned
+  `success` / "Has been skipped". An earlier entry (#2) said requiring it would deadlock every PR.
+  Wrong — it would pass **vacuously**, forever, while nothing ran. Under HARD RULE 4 that is worse
+  than a deadlock, because a deadlock is loud. Never require it; require
+  `gate / gate (pull_request)` alone. Corrected in #3.
+- **`gate.yml`'s push trigger omits `feat/**`** (it lists `feature/** fix/** docs/** chore/**`).
+  With `act_runner` capacity 1, a branch prefix that IS in that list fires a second redundant run
+  which serialises ahead of the PR run and doubles wall-clock. Exploited deliberately in
+  `infra/plex/claim-ship.sh`; the honest fix is to align the trigger list.
+- **The "~4-minute TTL"** asserted in four places was **uncited**, and upstream `pms-docker`
+  documents no expiry at all. The authoritative reading is the countdown plex.tv/claim renders at
+  issue time. Removed from `runbooks/plex-claim.md`; the same claim still sits in
+  `catalog/plex.yaml` and the Secret's header comment (left alone — editing a `.sops.yaml`
+  comment risks the MAC, and touching `catalog/plex.yaml` re-renders a golden).
+
+### Measured, once the pipeline actually worked
+
+`gate / gate (pull_request)` 1m34s–1m58s. Full claim loop end to end **~2m45s** — comfortably
+inside the window, which retires the theory that a short-lived token cannot survive a PR flow.
+That theory was mine and it was wrong: the pipeline was never too slow, it simply did not exist.
+
+### State now
+
+- `main` **protected**, `user_can_push:false`. Runner `incluster-dind` online.
+- `CLAUDE.md` is the agent delivery contract; `infra/gitea/` holds the protection + runner scripts.
+- **All 37 runbooks** that instructed `git push origin main` now describe the PR flow — that
+  instruction had become actively blocked, not merely stale.
+- `infra/plex/claim-ship.sh` drives the claim loop end to end.
+
+### Next steps
+
+1. **`enable_status_check: true`** with `gate / gate (pull_request)` as the only context. The
+   prerequisite is met: the gate has been observed green on real PRs.
+2. Add `jakman85` as a repo **collaborator**, then `required_approvals: 1`. (Currently the only
+   org member is `gitea-admin`, which agents authenticate as — requiring approval before that is
+   fixed would leave nobody able to approve.)
+3. Run the **push probe** against protection: `gitea-admin` is a site admin and Gitea may let it
+   bypass. `user_can_push:false` is the API's opinion, not proof.
+4. Align `gate.yml`'s push trigger with this repo's `feat/**` prefix.
+5. Convert the `dind` container to a **native sidecar** with a `startupProbe`, per the standing
+   FOLLOW-UP note in `apps/wave-1/gitea-actions.yaml`. It deletes both hand-rolled wait loops and
+   the envsubst hazard. Land it ALONE — it is the merge gate for the whole repo.
+6. Create the Plex libraries over `/media/movies`, `/media/tv`, `/media/photos`.
@@ -17,6 +17,7 @@ resources:
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
+- plex-claim-secret.sops.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
@@ -0,0 +1,69 @@
+# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
+# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
+#
+# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
+# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
+# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
+# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
+# not prose, so the upstream conditions below are described in words.
+#
+# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string
+# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has
+# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token
+# committed here is ever still valid when the pod starts. This file exists to make the Secret — and
+# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token.
+# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md.
+#
+# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
+# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
+# the same file extracted from the running `plexinc/pms-docker:latest` image):
+#   - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
+#     PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
+#     skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
+#     README: "If server is already logged in, this parameter is ignored."
+#   - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
+#     empty body. curl's exit status is never checked, the sed for the authentication-token element
+#     yields the empty string, and the success branch at line 120 simply does not fire. There is no
+#     else branch and no non-zero exit; the script has no `set -e` and ends by touching
+#     /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
+#     It does NOT crashloop.
+#   - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
+#     if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
+# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
+# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
+# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
+# by hand at claim time.
+#
+# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
+#   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+# Never decrypt this file into a shell, a log, or a shared terminal.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: plex-claim
+    namespace: plex
+    labels:
+        app.kubernetes.io/name: plex
+type: Opaque
+stringData:
+    PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
+            cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
+            THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
+            QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
+            RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-29T06:31:16Z"
+    mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4
@@ -132,6 +132,11 @@ spec:
          value: Etc/UTC
        - name: ALLOWED_NETWORKS
          value: 10.0.0.0/8,192.168.0.0/16
+        - name: PLEX_CLAIM
+          valueFrom:
+            secretKeyRef:
+              name: plex-claim
+              key: PLEX_CLAIM
        volumeMounts:
        - name: config
          mountPath: /config
@@ -14,7 +14,7 @@ metadata:
        app.kubernetes.io/name: act-runner
 type: Opaque
 stringData:
-    token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
+    token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
 sops:
    kms: []
    gcp_kms: []
@@ -30,8 +30,8 @@ sops:
            MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
            UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-08-26T04:48:14Z"
-    mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
+    lastmodified: "2026-09-29T05:48:35Z"
+    mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4
@@ -32,11 +32,40 @@ env:
  # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
  - name: ALLOWED_NETWORKS
    value: 10.0.0.0/8,192.168.0.0/16
-  # PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
-  # shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
-  # is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
-  # single API POST) against the running server; the claim then persists on the /config PVC and
-  # survives pod rolls (including the roll this env change causes), so it never belongs in git.
+  # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
+  # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
+  # workload). Plex cannot create libraries over the mounted bulk share while the server is
+  # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
+  # half of that reasoning survived contact with the upstream source.
+  #
+  # THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
+  # abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
+  # Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
+  # documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
+  # explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
+  #
+  # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
+  # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+  # the running `:latest` image), NOT assumed:
+  #   - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
+  #     PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
+  #     no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
+  #     logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
+  #   - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
+  #     status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
+  #     there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
+  #     SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
+  #     default 0, which continues startup even when a cont-init script fails.
+  # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
+  # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
+  # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
+  # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
+  # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
+  - name: PLEX_CLAIM
+    valueFrom:
+      secretKeyRef:
+        name: plex-claim
+        key: PLEX_CLAIM
  # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
  # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
  # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
@@ -56,10 +85,12 @@ storage:
                              # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
                              # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
                              # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
-                              # makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
-                              # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
-                              # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
-                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
+                              # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
+                              # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
+                              # exchange is then skipped on every later start. (The older note here claimed a
+                              # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
+                              # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
+                              # failure, and that is why one is committed.)
  # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
  # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
  # downloads/ movies/ photos/ tv/).
@@ -1844,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
    convention:
      - `<svc>.yaml`                        -> wave(svc)
      - `<svc>-bootstrap-secret.sops.yaml`  -> wave(svc)   (consumed by svc's env, same apply)
+      - `<svc>-claim-secret.sops.yaml`      -> wave(svc)   (a one-shot upstream-service claim token
+        consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
+        bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
+        in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
      - `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc)  (forward-auth sidecar OIDC creds)
      - `<svc>-db.yaml`                     -> wave(svc-db) (a strictly-lower wave; CNPG DB)
      - `<svc>-db-app.sops.yaml`            -> wave(svc-db) (CNPG initdb owner secret)
@@ -1870,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
        w = waves[name]
        add(w, f"{name}.yaml")
        add(w, f"{name}-bootstrap-secret.sops.yaml")
+        # A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
+        # app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
+        # so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
+        # Placed only when the file exists, so services without one are unaffected.
+        add(w, f"{name}-claim-secret.sops.yaml")
        # forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
        # Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
        # so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
@@ -2548,7 +2557,8 @@ def _selftest() -> int:
    # 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
    #     EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
    present = {
-        "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
+        "a.yaml", "a-claim-secret.sops.yaml",
+        "svc.yaml", "svc-bootstrap-secret.sops.yaml",
        "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
        "svc-db-backup-credentials-prod.sops.yaml",
        "gitea.yaml",
@@ -2559,6 +2569,10 @@ def _selftest() -> int:
    plan = plan_apps_waves(plan_sds, present)
    flat = {f: wv for wv, fs in plan.items() for f in fs}
    check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
+    check(flat.get("a-claim-secret.sops.yaml") == 0,
+          "a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
+          "as its app — an env secretKeyRef has no `optional`, so a later wave means "
+          "CreateContainerConfigError")
    check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
          and flat.get("svc-db-backup-credentials.sops.yaml") == 0
          and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
@@ -136,6 +136,65 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
 assert_present "R-ENV-02"     "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
 assert_present "R-GITOPS-01"  "$OUT4" "clusters/* attributes R-GITOPS-01"

+# ---------------------------------------------------------------------------------------
+# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
+#
+# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
+# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
+# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
+# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
+# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
+# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
+#
+# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
+# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
+#
+# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
+# from the PATH and not from the content scan.
+
+# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
+# under test here, not just the presence of both arms.
+git -C "$TMP" checkout -q -b feature-apps-vms main
+mkdir -p "$TMP/apps/vms/demo"
+printf 'kind: VirtualMachine\nmetadata:\n  name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
+OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
+assert_present "R-PLAT-03"  "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
+assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
+assert_absent  "R-PLAT-02"  "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
+
+# A plain ServiceDefinition under apps/ gets the container-workload set.
+git -C "$TMP" checkout -q -b feature-apps main
+mkdir -p "$TMP/apps/wave-1"
+printf 'kind: ServiceDefinition\nmetadata:\n  name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
+OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
+assert_present "R-PLAT-02"  "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
+assert_present "R-NET-01"   "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
+assert_present "R-NET-06"   "$OUT_APPS" "apps/* attributes the egress-class requirements"
+assert_present "R-IDP-02"   "$OUT_APPS" "apps/* attributes the auth-mode requirements"
+
+# homepage / headlamp ARE the command-centre surface.
+git -C "$TMP" checkout -q -b feature-c2 main
+mkdir -p "$TMP/platform/headlamp"
+printf 'kind: Deployment\nmetadata:\n  name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
+OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
+assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
+assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
+
+# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
+git -C "$TMP" checkout -q -b feature-assembly main
+mkdir -p "$TMP/platform"
+printf 'resources:\n  - cilium\n  - longhorn\n' > "$TMP/platform/kustomization.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
+OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
+assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
+
 # ---------------------------------------------------------------------------------------
 # THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
 #
@@ -177,6 +177,23 @@ path_requirements() {
  case "$1" in
    catalog/*)                 echo "R-GITOPS-02" ;;
    schemas/*)                 echo "R-GITOPS-02" ;;
+    # apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
+    # manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
+    # adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
+    # checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
+    # class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
+    # violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
+    # catch it at apply time" is post-merge, and the gate exists to catch it before.
+    #
+    # ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
+    # apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
+    # subject to default-deny (R-NET-01).
+    apps/vms/*)                echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
+    # Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
+    # workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
+    # (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
+    # default-deny (R-NET-01).
+    apps/*)                    echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
    platform/kyverno/*)        echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
    platform/cilium/*)         echo "R-NET-01 R-NET-02 R-NET-05" ;;
    platform/longhorn/*)       echo "R-STATE-01 R-STATE-02" ;;
@@ -208,6 +225,39 @@ path_requirements() {
    platform/keycloak/*)       echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
    platform/oauth2-proxy/*)   echo "R-IDP-02 R-IDP-06" ;;
    platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
+    # The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
+    # rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
+    # platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
+    # itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
+    # docs/12-command-center-ui.md), so editing them must select the C2 tests.
+    platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
+    # KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
+    platform/kubevirt/*)       echo "R-PLAT-03 R-OBS-01" ;;
+    # The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
+    # TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
+    platform/gateway/*)        echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
+    # CoreDNS is cluster DNS resolution (R-DNS-01).
+    platform/coredns/*)        echo "R-DNS-01" ;;
+    # crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
+    # its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
+    platform/crafty/*)         echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
+    # Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
+    platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
+    # runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
+    # resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
+    # is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
+    # runbook change is an observability change.
+    runbooks/*)                echo "R-OBS-04" ;;
+    # platform-config/ is the env-specific configuration Flux substitutes into and applies.
+    platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
+    # platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
+    # Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
+    # file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
+    # change. That is the most consequential single-line edit in the repo and it attributed nothing.
+    platform/kustomization.yaml) echo "R-GITOPS-01" ;;
+    # .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
+    # here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
+    .sops.yaml)                echo "R-SEC-01 R-GITOPS-01" ;;
    conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
    # infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
    # nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
@@ -0,0 +1,6 @@
+[1m[run.sh][0m Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
+[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
+Install the pinned binary or run via the in-cluster Job (incluster/).
@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="115.855688" />
+  </testsuite>
+</testsuites>
@@ -0,0 +1,232 @@
+[1m[run.sh][0m Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[1m[run.sh][0m LIGHT pass: no tests selected.
+[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 05:22:26 | [32;1mnode-loss[0;22m | [32;1m@chainsaw                              [0;22m | [32;1mCREATE[0;22m    | [32;1mOK[0;22m    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m       | [36;1mBEGIN[0;22m |
+    l.go:52: | 05:22:26 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m       | [36;1mRUN[0;22m   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m    | [36;1mLOG[0;22m   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1789795346371286043
+        victim node (currently hosts the attached volume): k3s-prod-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-prod-2 labeled
+        node/k3s-prod-2 cordoned
+        cordoned + labelled victim node: k3s-prod-2
+        pod "writer" deleted from chainsaw-advanced-boxer namespace
+        evicted writer off the victim node; volume detached from k3s-prod-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
+        recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m    | [32;1mDONE[0;22m  |
+    l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m       | [36;1mEND[0;22m   |
+    l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m   | [36;1mBEGIN[0;22m |
+    l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m       | [36;1mRUN[0;22m   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m    | [36;1mLOG[0;22m   |
+        === STDOUT
+        node/k3s-prod-2 uncordoned
+        node/k3s-prod-2 unlabeled
+        restored node to service: k3s-prod-2
+    l.go:52: | 05:23:37 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m    | [32;1mDONE[0;22m  |
+    l.go:52: | 05:23:37 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m       | [36;1mRUN[0;22m   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m    | [36;1mLOG[0;22m   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-advanced-boxer namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m    | [32;1mDONE[0;22m  |
+    l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m   | [36;1mEND[0;22m   |
+    l.go:52: | 05:23:38 | [36;1mnode-loss[0;22m | [36;1m@chainsaw                              [0;22m | [36;1mCLEANUP[0;22m   | [36;1mBEGIN[0;22m |
+    l.go:52: | 05:23:38 | [32;1mnode-loss[0;22m | [32;1m@chainsaw                              [0;22m | [32;1mDELETE[0;22m    | [32;1mOK[0;22m    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:24:21 | [36;1mnode-loss[0;22m | [36;1m@chainsaw                              [0;22m | [36;1mCLEANUP[0;22m   | [36;1mEND[0;22m   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (115.86s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
@@ -5,3 +5,4 @@
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}
@@ -256,7 +256,13 @@ SCRAPE_OWNERS: list[dict] = [
        "file": KPS,
        "require": [
            r"metricLabelsAllowlist:",
-            r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
+              # The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
+              # end in a literal `]`, asserting the list held exactly this one label — so adding a
+              # second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
+              # while the property it guards stayed perfectly satisfied. A guard that breaks when you
+              # correctly extend the thing it guards teaches people to edit the guard, which is how it
+              # stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
+              r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
        ],
        "forbid": [],
        "why": "KSM exposes NO custom labels by default. Without the allowlist entry "
@@ -179,6 +179,10 @@ UNTESTED_FILES = {
 # and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
 # that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
 # detectors are no longer unasserted.
+# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
+# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
+# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
+# had to become an assertion rather than a sentence in a comment.
 UNTESTED_ALERTS = {
    ("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
    ("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
@@ -195,7 +199,6 @@ UNTESTED_ALERTS = {
    ("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
    ("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
    ("platform/monitoring/alerts.yaml", "CertificateNotReady"),
-    ("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
    ("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
    ("platform/monitoring/alerts.yaml", "KeycloakDown"),
    ("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
@@ -478,7 +481,34 @@ for rel in sorted(alert_files):
          f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
          f"shrink-only: delete the line so the next missing test is caught")
    text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
-    covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
+    # "COVERED" MUST MEAN ASSERTED, NOT MENTIONED (tightened 2026-09-19).
+    # This used to be a bare word match over the whole test file, so an alert named anywhere —
+    # including in a PROSE COMMENT — counted as tested. Hit for real while adding
+    # platform/monitoring/tests/vpn-egress.test.yaml: its header comment explains that VPNTunnelDown
+    # stays quiet during an egress-steering failure, and that sentence alone was enough to make this
+    # check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Deleting
+    # the register entry would then have recorded coverage that did not exist — the exact "looks fixed"
+    # outcome this check exists to prevent, produced BY the check.
+    # Now an alert counts only when a test file names it in an `alertname:` position, with comment
+    # lines stripped first so a commented-out assertion does not count either.
+    # MEASURED BLAST RADIUS BEFORE TIGHTENING: of 61 alerts across the PrometheusRule manifests, 32 are
+    # asserted in one of the two idioms below, 29 are not mentioned at all (already declared in the
+    # registers), and ZERO are prose-only — so this change failed nothing on the day it landed. That
+    # mattered because the registers are shrink-only: had any alert been prose-only, tightening would
+    # have needed a register entry it is not allowed to add. (The first measurement said 31/30/0 because
+    # it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companions.)
+    text_no_comments = "\n".join(
+        l for l in text.split("\n") if not l.lstrip().startswith("#")
+    )
+    # BOTH promtool idioms count, because both are real assertions:
+    #   alert_rule_test:   - alertname: CNPGPrimariesCoLocated
+    #   promql_expr_test:  - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
+    # Matching only the first produced a FALSE NEGATIVE on CNPGPrimariesCoLocated, which
+    # platform/cnpg/alerts.promtool-test.yaml pins across FOUR scenarios entirely through the second
+    # form. Had that been believed, the "fix" would have been a redundant duplicate test for an alert
+    # already better covered than most.
+    covered = [n for n in names
+               if re.search(rf'(?:alertname:\s*|alertname="){re.escape(n)}\b', text_no_comments)]
    covered_total += len(covered)
    for n in names:
        if n in covered:
@@ -0,0 +1,82 @@
+# infra/gitea — platform settings that are NOT GitOps-managed
+
+Gitea's own repo settings (branch protection, required status checks, approvals) live
+in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
+and invisible to the gate** — exactly how the state below went unnoticed.
+
+## The gap this closes
+
+Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
+(2026-08-31) asserting the opposite:
+
+| Claimed | Measured |
+|---|---|
+| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
+| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
+
+So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
+rule; this directory is how the rule gets enforced by the machine.
+
+## `branch-protection.sh`
+
+Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
+do not trust the API response**, since Gitea may let site admins bypass protection.
+
+    ./infra/gitea/branch-protection.sh
+
+It authenticates with the credential `git` already has stored for this host, and never
+prints the secret.
+
+## `runner-token-rotate.sh`
+
+Mints a fresh instance-level registration token from the Gitea API and writes it into
+`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
+never printed.**
+
+    ./infra/gitea/runner-token-rotate.sh
+
+A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
+and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
+longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
+and polls with an identity Gitea does not recognise — so it never appears in the runner
+list. That is precisely the observed symptom: zero runners, pod running, no error anyone
+looks at. Clearing the PVC is what forces a real re-registration; the script prints the
+commands.
+
+## Sequence — order matters
+
+1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
+   the runner actually re-registers, then confirm it shows `online` in the API. Do not
+   skip the confirmation: the whole reason this went unnoticed is that a runner was
+   *recorded* as online without anyone checking.
+2. **Watch the gate actually report on a real PR**, and note the context name it produced.
+3. **Only then set `enable_status_check: true`** with that exact context.
+4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
+
+> **Do not do step 3 before step 1.** A required status check with no runner leaves every
+> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
+> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
+> reported.
+
+### Measured, once the runner came up (2026-09-29)
+
+The first CI jobs this repo ever ran, on PR #1 and PR #2:
+
+| Context | Result | Time |
+|---|---|---|
+| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
+| `gate / gate (push)` | success | 1m32s (PR #2 only) |
+| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
+
+**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
+but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
+does not hang a status check: Gitea reports it as **`success`** with the description
+"Has been skipped". So requiring it would not deadlock anything — it would pass
+*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
+a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
+exactly the fail-open shape this repo keeps finding.
+
+The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
+(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
+is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
+`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# Enable branch protection on main for home-network-k3s-impl.
+#
+# Sets enable_push:false  -> no direct pushes to main; changes land only by merging a PR.
+#
+# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner
+# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the
+# gate context would deadlock EVERY PR permanently -- including the PR that fixes it.
+# Turn it on only after a runner is live and you have seen the gate actually report on
+# a real PR, using the context name CI itself produced.
+#
+# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT
+# a repo collaborator, so requiring an approval now could leave nobody able to approve.
+# Add the collaborator first, then raise this to 1.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev
+OWNER=Home-Network
+REPO=home-network-k3s-impl
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"
+P="$(sed -n 's/^password=//p' <<<"$creds")"
+[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
+echo "authenticating as '${U}' (secret not shown)"
+
+echo "==> creating branch protection on 'main'"
+jq -n '{
+  branch_name: "main",
+  enable_push: false,
+  enable_status_check: false,
+  status_check_contexts: [],
+  required_approvals: 0,
+  block_on_rejected_reviews: true,
+  block_on_outdated_branch: false,
+  dismiss_stale_approvals: true
+}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
+  -X POST -H 'Content-Type: application/json' --data @- \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \
+| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end'
+
+echo "==> verifying (this is the part that matters)"
+curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
+| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}'
+
+cat <<'NOTE'
+
+Now prove it actually blocks you -- do not take the API's word for it. From a clean
+checkout on main, with nothing to lose:
+
+    git commit --allow-empty -m 'protection probe'
+    git push origin main          # EXPECT: rejected by the remote
+    git reset --hard HEAD~1       # tidy up either way
+
+If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not
+yet doing what you want.
+NOTE
@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+# Mint a fresh Gitea Actions registration token and write it into the SOPS secret.
+#
+# The token value is NEVER printed. It goes from the Gitea API into `sops set` and
+# nowhere else. (It is briefly visible in this host's process list as a sops argument;
+# acceptable on a single-operator workstation, and far better than echoing it.)
+#
+# This ALONE may not bring the runner back -- see "IMPORTANT" at the end of the output.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev
+SECRET=apps/wave-1/gitea-actions-runner-secret.sops.yaml
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+cd "$REPO_ROOT"
+[ -f "$SECRET" ] || { echo "not in the repo root, or $SECRET is missing" >&2; exit 1; }
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"
+P="$(sed -n 's/^password=//p' <<<"$creds")"
+[ -n "$U" ] && [ -n "$P" ] || { echo "no stored git credential for $HOST" >&2; exit 1; }
+echo "==> minting an instance-level registration token as '${U}'"
+
+TOKEN="$(curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
+  "https://${HOST}/api/v1/admin/runners/registration-token" | jq -r '.token // empty')"
+[ -n "$TOKEN" ] || { echo "Gitea did not return a token" >&2; exit 1; }
+echo "==> got a ${#TOKEN}-character token (value not shown)"
+
+echo "==> writing it into ${SECRET} via sops"
+DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run \
+  sops set "$SECRET" '["stringData"]["token"]' "\"${TOKEN}\""
+unset TOKEN
+
+echo "==> done. Verify the file is still encrypted (no plaintext token):"
+grep -q 'ENC\[' "$SECRET" && echo "    OK - stringData is still SOPS-encrypted" \
+  || { echo "    REFUSING TO CONTINUE: $SECRET is not encrypted" >&2; exit 1; }
+
+cat <<'NOTE'
+
+Next: commit this on a branch, open a PR, and merge it so Flux applies the new Secret.
+
+IMPORTANT -- a new token is usually NOT sufficient on its own.
+act_runner registers exactly ONCE and then persists its identity in /data/.runner on the
+act-runner-data PVC. If Gitea no longer has a matching runner row (DB restore, the row was
+deleted, the instance was rebuilt), the pod starts, sees .runner, SKIPS registration, and
+polls with an identity Gitea does not recognise -- so it never appears in the runner list,
+which is exactly the observed symptom (zero runners, pod running).
+
+So after the Secret lands, force a genuine re-registration by clearing that state:
+
+    kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=0
+    kubectl --context k3s-prod -n gitea-actions delete pvc act-runner-data
+    kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=1
+
+(The PVC holds only runner registration state -- no build artifacts, nothing durable.
+Deleting it is the supported way to re-register. Flux recreates it from the manifest.)
+
+Then confirm it actually registered -- do not assume:
+
+    curl -s -u '<user>' https://gitea.home.keithjackson.dev/api/v1/admin/actions/runners | jq
+
+A runner must appear with status "online" and the labels ubuntu-latest / homelab.
+NOTE
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
+# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
+# stopwatch so you can see the budget being spent.
+#
+#   branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
+#
+# It never sees, prints, or handles the token: the file is already encrypted by sops.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev
+OWNER=Home-Network
+REPO=home-network-k3s-impl
+SECRET=apps/wave-0/plex-claim-secret.sops.yaml
+T0=$(date +%s)
+el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
+
+cd "$(git rev-parse --show-toplevel)"
+
+# --- preflight: refuse to ship a file that is not what we think it is -------------
+git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
+  echo "  infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
+grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
+[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+# feat/ prefix ON PURPOSE. act_runner capacity is 1, and gate.yml's PUSH trigger covers
+# feature/** fix/** docs/** chore/** but NOT feat/** — so a chore/ branch would fire a
+# second, redundant run that SERIALISES ahead of the PR run and doubles the wall clock.
+BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
+
+el "committing on $BRANCH"
+git checkout -q -b "$BRANCH"
+git add "$SECRET"
+git commit -q -m 'chore(plex): fresh plex.tv claim token
+
+Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
+SHA=$(git rev-parse HEAD)
+
+el "pushing"
+git push -q -u origin "$BRANCH"
+
+el "opening PR"
+PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
+       '{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
+     | api -X POST -H 'Content-Type: application/json' --data @- \
+         "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
+[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
+echo "  PR #$PR"
+
+el "waiting for the gate (expect ~1m40s)"
+for i in $(seq 1 60); do
+  st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
+  printf '\r  gate=%s  [T+%ss]   ' "$st" "$(( $(date +%s)-T0 ))"
+  case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
+  sleep 5
+done
+[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
+
+el "merging PR #$PR"
+api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
+git checkout -q main && git pull -q --ff-only origin main
+
+el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
+./infra/devcontainer/dev.sh run bash -c \
+  'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
+
+# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
+# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
+# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
+el "rolling the pod (mandatory — a changed Secret restarts nothing)"
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
+   kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
+
+el "verdict"
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
+cat <<'NOTE'
+
+  "Attempting to obtain server token..." + "Token obtained successfully"  -> CLAIMED
+  "Attempting..." with no success line                                    -> token expired; retry, nothing is broken
+  neither line                                                            -> already claimed, or env did not reach the container
+
+Confirm it persisted (prints the attribute NAME only, never its value):
+
+  infra/devcontainer/dev.sh run bash -c \
+    'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
+       "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
+NOTE
@@ -904,6 +904,132 @@ spec:
            summary: "Egress VPN tunnel down"
            description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
+        # -----------------------------------------------------------------------------------------
+        # THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
+        #
+        # `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
+        # having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
+        # exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
+        # re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
+        # at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
+        # 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
+        # carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
+        # exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
+        #
+        # The datapath now self-heals: `roles/vpn_egress_gateway` publishes
+        # `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
+        # OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
+        # and that node write is what makes Cilium re-resolve. These two rules watch whether it
+        # ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
+        #
+        # WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
+        # `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
+        # observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
+        #   * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
+        #     `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
+        #     there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
+        #     metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
+        #     `cilium_*` and `hubble_*` query returns an empty vector today (which also means
+        #     `HubbleDeniedFlowSpike` above cannot fire yet).
+        #   * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
+        #     CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
+        #     map, not an exporter. No metric name is invented here to paper over that.
+        # So the observable is the label: the input to the re-steer, compared against the live tun0
+        # identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
+        # ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
+        # `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
+        #
+        # WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
+        # firing, the label write failing or never being deployed, the convergence timer being dead —
+        # i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
+        # "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
+        # until either the re-steer script exports the map's ifindex through a node-exporter textfile
+        # collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
+        # A probe was considered and rejected for now: there is no blackbox-exporter and no probe
+        # CronJob in this repo, so it means a NEW always-on component whose own failures become a new
+        # alerting surface — worth doing, but not as the thing that closes a hole today.
+        #
+        # WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
+        # VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
+        # label-to-number conversion, so the level-triggered equality test is not expressible. What IS
+        # expressible is the same invariant counted over a window: the published label must change at
+        # least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
+        # is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
+        # that held still, 2 for one that changed value inside the window); the `max by (<label>)`
+        # wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
+        # restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
+        # which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
+        #
+        # Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
+        # +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
+        # trade against the repo's alert-fatigue history (an unbounded critical on the one channel
+        # that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
+        # email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
+        - alert: VPNEgressSteeringStale
+          # FIRES when the live tun0 was re-created more times in the last hour than the gateway
+          # node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
+          # tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
+          # and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
+          # seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
+          # cold boot), so `for: 15m` cannot fire on a healthy reconnect.
+          expr: |
+            (
+                changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+              >
+                (
+                  count(
+                    count_over_time(
+                      max by (label_homelab_keithjackson_dev_egress_uplink) (
+                        kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
+                      )[1h:1m]
+                    )
+                  ) - 1
+                )
+            )
+            and on()
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "VPN egress steering is stale — vpn-class pods have no egress"
+            description: >-
+              The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
+              Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
+              being dropped, silently, while tun0 still looks healthy. Compare
+              `kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
+              `cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
+              (or setting any node label) forces a fresh resolution.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        - alert: VPNEgressUplinkLabelMissing
+          # The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
+          # in kube-state-metrics, and if that label is missing its comparison degrades to an empty
+          # vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
+          # node with no published uplink means the re-steer mechanism is not in place at all
+          # (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
+          # kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
+          # the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
+          # rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
+          # that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
+          # on the way in, and so an Ansible run in progress does not either.
+          expr: |
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+            and on()
+            absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Egress gateway node publishes no egress-uplink label"
+            description: >-
+              A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
+              `homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
+              nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
+              `VPNEgressSteeringStale` is inert. Check the label on the node, then the
+              `vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
+              metricLabelsAllowlist.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        # -----------------------------------------------------------------------------------------
        # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
        # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
        # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
@@ -231,7 +231,7 @@ spec:
      # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
      # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
      metricLabelsAllowlist:
-        - "nodes=[homelab.keithjackson.dev/egress-gateway]"
+        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
      # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
      # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
      # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
@@ -0,0 +1,188 @@
+# promtool rule-unit tests for VPNEgressSteeringStale + VPNEgressUplinkLabelMissing
+# (platform/monitoring/alerts.yaml)
+#
+# Run with: platform/monitoring/tests/run.sh
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
+# `egress-class=vpn` pods leave through one gateway node's tun0. Cilium resolves that interface to an
+# IFINDEX and caches it; an OpenVPN reconnect destroys and re-creates tun0 with a NEW ifindex, and the
+# gateway node's up-script publishes `homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex>` so that
+# node write forces Cilium to re-resolve. If the label does not follow the tunnel, the cache points at a
+# destroyed interface and every vpn-class pod is dropped — while tun0 itself stays up, so VPNTunnelDown
+# says nothing. That shape ran ~80 minutes undetected on the dev gateway.
+#
+# VPNEgressSteeringStale detects it by comparing HOW MANY TIMES tun0's ifindex changed in the last hour
+# against HOW MANY DISTINCT `egress-uplink` label values appeared in the same hour. The subtlety worth
+# testing is that a label VALUE change is not a sample change: it RETIRES one series and STARTS another,
+# so "the label followed the tunnel" shows up as two overlapping series in the window, not as a
+# changing value. Tests 1 and 2 are the same tunnel reconnect with and without that second series, so
+# they isolate exactly that mechanism — anything that broke the series-counting half would make test 2
+# fire, and anything that broke the tun0 half would make test 1 quiet.
+#
+# VPNEgressUplinkLabelMissing is the WATCHDOG for the rule above: with no uplink label anywhere, the
+# staleness comparison has nothing to compare and is structurally INERT — present, evaluating, and
+# incapable of firing. Tests 3 and 4 pin that, because an inert critical alert is the failure mode this
+# whole group exists to prevent.
+#
+# Fixture values are prod's real shape: gateway k3s-prod-3, tun0 ifindex 139 with address
+# 10.150.192.86 (live label value `10.150.192.86-139`), reconnecting to ifindex 140.
+
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. THE BUG — tun0 was re-created, the label did NOT follow. Steering is stale: FIRES.
+  # =============================================================================================
+  - interval: 1m
+    name: tunnel reconnect with no label update - VPNEgressSteeringStale fires
+    input_series:
+      # tun0 re-created at t=41m: ifindex 139 -> 140. One change inside the trailing hour.
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '139+0x40 140+0x40'
+      # tun0 is UP and passing traffic the whole time — this is what makes the failure deceptive, and
+      # it is asserted below: VPNTunnelDown stays QUIET while egress is actually broken.
+      - series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '0+120000x80'
+      # The node label NEVER changes, so only ONE distinct uplink value exists in the window.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
+        values: '1+0x80'
+    alert_rule_test:
+      # 1 tunnel change > (1 distinct label value - 1) = 0, held for 15m.
+      - eval_time: 60m
+        alertname: VPNEgressSteeringStale
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+            exp_annotations:
+              summary: "VPN egress steering is stale — vpn-class pods have no egress"
+              description: >-
+                The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
+                Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
+                being dropped, silently, while tun0 still looks healthy. Compare `kubectl get nodes -L
+                homelab.keithjackson.dev/egress-uplink` against `cilium-dbg bpf egress list` on the
+                gateway node; re-applying the CiliumEgressGatewayPolicy (or setting any node label)
+                forces a fresh resolution.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+      # Not yet: at 45m the condition has been true only ~4m, under the 15m `for`.
+      - eval_time: 45m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      # No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
+      # no VPN gateway must not be told its tunnel is down forever.
+      - eval_time: 45m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+      # The watchdog must stay quiet — the label IS present here.
+      - eval_time: 60m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      # THE HEADLINE CLAIM, asserted rather than asserted-in-prose: the tunnel alert says NOTHING
+      # while vpn-class egress is completely broken. If VPNTunnelDown covered this failure, the
+      # staleness rule would not need to exist.
+      - eval_time: 60m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. THE HEALTHY RECONNECT — same tunnel change, but the label followed it. STAYS QUIET.
+  #    This is the half that proves the rule is not simply "any reconnect alerts".
+  # =============================================================================================
+  - interval: 1m
+    name: tunnel reconnect WITH the label following - stays quiet
+    input_series:
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '139+0x40 140+0x40'
+      - series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '0+120000x80'
+      # A label VALUE change retires one series and starts another. Both are inside the trailing
+      # hour, so the window holds TWO distinct uplink values: 2 - 1 = 1, which is NOT < 1 change.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
+        values: '1+0x40 _x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-140"}'
+        values: '_x41 1+0x39'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      - eval_time: 75m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+
+  # =============================================================================================
+  # 3. THE INERT-ALERT CASE — a gateway node publishing NO uplink label at all.
+  #    VPNEgressSteeringStale cannot fire in this state, which is exactly why this warning exists.
+  # =============================================================================================
+  - interval: 1m
+    name: gateway node with no egress-uplink label - VPNEgressUplinkLabelMissing fires
+    input_series:
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true"}'
+        values: '1+0x60'
+      # tun0 is reconnecting the whole time. The staleness rule STILL cannot fire, because it has no
+      # label series to compare against — that is the inertness being alerted on.
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '139+0x30 140+0x30'
+      - series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '0+120000x60'
+    alert_rule_test:
+      - eval_time: 35m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+            exp_annotations:
+              summary: "Egress gateway node publishes no egress-uplink label"
+              description: >-
+                A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
+                `homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without
+                it nothing can tell whether Cilium's egress steering followed the last tunnel reconnect,
+                and `VPNEgressSteeringStale` is inert. Check the label on the node, then the
+                `vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
+                metricLabelsAllowlist.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+      # Under the 30m `for`.
+      - eval_time: 20m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      # And the proof of inertness: the critical rule is silent despite a real reconnect.
+      - eval_time: 55m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      - eval_time: 55m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. NO GATEWAY NODE AT ALL — neither rule may fire. Both are guarded by an `and on()` on the
+  #    egress-gateway label so they stay silent on a cluster with no VPN gateway, rather than
+  #    alerting forever about an absent feature.
+  # =============================================================================================
+  - interval: 1m
+    name: no egress-gateway node - both rules silent
+    input_series:
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
+        values: '1+0x60'
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.10:9100",device="eth0"}'
+        values: '2+0x60'
+    alert_rule_test:
+      - eval_time: 45m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      - eval_time: 45m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      # No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
+      # no VPN gateway must not be told its tunnel is down forever.
+      - eval_time: 45m
+        alertname: VPNTunnelDown
+        exp_alerts: []
@@ -157,6 +157,31 @@ kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
 If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
 gateway node is the next stop.

+### And now something watches it (2026-09-12)
+
+The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
+silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
+
+- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
+  more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
+  about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
+  steering pointed at a destroyed ifindex.
+- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
+  `egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
+  in which the rule above would be inert. The watchdog for the watchdog.
+
+Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
+so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
+`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
+the label to exist as a metric at all — the two changes must ship together.
+
+**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
+exposes no metric carrying the egress map's contents (the map is readable only through
+`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
+place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
+map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
+probe from a vpn-class pod. Neither exists yet.
+
 ## Resilience — there is NO gateway failover (spec correction, escalated)

 **ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form

    runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"

-46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
+46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
 live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
 help. These pages are what makes the link resolve.

@@ -33,7 +33,8 @@ negative is how a CLI quirk becomes real data loss. Write `backups.velero.io`,
 `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`,
 `volumes.longhorn.io`, `nodes.longhorn.io`.

-`git push origin main` **is** the deploy; there is no other apply step. Durable changes go through git.
+Merging a PR into `main` **is** the deploy; there is no other apply step. `main` is protected, so a
+direct push is rejected — durable changes go through a branch, a PR, and a green gate.

 ## Alerts you should not trust the silence of (measured 2026-09-13)

@@ -84,8 +85,10 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`.

 ### Networking
- **`vpn-down`** — the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert
-  cannot see, and what the kill-switch actually does.
+- **`vpn-down`** — the egress VPN tunnel. Explains what the kill-switch actually does, and the silent
+  egress-gateway staleness this alert cannot see — which now has its own alerts and page below.
+- **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached
+  ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
 - **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.

 ### Backups and DR
@@ -106,6 +109,13 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
  every green signal missed.

+### Operator procedures (no alert links here)
+- **`plex-claim`** — claiming the Plex Media Server. **Claimed 2026-09-29**; the page now only
+  applies after a `/config` wipe. The claim token is short-lived — read the countdown on
+  plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
+  token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
+  Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
 - **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.
@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -78,8 +78,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -55,8 +55,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -63,8 +63,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -69,8 +69,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -71,8 +71,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -68,8 +68,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -7,7 +7,7 @@
 ## What this means here

 Something in the GitOps chain is stuck at Ready=False. On this cluster that is more consequential than
-it sounds, because **`git push origin main` IS the deploy** — there is no other apply step. A stuck
+it sounds, because **merging a PR into `main` IS the deploy** — there is no other apply step. A stuck
 Kustomization means git and the cluster have diverged and nothing will close the gap.

 The chain has **14 tiers**: `flux-system` → `cilium` → `platform` → `platform-config` →
@@ -61,8 +61,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -87,8 +87,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -67,8 +67,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -59,8 +59,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -57,8 +57,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -53,8 +53,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -62,8 +62,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -81,7 +81,13 @@ configMapGenerator:
      - node-io=node-io.md
      - node-memory=node-memory.md
      - node-memory-swap=node-memory-swap.md
+      # Operator procedure, not an alert page (no alert links to it — the
+      # alert-runbooks-resolvable check treats that as advisory). It is served
+      # because the plex.tv claim window is short and is not a thing to go looking
+      # for in git: /runbooks/plex-claim has to resolve the moment it is needed.
+      - plex-claim=plex-claim.md
      - pvc-capacity=pvc-capacity.md
      - targets-down=targets-down.md
      - update-rollback=update-rollback.md
      - vpn-down=vpn-down.md
+      - vpn-egress-stale=vpn-egress-stale.md
@@ -83,8 +83,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -74,8 +74,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -75,8 +75,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -49,8 +49,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -56,8 +56,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -227,8 +227,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -52,8 +52,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -0,0 +1,189 @@
+# plex-claim — claiming the Plex Media Server inside the token window
+
+**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
+it is the page you need at the moment you need it, with a stopwatch running.
+**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml`
+(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden).
+
+## What this means here
+
+Plex will not let you **create libraries** over the mounted bulk media share while the server is
+**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the
+server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable
+`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on
+the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
+`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
+
+**A `https://plex.tv/claim` token is short-lived — minutes, not hours.** Read the countdown the
+claim page shows you; do not budget against a number written here. That is shorter than PR → gate →
+merge → Flux's 10-minute reconcile interval → pod roll if you let those run on their own timers,
+which is why this page exists: the loop is scripted and the reconcile is **forced** rather than
+waited for. Measured end to end on 2026-09-29: **~2m45s**.
+
+**An expired or wrong token does not break anything.** Verified against the image's own first-run
+script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
+body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
+no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
+serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
+ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no
+`optional`, so the Secret must always exist even when its value is useless).
+
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that
+hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and that default has already changed once (k3s-dev until dev
+was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once.
+
+Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator
+work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change
+is still a git change.
+
+## First commands
+
+**Before spending a token, find out whether the server is already claimed.** This prints only the
+attribute NAME, never its value — do not widen it to print the token:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
+     "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
+```
+
+`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn.
+Empty output ⇒ unclaimed, continue.
+
+Is the workload healthy enough to claim at all, and is the wave reconciling?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
+```
+
+## The procedure
+
+> **STATUS: this server was CLAIMED on 2026-09-29** and `PlexOnlineToken` is present in
+> `/config` on the `plex-config` PVC. The upstream exchange is gated on that pref being
+> *absent*, so `PLEX_CLAIM` is now a **no-op on every start** whatever it holds. You only need
+> this page again after a `/config` wipe or a rebuild onto an empty PVC.
+
+**On the clock.** The token is short-lived — **read the countdown that plex.tv/claim shows you**
+rather than trusting a number written here. (Earlier revisions of this page asserted a "~4-minute
+TTL" in four places with no citation; upstream `pms-docker` documents no expiry at all. The page
+is the only authority, and it is on screen at T+0.)
+
+**A burnt token costs nothing.** An expired token gets a 403 with an empty body, `curl`'s status is
+never checked, the `if` does not fire, there is no `else` and no `exit 1`, and the script (no
+`set -e`) exits 0. The pod comes up unclaimed and serving. So retry freely.
+
+### One command
+
+1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
+   <https://plex.tv/claim>, copy the `claim-…` string.
+
+2. **Put it in the Secret** — in place, encrypted, never touching a plaintext file:
+
+   ```
+   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+   ```
+
+3. **Ship it.** This drives the whole delivery loop and prints a running stopwatch:
+
+   ```
+   ./infra/plex/claim-ship.sh
+   ```
+
+   branch → commit → push → PR → gate → merge → `flux reconcile --with-source` → `rollout restart`
+   → verdict. It refuses to run if the file is unchanged or not encrypted.
+
+**Measured end to end, 2026-09-29: ~2m45s.** Gate ~1m40s, merge + reconcile ~25s, pod roll and
+container start ~30s.
+
+### Why it is a script and not six pasted commands
+
+`main` is a **protected branch** — the direct push this page used to instruct is now rejected — so
+the deploy is a PR merged after a green gate. Three of those steps are easy to get wrong under time
+pressure, and two are silent when skipped:
+
+- **`--with-source` is load-bearing.** Without it the Kustomization applies the previously-fetched
+  revision, not your commit.
+- **The pod roll is mandatory.** Only the Secret's ciphertext changes, so the Deployment's pod
+  template is byte-identical and *nothing restarts on its own*. `PLEX_CLAIM` is read once, at
+  container start. Skip this and the new Secret sits in the cluster while the running pod keeps the
+  old value forever.
+- **The branch is named `feat/…` on purpose.** `act_runner` capacity is 1, and `gate.yml`'s push
+  trigger covers `feature/** fix/** docs/** chore/**` but *not* `feat/**`. A `chore/` branch fires a
+  second redundant run that serialises ahead of the PR run and roughly doubles the wall clock.
+
+### Reading the verdict
+
+* `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
+  ⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
+  create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
+* `Attempting…` with **no** success line ⇒ the token was expired or invalid. Nothing is broken; go
+  back to step 1 with a new one.
+* **Neither line** ⇒ the exchange was skipped: `/config` already had a `PlexOnlineToken` (already
+  claimed — the expected case now), or the env did not reach the container. Check by NAME only:
+
+  ```
+  infra/devcontainer/dev.sh run bash -c \
+    'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+       -o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
+  ```
+
+## Likely causes, most likely first
+
+When the claim does not take, in order of how often it is the answer here:
+
+1. **The token expired** — you exceeded the countdown, usually by reviewing the PR instead of
+   merging it the moment the gate went green. Symptom: the "Attempting…" line with no success line.
+   Remedy: a new token, and let `claim-ship.sh` drive the loop rather than pasting steps.
+2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
+   whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
+   container still holds the placeholder.
+3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4
+   finished, so the new pod read the OLD Secret. Check
+   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'`
+   against your commit, then roll again.
+4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a
+   server claimed earlier ignores every later token silently. This is the benign case.
+5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled
+   connection can leave the container sitting in cont-init with **nothing listening on 32400** — the
+   one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after
+   "Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly.
+6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at
+   step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".
+
+## When to escalate or stop
+
+- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt
+  is how people start editing live Secrets by hand, which leaves the cluster holding a value that is
+  in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from.
+- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never
+  `kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is
+  reverted (or adopted with the wrong value) on the next reconcile.
+- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the
+  `PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential.
+- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means
+  removing the server from that account on plex.tv and clearing `PlexOnlineToken` from
+  `Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git
+  change.
+- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata
+  and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a
+  two-line pref, not a reason to lose the DB.
+- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is
+  skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no
+  `optional` — deleting it would put the pod into `CreateContainerConfigError`.
+
+---
+Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
+Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
+`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
+the running `plexinc/pms-docker:latest` image.
@@ -58,8 +58,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -60,8 +60,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -54,8 +54,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -66,8 +66,10 @@ a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09
 `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

 Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
-agents here: every durable change goes through git (`git push origin main` **is** the deploy).
-Where this runbook gives a mutating command it is for an operator, and it says so.
+agents here: every durable change goes through git. **`main` is a protected branch — a direct push
+is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
+green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
+and it says so.

 ## First commands

@@ -0,0 +1,123 @@
+# vpn-egress-stale — VPN egress steering did not follow the tunnel
+
+**Alerts:** `VPNEgressSteeringStale` (critical, 15m) and `VPNEgressUplinkLabelMissing` (warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`
+**Related:** `/runbooks/vpn-down` is for the tunnel being DOWN. This page is for the tunnel being **up**
+while the pods that must use it have no egress at all.
+
+## What this means here
+
+`egress-class=vpn` pods leave the cluster through one gateway node's `tun0`, selected by the
+`vpn-egress-steering` CiliumEgressGatewayPolicy. Cilium resolves that interface to an **ifindex** and
+caches it. When OpenVPN reconnects it destroys and re-creates `tun0`, which gets a **new ifindex** — and
+Cilium does not re-resolve on its own. The gateway node publishes
+`homelab.keithjackson.dev/egress-uplink=<tun0 address>-<ifindex>` from the OpenVPN up-script on every
+connect, and that node write is what makes Cilium re-resolve.
+
+If the label write does not happen, the cache keeps pointing at a destroyed interface. **`tun0` looks
+perfectly healthy the whole time** — it is up, it has an address, `VPNTunnelDown` stays quiet — while
+every vpn-class pod is silently dropped. That exact shape ran for roughly **80 minutes** undetected on
+the dev gateway, which is why these two alerts exist.
+
+`VPNEgressSteeringStale` compares how many times `tun0`'s ifindex changed in the last hour against how
+many distinct `egress-uplink` label values appeared in the same hour. More tunnel changes than label
+changes means a reconnect happened that the label did not follow.
+
+**`VPNEgressUplinkLabelMissing` is the watchdog for that comparison.** If the label is absent entirely,
+the staleness rule has nothing to compare against and is **inert** — present, evaluating, and incapable
+of firing. So the warning matters more than its severity suggests: it means your critical alert is
+switched off.
+
+Today's gateway is **`k3s-prod-3`** (`homelab.keithjackson.dev/egress-gateway=true`), and the policy has
+been in place since early September. It is a **singleton** — there is no second gateway to fail over to.
+
+## First commands
+
+Run cluster commands through the pinned toolbox, which supplies the kube context:
+
+```
+./infra/devcontainer/dev.sh run kubectl --context k3s-prod get nodes \
+  -L homelab.keithjackson.dev/egress-gateway -L homelab.keithjackson.dev/egress-uplink
+```
+
+You want exactly one node carrying `egress-gateway=true`, and that same node carrying a non-empty
+`egress-uplink` of the form `<address>-<ifindex>`.
+
+```
+# 2. what Cilium currently believes, on the gateway node itself
+ssh k3s-prod-3 'sudo cilium-dbg bpf egress list'
+ssh k3s-prod-3 'ip -o link show tun0'          # the REAL ifindex, first field
+```
+
+Compare the ifindex in the label and in `bpf egress list` against `ip link`. **A mismatch is the whole
+bug.**
+
+```
+# 3. is the re-steer mechanism itself alive?
+ssh k3s-prod-3 'systemctl status vpn-egress-resteer.service vpn-egress-resteer.timer --no-pager'
+ssh k3s-prod-3 'journalctl -u vpn-egress-resteer --since -2h --no-pager | tail -30'
+```
+
+```
+# 4. are vpn-class pods actually broken, or only the metric?
+./infra/devcontainer/dev.sh run kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io \
+  vpn-egress-steering -o yaml | head -40
+```
+
+Prove egress from a pod the policy selects rather than inferring it — the observable symptom is
+"outbound traffic silently disappears", so test outbound traffic:
+
+```
+./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n qbittorrent exec deploy/qbittorrent -- \
+  sh -c 'curl -s --max-time 10 https://ifconfig.me || echo EGRESS-FAILED'
+```
+
+That should return the **VPN provider's** address. The gateway VIP or a timeout both mean egress is
+broken; your own ISP address means the kill-switch is not holding and traffic is leaking, which is worse
+than a drop — see `/runbooks/vpn-down`.
+
+## Likely causes, most likely first
+
+1. **A tunnel reconnect that the label did not follow.** The common case, and what the alert is built to
+   catch. The OpenVPN up-script either did not run or could not write the node label (RBAC, expired
+   kubeconfig on the node, API unreachable at that moment). Check `journalctl -u vpn-egress-resteer` and
+   the up-script's own log around the reconnect time.
+2. **Cilium cached a dead ifindex despite the label changing.** The label write happened but the
+   re-resolution did not take. Forcing any node-label write makes Cilium re-resolve — that is the same
+   lever the up-script pulls, so it is a safe nudge, not a workaround:
+
+       ./infra/devcontainer/dev.sh run kubectl --context k3s-prod annotate node k3s-prod-3 \
+         egress-resteer-nudge="$(date -u +%s)" --overwrite
+
+   Re-applying the CiliumEgressGatewayPolicy has the same effect. Re-check `bpf egress list` after.
+3. **`metricLabelsAllowlist` lost the label** (this fires `VPNEgressUplinkLabelMissing`, not the stale
+   alert). kube-state-metrics exposes **no** custom node labels by default; the allowlist entry in
+   `platform/monitoring/kube-prometheus-stack.yaml` is what publishes them. If that entry is edited or
+   dropped, `kube_node_labels` carries no `label_homelab_keithjackson_dev_egress_uplink` and the critical
+   alert goes quiet without anything looking broken. Confirm what is actually exported:
+
+       ./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
+         prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
+         /bin/promtool query instant http://localhost:9090 'kube_node_labels'
+
+   Note that KSM does **not** restart on a ConfigMap-only change; the deployment carries no checksum
+   annotation over it, so a `rollout restart -n monitoring deploy/kube-prometheus-stack-kube-state-metrics`
+   is required after editing the allowlist.
+4. **The gateway node was replaced or relabelled.** If `egress-gateway=true` moved, the up-script must be
+   installed on the new node (`infra/ansible/roles/vpn_egress_gateway`). A gateway label with no
+   up-script behind it produces exactly this alert on the next reconnect.
+
+## When to escalate or stop
+
+- **Stop and do not "fix" it by deleting the policy.** Removing `vpn-egress-steering` does not restore
+  egress; it sends vpn-class traffic out of your **own** ISP connection, which is the leak the class
+  exists to prevent. A broken tunnel that drops traffic is the intended failure mode.
+- **Escalate to a maintenance window** if the ifindex mismatch returns after every reconnect. That is the
+  up-script or its RBAC, not a transient, and chasing it live means repeatedly breaking egress for the
+  vpn-class workloads.
+- **This is a singleton gateway.** There is no failover. If `k3s-prod-3` needs to be drained while this is
+  unresolved, accept that vpn-class egress is down for the duration rather than improvising a second
+  gateway under time pressure.
+- If `VPNEgressUplinkLabelMissing` is firing and you cannot restore the label quickly, say so explicitly
+  in the incident notes: for as long as it is missing, **`VPNEgressSteeringStale` is not watching
+  anything**, so the absence of that critical alert proves nothing.