fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing

A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself: the announcer does not answer its own ARP request, so on the lease holder the neighbour entry settles at FAILED and packets to that VIP die before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced. platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and MASQUERADEs remote clients to the node's own address, so a VPN client takes the node's LAN path and inherits its ARP failures. The router's udp/51820 forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16 Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering (its VIP has two Services and k3s-prod-2 held the other lease). The tunnel handshook, names resolved, every app hung, and nothing reported it. on k3s-prod-1: ip neigh show 192.168.2.20 -> FAILED (its own VIP) on prod-2/3: ip neigh show 192.168.2.20 -> lladdr ... (prod-1), 443 OK Mark the node and exclude it from announcing: * node_baseline labels the forwarded node (node_baseline_vpn_ingress_node, k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes the label from any node that no longer matches so moving it MOVES the label. * l2-announcement-policy.yaml excludes that label, so the node ARP-resolves every VIP. Fixes all VIPs for VPN clients, not just the one that collided. Cost: announcement candidacy drops from 3 nodes to 2. * VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels (label added to the kube-state-metrics allowlist), with VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches, so a missing label would silently re-arm the whole failure AND leave the critical alert inert. Four promtool cases, including that inertness. A node LABEL rather than a cluster-vars knob because the alert must read it too, and extract_rules.py rightly refuses a ${...} placeholder in an alert expr -- the rule unit-tested would not be the rule deployed. One source of truth, and it touches no sensitive path. Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and "failover is this one line". Both false; the first caused this. UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the handoff entry's third section. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-17 21:58:12 -05:00
19 changed files with 900 additions and 641 deletions
@@ -1,106 +0,0 @@
---
-date: 2026-10-17
-branch: feat/graceful-cluster-shutdown
-title: "Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC"
---
-
-## What changed
-
-All three nodes had to be powered off to move them onto a UPS. There was **no procedure for it**:
-`infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster, so
-the whole sequence was improvised live. Two defects came out of doing it.
-
-**1. Kubelet graceful node shutdown was unconfigured on every prod node.** No `shutdownGracePeriod`
-anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and every pod
-was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25
-attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into
-`/etc/rancher/k3s/config.yaml` with `shutdown-grace-period=90s` and
-`shutdown-grace-period-critical-pods=20s`.
-
-The half that is easy to miss and silently defeats the other half: **graceful node shutdown is
-enforced by logind, not by the kubelet.** The kubelet holds an inhibitor *delay* lock and systemd
-waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace
-period above that cap is truncated with no error, so the feature looks configured and still
-hard-kills pods. The role now ships a `logind.conf.d` drop-in setting it to 120, and **asserts**
-`k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period` rather than documenting the
-relationship, because the failure is invisible.
-
-**2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume.** `monitoring/loki` carries
-`persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's
-`scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so
-that PVC was the only copy of its log data. This is the defect in this change set that actually
-destroyed something, and it was mine.
-
-New `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`) encodes the sequence that
-worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it
-running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect
-`degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed
-from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS)
-rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it.
-
-`runbooks/cluster-maintenance-shutdown.md` (new, registered in `runbooks/kustomization.yaml`) is the
-operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the
-un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section
-separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the
-self-recovering `keycloak-operator` from the one thing that genuinely needs a human (a stuck
-`oauth2-proxy` replica).
-
-## What I verified
-
-* **Graceful shutdown was genuinely absent**, not merely undocumented: `grep -rn
-  'shutdownGracePeriod|graceful-node-shutdown'` over `infra/ansible/` returns nothing, and no
-  `logind` or `InhibitDelay` setting exists anywhere in the tree.
-* **The whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster
-  came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready,
-  `platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact revision, all 3
-  CNPG clusters `healthy` with primaries elected, the UniFi VM `Running`, PiHole resolving
-  (`home.keithjackson.dev` → `192.168.2.20` via `192.168.2.23`), and the Gateway serving
-  `200/200/401` for apex/gitea/plex.
-* **The PVC-deletion mechanism is confirmed, not inferred:** `kubectl get sts loki -o
-  jsonpath={.spec.persistentVolumeClaimRetentionPolicy}` returns
-  `{"whenDeleted":"Delete","whenScaled":"Delete"}`, the PVC was `NotFound` after the cycle, and the
-  Longhorn volume count had gone 25 → 24.
-* **The damage was repaired and verified:** `scale sts loki --replicas=1` (restoring the replica
-  count the chart declares) recreated `storage-loki-0` as a fresh 50Gi Bound PVC, `loki-0` reached
-  `2/2 Running`, and the volume count returned to 25.
-* **The guard works against the live cluster**, tested both ways: `sts_deletes_pvc_on_scale
-  monitoring loki` returns true (would be skipped) and `pihole pihole` returns false (safe to
-  scale).
-* **The script's read-only paths run clean on prod:** `status` reports correctly, and the discovery
-  function returns the 13 workloads currently holding a PVC.
-* `bash -n` clean; the two edited Ansible YAML files parse.
-
-## What I did NOT verify
-
-* **The kubelet change has NEVER BEEN APPLIED OR TESTED.** This is the significant gap. I did not
-  run the role, did not restart k3s, and have **not** observed a node actually drain on
-  `shutdown -h`. Everything about it is reasoning from kubelet/systemd semantics plus the measured
-  absence of the setting. Applying it needs a **rolling** k3s restart (`serial: 1` — kube-vip keeps
-  the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). **Verify by draining
-  one node for real:** apply, then `shutdown -h now` on a single node and watch whether pods
-  terminate gracefully instead of vanishing.
-* **Whether 90s/20s are the right numbers.** Chosen to comfortably cover a Postgres fast shutdown
-  and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a
-  node takes noticeably longer to go down, this is why.
-* **Whether k3s accepts `kubelet-arg` in `config.yaml` as written.** The file already carries
-  `kube-apiserver-arg` as a list and `kubelet-arg` is the documented sibling, but I did not render
-  or apply it, so a typo would surface only at k3s restart — and a bad `config.yaml` is exactly the
-  kind of thing that makes k3s fail to come up. **Review this one before applying.**
-* **The `assert` task's `regex_replace` on the grace period** assumes a plain `90s`-style value; a
-  value expressed as `1m30s` would parse to the nonsense integer `130`. Fine for the committed
-  defaults, wrong if someone writes a compound duration.
-* **`resume`'s Flux-reconcile wait has never run**, and neither has `quiesce` as a script — the
-  2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence
-  that worked but has not itself been executed end to end. Only `status` and the discovery and guard
-  functions have run.
-* **Loki's lost log window is gone for good.** Up to 72h of logs (its `retention_period`), destroyed
-  by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was
-  deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway.
-* **No other `whenScaled: Delete` StatefulSet was audited.** The guard now catches them generically,
-  but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only
-  one at risk. The runbook gives the one-liner to list them.
-* **TrueNAS and the router are entirely outside this change.** TrueNAS is not in the Ansible
-  inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check.
-* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1
-  offline subset only.
-* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
@@ -0,0 +1,137 @@
+---
+date: 2026-10-17
+branch: fix/vpn-ingress-l2-lease-collision
+title: "A node cannot reach a VIP it announces — so the VPN ingress node must announce nothing"
+---
+
+## What changed
+
+The first device enrolled on the remote-access VPN handshook, resolved names, and could not open a
+single connection to any cluster web app. The tunnel was not at fault. The fault is a property of
+Cilium L2 announcement that nothing in this repo knew about:
+
+**A node cannot reach a LoadBalancer VIP whose `CiliumL2AnnouncementPolicy` lease it holds itself.**
+The announcer answers ARP for the VIP on behalf of other LAN hosts but not for its own request, so
+on the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP is dropped
+before it leaves the node.
+
+`platform/vpn-ingress` is uniquely exposed because it runs WireGuard with `hostNetwork: true` and
+MASQUERADEs remote clients to the node's own address — a VPN client's packets take the node's LAN
+path and inherit its ARP failures. Ordinary pods never do. The router's `udp/51820` forward pointed
+at `k3s-prod-1`, which held `cilium-l2announce-gateway-cilium-gateway-shared`, so **all 16 `shared`
+Gateway hostnames were dead over the VPN** — while PiHole DNS kept answering, because
+`192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. Names resolved
+and every app hung on connect.
+
+Seven files:
+
+* **`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`** — new
+  `node_baseline_vpn_ingress_node`; the matching node gets
+  `homelab.keithjackson.dev/vpn-ingress=true`, and a second task **removes** the label from any node
+  that no longer matches, so moving the value moves the label rather than adding a second one. Same
+  shape as `roles/vpn_egress_gateway`'s `egress-gateway` label. Tagged `vpn-ingress`; skipped
+  entirely when the value is empty.
+* **`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`** — set to `k3s-prod-1`, with a
+  `[parity]` note that it must equal the router's forward.
+* **`platform-config/cilium/l2-announcement-policy.yaml`** — `nodeSelector` becomes a
+  `matchExpressions` list keeping `kubernetes.io/os: linux` and adding
+  `homelab.keithjackson.dev/vpn-ingress NotIn ["true"]`. The VPN node then holds no lease, so it
+  ARP-resolves every VIP. This fixes **all** VIPs for VPN clients (Gateway, crafty, plex, pihole,
+  unifi), not just the one that happened to collide.
+* **`platform/monitoring/kube-prometheus-stack.yaml`** — the label added to kube-state-metrics'
+  `metricLabelsAllowlist`, which is what lets the invariant be watched at all.
+* **`platform/monitoring/alerts.yaml`** — `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) joins
+  `kube_lease_owner` against `kube_node_labels` via `label_replace`, plus
+  `VpnIngressNodeLabelMissing` (warning, 30m) as its watchdog.
+* **`platform/monitoring/tests/vpn-ingress-l2.test.yaml`** — four promtool cases: fires on the real
+  collision, quiet in the fixed state, quiet across a transient 11-minute lease move (pinning the
+  `for: 15m`), and the label-absent case asserting the critical alert is inert *and* the watchdog
+  fires.
+* **`runbooks/vpn-ingress-l2-collision.md`** (new, registered in `runbooks/kustomization.yaml`),
+  **`runbooks/vpn-remote-access.md`**, **`platform/vpn-ingress/{daemonset.yaml,README.md}`** — step 3
+  of the setup runbook said *"pick any node — all three run an identical listener"* and *"node loss =
+  failover is this one line"*. Both were false and the first caused this. Failover is now documented
+  as two ordered changes.
+
+**Design note — why a node label and not a `cluster-vars` knob.** The first attempt added
+`WG_INGRESS_NODE` to `clusters/base/cluster-vars.yaml` and substituted it into both the policy and
+the alert. The alert half is **impossible**: `platform/monitoring/tests/extract_rules.py` rejects any
+alert whose `expr` contains a `${...}` placeholder, because the rule it would unit-test is not the
+rule deployed. Rather than duplicate the node name into the alert as a literal (two sources that can
+skew), the label is readable from both sides — the selector matches on it and `kube_node_labels`
+carries it into Prometheus. One source of truth, and as a side effect the change touches no
+sensitive path, so it merges on a green gate rather than needing a human review.
+
+The DaemonSet deliberately gained **no** probe for this. A readinessProbe on VIP reachability would
+wedge a rollout behind an unrelated Gateway outage (`maxUnavailable: 1`); a liveness probe would tear
+down a tunnel carrying LAN traffic correctly.
+
+## What I verified
+
+Measured live against prod, `--context k3s-prod`, on 2026-10-08:
+
+* **The whole tunnel is healthy**, which is what made this hard: `wg show` on `k3s-prod-1` shows the
+  peer with a real endpoint and bidirectional transfer (346 KiB in / 200 KiB out);
+  `net.ipv4.ip_forward=1`; `wg0` up at `10.90.0.1/24` MTU 1420; `iptables -t nat -S POSTROUTING`
+  carries the `MASQUERADE` rule with 26 packets; `FORWARD -i/-o wg0` ACCEPT with 108 / 171 packets;
+  DaemonSet generation 2, 3/3 ready on the content-hashed `wg-peers-m9kcckcb5t`.
+* **The ARP failure and its symmetry.** On `k3s-prod-1`: `ip neigh show 192.168.2.20` → `FAILED`,
+  while `.21`/`.23` resolve to prod-2's MAC `04:0e:3c:96:b0:94`. On prod-2/prod-3, `.20` resolves to
+  prod-1's `bc:e9:2f:7e:c4:a6`. TCP agrees: `.20:443` and `.20:80` FAIL from prod-1, OK from the
+  other two. Repeated across 3 nodes × 5 VIPs — each node reached every VIP except the ones it alone
+  announced (prod-2 ✗ `.22`/`.21`; prod-3 ✗ `.24`).
+* **Not an artefact of probing from the host netns.** `curl --interface wg0` from prod-1 fails the
+  same way, and a neighbour entry is node-global state, so the forwarded path cannot differ in ARP
+  resolution.
+* **The Gateway itself was never broken:** from the workstation (192.168.2.133)
+  `home.keithjackson.dev` → `200`, `plex.home.keithjackson.dev` → `401`, both at `192.168.2.20`.
+  From prod-1 the router (`.1`) and TrueNAS (`.30`) are `REACHABLE`.
+* **`NotIn` semantics, empirically** — the basis for the fail-back claim in the policy's comment:
+  `kubectl get nodes -l 'kubernetes.io/hostname notin (k3s-prod-1)'` returns prod-2 and prod-3, and
+  `NotIn` on a key no node carries returns **all three**. So an unapplied label excludes nothing and
+  degrades to the old behaviour rather than taking every VIP off the LAN.
+* **The alert metric is really scraped:** `kube_lease_owner` is served by kube-state-metrics (55
+  series, every `cilium-l2announce-*` lease with its holder in `lease_holder`), and `kube_node_labels`
+  already carries the two `egress-*` labels through the same allowlist (`k3s-prod-3` shows
+  `egress-gateway=true`), so the third entry follows a proven path.
+* **The four promtool cases pass** (`platform/monitoring/tests/run.sh` → `SUCCESS`, 55 rules
+  checked), including that `VpnIngressNodeAnnouncesL2VIP` is inert when the label is absent.
+* All edited YAML parses; the rendered `nodeSelector` is the intended two-expression list.
+
+## What I did NOT verify
+
+* **The fix is UNAPPLIED and therefore UNPROVEN end to end.** Applying it needs a mutating
+  `kubectl`/`flux reconcile` plus an Ansible run, which are operator actions (CLAUDE.md §5). I have
+  **not** observed `k3s-prod-1` resolve `192.168.2.20` after the exclusion lands, nor a VPN client
+  reaching an app. The prediction rests on the measured symmetry (non-announcing nodes resolve every
+  VIP, 15/15 data points), not on a test of the changed state. **Verify after merge, in this order:**
+  re-run `node_baseline --tags vpn-ingress`; `kubectl get nodes -L
+  homelab.keithjackson.dev/vpn-ingress` (exactly one node); `kubectl -n kube-system get lease | grep
+  l2announce` (no holder is that node); `ip neigh show 192.168.2.20` from the prod-1 wireguard pod
+  (expect a `lladdr`, not `FAILED`); then a client.
+* **That Cilium actually releases a lease when its holder stops matching the `nodeSelector`.** I
+  assumed re-election and did not test it. If Cilium keeps renewing, the affected Leases must be
+  deleted once by hand to force an election. Check this first if the fix appears not to work.
+* **The brief interruption on apply.** Moving the Gateway and `pihole-dns-udp` leases off prod-1 will
+  blip `192.168.2.20` and `192.168.2.23` for as long as re-election plus gratuitous ARP takes. Not
+  measured, and I did not test whether in-flight LAN sessions survive it.
+* **The Ansible tasks have never been executed.** They parse as YAML and follow
+  `roles/vpn_egress_gateway`'s shape, but neither the label task nor the label-removal task has run,
+  so the `changed_when`/`failed_when` conditions and the `k3s kubectl label ... -` removal path are
+  unproven. The removal task's `failed_when` tolerates `'not found'` in stderr; that string was
+  assumed from kubectl's behaviour, not observed.
+* **The residual off-repo coupling is NOT closed and cannot be, as built.** The router's forward is
+  unreadable from the cluster. If someone re-points it without moving the label, the alert watches
+  the wrong node and stays silent while the VPN breaks identically against a different set of VIPs.
+  Only the runbook guards it. Closing it needs a WireGuard exporter publishing which node is
+  actually terminating peer traffic (`wg show` knows — only the forwarded node has a peer
+  `endpoint`); listed as "Not built" in the component README and **recommended follow-up, not done
+  here**.
+* **Why Cilium behaves this way** — intended, a known upstream issue, or fixed in a later version. I
+  measured the behaviour and did not read Cilium's source or issue tracker, so I cannot say whether
+  an upgrade would remove the need for this exclusion.
+* **`192.168.2.21:8443` (unifi) fails from all three nodes**, including ones where ARP resolves, so
+  it is a different, pre-existing fault. Not investigated, not addressed.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline
+  subset only; none of the above is covered by a test that runs in the gate.
+* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).
@@ -420,3 +420,22 @@ flux_bootstrap_env_path: "clusters/prod"
 # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
 # for it.
 etcd_io_priority_enabled: true
+
+# --- The node terminating remote-access WireGuard — DECIDED (measured 2026-10-08) ---------------
+# k3s-prod-1 (192.168.2.10) is what the SOHO router forwards WAN udp/51820 to, confirmed from the
+# cluster rather than assumed: only prod-1's `wg show` had a peer with an `endpoint` and a recent
+# handshake, the other two listeners were idle.
+#
+# node_baseline labels it `homelab.keithjackson.dev/vpn-ingress=true`, and
+# platform-config/cilium/l2-announcement-policy.yaml excludes that label from announcing any
+# LoadBalancer VIP -- because a node CANNOT reach a VIP whose Cilium L2 announcement lease it holds
+# itself (the announcer does not answer its own ARP request; the neighbour entry goes to `FAILED`).
+# prod-1 held the `shared` Gateway lease, and since vpn-ingress MASQUERADEs clients to the node's
+# own address, all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved.
+#
+# [parity] MUST equal the node named by the router's forward (runbooks/vpn-remote-access.md step 3).
+# Nothing in CI can read that forward, so re-pointing it means changing BOTH: this value plus a
+# node_baseline re-run, and the router line. `VpnIngressNodeAnnouncesL2VIP` catches the case where
+# the label lands on a node that then takes a lease; it CANNOT catch the forward being moved
+# without this value following, which is why the runbook spells the two-step out.
+node_baseline_vpn_ingress_node: k3s-prod-1
@@ -107,31 +107,3 @@ k3s_base_domain: "home.keithjackson.dev"

 # Derived, so the realm path is stated once. Override k3s_base_domain, not this.
 k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
-
-# --- Graceful node shutdown (kubelet) ----------------------------------------------------------
-# MEASURED 2026-10-17, during the UPS move: graceful node shutdown was NOT configured on any prod
-# node, so `shutdown -h` did NOT drain pods. systemd stops k3s, containerd goes with it, and every
-# pod is SIGKILLed where it stands -- including 9 CloudNativePG instances mid-checkpoint, the UniFi
-# VM, and 25 attached Longhorn volumes. The whole-cluster shutdown that day had to quiesce
-# everything BY HAND first (runbooks/cluster-maintenance-shutdown.md), and the one volume that did
-# not finish detaching came back `degraded` and had to rebuild.
-#
-# With these set, the kubelet takes a systemd inhibitor lock and, on shutdown, evicts ordinary pods
-# first and critical ones last, so a plain `shutdown -h`/reboot drains the node. That makes the
-# manual scale-down in the runbook a belt-and-braces step for a PLANNED whole-cluster shutdown
-# rather than the only thing standing between a reboot and a hard kill.
-#
-# 90s/20s, not larger: the budget is spent on every reboot and node upgrade, and the kubelet waits
-# the FULL ordinary period before it starts on critical pods. Postgres does a fast shutdown in
-# well under 90s here, and Longhorn needs only enough time to flush and detach.
-k3s_kubelet_shutdown_grace_period: "90s"
-k3s_kubelet_shutdown_grace_period_critical_pods: "20s"
-
-# *** THE HALF THAT IS EASY TO MISS, AND SILENTLY BREAKS THE OTHER HALF. ***
-# Graceful node shutdown works by the kubelet holding a logind INHIBITOR DELAY lock. systemd caps
-# how long any inhibitor may delay shutdown at `InhibitDelayMaxSec`, which DEFAULTS TO 5 SECONDS.
-# If that cap is below the grace period above, systemd proceeds anyway and the drain is truncated
-# to ~5s -- i.e. the feature appears configured, reports no error, and still hard-kills pods. So
-# this MUST be >= k3s_kubelet_shutdown_grace_period; the role asserts it rather than trusting it.
-# Set a little above the kubelet's total (90s + 20s) to leave headroom.
-k3s_logind_inhibit_delay_max_sec: 120
@@ -15,12 +15,3 @@
    port: "{{ k3s_api_port }}"
    timeout: 180
  listen: Restart k3s
-
-# logind re-reads InhibitDelayMaxSec only on restart. Restarting logind does NOT log users out or
-# disturb k3s -- it manages login sessions and inhibitor locks, not workloads. The kubelet picks up
-# the new cap when it next takes its lock, so no k3s restart is needed for THIS half.
- name: Restart logind
-  ansible.builtin.systemd_service:
-    name: systemd-logind
-    state: restarted
-  listen: Restart logind
@@ -86,16 +86,6 @@
      # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
      # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
      # config.yaml is unchanged and the Restart handler does not fire.
-      # ── Graceful node shutdown ──────────────────────────────────────────────────────────────
-      # Without these the kubelet takes no systemd inhibitor lock, so `shutdown -h` stops
-      # containerd and SIGKILLs every pod where it stands -- MEASURED on all three prod nodes
-      # 2026-10-17 (defaults/main.yml has the detail). The companion logind drop-in below raises
-      # InhibitDelayMaxSec, WITHOUT WHICH systemd caps the drain at ~5s and these values are
-      # quietly ignored.
-      kubelet-arg:
-        - "shutdown-grace-period={{ k3s_kubelet_shutdown_grace_period }}"
-        - "shutdown-grace-period-critical-pods={{ k3s_kubelet_shutdown_grace_period_critical_pods }}"
-
      kube-apiserver-arg:
        - "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
        - "oidc-client-id=kubernetes"
@@ -193,46 +183,3 @@
    host: "127.0.0.1"
    port: "{{ k3s_api_port }}"
    timeout: 180
-
-# ── Graceful node shutdown: the systemd half — tag `shutdown` ──────────────────────────────────
-# The kubelet's shutdown-grace-period is ENFORCED BY logind, not by the kubelet: the kubelet holds
-# an inhibitor DELAY lock, and systemd will wait at most `InhibitDelayMaxSec` for it (default 5s)
-# before proceeding regardless. A grace period above that cap is therefore silently truncated --
-# the feature looks configured, logs no error, and still hard-kills pods. Assert the relationship
-# instead of documenting it, because the failure is invisible.
- name: Assert the logind inhibitor cap can actually cover the kubelet grace period
-  ansible.builtin.assert:
-    that:
-      - (k3s_logind_inhibit_delay_max_sec | int) >= (k3s_kubelet_shutdown_grace_period | regex_replace('[^0-9]', '') | int)
-    fail_msg: >-
-      k3s_logind_inhibit_delay_max_sec ({{ k3s_logind_inhibit_delay_max_sec }}s) is below
-      k3s_kubelet_shutdown_grace_period ({{ k3s_kubelet_shutdown_grace_period }}). systemd would
-      stop waiting for the kubelet's inhibitor lock before the drain finished, so pods would be
-      SIGKILLed anyway and graceful node shutdown would be configured but ineffective.
-    quiet: true
-  tags: [shutdown]
-
- name: Ensure /etc/systemd/logind.conf.d exists
-  ansible.builtin.file:
-    path: /etc/systemd/logind.conf.d
-    state: directory
-    owner: root
-    group: root
-    mode: "0755"
-  tags: [shutdown]
-
-# A drop-in, not an edit of logind.conf: the distro file stays pristine and `state: absent` here
-# would fully revert the change.
- name: Raise the logind inhibitor delay cap so the kubelet can finish draining
-  ansible.builtin.copy:
-    dest: /etc/systemd/logind.conf.d/10-k3s-graceful-shutdown.conf
-    owner: root
-    group: root
-    mode: "0644"
-    content: |
-      # Managed by Ansible (roles/k3s_server). See defaults/main.yml.
-      # MUST stay >= kubelet shutdown-grace-period, or the drain is truncated to this value.
-      [Login]
-      InhibitDelayMaxSec={{ k3s_logind_inhibit_delay_max_sec }}
-  notify: Restart logind
-  tags: [shutdown]
@@ -259,3 +259,30 @@ node_baseline_dns_options:
 # Written as ONE marked block, so removing an entry here removes it from the node, and
 # an empty list removes the block entirely.
 node_baseline_hosts_pins: []
+
+# --- The node that terminates remote-access WireGuard (tags: vpn-ingress) ----------------------
+# Hostname of the node the SOHO router forwards WAN udp/51820 to, i.e. the one platform/vpn-ingress
+# actually serves remote clients from. Set it in group_vars for the environment; EMPTY by default,
+# because dev has no such forward and because a wrong value is worse than none.
+#
+# WHAT IT IS FOR, AND WHY IT IS NOT COSMETIC. The matching node gets
+# `homelab.keithjackson.dev/vpn-ingress=true`, and
+# platform-config/cilium/l2-announcement-policy.yaml EXCLUDES that label from announcing any
+# LoadBalancer VIP. MEASURED on prod 2026-10-08: a node cannot reach a VIP whose Cilium L2
+# announcement lease it holds itself (the announcer does not answer its own ARP request -- the
+# neighbour entry goes to `FAILED`). Because vpn-ingress runs on `hostNetwork` and MASQUERADEs
+# clients to the node's own address, a VPN client inherits that: with the forward on the node
+# holding the `shared` Gateway lease, all 16 Gateway hostnames were dead over the VPN while PiHole
+# DNS still resolved. Names worked, every app hung, nothing alerted.
+#
+# [parity] MUST name the node the router's forward actually points at. The forward is off-repo and
+# no check can read it (runbooks/vpn-remote-access.md step 3). Changing the forward is therefore a
+# two-part change: this value (then re-run the role), and the router line.
+#
+# Only ONE node should ever carry the label: it removes that node from VIP announcement candidacy,
+# so labelling all three would leave nothing announcing any VIP at all. The task below removes the
+# label from a node that no longer matches, so moving the value moves the label.
+node_baseline_vpn_ingress_node: ""
+node_baseline_vpn_ingress_label_key: homelab.keithjackson.dev/vpn-ingress
+# k3s ships its kubeconfig here; every prod node is a k3s server, so `k3s kubectl` works locally.
+node_baseline_vpn_ingress_kubeconfig: /etc/rancher/k3s/k3s.yaml
@@ -1109,3 +1109,45 @@
    state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
    backup: true
  tags: [dns, hosts]
+
+# ── vpn-ingress node label — tag `vpn-ingress` ────────────────────────────────────────────────
+# Marks the node terminating remote-access WireGuard so Cilium's L2 announcement policy can EXCLUDE
+# it from announcing LoadBalancer VIPs (a node cannot reach a VIP it announces itself -- see
+# defaults/main.yml). Same shape as roles/vpn_egress_gateway's egress-gateway label, and for the
+# same reason: the label is the one place BOTH a nodeSelector and a PromQL alert can read it from.
+#
+# Skipped entirely when node_baseline_vpn_ingress_node is empty (dev, or before the forward exists),
+# so this adds nothing to an environment that has no WAN forward.
+- name: Label this node as the vpn-ingress node
+  ansible.builtin.command:
+    cmd: >-
+      k3s kubectl label node {{ inventory_hostname }}
+      {{ node_baseline_vpn_ingress_label_key }}=true --overwrite
+  environment:
+    KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
+  register: node_baseline_vpn_ingress_label_apply
+  changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_apply.stdout"
+  when:
+    - node_baseline_vpn_ingress_node | length > 0
+    - inventory_hostname == node_baseline_vpn_ingress_node
+  tags: [vpn-ingress]
+
+# The other half: moving the knob must MOVE the label, not add a second one. Two labelled nodes
+# would remove two of three nodes from announcement candidacy; three would leave no announcer at
+# all and take every VIP off the LAN.
+- name: Remove the vpn-ingress label from a node that no longer holds the forward
+  ansible.builtin.command:
+    cmd: >-
+      k3s kubectl label node {{ inventory_hostname }}
+      {{ node_baseline_vpn_ingress_label_key }}-
+  environment:
+    KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
+  register: node_baseline_vpn_ingress_label_remove
+  changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_remove.stdout"
+  failed_when:
+    - node_baseline_vpn_ingress_label_remove.rc != 0
+    - "'not found' not in node_baseline_vpn_ingress_label_remove.stderr"
+  when:
+    - node_baseline_vpn_ingress_node | length > 0
+    - inventory_hostname != node_baseline_vpn_ingress_node
+  tags: [vpn-ingress]
@@ -1,265 +0,0 @@
-#!/usr/bin/env bash
-# =================================================================================================
-# quiesce.sh — bring the prod cluster to a safe standstill before powering nodes off, and bring it
-#              back afterwards. Companion to runbooks/cluster-maintenance-shutdown.md.
-# =================================================================================================
-# WHY THIS EXISTS. On 2026-10-17 all three nodes had to be powered off to move them onto a UPS, and
-# there was NO procedure for it: `infra/cluster.sh stop` is `virsh`-based and only ever applied to
-# the decommissioned dev cluster. The whole sequence was improvised live. This script is that
-# sequence, so the next person does not improvise it.
-#
-# WHY IT IS NOT JUST `shutdown -h`. Graceful node shutdown was unconfigured on every prod node that
-# day, so systemd stopped containerd and SIGKILLed every pod where it stood. roles/k3s_server now
-# sets shutdown-grace-period (+ the logind InhibitDelayMaxSec cap that actually enforces it), which
-# makes a bare reboot safe. This script is still the right tool for a PLANNED whole-cluster
-# shutdown, for three reasons the kubelet's drain does not cover:
-#   1. The UniFi VM needs an ACPI guest shutdown, not an eviction.
-#   2. CloudNativePG wants hibernation, not pod deletion -- and hibernation is IMPERATIVE state
-#      that Flux will not set or clear for you.
-#   3. Draining three nodes one at a time makes pods RESCHEDULE onto the survivors, churning
-#      Longhorn replicas for no reason when the whole cluster is going down anyway. Scaling in
-#      place avoids all of it.
-#
-# SAFE TO RE-RUN. Every step is idempotent and `resume` is the exact inverse of `quiesce`.
-#
-# WHAT IT DELIBERATELY DOES NOT DO: power anything off. Node poweroff is an operator action
-# (CLAUDE.md §5) and needs the become password; the runbook gives the ansible one-liner. It also
-# does not touch TrueNAS (not in the inventory, SSH refused -- web UI only) or the router.
-set -euo pipefail
-
-CONTEXT="${CONTEXT:-k3s-prod}"
-KUBECTL=(kubectl --context "$CONTEXT" --request-timeout=30s)
-TIMEOUT="${TIMEOUT:-600}"
-
-# The three CloudNativePG clusters, as namespace/name. Hibernation is per-Cluster.
-CNPG=(keycloak/keycloak-db nextcloud-db/nextcloud-db semaphore-db/semaphore-db)
-
-log()  { printf '\033[34m[quiesce]\033[0m %s\n' "$*"; }
-ok()   { printf '\033[32m[ ok ]\033[0m %s\n' "$*"; }
-warn() { printf '\033[33m[warn]\033[0m %s\n' "$*"; }
-die()  { printf '\033[31m[fail]\033[0m %s\n' "$*" >&2; exit 1; }
-
-need() { command -v "$1" >/dev/null || die "$1 not on PATH — run this inside ./infra/devcontainer/dev.sh"; }
-
-# ---------------------------------------------------------------- discovery
-# Every Deployment/StatefulSet whose pods mount a PVC — Longhorn OR the TrueNAS NFS bulk tier.
-# Computed from the LIVE cluster rather than hardcoded, so a newly onboarded service is covered
-# without anyone remembering to add it here. PROTECTED namespaces are excluded because scaling
-# them down would remove the very machinery that performs the detach (longhorn-system), keep the
-# API reachable (kube-system), or finish the hibernation and the VM shutdown (cnpg-system,
-# kubevirt).
-pvc_workloads() {
-  "${KUBECTL[@]}" get pods -A -o json | python3 -c '
-import json,sys
-PROTECT={"longhorn-system","kube-system","cilium-secrets","cnpg-system","kubevirt","flux-system","cdi"}
-out={}
-for p in json.load(sys.stdin)["items"]:
-    ns=p["metadata"]["namespace"]
-    if ns in PROTECT: continue
-    if not any(v.get("persistentVolumeClaim") for v in (p["spec"].get("volumes") or [])): continue
-    ow=(p["metadata"].get("ownerReferences") or [])
-    if not ow: continue
-    k,n=ow[0]["kind"],ow[0]["name"]
-    if k=="ReplicaSet": k,n="deployment",n.rsplit("-",1)[0]
-    elif k=="StatefulSet": k="statefulset"
-    else: continue          # Jobs, and CNPG Clusters, are handled separately
-    out[(ns,k,n)]=1
-for ns,k,n in sorted(out): print(ns,k,n)
-'
-}
-
-# ---------------------------------------------------------------- the PVC-deletion guard
-# *** DO NOT REMOVE THIS CHECK. IT EXISTS BECAUSE SCALING A StatefulSet DESTROYED DATA. ***
-# A StatefulSet may carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and then
-# scaling it to 0 DELETES ITS PVCs -- and with them the Longhorn volumes. MEASURED 2026-10-17: the
-# quiesce for the UPS move scaled `monitoring/loki` (whenScaled: Delete) to 0 and its
-# `storage-loki-0` PVC and volume were deleted outright. Loki stores with `type: filesystem`, so
-# that PVC was the only copy; 72h retention bounded the loss to a rolling window, but nothing about
-# the mechanism is bounded -- the same action on a Delete-policy StatefulSet holding real state
-# would be unrecoverable. The volume count going 25 -> 24 was the only visible symptom, and it was
-# first misread as a miscount.
-#
-# So: such a StatefulSet is NEVER scaled. It is left RUNNING and its volume hard-detaches at
-# poweroff, which costs a replica rebuild on restart (degraded -> healthy, automatic). A rebuild is
-# an enormously better outcome than a deletion, and that is the whole trade being made here.
-sts_deletes_pvc_on_scale() { # ns name -> 0 if scaling it would delete its PVCs
-  local policy
-  policy=$("${KUBECTL[@]}" -n "$1" get statefulset "$2" \
-            -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy.whenScaled}' 2>/dev/null || true)
-  [ "$policy" = "Delete" ]
-}
-
-wait_for() { # wait_for <description> <shell-condition>
-  local desc="$1" cond="$2" waited=0
-  log "waiting: $desc (up to ${TIMEOUT}s)"
-  until eval "$cond"; do
-    (( waited += 10 )); (( waited >= TIMEOUT )) && { warn "TIMED OUT waiting: $desc"; return 1; }
-    sleep 10
-  done
-  ok "$desc"
-}
-
-# ---------------------------------------------------------------- quiesce
-do_quiesce() {
-  need kubectl; need python3
-
-  # Refuse to start if Longhorn is already unhealthy. Powering off on top of a degraded volume is
-  # how one replica becomes zero; better to stop here and let it rebuild first.
-  local unhealthy
-  unhealthy=$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io \
-    -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)
-  [ "$unhealthy" = "0" ] || die "$unhealthy Longhorn volume(s) are not healthy — fix or wait for rebuild BEFORE powering off"
-  ok "Longhorn: all volumes healthy"
-
-  # 1. Flux first, or it reverts every scale-down below within its reconcile interval.
-  log "suspending Flux Kustomizations"
-  local k
-  for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
-    "${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":true}}' >/dev/null
-  done
-  ok "all Kustomizations suspended"
-
-  # 2. The VM needs an ACPI shutdown of the GUEST. KubeVirt does that on running:false; a plain
-  #    pod eviction would be a power cut to the VM.
-  if "${KUBECTL[@]}" -n unifi get vm unifi >/dev/null 2>&1; then
-    log "ACPI-stopping the UniFi VM"
-    "${KUBECTL[@]}" -n unifi patch vm unifi --type=merge -p '{"spec":{"running":false}}' >/dev/null
-  fi
-
-  # 3. Postgres: hibernation is a clean smart shutdown that KEEPS the PVCs. CNPG does one instance
-  #    at a time, primary last-ish, so this is not instant.
-  log "hibernating CloudNativePG clusters"
-  local c ns name
-  for c in "${CNPG[@]}"; do
-    ns="${c%%/*}"; name="${c##*/}"
-    "${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
-      cnpg.io/hibernation=on --overwrite >/dev/null 2>&1 || warn "could not hibernate $c"
-  done
-
-  # 4. Everything holding a PVC, scaled in place. NFS consumers first so nothing holds a hard
-  #    TrueNAS mount; PiHole last because it is the LAN resolver.
-  log "scaling down PVC-holding workloads"
-  local line ns2 kind nm deferred=() skipped_delete_policy=()
-  while read -r ns2 kind nm; do
-    [ -z "${ns2:-}" ] && continue
-    if [ "$kind" = "statefulset" ] && sts_deletes_pvc_on_scale "$ns2" "$nm"; then
-      warn "SKIPPING $ns2/$nm — whenScaled: Delete would DELETE its PVC (see the guard above)."
-      warn "  Left running; its volume hard-detaches at poweroff and rebuilds on restart."
-      skipped_delete_policy+=("$ns2/$nm")
-      continue
-    fi
-    if [ "$ns2" = "pihole" ]; then deferred+=("$ns2 $kind $nm"); continue; fi
-    "${KUBECTL[@]}" -n "$ns2" scale "$kind" "$nm" --replicas=0 >/dev/null 2>&1 \
-      && printf '    scaled 0  %s/%s %s\n' "$ns2" "$kind" "$nm" \
-      || warn "could not scale $ns2/$kind/$nm"
-  done < <(pvc_workloads)
-  for line in "${deferred[@]:-}"; do
-    [ -z "${line:-}" ] && continue
-    # shellcheck disable=SC2086
-    set -- $line
-    "${KUBECTL[@]}" -n "$1" scale "$2" "$3" --replicas=0 >/dev/null 2>&1 \
-      && printf '    scaled 0  %s/%s %s  (LAN DNS — expect name resolution to stop)\n' "$1" "$2" "$3"
-  done
-
-  # 5. The only assertion that matters. An attached volume at poweroff comes back degraded and
-  #    rebuilds; this is what we are buying with all of the above.
-  wait_for "UniFi VMI gone" \
-    '[ "$('"${KUBECTL[*]}"' -n unifi get vmi --no-headers 2>/dev/null | wc -l)" = "0" ]' || true
-  wait_for "all Postgres instances stopped" \
-    '[ "$('"${KUBECTL[*]}"' get pods -A --no-headers 2>/dev/null | grep -cE "(keycloak|nextcloud|semaphore)-db-[0-9]")" = "0" ]' || true
-  wait_for "all Longhorn volumes detached" \
-    '[ "$('"${KUBECTL[*]}"' -n longhorn-system get volumes.longhorn.io -o jsonpath="{range .items[*]}{.status.state}{\"\n\"}{end}" 2>/dev/null | grep -c attached)" = "0" ]' || \
-      warn "some volumes are STILL ATTACHED — they will come back degraded and rebuild; acceptable, but note which"
-
-  if [ "${#skipped_delete_policy[@]}" -gt 0 ]; then
-    echo
-    warn "NOT scaled down (whenScaled: Delete — scaling them would delete their PVCs):"
-    printf '      %s\n' "${skipped_delete_policy[@]}"
-    warn "Expect these volumes to come back 'degraded' and rebuild. That is correct and automatic."
-  fi
-  echo
-  ok "QUIESCED. Safe to power off."
-  cat <<'NEXT'
-
-  Next, from inside the devcontainer (it holds the become password):
-    cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
-      -m shell -a 'shutdown -h +1 "maintenance"'
-
-  THEN, in this order and not before:
-    1. TrueNAS (192.168.2.30) — web UI only; not in the ansible inventory and SSH is refused.
-    2. Router / switch last.
-NEXT
-}
-
-# ---------------------------------------------------------------- resume
-do_resume() {
-  need kubectl
-
-  # THE STEP THAT IS ALWAYS FORGOTTEN. Hibernation is imperative state: Flux neither sets nor
-  # clears it, so resuming Flux alone leaves all three databases down and every app that needs
-  # one looking broken for reasons that point nowhere near Postgres.
-  log "un-hibernating CloudNativePG clusters"
-  local c ns name
-  for c in "${CNPG[@]}"; do
-    ns="${c%%/*}"; name="${c##*/}"
-    "${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
-      cnpg.io/hibernation=off --overwrite >/dev/null 2>&1 || warn "could not un-hibernate $c"
-  done
-
-  # Flux restores every replica count AND the VM's running:true from git, so nothing here needs to
-  # undo the scale-downs by hand.
-  log "resuming Flux Kustomizations"
-  local k
-  for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
-    "${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":false}}' >/dev/null
-  done
-  ok "all Kustomizations resumed"
-
-  wait_for "Flux fully reconciled" \
-    '[ "$('"${KUBECTL[*]}"' -n flux-system get kustomizations --no-headers 2>/dev/null | grep -vc True)" = "0" ]' || \
-      warn "some Kustomizations are not Ready — the graph serialises behind platform-config, so give it a few minutes before worrying"
-
-  echo
-  ok "RESUMED."
-  cat <<'NEXT'
-
-  Known cold-start race, not a fault in this script: oauth2-proxy performs OIDC discovery at
-  startup and exits 1 on the 503 it gets while Keycloak is still coming up, then sits in
-  exponential CrashLoopBackOff. If a replica is stuck, delete the pod once:
-    kubectl --context k3s-prod -n oauth2-proxy delete pod <name>
-NEXT
-}
-
-do_status() {
-  need kubectl
-  printf 'nodes:      %s Ready\n' "$("${KUBECTL[@]}" get nodes --no-headers | grep -c ' Ready')"
-  printf 'flux:       %s/%s Ready\n' \
-    "$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | grep -c True)" \
-    "$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | wc -l)"
-  printf 'longhorn:   %s attached, %s detached, %s not healthy\n' \
-    "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c attached)" \
-    "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c detached)" \
-    "$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)"
-  printf 'postgres:   %s instance pod(s)\n' \
-    "$("${KUBECTL[@]}" get pods -A --no-headers | grep -cE '(keycloak|nextcloud|semaphore)-db-[0-9]' || true)"
-  printf 'unifi VM:   %s VMI\n' "$("${KUBECTL[@]}" -n unifi get vmi --no-headers 2>/dev/null | wc -l)"
-}
-
-case "${1:-}" in
-  quiesce) do_quiesce ;;
-  resume)  do_resume ;;
-  status)  do_status ;;
-  *) cat >&2 <<USAGE
-usage: $0 {quiesce|resume|status}
-
-  quiesce  suspend Flux, ACPI-stop the UniFi VM, hibernate Postgres, scale down every
-           PVC-holding workload, and wait for all Longhorn volumes to detach.
-           Does NOT power anything off — see runbooks/cluster-maintenance-shutdown.md.
-  resume   the exact inverse: un-hibernate Postgres, resume Flux, wait for reconcile.
-  status   one-screen readout of nodes / Flux / Longhorn / Postgres / VM.
-
-Run it inside the pinned toolbox, which supplies kubectl and the context:
-  ./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
-USAGE
-     exit 2 ;;
-esac
@@ -16,6 +16,71 @@ spec:
  externalIPs: false
  interfaces:
    - ^e(n|th)
+  # ===============================================================================================
+  # WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}`
+  # ===============================================================================================
+  # The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true`
+  # from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to.
+  # It is not tuning; it is the fix for a measured, silent, total failure of remote access.
+  #
+  # MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2
+  # announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node
+  # itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP
+  # is dropped before it leaves the node:
+  #
+  #   k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared:
+  #     ip neigh show 192.168.2.20  ->  192.168.2.20 dev eno1 FAILED       (its own VIP)
+  #     nc -z 192.168.2.20 443      ->  FAIL          nc -z 192.168.2.20 80 -> FAIL
+  #   k3s-prod-2 and k3s-prod-3, holding no lease for it:
+  #     ip neigh show 192.168.2.20  ->  lladdr bc:e9:2f:7e:c4:a6 (prod-1)   nc 443/80 -> OK
+  #
+  # Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone
+  # announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not
+  # reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service.
+  #
+  # WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services
+  # through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception --
+  # it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own
+  # address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP
+  # failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all
+  # 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names
+  # resolved and every app hung. Nothing in the cluster reported it; the alerts below now do.
+  #
+  # WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which
+  # is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded
+  # invariant here is worth less than a reviewable one. The alert harness
+  # (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a
+  # dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself
+  # envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the
+  # WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the
+  # rule it would unit-test is not the rule that gets
+  # deployed. A label is readable from BOTH sides: this selector matches on it, and
+  # `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease
+  # holders against it. One source of truth, no git-vs-cluster skew, and the policy stays
+  # env-agnostic as its header above claims. The label is set by
+  # infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape
+  # roles/vpn_egress_gateway already uses for `egress-gateway`.
+  #
+  # THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant,
+  # and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP
+  # matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs
+  # go dark where previously one node might have served them, but at that point the cluster has one
+  # node left and far worse problems.
+  #
+  # `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so
+  # the other two nodes need no label and nothing has to be applied to them. If the label is never
+  # applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour
+  # -- the status quo, not an outage: it fails back, not down. That silent-degradation path is
+  # exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below
+  # would be structurally INERT (its join would match nothing) and would look identical to healthy.
  nodeSelector:
-    matchLabels:
-      kubernetes.io/os: linux
+    matchExpressions:
+      - key: kubernetes.io/os
+        operator: In
+        values:
+          - linux
+      # The node terminating remote-access WireGuard must announce nothing -- see above.
+      - key: homelab.keithjackson.dev/vpn-ingress
+        operator: NotIn
+        values:
+          - "true"
@@ -1147,6 +1147,97 @@ spec:
              metricLabelsAllowlist.
            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
        # -----------------------------------------------------------------------------------------
+        # REMOTE-ACCESS VPN (platform/vpn-ingress) -- NOT the egress tunnel above. Different
+        # direction, different workload: this is inbound WireGuard for remote devices.
+        # -----------------------------------------------------------------------------------------
+        - alert: VpnIngressNodeAnnouncesL2VIP
+          # THE FAILURE THIS CATCHES. MEASURED on prod 2026-10-08: a node CANNOT reach a
+          # LoadBalancer VIP whose Cilium L2 announcement lease it holds itself -- the announcer
+          # does not answer its own ARP request, so the neighbour entry settles at `FAILED` and
+          # packets die before leaving the node (`ip neigh show 192.168.2.20` -> FAILED on the
+          # lease holder, resolved on both others). platform/vpn-ingress runs WireGuard on
+          # `hostNetwork` and MASQUERADEs remote clients to the node's own address, so if the node
+          # the router forwards udp/51820 to is also a lease holder, VPN clients cannot reach the
+          # VIPs it announces. On 2026-10-08 that was the `shared` Gateway: all 16 app hostnames
+          # dead over the VPN, while PiHole DNS kept answering (its VIP has two Services and the
+          # other lease sat on prod-2) -- so the tunnel handshook, names resolved, and every app
+          # hung. Nothing reported it for ~2 days.
+          #
+          # platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
+          # `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so this should be
+          # permanently unsatisfiable. The alert exists because the exclusion has ways to stop
+          # working that no static check on that file's TEXT can see: the label being removed or
+          # moved to the wrong node, a Cilium upgrade changing `NotIn` semantics or the
+          # nodeSelector field, and the selector silently failing to match. This watches the
+          # OUTCOME -- who actually holds the lease -- so all of them surface the same way.
+          #
+          # WHY THE LABEL JOIN RATHER THAN A NODE NAME. The node's identity is NOT written into
+          # this expr, because the harness (tests/extract_rules.py) refuses any alert whose expr
+          # contains a dollar-brace placeholder -- spelled in words because a literal one here is
+          # unparseable to envsubst and would fail this whole Kustomization. Rightly so: the rule it
+          # unit-tests would not be the
+          # rule deployed. Joining `kube_lease_owner` against `kube_node_labels` keeps ONE source
+          # of truth (the label itself, which is also what the policy selects on), so this alert
+          # cannot drift out of agreement with the exclusion it guards. `label_replace` is needed
+          # because kube_lease_owner names the holder `lease_holder` while kube_node_labels names
+          # it `node`; the join is then on a shared `node` label. The label is in
+          # kube-state-metrics' metricLabelsAllowlist (kube-prometheus-stack.yaml) -- without that
+          # line this rule is inert, which `VpnIngressNodeLabelMissing` below exists to catch.
+          #
+          # `for: 15m` because leases legitimately move during a node reboot or a Cilium roll and
+          # Cilium re-elects within seconds; a real collision persists, since the lease holder
+          # keeps renewing and it never self-heals.
+          expr: |
+            max by (lease) (
+              label_replace(
+                kube_lease_owner{namespace="kube-system", lease=~"cilium-l2announce-.*"},
+                "node", "$1", "lease_holder", "(.+)"
+              )
+              * on (node) group_left ()
+              kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"}
+            ) == 1
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "VPN ingress node is announcing a LoadBalancer VIP ({{ $labels.lease }})"
+            description: >-
+              The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
+              WireGuard, and it now holds the Cilium L2 announcement lease `{{ $labels.lease }}`. A
+              node cannot reach a VIP it announces itself, so every remote VPN client has just lost
+              all access to that VIP's Service -- silently, with DNS still resolving. Check that
+              platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
+              the label is still on the node the router forwards udp/51820 to.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
+        - alert: VpnIngressNodeLabelMissing
+          # THE WATCHDOG for the rule above, and the reason it is not optional. The exclusion in
+          # l2-announcement-policy.yaml is a `NotIn` on this label, and `NotIn` with an ABSENT key
+          # MATCHES -- so if the label is on NO node, the policy excludes nothing, every node is an
+          # announcement candidate again, and the 2026-10-08 failure is fully re-armed. The alert
+          # above would not fire either: its join would match nothing and return no series. Present,
+          # evaluating, and incapable of firing -- the exact "inert critical alert" shape that
+          # VPNEgressUplinkLabelMissing exists for on the egress side, and that this repo has been
+          # burned by before.
+          #
+          # The label is applied by infra/ansible/roles/node_baseline
+          # (node_baseline_vpn_ingress_node); the most likely cause of it going absent is a node
+          # rebuild where that role has not been re-run.
+          expr: absent(kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"})
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
+            description: >-
+              No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
+              l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
+              any node may now announce a LoadBalancer VIP, including the one terminating
+              remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
+              the warning matters more than its severity suggests. Re-run the node_baseline role on
+              the forwarded node, and confirm the label is in kube-state-metrics'
+              metricLabelsAllowlist.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
+        # -----------------------------------------------------------------------------------------
        # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
        # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
        # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
@@ -281,8 +281,20 @@ spec:
      # without anyone having to remember to re-enable it later. Scoped to one label on one resource:
      # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
      # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
+      # `vpn-ingress` (added 2026-10-17) marks the node the SOHO router forwards WAN udp/51820 to.
+      # platform-config/cilium/l2-announcement-policy.yaml excludes that node from announcing any
+      # LoadBalancer VIP, because a node cannot reach a VIP whose L2 announcement lease it holds
+      # itself and vpn-ingress MASQUERADEs remote clients to the node's own address (measured
+      # 2026-10-08: all 16 Gateway hostnames dead over the VPN while DNS still resolved). It is
+      # allowlisted HERE because that is the only way the invariant can be WATCHED:
+      # `VpnIngressNodeAnnouncesL2VIP` joins `kube_lease_owner` against this label, and the alert
+      # unit-test harness refuses a dollar-brace placeholder in an alert expr (written out in words:
+      # a literal one here is unparseable to envsubst and fails the whole Kustomization), so a knob
+      # could not have been used. Without this line BOTH that alert and its
+      # `VpnIngressNodeLabelMissing` watchdog are structurally inert. Same reasoning, and the same
+      # 3-extra-series cost, as the two egress labels below.
      metricLabelsAllowlist:
-        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
+        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink,homelab.keithjackson.dev/vpn-ingress]"
      # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
      # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
      # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
@@ -0,0 +1,203 @@
+# promtool rule-unit tests for VpnIngressNodeAnnouncesL2VIP + VpnIngressNodeLabelMissing
+# (platform/monitoring/alerts.yaml)
+#
+# Run with: platform/monitoring/tests/run.sh
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
+# MEASURED on prod 2026-10-08: a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
+# lease it holds ITSELF -- the announcer does not answer its own ARP request, so on the lease holder
+# the neighbour entry settles at `FAILED` and packets to that VIP die before leaving the node.
+# platform/vpn-ingress runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
+# own address, so a VPN client takes that node's LAN path and inherits the failure. With the
+# router's udp/51820 forward on k3s-prod-1, which held the `shared` Gateway lease, all 16 Gateway
+# hostnames were dead over the VPN -- while PiHole DNS kept answering, because its VIP has two
+# Services and k3s-prod-2 held the other lease. The tunnel handshook, names resolved, every app
+# hung, and nothing reported it for ~2 days.
+#
+# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
+# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so VpnIngressNodeAnnouncesL2VIP
+# should be permanently unsatisfiable. THAT IS EXACTLY WHY IT NEEDS A TEST: an alert expected never
+# to fire is indistinguishable, in production, from an alert that CANNOT fire. Nothing would ever
+# reveal a mistyped label, a broken `label_replace`, or a join on the wrong label. Test 1 proves it
+# fires on the real failure; test 2 proves it is quiet in the fixed state, so it is not a rule that
+# fires on anything; test 3 pins the `for: 15m` against routine lease churn.
+#
+# THE JOIN IS THE FRAGILE PART AND IS TESTED DELIBERATELY. `kube_lease_owner` names the holder in
+# `lease_holder`; `kube_node_labels` names it `node`. The rule bridges them with `label_replace`
+# and joins `on (node) group_left ()`. Fixtures therefore carry BOTH series shapes with real label
+# sets, because a join that silently matches nothing returns no series -- which looks exactly like
+# "healthy" and is the failure mode this whole group exists to prevent.
+#
+# Fixture values are prod's real shape (measured 2026-10-08): leases
+# `cilium-l2announce-gateway-cilium-gateway-shared` (k3s-prod-1) and
+# `cilium-l2announce-pihole-pihole-dns-tcp` (k3s-prod-2), in namespace kube-system, with
+# k3s-prod-1 the vpn-ingress node.
+
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. THE BUG, EXACTLY AS IT HAPPENED -- the vpn-ingress node holds the Gateway lease: FIRES.
+  # =============================================================================================
+  - interval: 1m
+    name: vpn-ingress node holds the shared Gateway L2 lease - VpnIngressNodeAnnouncesL2VIP fires
+    input_series:
+      # The collision. A lease holder keeps renewing, so this series does not lapse on its own --
+      # which is why the condition never self-heals and why a 15m `for:` is safe.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
+        values: '1+0x40'
+      # A second lease held by a DIFFERENT node, present throughout. This is the discriminator that
+      # stops the rule from being "any l2announce lease exists anywhere": if the join were dropped,
+      # this series alone would fire the alert, and test 2 would fail.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-pihole-pihole-dns-tcp",lease_holder="k3s-prod-2"}'
+        values: '1+0x40'
+      # k3s-prod-1 is the vpn-ingress node. Only this node carries the label.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
+        values: '1+0x40'
+      # The other two nodes publish kube_node_labels WITHOUT the vpn-ingress label. Present so the
+      # join is exercised against non-matching nodes rather than against nothing at all.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
+        values: '1+0x40'
+    alert_rule_test:
+      # Held 15m, so nothing before then.
+      - eval_time: 14m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      - eval_time: 20m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts:
+          # EXACTLY ONE alert: the pihole lease on prod-2 must NOT produce a second one.
+          - exp_labels:
+              severity: critical
+              lease: cilium-l2announce-gateway-cilium-gateway-shared
+            exp_annotations:
+              summary: "VPN ingress node is announcing a LoadBalancer VIP (cilium-l2announce-gateway-cilium-gateway-shared)"
+              description: >-
+                The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
+                WireGuard, and it now holds the Cilium L2 announcement lease `cilium-l2announce-gateway-cilium-gateway-shared`. A
+                node cannot reach a VIP it announces itself, so every remote VPN client has just lost
+                all access to that VIP's Service -- silently, with DNS still resolving. Check that
+                platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
+                the label is still on the node the router forwards udp/51820 to.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
+      # The label IS present, so the watchdog must stay quiet here.
+      - eval_time: 40m
+        alertname: VpnIngressNodeLabelMissing
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. THE FIXED STATE -- every lease is held by a node that is NOT the vpn-ingress node: QUIET.
+  # This is what l2-announcement-policy.yaml's exclusion produces, and the assertion that keeps
+  # test 1 meaningful rather than an alert that fires on anything.
+  # =============================================================================================
+  - interval: 1m
+    name: all l2 leases held by other nodes - VpnIngressNodeAnnouncesL2VIP stays quiet
+    input_series:
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-2"}'
+        values: '1+0x40'
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-plex-plex-l4-tcp",lease_holder="k3s-prod-3"}'
+        values: '1+0x40'
+      # A lease the vpn-ingress node DOES legitimately hold, in a different family -- proves the
+      # `cilium-l2announce-.*` matcher is doing work. Every node holds operator/controller leases;
+      # only the ANNOUNCEMENT ones matter here, and a matcher broad enough to catch this would page
+      # on a healthy cluster forever.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-operator-resource-lock",lease_holder="k3s-prod-1"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
+        values: '1+0x40'
+    alert_rule_test:
+      - eval_time: 40m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      - eval_time: 40m
+        alertname: VpnIngressNodeLabelMissing
+        exp_alerts: []
+
+  # =============================================================================================
+  # 3. A LEGITIMATE, TRANSIENT LEASE MOVE -- the vpn-ingress node holds it for 11m during a node
+  # reboot or a Cilium roll, then Cilium re-elects away: QUIET. This pins the `for: 15m` as
+  # deliberate. Without it, every rolling restart of the cilium DaemonSet would page, and a paging
+  # alert that cries wolf on routine maintenance gets muted -- which is how the real collision
+  # would go unseen again.
+  # =============================================================================================
+  - interval: 1m
+    name: transient lease move shorter than for 15m - VpnIngressNodeAnnouncesL2VIP stays quiet
+    input_series:
+      # Held by the vpn-ingress node for 11 samples, then the series goes absent (the lease moved,
+      # so kube-state-metrics retires this label set and starts a new one).
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
+        values: '1+0x10 _x30'
+      # Re-elected onto prod-3 from t=11m. A label VALUE change retires one series and starts
+      # another, which is why this is a second series and not a changing value.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-3"}'
+        values: '_x11 1+0x29'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
+        values: '1+0x40'
+    alert_rule_test:
+      # Inside the transient window: pending at most, never firing.
+      - eval_time: 10m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      # Well past it, with the lease settled on another node.
+      - eval_time: 30m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. THE WATCHDOG -- no node carries the label at all. The policy's `NotIn` then excludes
+  # NOTHING (a missing key matches), so the 2026-10-08 failure is fully re-armed, AND
+  # VpnIngressNodeAnnouncesL2VIP is structurally INERT: its join matches nothing and returns no
+  # series, which is indistinguishable from healthy. Both halves are asserted here, because an
+  # inert critical alert is the exact shape this repo has been burned by before.
+  # =============================================================================================
+  - interval: 1m
+    name: vpn-ingress label absent everywhere - watchdog fires and the critical alert is inert
+    input_series:
+      # A lease held by the node that IS (in reality) the forwarded one -- but with no label
+      # anywhere, nothing can tell. This is the dangerous state.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
+        values: '1+0x60'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
+        values: '1+0x60'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
+        values: '1+0x60'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
+        values: '1+0x60'
+    alert_rule_test:
+      # The critical alert CANNOT fire -- this is the inertness being pinned, not an aside.
+      - eval_time: 45m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      # ... which is precisely why the watchdog must.
+      - eval_time: 45m
+        alertname: VpnIngressNodeLabelMissing
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              # absent() propagates the matcher's own labels onto the series it returns, so the
+              # alert inherits this one. Asserted rather than worked around: it is harmless, and
+              # pinning it means a future change to the matcher cannot alter the alert's identity
+              # (and therefore its Alertmanager grouping) unnoticed.
+              label_homelab_keithjackson_dev_vpn_ingress: "true"
+            exp_annotations:
+              summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
+              description: >-
+                No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
+                l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
+                any node may now announce a LoadBalancer VIP, including the one terminating
+                remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
+                the warning matters more than its severity suggests. Re-run the node_baseline role on
+                the forwarded node, and confirm the label is in kube-state-metrics'
+                metricLabelsAllowlist.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
@@ -117,8 +117,19 @@ boundary around this port is WireGuard's peer authentication, full stop.
 The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
 forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
 A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
-keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
-re-issued client profiles. The two idle listeners cost ~15 MiB each.
+keys — so losing the forwarded node never costs a re-issued client profile. The two idle listeners
+cost ~15 MiB each.
+
+> **This section used to say "losing the forwarded node is a one-line router change, with no cluster
+> work". It was wrong, and it cost two days of silently broken remote access on 2026-10-08.** The
+> nodes are warm, but they are **not interchangeable**: the node the forward points at must be the
+> one labelled `homelab.keithjackson.dev/vpn-ingress=true`, because that is the node
+> `platform-config/cilium/l2-announcement-policy.yaml` excludes from announcing LoadBalancer VIPs.
+> **A node cannot reach a VIP whose Cilium L2 announcement lease it holds itself** (see "Measured on
+> prod, 2026-10-08" below), and since this pod MASQUERADEs clients to the node's own address, a VPN
+> client inherits that. Failover is therefore **two** changes: move
+> `node_baseline_vpn_ingress_node` and re-run that role, then re-point the router —
+> `runbooks/vpn-remote-access.md` step 3.

 A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
 replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
@@ -186,6 +197,45 @@ right way; one was a defect in my own manifest.
 | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
 | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |

+## Measured on prod, 2026-10-08 — why the forwarded node cannot be any node
+
+The first device enrolled, handshook, resolved names, and could not open a single connection to any
+cluster web app. The tunnel was not at fault — every part of it measured healthy:
+`net.ipv4.ip_forward=1`, `wg0` up at `10.90.0.1/24` MTU 1420, the peer authorised with real
+bidirectional transfer (346 KiB in / 200 KiB out), `MASQUERADE` present and counting 26 flows,
+`FORWARD -i/-o wg0 ACCEPT` counting 108 / 171 packets, and PiHole DNS answering from the node.
+
+**The fault was that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
+holds itself.** Cilium's announcer answers ARP for the VIP on behalf of other LAN hosts, but not for
+the announcing node's own request, so the neighbour entry settles at `FAILED`:
+
+| from | `192.168.2.20` (Gateway) | `192.168.2.22` (crafty) | `192.168.2.24` (plex) | announces |
+|---|---|---|---|---|
+| `k3s-prod-1` | **FAILED** | — | — | `.20` |
+| `k3s-prod-2` | ok | **INCOMPLETE** | ok | `.21`, `.22`, `.23-tcp` |
+| `k3s-prod-3` | ok | ok | **INCOMPLETE** | `.24` |
+
+Symmetric across 3 nodes × 5 VIPs: each node reached every VIP **except** the ones it alone
+announced. It is a property of L2 announcement, not of any one Service, and it is harmless for
+ordinary pods — they reach Services through the cluster datapath and never ARP for a VIP. This
+component is the exception, because `hostNetwork` + `MASQUERADE` put remote clients on the node's
+own LAN path.
+
+The forward pointed at `k3s-prod-1`, which held the `shared` Gateway lease, so **all 16 Gateway
+hostnames were dead over the VPN**. Two things hid it:
+
+* **PiHole survived by accident.** `192.168.2.23` is announced by *two* Services
+  (`pihole-dns-udp` on prod-1, `pihole-dns-tcp` on prod-2), so prod-2 answered the ARP prod-1 could
+  not. Names resolved perfectly and every app hung — the hardest shape to diagnose from a client.
+* **Nothing reported it.** No probe in this repo compares the forwarded node against the lease
+  holders, and from outside WireGuard is silent to unauthenticated traffic anyway.
+
+Closed by labelling the forwarded node `homelab.keithjackson.dev/vpn-ingress=true` and excluding
+that label in `platform-config/cilium/l2-announcement-policy.yaml` (the node then ARP-resolves every
+VIP), with `VpnIngressNodeAnnouncesL2VIP` watching the lease holders so a recurrence is loud and
+`VpnIngressNodeLabelMissing` watching for the label going absent, which would silently re-arm the
+whole failure. Cost: VIP announcement candidacy drops from three nodes to two.
+
 ## Prerequisites outside this directory

 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
@@ -62,9 +62,39 @@
 # as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
 # whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
 # DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
-# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
-# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
-# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
+# them, and if that node is lost every other node is already warm and answers the same keys, so no
+# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict,
+# and the two idle listeners cost ~15 MiB each.
+#
+# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE.
+# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the
+# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node
+# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer
+# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to
+# that VIP die before leaving the node:
+#
+#   k3s-prod-1, holding the shared Gateway lease:  ip neigh show 192.168.2.20 -> FAILED
+#   k3s-prod-2 / k3s-prod-3:                       ip neigh show 192.168.2.20 -> lladdr …(prod-1)
+#
+# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client
+# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway
+# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that
+# handshakes, resolves names, and then hangs on every single app.
+#
+# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled
+# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from
+# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes
+# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a
+# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test
+# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move
+# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router
+# (runbooks/vpn-remote-access.md step 3).
+# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again.
+#
+# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge
+# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness
+# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a
+# property of the CLUSTER's announcement layout, so it is watched there.
 #
 # All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
 # DaemonSet to cover the cluster; add them if a node is ever tainted.
@@ -1,162 +0,0 @@
-# cluster-maintenance-shutdown — powering the whole cluster off, and back on
-
-**Operator procedure, not an alert page.** Nothing links here from an alert.
-**Script:** `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`)
-**Related:** `/runbooks/update-rollback` is for rolling node work where the cluster stays up. This
-page is for taking **all three nodes down at once**.
-
-First written after the 2026-10-17 UPS move, which was improvised live because no procedure
-existed — `infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev
-cluster, so it does nothing for these three baremetal nodes.
-
-## Why you cannot just run `shutdown -h`
-
-You mostly can now, and could not before.
-
-Until 2026-10-17 **kubelet graceful node shutdown was unconfigured on every prod node**, so systemd
-stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 Postgres
-instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes.
-`roles/k3s_server` now sets `shutdown-grace-period` (and, critically, the
-`InhibitDelayMaxSec` logind cap that actually enforces it — without that systemd stops waiting
-after 5s and the drain is silently truncated). A plain reboot of one node is therefore safe.
-
-**A planned whole-cluster shutdown still wants `quiesce`**, for three things the kubelet's drain
-does not do:
-
-1. **The UniFi VM needs an ACPI guest shutdown**, not an eviction. KubeVirt does that on
-   `running: false`; an eviction is a power cut to the guest.
-2. **CloudNativePG wants hibernation**, not pod deletion — and hibernation is **imperative** state
-   that Flux will neither set nor clear.
-3. **Draining three nodes one at a time makes pods reschedule onto the survivors**, churning
-   Longhorn replicas for no reason when everything is going down anyway. Scaling in place avoids it.
-
-## The hazard that bit us — read this before scaling anything by hand
-
-**A StatefulSet can carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and scaling it
-to 0 then DELETES its PVCs and their Longhorn volumes.**
-
-On 2026-10-17 the hand-rolled quiesce scaled `monitoring/loki` (which has that policy) to 0, and
-`storage-loki-0` was deleted outright. Loki stores with `type: filesystem`, so that PVC was the only
-copy. Its 72h retention bounded the loss to a rolling window — nothing about the *mechanism* is
-bounded. The only visible symptom was the Longhorn volume count going 25 → 24, which was first
-misread as a miscount.
-
-`quiesce.sh` now refuses to scale any such StatefulSet, leaves it running, and tells you which
-volumes to expect `degraded` on restart. **If you quiesce by hand, check the policy first:**
-
-```bash
-kubectl --context k3s-prod get statefulset -A \
-  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,WHEN_SCALED:.spec.persistentVolumeClaimRetentionPolicy.whenScaled'
-```
-
-A hard-detached volume comes back `degraded` and rebuilds itself automatically. That is a vastly
-better outcome than a deleted one — take the rebuild.
-
-## Shutting down
-
-Run everything through the pinned toolbox; it supplies `kubectl` and the context.
-
-```bash
-# 0. Look before you leap. Refuses to proceed if any Longhorn volume is unhealthy —
-#    powering off on top of a degraded volume is how one replica becomes zero.
-./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
-
-# 1. Quiesce: suspend Flux, ACPI-stop the VM, hibernate Postgres, scale down every
-#    PVC-holding workload (NFS consumers first, PiHole last), wait for detach.
-./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh quiesce'
-```
-
-**Expect DNS to stop** when PiHole scales down — it is the LAN resolver. `kubectl` is unaffected
-because the kubeconfig points at the kube-vip IP `192.168.2.9`, not a name.
-
-```bash
-# 2. Power the nodes off. Inside the devcontainer, which holds the become password.
-./infra/devcontainer/dev.sh run bash -c \
-  'cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
-     -m shell -a "shutdown -h +1 \"maintenance\""'
-```
-
-`+1` gives Ansible time to disconnect cleanly. The group resolves to all three nodes; confirm with
-`--list-hosts` if you want to see them first.
-
-```bash
-# 3. Confirm they are actually down before touching anything else.
-for h in 192.168.2.10 192.168.2.11 192.168.2.12; do ping -c1 -W2 $h >/dev/null && echo "$h UP" || echo "$h down"; done
-```
-
-**Then, in this order and not before:**
-
-| # | Machine | How |
-|---|---|---|
-| 4 | **TrueNAS** `192.168.2.30` | **Web UI only.** Not in the ansible inventory, and SSH is **refused** (service disabled). `https://192.168.2.30` → power icon → Shut Down. Safe once the nodes are down, because every NFS consumer (`plex-media`, `*-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive`) is already scaled to zero and nothing holds a hard mount. |
-| 5 | **Router / switch** | Last. |
-
-TrueNAS must go **after** the k3s nodes: a hard NFS mount on a live node wedges kubelet in
-D-state (`/runbooks/nfs-export-latency`, and the load-average symptom in the 2026-10-?? note).
-
-## Bringing it back
-
-**Power on in the reverse order: router/switch → TrueNAS → all three nodes together.** etcd needs
-2 of 3 for quorum, so start the nodes as a group rather than one at a time.
-
-```bash
-# 1. Nodes Ready, etcd quorum, storage reachable.
-./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
-./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get --raw=/healthz/etcd'
-
-# 2. Resume: un-hibernate Postgres, then resume Flux.
-./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh resume'
-```
-
-**Un-hibernating Postgres is the step everyone forgets.** The annotation is imperative and survives
-a Flux resume, so resuming Flux alone leaves all three databases down and Keycloak, Nextcloud and
-Semaphore looking broken for reasons that point nowhere near Postgres. `resume` does it first, for
-exactly that reason.
-
-Flux restores every scaled-down replica count **and** the VM's `running: true` from git, so nothing
-needs undoing by hand.
-
-### What "still broken" looks like when it is actually fine
-
-* **`4/19` Kustomizations Ready for several minutes.** The dependency graph serialises behind
-  `platform-config`; 15 of them report `dependency ... is not ready` until it finishes. Check
-  `platform-config` itself is `Ready=True` and its `lastAppliedRevision` matches the GitRepository
-  artifact — if it is, wait.
-* **One or more Longhorn volumes `degraded`.** Any volume that was still attached at poweroff
-  rebuilds its lagging replica automatically. On 2026-10-17 it was Prometheus's 50Gi TSDB; it went
-  healthy on its own. Worry only if it is still degraded after ~30 minutes.
-* **`keycloak-operator` crashlooping, then recovering.** It is the CRD reconciler, not in the
-  request path; Keycloak's StatefulSet stays 2/2 and auth never goes down. See
-  `/runbooks/keycloak-*` and the known OOM-at-450Mi behaviour.
-
-### What is actually broken and needs you
-
-* **An `oauth2-proxy` replica stuck in CrashLoopBackOff.** It performs OIDC discovery at startup and
-  **exits 1** on the `503 no healthy upstream` it gets while Keycloak is still coming up, then sits
-  in exponential backoff long after Keycloak is fine. It has no startup dependency on Keycloak.
-  Delete the pod once and it comes up clean:
-
-  ```bash
-  ./infra/devcontainer/dev.sh run bash -c \
-    'kubectl --context k3s-prod -n oauth2-proxy delete pod <name>'
-  ```
-
-* **Orphaned `Unknown`/`Error` Job pods** from CronJobs that were mid-run at poweroff. They never
-  recover; delete them.
-
-## Verifying you are actually back
-
-```bash
-./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
-# nodes 3 Ready | flux 19/19 | longhorn 0 not healthy | postgres 9 pods | unifi 1 VMI
-
-# DNS, through PiHole's VIP
-./infra/devcontainer/dev.sh run bash -c 'nslookup home.keithjackson.dev 192.168.2.23'
-
-# The Gateway actually serving (401 on an auth-gated host is CORRECT)
-for h in home.keithjackson.dev gitea.home.keithjackson.dev plex.home.keithjackson.dev; do
-  curl -s -o /dev/null -w "$h %{http_code}\n" -k --resolve $h:443:192.168.2.20 https://$h/
-done
-```
-
-Expect `200`, `200`, `401`.
@@ -52,13 +52,6 @@ configMapGenerator:
      - backup-failure=backup-failure.md
      - backup-missed=backup-missed.md
      - backup-partial=backup-partial.md
-      # Operator procedure, not an alert page — nothing links here from an alert. Served because
-      # the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is
-      # virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was
-      # improvised live, and the one hazard it exposed -- a StatefulSet with
-      # persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 --
-      # destroyed monitoring/loki's volume before anyone noticed.
-      - cluster-maintenance-shutdown=cluster-maintenance-shutdown.md
      - cdi-import-stuck=cdi-import-stuck.md
      - cert-expiry=cert-expiry.md
      - cert-issuance=cert-issuance.md
@@ -126,6 +119,11 @@ configMapGenerator:
      - update-rollback=update-rollback.md
      - vpn-down=vpn-down.md
      - vpn-egress-stale=vpn-egress-stale.md
+      # `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two
+      # pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease
+      # it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the
+      # forwarded node must announce nothing (measured 2026-10-08).
+      - vpn-ingress-l2-collision=vpn-ingress-l2-collision.md
      # Operator procedure, not an alert page. Served because the component it completes is INERT
      # until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
      # and because the one check that actually proves it cannot be run from inside the house.
@@ -0,0 +1,165 @@
+# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP
+
+**Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing`
+(warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`
+**Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component.
+`/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA
+tunnel — and have nothing to do with this.
+
+## What this means here
+
+**A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
+announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's
+own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that
+VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three
+nodes and five VIPs — each node reached every VIP **except** the ones it alone announced.
+
+For an ordinary workload this is harmless: pods reach Services through the cluster datapath and
+never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with
+`hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's
+packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards
+`udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the
+Services behind the VIPs it announces.**
+
+The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and
+`platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything,
+so this should be unreachable.
+
+**The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat
+on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over
+the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and
+`k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve,
+and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is
+down".
+
+Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the
+alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at
+once.
+
+### If it is `VpnIngressNodeLabelMissing` that fired
+
+That is the watchdog, and it means the protection is **switched off**, not that something is merely
+untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so
+with the label on no node, the policy excludes nothing and every node is an announcement candidate
+again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and
+returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as
+meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has
+not been re-run on it.
+
+## First commands
+
+Run cluster commands through the pinned toolbox, which supplies the kube context:
+
+```bash
+# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines.
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n kube-system get lease | grep l2announce'
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress'
+# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it.
+
+# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a
+#    homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels.
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo'
+
+# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when
+#    the label is on the node (kube-state-metrics metricLabelsAllowlist).
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \
+     wget -qO- localhost:8080/metrics | grep vpn_ingress | head'
+```
+
+**Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard
+pod on the affected node (it shares the host netns, so this is the path a VPN client takes):
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress get pods -o wide'   # pick the pod on that node
+
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- \
+     sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"'
+# BROKEN:  192.168.2.20 dev eno1 FAILED
+# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY
+```
+
+The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so
+**ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs
+to `{{ $labels.lease }}` if the Gateway is not the one that moved.
+
+**Which node actually holds the forward** (the router cannot be queried, but the cluster knows):
+only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`.
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do
+     echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done'
+```
+
+## Likely causes, most likely first
+
+1. **The label is on the wrong node** — someone re-pointed the router's forward during a node
+   failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a
+   wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show`
+   loop above: the node with the live peer endpoint is the real forward target.
+   *Fix:* merge a PR setting `node_baseline_vpn_ingress_node`
+   (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role —
+   `ansible-playbook -i inventory/hosts.ini <playbook> -e k3s_hosts=k3s_prod --tags vpn-ingress`.
+   The role labels the new node and **removes** the label from the old one. Then confirm Cilium
+   re-elected the lease off the newly-labelled node.
+
+2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the
+   file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this
+   directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is
+   shaped that way.
+
+3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without
+   a `node_baseline` re-run. *Fix:* re-run the role as above.
+
+4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in
+   `platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical
+   alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3.
+
+5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct
+   and in place, but the lease never re-elected. This was never verified when the fix was authored
+   (see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease
+   object to force an election — `kubectl -n kube-system delete lease <name>` — and watch it come
+   back on a different node.
+
+6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it
+   is the reason this is a runtime alert rather than a static check on the policy text.
+
+## When to escalate or stop
+
+**Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this
+is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists
+indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node
+reboot or a Cilium roll resolves on its own well inside it.
+
+**The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's
+`udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no
+cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the
+`node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert
+is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08,
+prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex
+`192.168.2.24`, so neither is a cost-free target.
+
+**Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert
+you, and the drift is the kind that outlives the memory of making it. The two acceptable hand
+actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a
+*temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo
+a hand-applied label, so it is not a fix.
+
+**Stop and escalate to the operator if** the leases will not leave the excluded node after a forced
+re-election, or if the intended forward target is unclear. Both need a decision about which node
+carries remote access, which is an operator call, not a runbook step.
+
+**Known gap, so you are not misled:** if someone re-points the router without moving the label,
+`VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in
+exactly the way described above. There is no in-cluster signal for that today; closing it needs a
+WireGuard exporter publishing which node is actually terminating peer traffic
+(`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute.
+Until then, if remote access is broken and both alerts are quiet, **check the forward against the
+label by hand before believing the alerts.**
@@ -171,14 +171,57 @@ server without a `wg0` "ready".
 On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
 `192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.

-Pick any node — all three run an identical listener. Note which one you chose here:
+> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node.
+>
+> This step used to say "pick any node — all three run an identical listener". **That sentence
+> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a
+> cluster-vars knob instead of a free choice here.
+>
+> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
+> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease
+> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they
+> leave the node. Measured on prod 2026-10-08:
+>
+> ```text
+> on k3s-prod-1 (held the shared Gateway lease):   ip neigh show 192.168.2.20 -> FAILED
+> on k3s-prod-2 / k3s-prod-3:                      ip neigh show 192.168.2.20 -> lladdr … (prod-1)
+> ```
+>
+> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN
+> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway
+> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so
+> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported
+> it.
+>
+`infra/ansible/roles/node_baseline` now labels the forwarded node
+> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in
+> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes
+> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI
+> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a
+> labelled node taking a lease, but it CANNOT catch the forward being moved without the label.
+
+Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`).
+Note it here:

 ```
 # WAN udp/51820 -> 192.168.2.__  (k3s-prod-_)   set <date>
+# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml
 ```

-**Node loss = failover is this one line.** Re-point the forward at another node address; every
-device keeps working with the same profile, because all three nodes answer the same keys.
+**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that
+was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and
+reintroduces the failure above against a different set of addresses (prod-2 announces crafty and
+unifi; prod-3 announces plex). Do both, in this order:
+
+1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role
+   (`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium
+   then moves the leases off the newly-labelled node and onto its peers.
+2. **Re-point the router** forward at the new node's address.
+
+Every device keeps working with the same profile either way; all three nodes answer the same keys,
+and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that
+handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because
+that alert watches the node the knob names, not the node your router actually points at.

 Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
 already silent to unauthenticated probes (README, "Why this is safe to expose").