docs(vpn-ingress): withdraw the L2-lease diagnosis — it was falsified by measurement

The claim this branch originally shipped (a node cannot reach a LoadBalancer VIP
whose Cilium L2 announcement lease it holds, so VPN clients inherit that via
hostNetwork MASQUERADE) is FALSE. Re-measured with all 8 l2announce leases on
k3s-prod-1:

  ip neigh show 192.168.2.20   -> INCOMPLETE   (cause present)
  nc -z 192.168.2.20 443       -> OK
  curl --resolve ... :192.168.2.20  -> 200 in 25ms   (effect absent)

Cause present, effect absent. prod-2 and prod-3 return the same 200.

Why it was never going to bite: cilium-dbg status shows
`KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1]` and
`Masquerading: BPF [eno1, wg0]` -- Cilium's LB datapath is attached to wg0, so a
client packet for a VIP is DNAT'd in BPF BEFORE any neighbour lookup, and the
Gateway backend is a node-local Envoy socket anyway.

The methodological error, which is the part worth keeping: every probe was run
from the wireguard pod on the grounds that sharing the host netns made it "the
path a MASQUERADEd client takes". False. A MASQUERADEd client is FORWARDED
(netfilter FORWARD + routing); a probe started in the pod is SOCKET-ORIGINATED
(Cilium socket-LB). Different hooks -- sharing a namespace is not sharing a hook.
The `curl --interface wg0` control cited to rule out a host-netns artefact fails
on ALL THREE nodes including ones whose ARP resolves, so it discriminated
nothing. A raw-IP `curl -k https://192.168.2.20/` returning rc=35 is missing SNI,
not unreachability.

Also corrected: new-peer.sh emits AllowedIPs = 0.0.0.0/0 (FULL tunnel) by
default; `--split` is the opt-in. An earlier reading had the branches backwards.

All code is withdrawn -- the nodeSelector exclusion, the node_baseline label
tasks, both alerts, the promtool tests and the KSM allowlist entry are gone.
Net change is three documentation files, purely additive (356 insertions, 0
deletions), recording what was measured so the next person does not redo it.

The reported internet-routing problem is NOT diagnosed here: no client had
connected since the power-cycle (no peer endpoint, 0 packets through FORWARD and
MASQUERADE), so the forwarded path was unmeasurable.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

decisions/2026-10-17-automated-upgrades.md

@@ -185,6 +185,55 @@ Measured against the live cluster and nodes, read-only, on 2026-10-17.
 - **`renovate.json` parses** as valid JSON after my edits (`json.load`).
 - **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.
 
+### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong
+
+`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
+nodes at the package task:
+
+```
+fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
+```
+
+**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
+`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
+package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
+role could never have worked. This was the exact risk the first version of this entry flagged ("the
+`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
+not a wrong glob, a nonexistent package.
+
+Three further measurements settled what to do instead:
+
+* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
+* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
+  patch, so merely placing a script at that path would never be called.
+* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
+`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
+installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
+`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
+`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
+self-clears on boot and needs no cleanup task.
+
+The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
+`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
+than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
+reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
+role owns and requires the script to be executable.
+
+**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
+unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
+`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
+running kernel *is* the newest. `sh -n` and `bash -n` clean.
+
+**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
+three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
+nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
+whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
+runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
+a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
+produced the right answer with the current set; a future set could differ.
+
 ## What I did NOT verify
 
 The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.

decisions/2026-10-17-vpn-ingress-l2-lease-collision.md

@@ -0,0 +1,198 @@
+---
+date: 2026-10-17
+branch: fix/vpn-ingress-l2-lease-collision
+title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong"
+---
+
+## What changed
+
+**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it
+was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is
+not re-derived — it is very easy to re-derive from `ip neigh` alone.
+
+### The claim, and how it died
+
+The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2
+announcement lease it holds itself* — the announcer does not answer its own ARP request, the
+neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because
+`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
+own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded
+that label from announcing, with two alerts and four promtool tests behind it.
+
+The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*`
+leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) —
+the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five
+VIPs and the other two nodes should reach **all**.
+
+**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node:
+
+| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
+|---|---|---|
+| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
+| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms |
+| `nc -z 192.168.2.20 443` | **OK** | OK |
+| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` |
+| `192.168.2.21:8443` | FAIL | **FAIL** (both) |
+
+The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the
+asserted **effect is absent**. That is sufficient to falsify the causal claim without having to
+explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**.
+
+### Why it is inert
+
+No traffic path resolves a LoadBalancer VIP by ARP:
+
+* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
+  `Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath
+  is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP
+  is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**.
+* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's
+  backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet.
+* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
+  which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a
+  bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT,
+  not wire traffic.
+
+### The two probes that misled me
+
+Both were cited as evidence in the previous version of this entry.
+
+* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing
+  artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong.
+* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a
+  host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including
+  the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it
+  discriminates nothing.
+
+**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then
+routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns
+probe cannot measure the forwarded path in either direction — it reports success where the forwarded
+path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not
+sharing a hook, and that is the assumption the original diagnosis rested on.
+
+### Files
+
+Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`,
+`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`,
+`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`,
+`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted:
+`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`.
+
+Kept, rewritten from scratch:
+
+* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is
+  inert and which probes lie, and the internet-egress facts below.
+* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster
+  work, internet does not") that works outwards from the client profile, and a short section saying
+  the L2 lease is a dead end, with the unifi `.21` note.
+
+**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is
+proposed by this branch any more.**
+
+### The internet-routing report: no server-side fix was needed, and none was made
+
+The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full
+one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has
+emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`,
+2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and
+`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel
+feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that
+already exists. **I deliberately shipped no code change here** — writing one would have meant
+inventing a fix for a symptom I could not reproduce (see below).
+
+The server side was measured end to end and is capable and correct, so the cause is in the client
+profile or the client. The triage section names the two candidates that fit the symptom exactly —
+a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and
+leak public traffic outside the tunnel — each with a one-line test that settles it.
+
+## What I verified
+
+Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on
+2026-10-17:
+
+* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1`
+  (`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again.
+* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a
+  real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere.
+* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three
+  nodes — so the successes are service-aware DNAT, not wire delivery.
+* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with
+  `REACHABLE` neighbour entries. This is the measurement that destroys the original control.
+* **The datapath facts** quoted above from `cilium-dbg status --verbose` and
+  `cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`:
+  `kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`,
+  `routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three.
+* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` →
+  `dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP.
+  So the 200 is not loopback delivery by the host stack — it is BPF.
+* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1`
+  (read, never written); `iptables -t nat -S POSTROUTING` carries
+  `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0
+  packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both
+  present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it;
+  `curl http://1.1.1.1/` returns `301` from each node.
+* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by
+  `podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod
+  `egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on
+  `-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the
+  home WAN IP, not PIA's.
+* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned
+  `142.250.113.138`.
+* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one
+  enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both
+  `FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The
+  `wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters.
+* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f`
+  (2026-10-05) and never changed.
+* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts.
+
+## What I did NOT verify
+
+* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could
+  not test what a real remote client does, and — as established above — **no probe runnable from the
+  pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a
+  VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a
+  LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one
+  remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in
+  the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this
+  unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show`
+  shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the
+  client loads a cluster app — and repeat it on a node that holds leases and one that does not.*
+* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same
+  reason. **I need a client connected** to learn it; the router is unreadable from the cluster.
+* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP
+  failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes.
+  My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**,
+  rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A
+  competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and
+  `wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is
+  needed for the falsification** — the asserted cause is present today without its effect. I did not
+  retro-measure anything, and I did not read Cilium's source or issue tracker.
+* **The operator's report cannot be a description of "right now."** No client has connected since
+  the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster
+  services and the LAN but not the internet" must describe a session before the power-cycle or one
+  that has since dropped. **I did not establish when it was observed, or with which profile.**
+* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not**
+  find the fault, and I shipped no fix for it. The two candidates in the triage section (a
+  `--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not
+  measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line
+  decides split vs full and halves the search — and a connected client for step 2.
+* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and
+  lives only on the operator's machine and the device; it is not in the repo, and I did not look for
+  it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing.
+* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without
+  a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU
+  would see large transfers hang while small requests succeed. The runbook's existing MTU note
+  (drop to 1280) already covers the remedy.
+* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole
+  IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a
+  client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an
+  interface with no IPv6 address.
+* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate,
+  pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1
+  offline subset only. Nothing above is covered by a test that runs in CI.
+* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there
+  is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating
+  `kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material).
+* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).

infra/ansible/roles/unattended_upgrades/README.md

@@ -20,10 +20,19 @@ restarted, and reboots are delegated to **kured** (`platform/kured`).
 
 Deploying either alone produces a mechanism that looks healthy and does nothing:
 
-- **kured alone** watches a file that nothing creates. Measured 2026-10-17:
-  `update-notifier-common` — the package that owns the hook which writes
-  `/var/run/reboot-required` — was **not installed on any node**. kured would sit at 1/1 Running
-  reporting `kured_reboot_required=0` forever.
+- **kured alone** watches a file that **nothing on Debian ever creates**. Measured 2026-10-18,
+  after a first attempt to install Ubuntu's `update-notifier-common` failed on all three nodes
+  with *"No package matching 'update-notifier-common' is available"*:
+  - `apt-cache policy update-notifier-common` → `Candidate: (none)` — the package does not exist
+    in Debian; it is Ubuntu-only.
+  - `/usr/share/update-notifier/notify-reboot-required` → absent.
+  - **No `linux-image-*` postinst references `notify-reboot-required`** — calling it is an Ubuntu
+    patch, so even dropping a script at that path would never be invoked.
+  - No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+  So **this role ships the writer itself** (`/usr/local/sbin/k3s-notify-reboot-required` plus a
+  `DPkg::Post-Invoke` hook) rather than installing a package. Without it kured would sit at 1/1
+  Running reporting `kured_reboot_required=0` forever.
 - **this role alone** installs patches that a long-running process never picks up, and when a
   reboot *is* needed there is nothing safe to take it. `Automatic-Reboot "true"` is not the
   answer: three control-plane nodes with etcd quorum 2/3, each rebooting at its own time, is how
@@ -36,7 +45,7 @@ Measured on all three prod nodes, 2026-10-17:
 | | |
 |---|---|
 | `unattended-upgrades` | **not installed** |
-| `update-notifier-common` | **not installed** |
+| `update-notifier-common` | **not installable** — Ubuntu-only, `Candidate: (none)` on trixie |
 | `needrestart` | **not installed** |
 | `apt-mark showhold` | **empty** — no package held on any node |
 | `/etc/apt/preferences.d/` | **empty** — no apt pin declares the backports-kernel intent |
@@ -157,7 +166,9 @@ first two):
 ```bash
 ssh [email protected] 'apt-config dump | grep -iE "unattended-upgrade::(automatic-reboot|package-blacklist|origins)"'
 ssh [email protected] 'systemctl list-timers apt-daily-upgrade.timer --no-pager'
-ssh [email protected] 'ls -la /var/run/reboot-required 2>&1; dpkg -l update-notifier-common needrestart unattended-upgrades | tail -4'
+ssh [email protected] 'ls -la /var/run/reboot-required* 2>&1 | head -2
+  ls -l /usr/local/sbin/k3s-notify-reboot-required /etc/apt/apt.conf.d/99-k3s-reboot-required 2>&1
+  dpkg -l needrestart unattended-upgrades | tail -3'
 ```
 
 Expect: `Automatic-Reboot "false"`, a blacklist containing `^linux-`, an `Origins-Pattern`

infra/ansible/roles/unattended_upgrades/defaults/main.yml

@@ -92,13 +92,11 @@ unattended_upgrades_enabled: false
 unattended_upgrades_packages:
   # The upgrader itself.
   - unattended-upgrades
-  # THE SENTINEL WRITER, AND THE WHOLE REASON kured CAN WORK. On Debian, /var/run/reboot-required is
-  # not created by the kernel, by apt, or by unattended-upgrades — it is created by a hook shipped in
-  # update-notifier-common. Measured ABSENT on all three nodes. Without this package the file is
-  # never written, kured correctly never reboots, every dashboard stays green, and the reboot half of
-  # this PR is an elaborate no-op. The role asserts the hook exists after install rather than
-  # assuming the package shipped one (see tasks/main.yml).
-  - update-notifier-common
+  # NOTE: `update-notifier-common` was listed here and REMOVED (2026-10-18). It is an UBUNTU
+  # package and does not exist in Debian at all -- `apt-cache policy update-notifier-common` on
+  # trixie returns `Candidate: (none)`, so the play failed outright with "No package matching
+  # 'update-notifier-common' is available" on all three nodes. The sentinel writer is now provided
+  # by this role itself; see unattended_upgrades_sentinel_* below and tasks/main.yml.
   # Restarts services still mapped to a deleted library after a security upgrade. Without it a
   # security-only policy patches bytes on disk that no running process reads — see DECISION 3.
   - needrestart
@@ -170,3 +168,32 @@ unattended_upgrades_needrestart_blacklist:
 unattended_upgrades_download_time: "03:30"
 unattended_upgrades_upgrade_time: "04:00"
 unattended_upgrades_randomised_delay: 60m
+
+
+# ── THE REBOOT SENTINEL, PROVIDED BY THIS ROLE ──────────────────────────────────────────────────
+# /var/run/reboot-required is the ONLY thing kured watches, and on Debian trixie **nothing creates
+# it**. Measured on all three nodes 2026-10-18:
+#   * `update-notifier-common` (which owns that hook on Ubuntu) has `Candidate: (none)` — the
+#     package does not exist in Debian. Installing it is what broke the first run of this play.
+#   * `/usr/share/update-notifier/notify-reboot-required` is absent.
+#   * NO `linux-image-*` postinst references `notify-reboot-required` — calling it is an Ubuntu
+#     patch, not Debian behaviour, so merely dropping a script at that path would never be invoked.
+#   * No hook under /etc/apt/apt.conf.d or /etc/kernel/ mentions reboot-required.
+# So the writer has to be ours. Without it kured is a permanent, INVISIBLE no-op: it stays 1/1
+# Running and reports kured_reboot_required=0 forever, which is the "failure mode is a green tick"
+# shape this repo keeps getting burned by.
+#
+# MECHANISM: a dpkg Post-Invoke hook runs a script that compares the RUNNING kernel against the
+# newest INSTALLED one and touches the sentinel when they differ. Deliberately kernel-only, which
+# matches the design: unattended-upgrades is security-only with `^linux-` blacklisted, so it will
+# essentially never set this — the sentinel appears when the OPERATOR deliberately runs the
+# node_baseline kernel play, and kured then automates only the mechanical reboot.
+#
+# It uses `linux-version` (from `linux-base`, a dependency of every linux-image package, so
+# guaranteed present — measured `linux-base 4.12.1` installed) rather than parsing `dpkg -l`
+# output, because kernel version ordering is not string ordering.
+#
+# /run is a tmpfs (measured), so the sentinel self-clears on boot: after a reboot the running
+# kernel IS the newest and the script writes nothing. No cleanup task is needed, and none exists.
+unattended_upgrades_sentinel_script: /usr/local/sbin/k3s-notify-reboot-required
+unattended_upgrades_sentinel_hook: /etc/apt/apt.conf.d/99-k3s-reboot-required

infra/ansible/roles/unattended_upgrades/tasks/main.yml

@@ -35,9 +35,9 @@
       tags: [packages, config, verify]
 
     # ── PACKAGES ────────────────────────────────────────────────────────────────────────────────
-    - name: Install unattended-upgrades, the reboot-sentinel writer and needrestart
-      # update-notifier-common is NOT optional padding: it owns the hook that creates
-      # /var/run/reboot-required, which is the only thing kured watches. See defaults/main.yml.
+    - name: Install unattended-upgrades and needrestart
+      # The reboot-sentinel writer is NOT a package here. On Debian nothing ships one -- see
+      # defaults/main.yml -- so this role installs its own script and dpkg hook, two tasks below.
       ansible.builtin.apt:
         name: "{{ unattended_upgrades_packages }}"
         state: present
@@ -45,6 +45,60 @@
         cache_valid_time: 3600
       tags: [packages]
 
+    # ── THE REBOOT SENTINEL WRITER (ours, because Debian ships none) ────────────────────────────
+    # Full rationale and the four measurements behind it: defaults/main.yml. Short version: kured
+    # watches /var/run/reboot-required and nothing on Debian creates it, so without these two files
+    # the entire reboot half is an invisible no-op.
+    - name: Install the reboot-sentinel writer
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_script }}"
+        owner: root
+        group: root
+        mode: "0755"
+        content: |
+          #!/bin/sh
+          # MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits are reverted.
+          #
+          # Touch /var/run/reboot-required when the RUNNING kernel is not the newest INSTALLED one.
+          # kured watches that path; see the role's defaults/main.yml for why this is ours to ship.
+          #
+          # `linux-version` comes from linux-base (a dependency of every linux-image package), and
+          # is used rather than sorting `dpkg -l` output because kernel version ordering is not
+          # string ordering. If it is somehow missing, exit 0 and write nothing: a dpkg hook must
+          # never fail an apt transaction, and a missing sentinel is caught by the verify assert.
+          set -eu
+          command -v linux-version >/dev/null 2>&1 || exit 0
+
+          running="$(uname -r)"
+          newest="$(linux-version list 2>/dev/null | linux-version sort --reverse 2>/dev/null | head -n1)"
+
+          # No newest (nothing installed we can read) -> nothing to say.
+          [ -n "$newest" ] || exit 0
+
+          if [ "$running" != "$newest" ]; then
+            touch /var/run/reboot-required 2>/dev/null || exit 0
+            # kured only needs the file, but the .pkgs companion is what an operator reads to find
+            # out WHY a node is waiting, so write a reason rather than an empty marker.
+            printf 'linux-image-%s\n' "$newest" > /var/run/reboot-required.pkgs 2>/dev/null || true
+          fi
+          exit 0
+      tags: [packages, config]
+
+    - name: Install the dpkg hook that runs the sentinel writer
+      # Post-Invoke, so it runs after dpkg has finished and a newly installed kernel is already
+      # visible to `linux-version list`. `|| true` so a failure here can never abort an apt
+      # transaction -- an unwritten sentinel is a missed reboot, but a broken apt is a broken node.
+      ansible.builtin.copy:
+        dest: "{{ unattended_upgrades_sentinel_hook }}"
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          // MANAGED BY ANSIBLE -- infra/ansible/roles/unattended_upgrades. Local edits reverted.
+          // Writes /var/run/reboot-required when the running kernel is not the newest installed.
+          DPkg::Post-Invoke { "{{ unattended_upgrades_sentinel_script }} || true"; };
+      tags: [packages, config]
+
     # ── CONFIG ──────────────────────────────────────────────────────────────────────────────────
     # Written as a whole file rather than patched in place, for the reason node_baseline gives for
     # its own apt source: an idempotent full-file write with a managed-by header is reviewable and
@@ -246,21 +300,24 @@
           Effective apt config confirmed: kernel blacklisted, automatic reboot disabled.
       tags: [verify]
 
-    - name: Find the hooks that can create /var/run/reboot-required
-      # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN, and
-      # it is the one part that cannot be asserted from the repo — it depends on what
-      # update-notifier-common actually ships on this Debian release. Rather than hard-code a path
-      # that may not exist, discover the candidates and assert that at least one is present.
-      ansible.builtin.find:
-        paths:
-          - /etc/kernel/postinst.d
-          - /etc/apt/apt.conf.d
-        patterns:
-          - "*update-notifier*"
-          - "*unattended-upgrades*"
-          - "99update-notifier"
-        file_type: any
-      register: unattended_upgrades_sentinel_hooks
+    # THE SENTINEL IS THE SINGLE POINT OF FAILURE FOR THE ENTIRE REBOOT HALF OF THIS DESIGN.
+    # This was originally a `find` over /etc/kernel/postinst.d and /etc/apt/apt.conf.d, on the
+    # theory that the writer came from a package and the role could only discover whatever that
+    # package happened to ship. That theory was wrong: no such package exists on Debian (see
+    # defaults/main.yml), so the writer is now OURS and lives at two known paths. Checking two
+    # exact paths is strictly stronger than a glob -- a glob would have matched Debian's own
+    # unattended-upgrades apt conffile and passed while no sentinel writer existed at all.
+    - name: Stat the sentinel writer script
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_script }}"
+      register: unattended_upgrades_sentinel_script_stat
+      changed_when: false
+      tags: [verify]
+
+    - name: Stat the sentinel dpkg hook
+      ansible.builtin.stat:
+        path: "{{ unattended_upgrades_sentinel_hook }}"
+      register: unattended_upgrades_sentinel_hook_stat
       changed_when: false
       tags: [verify]
 
@@ -272,18 +329,21 @@
       # before, so it is an assert and not a debug message.
       ansible.builtin.assert:
         that:
-          - unattended_upgrades_sentinel_hooks.matched | int > 0
+          - unattended_upgrades_sentinel_script_stat.stat.exists
+          - unattended_upgrades_sentinel_script_stat.stat.executable | default(false)
+          - unattended_upgrades_sentinel_hook_stat.stat.exists
         fail_msg: >-
-          NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required. No
-          update-notifier / unattended-upgrades hook was found under /etc/kernel/postinst.d or
-          /etc/apt/apt.conf.d. kured watches that path and reboots only when it exists, so with no
-          writer the reboot automation is permanently and INVISIBLY idle -- kured stays 1/1 Running
-          and reports kured_reboot_required=0 on every node forever. Confirm
-          update-notifier-common is installed (`dpkg -l update-notifier-common`) and inspect
-          `dpkg -L update-notifier-common | grep -E 'postinst.d|apt.conf.d'`.
+          NOTHING ON {{ inventory_hostname }} CAN CREATE /var/run/reboot-required.
+          {{ unattended_upgrades_sentinel_script }} (executable) and
+          {{ unattended_upgrades_sentinel_hook }} must both exist -- this role ships both, because
+          Debian provides no sentinel writer of its own (update-notifier-common is an UBUNTU
+          package and is not installable here). kured watches that path and reboots only when it
+          exists, so with no writer the reboot automation is permanently and INVISIBLY idle --
+          kured stays 1/1 Running and reports kured_reboot_required=0 on every node forever.
+          Re-run this role without `--tags verify` to install them.
         success_msg: >-
-          Reboot sentinel writer present ({{ unattended_upgrades_sentinel_hooks.matched }} hook(s)
-          found) -- a kernel or flagged package upgrade will signal kured.
+          Reboot sentinel writer present ({{ unattended_upgrades_sentinel_script }} + dpkg hook)
+          -- a kernel change will signal kured on the next apt transaction.
       tags: [verify]
 
     - name: Report the pending-reboot state of this node

platform/vpn-ingress/README.md

@@ -186,6 +186,86 @@ right way; one was a defect in my own manifest.
 | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
 | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
 
+## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach
+
+A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose
+Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from
+announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is
+real but inert, and the distinction matters enough to record, because the wrong version is very
+easy to re-derive from `ip neigh` alone.
+
+Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case,
+after the UPS power-cycle — from the host netns of the `wireguard` pod on each node:
+
+| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
+|---|---|---|
+| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
+| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms |
+| `nc -z 192.168.2.20 443` | **OK** | OK |
+| plex `192.168.2.24:32400` | **`200`** | `200` |
+
+So the announcer really does **not** answer its own ARP request — that part of the observation is
+reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**:
+
+* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
+  `Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**,
+  so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before
+  any route or neighbour lookup for the VIP happens**.
+* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the
+  shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without
+  a packet crossing the LAN at all.
+* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
+  which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real**
+  service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes —
+  that is service-aware DNAT, not wire traffic.
+
+**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis:
+
+* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an
+  artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not
+  anything is wrong.
+* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the
+  two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it
+  discriminates nothing — and it is not the client path either (see below).
+
+**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are
+**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is
+**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the
+forwarded path in either direction: it reports success where the forwarded path might fail, and
+`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with
+a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE`
+counters move.
+
+## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses
+
+`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the
+component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature
+to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel
+client cannot reach the internet, nothing in the list below is the reason:
+
+| checked | result on all three nodes |
+|---|---|
+| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` |
+| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change |
+| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE |
+| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present |
+| default route | `default via 192.168.2.1 dev eno1 onlink`, intact |
+| node's own internet reachability | `http://1.1.1.1/` returns `301` |
+
+**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is
+closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by
+`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled
+`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity
+traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway
+node. A remote client's apparent public address is therefore the **home WAN IP**, which is the
+intended behaviour and the thing to state when enrolling a device.
+
+**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one
+enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and
+the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the
+power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the
+forwarded node shows a peer with an `endpoint` and a recent `latest handshake`.
+
 ## Prerequisites outside this directory
 
 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`

runbooks/vpn-remote-access.md

@@ -273,6 +273,84 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
 by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
 if the router is not delivering the packets, and a `handshake` with no completed session if it is.
 
+## Triage — "the client reaches the LAN and cluster apps, but not the internet"
+
+Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
+almost always lives, and because the server side has been measured and is not the cause.
+
+**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**
+
+| the profile says | meaning |
+|---|---|
+| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
+| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |
+
+`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
+`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
+profile only**, so an already-issued profile keeps whatever it was rendered with until it is
+re-rendered and re-imported on the device. The two forms differ by one flag:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>'           # full tunnel
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split'   # LAN only
+```
+
+Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
+`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
+editing `AllowedIPs` in the profile **on the device** — the server does not care.
+
+**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
+client connected, then watch the two counters that only forwarded traffic moves:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
+     wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
+     iptables -t nat -L POSTROUTING -n -v | head -5"'
+```
+
+* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
+  node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
+* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
+  upstream of the cluster (router forward, client, or carrier), not here.
+* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
+  failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
+  which breaks internet while node-local cluster VIPs still work. It has not been observed.
+
+**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:
+
+* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
+  client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
+  so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
+  `curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
+  and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
+  blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
+* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
+  2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
+  blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
+  a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.
+
+**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
+`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
+**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
+`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
+PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
+`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
+**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).
+
+## Triage — which L2 announcement lease a node holds does NOT matter
+
+Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
+could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
+that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
+for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
+VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
+is a node-local socket. Full measurements and the two misleading probes
+(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.
+
+`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
+separate, pre-existing fault in that service and has nothing to do with the VPN.
+
 ## Revoking a device
 
 Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the