fix(renovate): pin renovate-version — the action's default (39) rejects renovate.json5

Co-Authored-By: Claude Opus 5.5 <[email protected]>

.gitea/workflows/renovate.yaml

@@ -74,12 +74,26 @@ name: renovate
 
 on:
   schedule:
-    # 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
-    # America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
-    # will decline to open PRs outside it. The two are deliberately both present: this cron decides
-    # when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
-    # of the repo window means a run is always in flight when the window opens.
-    - cron: "15 3 * * *"
+    # 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
+    # `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
+    # it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
+    # merely near it.
+    #
+    # WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
+    # run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
+    # 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
+    # Every nightly run would have started, logged "not within schedule", opened zero PRs and
+    # exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
+    # green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
+    # needed updating".
+    #
+    # THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
+    #   EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
+    #   EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
+    # 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
+    # schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
+    # early-morning-UTC value without redoing this arithmetic.
+    - cron: "15 8 * * *"
   workflow_dispatch:
     inputs:
       dry_run:
@@ -138,8 +152,17 @@ jobs:
           # Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
           # includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
           # This is the one dependency in the repo whose updater is its own updater.
-          configurationFile: renovate.json
+          # MUST match the file's real name. #79 renamed it to renovate.json5 and left this line
+          # saying `renovate.json`; every scheduled run from 2026-10-18 then died in ~1s with
+          # ENOENT before Renovate started, and nothing in the gate noticed. ci/lint.sh now
+          # asserts that every `configurationFile:` here names a file that exists.
+          configurationFile: renovate.json5
           token: ${{ secrets.RENOVATE_TOKEN }}
+          # PINNED 2026-10-29. Unset, this action version runs Renovate `39` (action.yml
+          # `renovate-version` default at v41.0.9), i.e. 39.264.0, and renovate.json5 uses
+          # `managerFilePatterns` (>= 40.2.0): under 39.x validation fails with five errors and
+          # the run stops at CONFIG_VALIDATION. #79 validated against `latest`, not this.
+          renovate-version: 44.167.0
         env:
           # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
           # header said were "set at runtime by the operator's renovate runner" — this is that

apps/wave-0/pihole.yaml

@@ -73,7 +73,7 @@ spec:
         - name: FTLCONF_dns_rateLimit_count
           value: '0'
         - name: FTLCONF_misc_dnsmasq_lines
-          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
+          value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
         - name: FTLCONF_webserver_api_password
           valueFrom:
             secretKeyRef:

catalog/pihole.yaml

@@ -143,8 +143,39 @@ env:
   # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
   # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
   # every client that takes the ADVERTISED connection cannot reach the server at all.
+  #
+  # ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
+  # MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
+  #
+  #   dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+  #   dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+  #   dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+  #
+  # dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
+  # bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
+  #   * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
+  #     single lookup, which is maximum exposure to any momentary gap; while
+  #   * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
+  #     upstream (standard forwarder behaviour when the forward target does not answer) it latches
+  #     the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
+  # The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
+  # live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
+  # flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
+  # That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
+  #
+  # 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
+  # re-election (both seconds), short enough that a genuine VIP change still propagates inside a
+  # minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
+  # propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
+  # LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
+  # the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
+  # cache expires, the router falls back, and the public answer latches again. That is a real
+  # outage, and the fix for it is PiHole availability, not DNS caching.
+  #
+  # Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
+  # configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
   - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
+    value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
   # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
   # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
   # carries the full rationale and the measurement that found it.

ci/lint.sh

@@ -27,3 +27,18 @@ fi
 
 yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
 ok "yaml/lint passed (${#files[@]} files)"
+
+# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
+# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
+# while the gate stayed green. Valid YAML, so yamllint cannot see it.
+missing=0
+while IFS=: read -r wf _ value; do
+  value="${value#*configurationFile:}"
+  value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
+  if [ -n "$value" ] && [ ! -e "$value" ]; then
+    err "${wf}: configurationFile '${value}' does not exist in the repo"
+    missing=1
+  fi
+done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
+[ "$missing" -eq 0 ] || exit 1
+ok "workflow configurationFile references resolve"

decisions/2026-10-17-vpn-ingress-l2-lease-collision.md

@@ -1,198 +0,0 @@
----
-date: 2026-10-17
-branch: fix/vpn-ingress-l2-lease-collision
-title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong"
----
-
-## What changed
-
-**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it
-was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is
-not re-derived — it is very easy to re-derive from `ip neigh` alone.
-
-### The claim, and how it died
-
-The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2
-announcement lease it holds itself* — the announcer does not answer its own ARP request, the
-neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because
-`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
-own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded
-that label from announcing, with two alerts and four promtool tests behind it.
-
-The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*`
-leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) —
-the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five
-VIPs and the other two nodes should reach **all**.
-
-**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node:
-
-| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
-|---|---|---|
-| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
-| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms |
-| `nc -z 192.168.2.20 443` | **OK** | OK |
-| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` |
-| `192.168.2.21:8443` | FAIL | **FAIL** (both) |
-
-The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the
-asserted **effect is absent**. That is sufficient to falsify the causal claim without having to
-explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**.
-
-### Why it is inert
-
-No traffic path resolves a LoadBalancer VIP by ARP:
-
-* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
-  `Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath
-  is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP
-  is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**.
-* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's
-  backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet.
-* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
-  which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a
-  bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT,
-  not wire traffic.
-
-### The two probes that misled me
-
-Both were cited as evidence in the previous version of this entry.
-
-* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing
-  artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong.
-* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a
-  host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including
-  the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it
-  discriminates nothing.
-
-**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then
-routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns
-probe cannot measure the forwarded path in either direction — it reports success where the forwarded
-path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not
-sharing a hook, and that is the assumption the original diagnosis rested on.
-
-### Files
-
-Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`,
-`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`,
-`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`,
-`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted:
-`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`.
-
-Kept, rewritten from scratch:
-
-* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is
-  inert and which probes lie, and the internet-egress facts below.
-* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster
-  work, internet does not") that works outwards from the client profile, and a short section saying
-  the L2 lease is a dead end, with the unifi `.21` note.
-
-**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is
-proposed by this branch any more.**
-
-### The internet-routing report: no server-side fix was needed, and none was made
-
-The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full
-one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has
-emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`,
-2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and
-`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel
-feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that
-already exists. **I deliberately shipped no code change here** — writing one would have meant
-inventing a fix for a symptom I could not reproduce (see below).
-
-The server side was measured end to end and is capable and correct, so the cause is in the client
-profile or the client. The triage section names the two candidates that fit the symptom exactly —
-a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and
-leak public traffic outside the tunnel — each with a one-line test that settles it.
-
-## What I verified
-
-Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on
-2026-10-17:
-
-* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1`
-  (`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again.
-* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a
-  real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere.
-* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three
-  nodes — so the successes are service-aware DNAT, not wire delivery.
-* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with
-  `REACHABLE` neighbour entries. This is the measurement that destroys the original control.
-* **The datapath facts** quoted above from `cilium-dbg status --verbose` and
-  `cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`:
-  `kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`,
-  `routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three.
-* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` →
-  `dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP.
-  So the 200 is not loopback delivery by the host stack — it is BPF.
-* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1`
-  (read, never written); `iptables -t nat -S POSTROUTING` carries
-  `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0
-  packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both
-  present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it;
-  `curl http://1.1.1.1/` returns `301` from each node.
-* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by
-  `podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod
-  `egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on
-  `-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the
-  home WAN IP, not PIA's.
-* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned
-  `142.250.113.138`.
-* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one
-  enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both
-  `FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The
-  `wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters.
-* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f`
-  (2026-10-05) and never changed.
-* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts.
-
-## What I did NOT verify
-
-* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could
-  not test what a real remote client does, and — as established above — **no probe runnable from the
-  pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a
-  VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a
-  LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one
-  remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in
-  the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this
-  unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show`
-  shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the
-  client loads a cluster app — and repeat it on a node that holds leases and one that does not.*
-* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same
-  reason. **I need a client connected** to learn it; the router is unreadable from the cluster.
-* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP
-  failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes.
-  My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**,
-  rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A
-  competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and
-  `wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is
-  needed for the falsification** — the asserted cause is present today without its effect. I did not
-  retro-measure anything, and I did not read Cilium's source or issue tracker.
-* **The operator's report cannot be a description of "right now."** No client has connected since
-  the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster
-  services and the LAN but not the internet" must describe a session before the power-cycle or one
-  that has since dropped. **I did not establish when it was observed, or with which profile.**
-* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not**
-  find the fault, and I shipped no fix for it. The two candidates in the triage section (a
-  `--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not
-  measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line
-  decides split vs full and halves the search — and a connected client for step 2.
-* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and
-  lives only on the operator's machine and the device; it is not in the repo, and I did not look for
-  it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing.
-* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without
-  a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU
-  would see large transfers hang while small requests succeed. The runbook's existing MTU note
-  (drop to 1280) already covers the remedy.
-* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole
-  IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a
-  client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an
-  interface with no IPv6 address.
-* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate,
-  pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such.
-* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1
-  offline subset only. Nothing above is covered by a test that runs in CI.
-* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there
-  is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating
-  `kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material).
-* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).

decisions/2026-10-18-pihole-local-ttl.md

@@ -0,0 +1,82 @@
+---
+date: 2026-10-18
+branch: fix/pihole-local-ttl
+title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
+---
+
+## What changed
+
+The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
+PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
+**asymmetry in cacheability** that we created.
+
+Measured on prod 2026-10-18:
+
+```
+dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+```
+
+dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
+LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
+rather than forwarding live. So:
+
+* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
+  lookup, which is maximum exposure to any momentary gap;
+* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
+  (standard forwarder behaviour when the forward target does not answer) it latches the public WAN
+  address and serves it to **every LAN client** for five minutes.
+
+So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
+then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
+That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
+
+Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
+private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
+regenerated with the canonical render command.
+
+## What I verified
+
+* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
+  the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
+* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
+  143 (decrementing), which a pure forwarder could not do.
+* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
+* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
+  address), so a fallback always has a wrong answer available.
+* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
+  `/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
+  standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
+  `FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
+* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
+  --no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
+  deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
+  `apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
+
+## What I did NOT verify
+
+* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
+  nor the router caching the private answer, nor the flap stopping. **Verify after merge with
+  `dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
+  router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
+* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
+  directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
+  records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
+  — so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
+  cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
+* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
+  seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
+  public answer latches again. That is a real availability problem, not a caching one.
+* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
+  (`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
+  two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
+  (tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
+  Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
+  that decision, and it is deliberately **not** in this change.
+* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
+  DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
+  there. Only this one workstation's resolver chain was inspected.
+* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
+  `home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
+  next recurrence would again be found by a human noticing. That gap is not closed here.

decisions/2026-10-18-renovate-cron-window.md

@@ -0,0 +1,78 @@
+---
+date: 2026-10-18
+branch: fix/renovate-cron-window
+title: "The Renovate cron fired outside Renovate's own schedule, so it would never have opened a PR"
+---
+
+## What changed
+
+`.gitea/workflows/renovate.yaml` shipped (PR #73) with `cron: "15 3 * * *"`. **That schedule could
+never have produced a single PR.**
+
+Gitea Actions cron is **UTC**. `renovate.json` carries `"schedule": ["after 3am and before 6am"]`
+with `"timezone": "America/New_York"`, and that inner schedule is **authoritative** — Renovate
+re-checks it and declines to open PRs outside it. So:
+
+```
+cron 03:15 UTC  ->  23:15 EDT, the PREVIOUS EVENING in New York
+window          ->  03:00-06:00 local
+inside?         ->  NO (3h45m before the window opens)
+```
+
+The original comment claimed the cron was "slightly ahead of the repo window ... so a run is always
+in flight when the window opens". It is 3h45m *behind*, on the previous day. Every nightly run would
+have started, logged "not within schedule", opened **zero** PRs and exited **successfully** — a
+green tick forever, indistinguishable from "nothing needed updating". That is the same
+"failure-mode-is-a-green-tick" shape this repo keeps hitting, and it would have hidden the ~30
+pending updates PR #73 was written to surface.
+
+Changed to `cron: "15 8 * * *"`, chosen to survive DST rather than to be tidy. The window's UTC
+position moves:
+
+| | local window | in UTC |
+|---|---|---|
+| EDT (UTC-4, Mar–Nov) | 03:00–06:00 | 07:00–10:00 |
+| EST (UTC-5, Nov–Mar) | 03:00–06:00 | 08:00–11:00 |
+
+`08:15 UTC` is inside **both** (04:15 EDT / 03:15 EST), so no edit is needed at either switchover.
+A cron cannot follow DST, so this is the only way to get it right once.
+
+## What I verified
+
+* **The defect, by arithmetic rather than assertion:** `datetime(2026,10,18,03,15, UTC)` converted to
+  `America/New_York` via `zoneinfo` is `2026-10-17 23:15 EDT`; `3 <= hour < 6` is **False**.
+* **The fix, at both DST extremes:** `08:15 UTC` → `04:15 EDT` in July and `03:15 EST` in December,
+  both inside `03:00–06:00`. Computed, not reasoned.
+* **The inner schedule really is authoritative and really is set:** `renovate.json` on `main` carries
+  `timezone = America/New_York` and `schedule = ['after 3am and before 6am']`.
+* **`RENOVATE_TOKEN` now exists** as a repository Actions secret (listed by name via the Gitea API
+  alongside `SETTINGS_API_TOKEN`), so the workflow's fail-hard preflight will pass. It did not exist
+  when PR #73 was authored.
+* The edited workflow parses as YAML.
+
+## What I did NOT verify
+
+* **Renovate has still never executed.** This fixes *when* it runs, not *whether it works*. No run
+  has happened, so nothing here proves it extracts dependencies, authenticates to Gitea, or opens a
+  usable PR.
+* **`enabledManagers` may silently disable the custom managers — UNRESOLVED and the likelier
+  remaining bug.** `renovate.json` lists `enabledManagers: [..., 'regex', ...]` and defines
+  `customManagers` (correct modern key; `regexManagers` is absent, which is right). But modern
+  Renovate names the custom manager **`custom.regex`** in `enabledManagers`, not bare `regex`. If
+  the bare name is invalid or ignored, every inline `# renovate:` marker in the tree does nothing —
+  including the `external-snapshotter` `newTag` pin PR #73 flagged as probably inert. **I did not
+  change it**, because guessing at a config key that could fail validation and take the whole run
+  down is worse than leaving a question that one dry run answers definitively. The `dry_run: true`
+  dispatch log settles it: look for a per-manager dependency-extraction count, and for any config
+  validation warning naming `regex`.
+* **Whether a dry run inside vs outside the window differs usefully.** Renovate honours its schedule
+  even in dry-run, so a manual dispatch outside 03:00–06:00 local will also log "not within
+  schedule". That makes the first dispatch's output less informative than it looks; to see what
+  Renovate *would* do, either dispatch inside the local window or override the schedule for that
+  run. Not tested either way.
+* **No alert fires if the schedule stops firing.** PR #73 recorded this gap and it is still open: a
+  Renovate workflow that silently stops running (disabled Actions, expired token, deleted secret)
+  produces no signal at all. Closing it needs a check on last-successful-run age, which nothing
+  does.
+* No conformance test asserts the cron falls inside `renovate.json`'s window, so this exact bug can
+  return. A static check comparing the two is feasible and is not written.

decisions/2026-10-18-renovate-invalid-config.md

@@ -0,0 +1,78 @@
+---
+date: 2026-10-18
+branch: fix/renovate-invalid-config
+title: "renovate.json was invalid, so the first manual run errored out"
+---
+
+## What changed
+
+The operator ran the Renovate workflow manually and it **failed**. The cause is not the workflow:
+**`renovate.json` was never a valid Renovate config.**
+
+It carried five pseudo-comment keys — `_comment`, `_kubernetes_comment`, `_unmanageable_comment`,
+`_ignorePaths_comment`, `_docker_comment` — because JSON has no comments. Renovate **rejects unknown
+top-level options**, so the entire config failed validation. Proven with the official validator
+rather than inferred:
+
+```
+$ podman run --rm -v "$PWD/renovate.json:/tmp/renovate.json:ro,Z" \
+    ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json
+ERROR: Found errors in configuration
+  Invalid configuration option: _comment
+  Invalid configuration option: _docker_comment
+  Invalid configuration option: _ignorePaths_comment
+  Invalid configuration option: _kubernetes_comment
+  Invalid configuration option: _unmanageable_comment
+```
+
+This is the second independent reason Renovate had never worked, on top of PR #77's cron firing
+outside `renovate.json`'s own schedule window. Either alone was sufficient to produce "nothing ever
+happens"; #73 shipped both.
+
+**Converted `renovate.json` → `renovate.json5`.** JSON5 permits real `//` comments, and Renovate
+discovers `renovate.json5` natively — nothing in `.gitea/` or `ci/` pins the old filename (only
+prose inside comments), and no `RENOVATE_CONFIG_FILE` is set. Every word of the original rationale
+is preserved as comments positioned next to what it explains, which a flattened `description`
+string would have lost.
+
+A header on the new file records the failure, the validator command, and the instruction to run it
+before committing — this class of bug is invisible to the repo's own gate.
+
+## What I verified
+
+* **The exact failure**, from the official validator, quoted above. Not read off a log, not inferred
+  from the workflow.
+* **The fix validates**: the same validator against `renovate.json5` returns
+  `INFO: Config validated successfully against 1 file(s)`.
+* **Nothing was dropped in the conversion**, checked programmatically against `git show
+  HEAD:renovate.json`: 18 non-comment keys before, 18 after, **no missing keys, no extra keys, and
+  no key whose value changed**. `docker: {enabled: true}` confirmed still present — the validator
+  prints a normalisation diff that mentions it, which is its own migration suggestion, not a
+  removal.
+* **74 comment lines survive** in the new file, so the rationale is intact.
+* **`RENOVATE_TOKEN` exists** as a repository Actions secret (listed by name via the Gitea API), so
+  the workflow's fail-hard preflight is satisfied.
+* **A suspicion was DISPROVED:** `enabledManagers` containing bare `regex` was thought to need to be
+  `custom.regex`, which would have silently disabled every inline `# renovate:` marker. The
+  validator accepts `regex`, so that concern was unfounded and nothing here changes it.
+* Nothing functionally references `renovate.json` by name — `grep -rn "renovate\.json" .gitea/ ci/
+  conformance/` returns only comment prose.
+
+## What I did NOT verify
+
+* **Renovate has still never completed a run.** This removes one of two blockers; PR #77 removes the
+  other. Neither is merged, so **no successful Renovate run has been observed** and I cannot say the
+  bot works — only that these two specific defects are fixed.
+* **I did not re-run the workflow after the fix.** The honest test is a `workflow_dispatch` with
+  `dry_run: true` once both this and #77 are merged. Expect ~30 dependencies behind, rate-limited to
+  `prHourlyLimit: 2` / `prConcurrentLimit: 5`.
+* **Whether the managers actually extract anything.** A valid config that matches no files is still
+  a no-op. The dry-run log's per-manager dependency counts are the only thing that proves the
+  `managerFilePatterns` are right, and I have not seen them.
+* **Whether `renovate.json5` is honoured by the pinned Renovate version in the workflow.** JSON5
+  discovery is long-standing and the validator accepted the file, but the workflow's image tag was
+  not cross-checked against the version that added it.
+* **Nothing in the gate catches an invalid Renovate config.** This bug would have recurred silently.
+  A one-line L0 check shelling out to `renovate-config-validator` would close it; it is not written
+  here, and `conformance/**` is a sensitive path.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

decisions/2026-10-29-renovate-config-filename.md

@@ -0,0 +1,57 @@
+---
+date: 2026-10-29
+branch: fix/renovate-config-filename
+title: Renovate has failed every run since #79 — the workflow still named renovate.json
+---
+
+## What changed
+
+**`.gitea/workflows/renovate.yaml`** — `configurationFile: renovate.json` → `renovate.json5`.
+
+PR #79 (`fix/renovate-invalid-config`) renamed the config to `renovate.json5` and did not update
+the one line in the workflow that names it. Measured 2026-10-29 from the Gitea Actions log of run
+322 (task 513, schedule, 08:15 UTC):
+
+    Error: ENOENT: no such file or directory, stat '/workspace/Home-Network/home-network-k3s-impl/renovate.json'
+
+Runs 315–322 (2026-10-22 → 10-29, every one listed) are all `failure` on the `schedule` event,
+each failing in about one second. Renovate never started. `RENOVATE_TOKEN` **does** exist as a
+repository Actions secret (listed by name via the API; value not read), so the token preflight
+passes and this filename is the only blocker observed.
+
+**`ci/lint.sh`** — a new assertion after yamllint: every non-comment `configurationFile:` key in
+`.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see
+it, and the gate stayed green through 11 days of failures.
+
+**Second blocker, same PR:** the workflow did not pin a Renovate version, so
+`renovatebot/[email protected]` ran its default, Renovate **39** (`action.yml`:
+`renovate-version` `default: '39'`, read at that tag). `renovate.json5` uses `managerFilePatterns`,
+which needs >= 40.2.0. Under 39.264.0 the validator rejects the config with five errors. This was
+found and reproduced by the image-pipeline worker (PR #81). #79 had validated against `latest`. Pinned
+`renovate-version: 44.167.0`, the version the config validates clean against per that worker.
+
+## What I verified
+
+- The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and
+  no `renovate.json` (`git ls-tree`).
+- The new guard, both ways, run standalone: on this branch it prints
+  `workflow configurationFile references resolve`, rc 0. With the line reverted to `renovate.json`
+  it prints `configurationFile 'renovate.json' does not exist in the repo`, rc 1. My first version
+  also matched its own explanatory comment in the workflow and failed. That is why the grep is
+  anchored to non-comment lines.
+- Toolbox gate: result and rc are reported in the PR, not predicted here.
+
+## What I did NOT verify
+
+- **The 39.x rejection and the 44.167.0 pass were not reproduced by me.** I confirmed only the
+  action's default (`'39'`). The validation results are the PR #81 worker's.
+- **Renovate has still never completed a run.** This removes the observed blocker. Others may
+  follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is
+  the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`.
+- **The `enabledManagers: ["regex"]` vs `custom.regex` question** (#73, #77) is still open. The
+  same dry-run log answers it: look for the regex/custom manager's dependency-extraction count.
+- **No alert for "Renovate keeps failing".** The guard prevents this specific cause from recurring.
+  Nothing notices a scheduled workflow that is red every day. That gap stays open.
+- **Sensitive path.** `.gitea/**` and `ci/**` are both in `approval-policy.json`. CI will block
+  on `approval-policy` until the operator posts `/approve-override` as a review. The local gate
+  reports that stage as N/A.

platform/vpn-ingress/README.md

@@ -186,86 +186,6 @@ right way; one was a defect in my own manifest.
 | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
 | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
 
-## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach
-
-A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose
-Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from
-announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is
-real but inert, and the distinction matters enough to record, because the wrong version is very
-easy to re-derive from `ip neigh` alone.
-
-Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case,
-after the UPS power-cycle — from the host netns of the `wireguard` pod on each node:
-
-| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
-|---|---|---|
-| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
-| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms |
-| `nc -z 192.168.2.20 443` | **OK** | OK |
-| plex `192.168.2.24:32400` | **`200`** | `200` |
-
-So the announcer really does **not** answer its own ARP request — that part of the observation is
-reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**:
-
-* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
-  `Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**,
-  so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before
-  any route or neighbour lookup for the VIP happens**.
-* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the
-  shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without
-  a packet crossing the LAN at all.
-* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
-  which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real**
-  service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes —
-  that is service-aware DNAT, not wire traffic.
-
-**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis:
-
-* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an
-  artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not
-  anything is wrong.
-* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the
-  two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it
-  discriminates nothing — and it is not the client path either (see below).
-
-**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are
-**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is
-**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the
-forwarded path in either direction: it reports success where the forwarded path might fail, and
-`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with
-a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE`
-counters move.
-
-## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses
-
-`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the
-component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature
-to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel
-client cannot reach the internet, nothing in the list below is the reason:
-
-| checked | result on all three nodes |
-|---|---|
-| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` |
-| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change |
-| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE |
-| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present |
-| default route | `default via 192.168.2.1 dev eno1 onlink`, intact |
-| node's own internet reachability | `http://1.1.1.1/` returns `301` |
-
-**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is
-closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by
-`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled
-`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity
-traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway
-node. A remote client's apparent public address is therefore the **home WAN IP**, which is the
-intended behaviour and the thing to state when enrolling a device.
-
-**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one
-enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and
-the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the
-power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the
-forwarded node shows a peer with an `endpoint` and a recent `latest handshake`.
-
 ## Prerequisites outside this directory
 
 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`

renovate.json

@@ -1,99 +0,0 @@
-{
-  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
-  "_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
-  "extends": [
-    "config:recommended",
-    ":dependencyDashboard",
-    ":semanticCommits",
-    "helpers:pinGitHubActionDigests"
-  ],
-  "timezone": "America/New_York",
-  "schedule": ["after 3am and before 6am"],
-  "prConcurrentLimit": 5,
-  "prHourlyLimit": 2,
-  "commitMessagePrefix": "chore(deps):",
-  "labels": ["dependencies", "automated-update"],
-  "ignoreUnstable": true,
-  "respectLatest": true,
-  "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
-  "flux": {
-    "managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
-  },
-  "kustomize": {
-    "managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
-  },
-  "_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
-  "kubernetes": {
-    "managerFilePatterns": [
-      "/(^|/)platform/.+\\.ya?ml$/",
-      "/(^|/)platform-config/.+\\.ya?ml$/",
-      "/(^|/)apps/.+\\.ya?ml$/",
-      "/(^|/)catalog/.+\\.ya?ml$/",
-      "/(^|/)runbooks/serving/.+\\.ya?ml$/",
-      "/(^|/)infra/agent/.+\\.ya?ml$/"
-    ]
-  },
-  "customManagers": [
-    {
-      "customType": "regex",
-      "description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
-      "managerFilePatterns": [
-        "/(^|/)kustomization\\.ya?ml$/",
-        "/(^|/)versions\\.env$/"
-      ],
-      "matchStrings": [
-        "#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
-      ],
-      "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
-    }
-  ],
-  "packageRules": [
-    {
-      "description": "Auto-merge the allowed low-risk classes on green CI — no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
-      "matchUpdateTypes": ["minor", "patch", "digest"],
-      "automerge": true,
-      "automergeType": "pr",
-      "platformAutomerge": true
-    },
-    {
-      "description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
-      "matchUpdateTypes": ["major"],
-      "automerge": false,
-      "addLabels": ["needs-review", "breaking"]
-    },
-    {
-      "description": "Keep the KubeVirt + CDI operator pair on a compatible track — grouped so they bump together (platform/kubevirt/kustomization.yaml).",
-      "matchDepNames": ["kubevirt/kubevirt", "kubevirt/containerized-data-importer"],
-      "groupName": "kubevirt-cdi"
-    },
-    {
-      "description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface — label but do not auto-merge; a version bump there changes CI reproducibility.",
-      "matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
-      "automerge": false,
-      "addLabels": ["ci-tooling"]
-    },
-    {
-      "description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
-      "matchPackageNames": [
-        "kured",
-        "longhorn",
-        "cilium",
-        "cloudnative-pg",
-        "kube-prometheus-stack"
-      ],
-      "automerge": false,
-      "addLabels": ["needs-review", "blast-radius"]
-    }
-  ],
-  "_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
-  "ignorePaths": [
-    "**/*.example",
-    "**/*.sops.yaml",
-    "conformance/**",
-    "platform/image-automation/per-service/**",
-    "platform/kyverno/tests/**"
-  ],
-  "_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
-  "_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
-  "docker": { "enabled": true }
-}

renovate.json5

@@ -0,0 +1,187 @@
+// =================================================================================================
+// Renovate configuration — JSON5, so the rationale can live NEXT TO what it explains.
+// =================================================================================================
+// THIS FILE WAS `renovate.json` AND IT WAS INVALID. It carried five `_comment` / `_*_comment`
+// pseudo-comment keys, because JSON has no comments. Renovate REJECTS unknown top-level options, so
+// the whole config failed validation and the first manual run of the workflow ERRORED OUT. Proven
+// with the official validator (`ghcr.io/renovatebot/renovate renovate-config-validator`):
+//
+//   ERROR: Found errors in configuration
+//     Invalid configuration option: _comment
+//     Invalid configuration option: _docker_comment
+//     Invalid configuration option: _ignorePaths_comment
+//     Invalid configuration option: _kubernetes_comment
+//     Invalid configuration option: _unmanageable_comment
+//
+// Renovate discovers `renovate.json5` natively (no RENOVATE_CONFIG_FILE needed, and nothing in
+// .gitea/ or ci/ pins the old filename — only prose in comments), and JSON5 permits real `//`
+// comments. So the fix keeps every word of the original rationale, positioned, instead of
+// flattening it into a `description` string.
+//
+// VALIDATE BEFORE COMMITTING — it is one command and it is the only thing that catches this class:
+//   podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
+//     ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5
+//
+// NOTE on what was NOT wrong: `enabledManagers` containing bare `regex` was suspected of needing to
+// be `custom.regex`. The validator accepts `regex`, so that suspicion was unfounded and nothing
+// here changes it.
+{  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
+  // Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17
+  // RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags.
+  // It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009
+  // split that gave image tags to Flux image-automation is formally closed — see
+  // .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than
+  // opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with
+  // user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write
+  // anyway; and enabling it would need an in-cluster git WRITE credential for the repo that
+  // defines the cluster). image-automation has been commented out at
+  // platform/kustomization.yaml:52 since it was authored and the cluster holds zero
+  // ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down.
+  // CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid
+  // fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag`
+  // line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth
+  // sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER:
+  // .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.
+  "extends": ["config:recommended", ":dependencyDashboard", ":semanticCommits", "helpers:pinGitHubActionDigests"],
+  "timezone": "America/New_York",
+  "schedule": ["after 3am and before 6am"],
+  "prConcurrentLimit": 5,
+  "prHourlyLimit": 2,
+  "commitMessagePrefix": "chore(deps):",
+  "labels": ["dependencies", "automated-update"],
+  "ignoreUnstable": true,
+  "respectLatest": true,
+  "enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
+  "flux": {
+    "managerFilePatterns": [
+      "/(^|/)platform/.+\\.ya?ml$/",
+      "/(^|/)apps/.+\\.ya?ml$/"
+    ]
+  },
+  "kustomize": {
+    "managerFilePatterns": [
+      "/(^|/)kustomization\\.ya?ml$/"
+    ]
+  },
+  // ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes
+  // manifests, and it is NOT auto-detected — it only looks at files matched here, which is why
+  // this block is as broad as it is. Scope covers every place this repo writes a container image:
+  // platform components, the rendered app waves, the ServiceDefinition catalog that generates
+  // them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as
+  // `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the
+  // rendered output is reverted by the next render and would also fail ci/render-golden.sh.
+  // Renovate will raise BOTH, and both must land together.
+  "kubernetes": {
+    "managerFilePatterns": [
+      "/(^|/)platform/.+\\.ya?ml$/",
+      "/(^|/)platform-config/.+\\.ya?ml$/",
+      "/(^|/)apps/.+\\.ya?ml$/",
+      "/(^|/)catalog/.+\\.ya?ml$/",
+      "/(^|/)runbooks/serving/.+\\.ya?ml$/",
+      "/(^|/)infra/agent/.+\\.ya?ml$/"
+    ]
+  },
+  "customManagers": [
+    {
+      "customType": "regex",
+      "description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
+      "managerFilePatterns": [
+        "/(^|/)kustomization\\.ya?ml$/",
+        "/(^|/)versions\\.env$/"
+      ],
+      "matchStrings": [
+        "#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
+      ],
+      "versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
+    }
+  ],
+  "packageRules": [
+    {
+      "description": "Auto-merge the allowed low-risk classes on green CI \u2014 no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
+      "matchUpdateTypes": [
+        "minor",
+        "patch",
+        "digest"
+      ],
+      "automerge": true,
+      "automergeType": "pr",
+      "platformAutomerge": true
+    },
+    {
+      "description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
+      "matchUpdateTypes": [
+        "major"
+      ],
+      "automerge": false,
+      "addLabels": [
+        "needs-review",
+        "breaking"
+      ]
+    },
+    {
+      "description": "Keep the KubeVirt + CDI operator pair on a compatible track \u2014 grouped so they bump together (platform/kubevirt/kustomization.yaml).",
+      "matchDepNames": [
+        "kubevirt/kubevirt",
+        "kubevirt/containerized-data-importer"
+      ],
+      "groupName": "kubevirt-cdi"
+    },
+    {
+      "description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface \u2014 label but do not auto-merge; a version bump there changes CI reproducibility.",
+      "matchFileNames": [
+        "ci/versions.env",
+        "infra/devcontainer/versions.env"
+      ],
+      "automerge": false,
+      "addLabels": [
+        "ci-tooling"
+      ]
+    },
+    {
+      "description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check \u2014 it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
+      "matchPackageNames": [
+        "kured",
+        "longhorn",
+        "cilium",
+        "cloudnative-pg",
+        "kube-prometheus-stack"
+      ],
+      "automerge": false,
+      "addLabels": [
+        "needs-review",
+        "blast-radius"
+      ]
+    }
+  ],
+  // WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
+  // FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it
+  // and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest,
+  // lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak
+  // (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete
+  // version before any automation can maintain them, and until then they drift on every pull with
+  // nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12,
+  // e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it
+  // has been told not to leave; raising those ceilings is deliberate per-component work and is
+  // NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.
+  // platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest`
+  // and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed
+  // shape, not software to keep current. `conformance/**` was already ignored for the same
+  // reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise
+  // have opened.
+  "ignorePaths": [
+    "**/*.example",
+    "**/*.sops.yaml",
+    "conformance/**",
+    "platform/image-automation/per-service/**",
+    "platform/kyverno/tests/**"
+  ],
+  // The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
+  // BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
+  // over a running tag. That controller was never enabled and never will be here (see _comment),
+  // so the only thing those two flags were achieving was leaving every plain `image:` tag in the
+  // repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
+  // Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
+  "docker": {
+    "enabled": true
+  }
+}

runbooks/vpn-remote-access.md

@@ -273,84 +273,6 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
 by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
 if the router is not delivering the packets, and a `handshake` with no completed session if it is.
 
-## Triage — "the client reaches the LAN and cluster apps, but not the internet"
-
-Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
-almost always lives, and because the server side has been measured and is not the cause.
-
-**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**
-
-| the profile says | meaning |
-|---|---|
-| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
-| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |
-
-`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
-`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
-profile only**, so an already-issued profile keeps whatever it was rendered with until it is
-re-rendered and re-imported on the device. The two forms differ by one flag:
-
-```bash
-./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>'           # full tunnel
-./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split'   # LAN only
-```
-
-Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
-`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
-editing `AllowedIPs` in the profile **on the device** — the server does not care.
-
-**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
-client connected, then watch the two counters that only forwarded traffic moves:
-
-```bash
-./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
-     wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
-     iptables -t nat -L POSTROUTING -n -v | head -5"'
-```
-
-* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
-  node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
-* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
-  upstream of the cluster (router forward, client, or carrier), not here.
-* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
-  failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
-  which breaks internet while node-local cluster VIPs still work. It has not been observed.
-
-**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:
-
-* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
-  client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
-  so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
-  `curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
-  and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
-  blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
-* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
-  2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
-  blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
-  a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.
-
-**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
-`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
-**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
-`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
-PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
-`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
-**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).
-
-## Triage — which L2 announcement lease a node holds does NOT matter
-
-Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
-could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
-that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
-for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
-VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
-is a node-local socket. Full measurements and the two misleading probes
-(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.
-
-`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
-separate, pre-existing fault in that service and has nothing to do with the VPN.
-
 ## Revoking a device
 
 Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the