revert(vpn-ingress): withdraw the L2 lease-collision fix — a node CAN reach a VIP it announces
PR #69 claimed a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
lease it holds, and excluded the VPN ingress node from announcing on that basis.
The claim makes a sharp prediction; today's power-cycle put ALL EIGHT
cilium-l2announce-* leases on k3s-prod-1, and the prediction failed.
Measured 2026-10-17 from the wireguard pod's host netns on each node: the holder
reaches 192.168.2.20/.22/.23/.24 normally (curl 200 in 18ms on .20) while its
ip neigh for every one of them sits at FAILED. The asserted cause is present
without the asserted effect, which falsifies it. .21 (unifi) fails from all three
nodes and is a separate pre-existing fault.
It is inert because no path resolves a VIP by ARP: cilium-dbg reports
KubeProxyReplacement on [eno1, wg0] and programs 192.168.2.20:443 =>
127.0.0.1:13606, so VIPs are DNAT'd in BPF before any neighbour lookup, on the
forwarded path as well as the host one.
Both probes behind the original diagnosis were bad. `ip neigh` FAILED is
permanent and harmless, and a ping artefact. `curl --interface wg0` — the control
used to rule out a host-netns artefact — fails on all three nodes, including ones
whose ARP resolves, so it discriminates nothing. The deeper error: a MASQUERADEd
client is FORWARDED while every pod-runnable probe is socket-originated. Sharing
a netns is not sharing a hook.
Reverted the policy nodeSelector, the Ansible node label, both alerts, their
promtool tests and the kube-state-metrics allowlist entry. Kept documentation
only: the measurements, which probes lie, and the fact that the forwarded path
remains unmeasured without a connected client.
Also recorded, for the reported "LAN and cluster work, internet does not":
new-peer.sh has emitted a FULL tunnel (0.0.0.0/0) by default since f92707f, so
there is no full-tunnel option to add and no code change is made. The server side
is capable — ip_forward=1, the 10.90.0.0/24 MASQUERADE present with an empty
CILIUM_POST_nat ahead of it, FORWARD accepts, default route intact, nodes reach
1.1.1.1. A full tunnel egresses the HOUSE WAN, not PIA: the egress policy selects
pods by egress-class=vpn, this pod is labelled internet, and tun0 exists on
k3s-prod-3 only. Root cause NOT found; no client has connected since the
power-cycle (0 forwarded packets, no handshake), so it is not reproducible here.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
4656b07178
commit
e82dab5df5
13 files changed
+338
-871
No files matched your search
@@ -1,137 +1,198 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: fix/vpn-ingress-l2-lease-collision
|
||||
title: "A node cannot reach a VIP it announces — so the VPN ingress node must announce nothing"
|
||||
title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The first device enrolled on the remote-access VPN handshook, resolved names, and could not open a
|
||||
single connection to any cluster web app. The tunnel was not at fault. The fault is a property of
|
||||
Cilium L2 announcement that nothing in this repo knew about:
|
||||
**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it
|
||||
was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is
|
||||
not re-derived — it is very easy to re-derive from `ip neigh` alone.
|
||||
|
||||
**A node cannot reach a LoadBalancer VIP whose `CiliumL2AnnouncementPolicy` lease it holds itself.**
|
||||
The announcer answers ARP for the VIP on behalf of other LAN hosts but not for its own request, so
|
||||
on the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP is dropped
|
||||
before it leaves the node.
|
||||
### The claim, and how it died
|
||||
|
||||
`platform/vpn-ingress` is uniquely exposed because it runs WireGuard with `hostNetwork: true` and
|
||||
MASQUERADEs remote clients to the node's own address — a VPN client's packets take the node's LAN
|
||||
path and inherit its ARP failures. Ordinary pods never do. The router's `udp/51820` forward pointed
|
||||
at `k3s-prod-1`, which held `cilium-l2announce-gateway-cilium-gateway-shared`, so **all 16 `shared`
|
||||
Gateway hostnames were dead over the VPN** — while PiHole DNS kept answering, because
|
||||
`192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. Names resolved
|
||||
and every app hung on connect.
|
||||
The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2
|
||||
announcement lease it holds itself* — the announcer does not answer its own ARP request, the
|
||||
neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because
|
||||
`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
|
||||
own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded
|
||||
that label from announcing, with two alerts and four promtool tests behind it.
|
||||
|
||||
Seven files:
|
||||
The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*`
|
||||
leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) —
|
||||
the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five
|
||||
VIPs and the other two nodes should reach **all**.
|
||||
|
||||
* **`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`** — new
|
||||
`node_baseline_vpn_ingress_node`; the matching node gets
|
||||
`homelab.keithjackson.dev/vpn-ingress=true`, and a second task **removes** the label from any node
|
||||
that no longer matches, so moving the value moves the label rather than adding a second one. Same
|
||||
shape as `roles/vpn_egress_gateway`'s `egress-gateway` label. Tagged `vpn-ingress`; skipped
|
||||
entirely when the value is empty.
|
||||
* **`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`** — set to `k3s-prod-1`, with a
|
||||
`[parity]` note that it must equal the router's forward.
|
||||
* **`platform-config/cilium/l2-announcement-policy.yaml`** — `nodeSelector` becomes a
|
||||
`matchExpressions` list keeping `kubernetes.io/os: linux` and adding
|
||||
`homelab.keithjackson.dev/vpn-ingress NotIn ["true"]`. The VPN node then holds no lease, so it
|
||||
ARP-resolves every VIP. This fixes **all** VIPs for VPN clients (Gateway, crafty, plex, pihole,
|
||||
unifi), not just the one that happened to collide.
|
||||
* **`platform/monitoring/kube-prometheus-stack.yaml`** — the label added to kube-state-metrics'
|
||||
`metricLabelsAllowlist`, which is what lets the invariant be watched at all.
|
||||
* **`platform/monitoring/alerts.yaml`** — `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) joins
|
||||
`kube_lease_owner` against `kube_node_labels` via `label_replace`, plus
|
||||
`VpnIngressNodeLabelMissing` (warning, 30m) as its watchdog.
|
||||
* **`platform/monitoring/tests/vpn-ingress-l2.test.yaml`** — four promtool cases: fires on the real
|
||||
collision, quiet in the fixed state, quiet across a transient 11-minute lease move (pinning the
|
||||
`for: 15m`), and the label-absent case asserting the critical alert is inert *and* the watchdog
|
||||
fires.
|
||||
* **`runbooks/vpn-ingress-l2-collision.md`** (new, registered in `runbooks/kustomization.yaml`),
|
||||
**`runbooks/vpn-remote-access.md`**, **`platform/vpn-ingress/{daemonset.yaml,README.md}`** — step 3
|
||||
of the setup runbook said *"pick any node — all three run an identical listener"* and *"node loss =
|
||||
failover is this one line"*. Both were false and the first caused this. Failover is now documented
|
||||
as two ordered changes.
|
||||
**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node:
|
||||
|
||||
**Design note — why a node label and not a `cluster-vars` knob.** The first attempt added
|
||||
`WG_INGRESS_NODE` to `clusters/base/cluster-vars.yaml` and substituted it into both the policy and
|
||||
the alert. The alert half is **impossible**: `platform/monitoring/tests/extract_rules.py` rejects any
|
||||
alert whose `expr` contains a `${...}` placeholder, because the rule it would unit-test is not the
|
||||
rule deployed. Rather than duplicate the node name into the alert as a literal (two sources that can
|
||||
skew), the label is readable from both sides — the selector matches on it and `kube_node_labels`
|
||||
carries it into Prometheus. One source of truth, and as a side effect the change touches no
|
||||
sensitive path, so it merges on a green gate rather than needing a human review.
|
||||
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|
||||
|---|---|---|
|
||||
| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
|
||||
| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms |
|
||||
| `nc -z 192.168.2.20 443` | **OK** | OK |
|
||||
| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` |
|
||||
| `192.168.2.21:8443` | FAIL | **FAIL** (both) |
|
||||
|
||||
The DaemonSet deliberately gained **no** probe for this. A readinessProbe on VIP reachability would
|
||||
wedge a rollout behind an unrelated Gateway outage (`maxUnavailable: 1`); a liveness probe would tear
|
||||
down a tunnel carrying LAN traffic correctly.
|
||||
The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the
|
||||
asserted **effect is absent**. That is sufficient to falsify the causal claim without having to
|
||||
explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**.
|
||||
|
||||
### Why it is inert
|
||||
|
||||
No traffic path resolves a LoadBalancer VIP by ARP:
|
||||
|
||||
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
|
||||
`Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath
|
||||
is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP
|
||||
is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**.
|
||||
* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's
|
||||
backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet.
|
||||
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
|
||||
which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a
|
||||
bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT,
|
||||
not wire traffic.
|
||||
|
||||
### The two probes that misled me
|
||||
|
||||
Both were cited as evidence in the previous version of this entry.
|
||||
|
||||
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing
|
||||
artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong.
|
||||
* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a
|
||||
host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including
|
||||
the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it
|
||||
discriminates nothing.
|
||||
|
||||
**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then
|
||||
routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns
|
||||
probe cannot measure the forwarded path in either direction — it reports success where the forwarded
|
||||
path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not
|
||||
sharing a hook, and that is the assumption the original diagnosis rested on.
|
||||
|
||||
### Files
|
||||
|
||||
Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`,
|
||||
`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`,
|
||||
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`,
|
||||
`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted:
|
||||
`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`.
|
||||
|
||||
Kept, rewritten from scratch:
|
||||
|
||||
* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is
|
||||
inert and which probes lie, and the internet-egress facts below.
|
||||
* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster
|
||||
work, internet does not") that works outwards from the client profile, and a short section saying
|
||||
the L2 lease is a dead end, with the unifi `.21` note.
|
||||
|
||||
**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is
|
||||
proposed by this branch any more.**
|
||||
|
||||
### The internet-routing report: no server-side fix was needed, and none was made
|
||||
|
||||
The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full
|
||||
one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has
|
||||
emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`,
|
||||
2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and
|
||||
`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel
|
||||
feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that
|
||||
already exists. **I deliberately shipped no code change here** — writing one would have meant
|
||||
inventing a fix for a symptom I could not reproduce (see below).
|
||||
|
||||
The server side was measured end to end and is capable and correct, so the cause is in the client
|
||||
profile or the client. The triage section names the two candidates that fit the symptom exactly —
|
||||
a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and
|
||||
leak public traffic outside the tunnel — each with a one-line test that settles it.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured live against prod, `--context k3s-prod`, on 2026-10-08:
|
||||
Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on
|
||||
2026-10-17:
|
||||
|
||||
* **The whole tunnel is healthy**, which is what made this hard: `wg show` on `k3s-prod-1` shows the
|
||||
peer with a real endpoint and bidirectional transfer (346 KiB in / 200 KiB out);
|
||||
`net.ipv4.ip_forward=1`; `wg0` up at `10.90.0.1/24` MTU 1420; `iptables -t nat -S POSTROUTING`
|
||||
carries the `MASQUERADE` rule with 26 packets; `FORWARD -i/-o wg0` ACCEPT with 108 / 171 packets;
|
||||
DaemonSet generation 2, 3/3 ready on the content-hashed `wg-peers-m9kcckcb5t`.
|
||||
* **The ARP failure and its symmetry.** On `k3s-prod-1`: `ip neigh show 192.168.2.20` → `FAILED`,
|
||||
while `.21`/`.23` resolve to prod-2's MAC `04:0e:3c:96:b0:94`. On prod-2/prod-3, `.20` resolves to
|
||||
prod-1's `bc:e9:2f:7e:c4:a6`. TCP agrees: `.20:443` and `.20:80` FAIL from prod-1, OK from the
|
||||
other two. Repeated across 3 nodes × 5 VIPs — each node reached every VIP except the ones it alone
|
||||
announced (prod-2 ✗ `.22`/`.21`; prod-3 ✗ `.24`).
|
||||
* **Not an artefact of probing from the host netns.** `curl --interface wg0` from prod-1 fails the
|
||||
same way, and a neighbour entry is node-global state, so the forwarded path cannot differ in ARP
|
||||
resolution.
|
||||
* **The Gateway itself was never broken:** from the workstation (192.168.2.133)
|
||||
`home.keithjackson.dev` → `200`, `plex.home.keithjackson.dev` → `401`, both at `192.168.2.20`.
|
||||
From prod-1 the router (`.1`) and TrueNAS (`.30`) are `REACHABLE`.
|
||||
* **`NotIn` semantics, empirically** — the basis for the fail-back claim in the policy's comment:
|
||||
`kubectl get nodes -l 'kubernetes.io/hostname notin (k3s-prod-1)'` returns prod-2 and prod-3, and
|
||||
`NotIn` on a key no node carries returns **all three**. So an unapplied label excludes nothing and
|
||||
degrades to the old behaviour rather than taking every VIP off the LAN.
|
||||
* **The alert metric is really scraped:** `kube_lease_owner` is served by kube-state-metrics (55
|
||||
series, every `cilium-l2announce-*` lease with its holder in `lease_holder`), and `kube_node_labels`
|
||||
already carries the two `egress-*` labels through the same allowlist (`k3s-prod-3` shows
|
||||
`egress-gateway=true`), so the third entry follows a proven path.
|
||||
* **The four promtool cases pass** (`platform/monitoring/tests/run.sh` → `SUCCESS`, 55 rules
|
||||
checked), including that `VpnIngressNodeAnnouncesL2VIP` is inert when the label is absent.
|
||||
* All edited YAML parses; the rendered `nodeSelector` is the intended two-expression list.
|
||||
* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1`
|
||||
(`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again.
|
||||
* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a
|
||||
real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere.
|
||||
* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three
|
||||
nodes — so the successes are service-aware DNAT, not wire delivery.
|
||||
* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with
|
||||
`REACHABLE` neighbour entries. This is the measurement that destroys the original control.
|
||||
* **The datapath facts** quoted above from `cilium-dbg status --verbose` and
|
||||
`cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`:
|
||||
`kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`,
|
||||
`routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three.
|
||||
* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` →
|
||||
`dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP.
|
||||
So the 200 is not loopback delivery by the host stack — it is BPF.
|
||||
* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1`
|
||||
(read, never written); `iptables -t nat -S POSTROUTING` carries
|
||||
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0
|
||||
packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both
|
||||
present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it;
|
||||
`curl http://1.1.1.1/` returns `301` from each node.
|
||||
* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by
|
||||
`podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod
|
||||
`egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on
|
||||
`-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the
|
||||
home WAN IP, not PIA's.
|
||||
* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned
|
||||
`142.250.113.138`.
|
||||
* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one
|
||||
enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both
|
||||
`FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The
|
||||
`wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters.
|
||||
* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f`
|
||||
(2026-10-05) and never changed.
|
||||
* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **The fix is UNAPPLIED and therefore UNPROVEN end to end.** Applying it needs a mutating
|
||||
`kubectl`/`flux reconcile` plus an Ansible run, which are operator actions (CLAUDE.md §5). I have
|
||||
**not** observed `k3s-prod-1` resolve `192.168.2.20` after the exclusion lands, nor a VPN client
|
||||
reaching an app. The prediction rests on the measured symmetry (non-announcing nodes resolve every
|
||||
VIP, 15/15 data points), not on a test of the changed state. **Verify after merge, in this order:**
|
||||
re-run `node_baseline --tags vpn-ingress`; `kubectl get nodes -L
|
||||
homelab.keithjackson.dev/vpn-ingress` (exactly one node); `kubectl -n kube-system get lease | grep
|
||||
l2announce` (no holder is that node); `ip neigh show 192.168.2.20` from the prod-1 wireguard pod
|
||||
(expect a `lladdr`, not `FAILED`); then a client.
|
||||
* **That Cilium actually releases a lease when its holder stops matching the `nodeSelector`.** I
|
||||
assumed re-election and did not test it. If Cilium keeps renewing, the affected Leases must be
|
||||
deleted once by hand to force an election. Check this first if the fix appears not to work.
|
||||
* **The brief interruption on apply.** Moving the Gateway and `pihole-dns-udp` leases off prod-1 will
|
||||
blip `192.168.2.20` and `192.168.2.23` for as long as re-election plus gratuitous ARP takes. Not
|
||||
measured, and I did not test whether in-flight LAN sessions survive it.
|
||||
* **The Ansible tasks have never been executed.** They parse as YAML and follow
|
||||
`roles/vpn_egress_gateway`'s shape, but neither the label task nor the label-removal task has run,
|
||||
so the `changed_when`/`failed_when` conditions and the `k3s kubectl label ... -` removal path are
|
||||
unproven. The removal task's `failed_when` tolerates `'not found'` in stderr; that string was
|
||||
assumed from kubectl's behaviour, not observed.
|
||||
* **The residual off-repo coupling is NOT closed and cannot be, as built.** The router's forward is
|
||||
unreadable from the cluster. If someone re-points it without moving the label, the alert watches
|
||||
the wrong node and stays silent while the VPN breaks identically against a different set of VIPs.
|
||||
Only the runbook guards it. Closing it needs a WireGuard exporter publishing which node is
|
||||
actually terminating peer traffic (`wg show` knows — only the forwarded node has a peer
|
||||
`endpoint`); listed as "Not built" in the component README and **recommended follow-up, not done
|
||||
here**.
|
||||
* **Why Cilium behaves this way** — intended, a known upstream issue, or fixed in a later version. I
|
||||
measured the behaviour and did not read Cilium's source or issue tracker, so I cannot say whether
|
||||
an upgrade would remove the need for this exclusion.
|
||||
* **`192.168.2.21:8443` (unifi) fails from all three nodes**, including ones where ARP resolves, so
|
||||
it is a different, pre-existing fault. Not investigated, not addressed.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline
|
||||
subset only; none of the above is covered by a test that runs in the gate.
|
||||
* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could
|
||||
not test what a real remote client does, and — as established above — **no probe runnable from the
|
||||
pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a
|
||||
VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a
|
||||
LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one
|
||||
remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in
|
||||
the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this
|
||||
unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show`
|
||||
shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the
|
||||
client loads a cluster app — and repeat it on a node that holds leases and one that does not.*
|
||||
* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same
|
||||
reason. **I need a client connected** to learn it; the router is unreadable from the cluster.
|
||||
* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP
|
||||
failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes.
|
||||
My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**,
|
||||
rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A
|
||||
competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and
|
||||
`wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is
|
||||
needed for the falsification** — the asserted cause is present today without its effect. I did not
|
||||
retro-measure anything, and I did not read Cilium's source or issue tracker.
|
||||
* **The operator's report cannot be a description of "right now."** No client has connected since
|
||||
the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster
|
||||
services and the LAN but not the internet" must describe a session before the power-cycle or one
|
||||
that has since dropped. **I did not establish when it was observed, or with which profile.**
|
||||
* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not**
|
||||
find the fault, and I shipped no fix for it. The two candidates in the triage section (a
|
||||
`--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not
|
||||
measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line
|
||||
decides split vs full and halves the search — and a connected client for step 2.
|
||||
* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and
|
||||
lives only on the operator's machine and the device; it is not in the repo, and I did not look for
|
||||
it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing.
|
||||
* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without
|
||||
a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU
|
||||
would see large transfers hang while small requests succeed. The runbook's existing MTU note
|
||||
(drop to 1280) already covers the remedy.
|
||||
* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole
|
||||
IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a
|
||||
client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an
|
||||
interface with no IPv6 address.
|
||||
* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate,
|
||||
pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1
|
||||
offline subset only. Nothing above is covered by a test that runs in CI.
|
||||
* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there
|
||||
is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating
|
||||
`kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material).
|
||||
* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).
|
||||
Reference in new issue
Block a user