revert(vpn-ingress): withdraw the L2 lease-collision fix — a node CAN reach a VIP it announces
PR #69 claimed a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
lease it holds, and excluded the VPN ingress node from announcing on that basis.
The claim makes a sharp prediction; today's power-cycle put ALL EIGHT
cilium-l2announce-* leases on k3s-prod-1, and the prediction failed.
Measured 2026-10-17 from the wireguard pod's host netns on each node: the holder
reaches 192.168.2.20/.22/.23/.24 normally (curl 200 in 18ms on .20) while its
ip neigh for every one of them sits at FAILED. The asserted cause is present
without the asserted effect, which falsifies it. .21 (unifi) fails from all three
nodes and is a separate pre-existing fault.
It is inert because no path resolves a VIP by ARP: cilium-dbg reports
KubeProxyReplacement on [eno1, wg0] and programs 192.168.2.20:443 =>
127.0.0.1:13606, so VIPs are DNAT'd in BPF before any neighbour lookup, on the
forwarded path as well as the host one.
Both probes behind the original diagnosis were bad. `ip neigh` FAILED is
permanent and harmless, and a ping artefact. `curl --interface wg0` — the control
used to rule out a host-netns artefact — fails on all three nodes, including ones
whose ARP resolves, so it discriminates nothing. The deeper error: a MASQUERADEd
client is FORWARDED while every pod-runnable probe is socket-originated. Sharing
a netns is not sharing a hook.
Reverted the policy nodeSelector, the Ansible node label, both alerts, their
promtool tests and the kube-state-metrics allowlist entry. Kept documentation
only: the measurements, which probes lie, and the fact that the forwarded path
remains unmeasured without a connected client.
Also recorded, for the reported "LAN and cluster work, internet does not":
new-peer.sh has emitted a FULL tunnel (0.0.0.0/0) by default since f92707f, so
there is no full-tunnel option to add and no code change is made. The server side
is capable — ip_forward=1, the 10.90.0.0/24 MASQUERADE present with an empty
CILIUM_POST_nat ahead of it, FORWARD accepts, default route intact, nodes reach
1.1.1.1. A full tunnel egresses the HOUSE WAN, not PIA: the egress policy selects
pods by egress-class=vpn, this pod is labelled internet, and tun0 exists on
k3s-prod-3 only. Root cause NOT found; no client has connected since the
power-cycle (0 forwarded packets, no handshake), so it is not reproducible here.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
4656b07178
commit
e82dab5df5
13 files changed
+338
-871
No files matched your search
@@ -281,20 +281,8 @@ spec:
|
||||
# without anyone having to remember to re-enable it later. Scoped to one label on one resource:
|
||||
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
||||
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
||||
# `vpn-ingress` (added 2026-10-17) marks the node the SOHO router forwards WAN udp/51820 to.
|
||||
# platform-config/cilium/l2-announcement-policy.yaml excludes that node from announcing any
|
||||
# LoadBalancer VIP, because a node cannot reach a VIP whose L2 announcement lease it holds
|
||||
# itself and vpn-ingress MASQUERADEs remote clients to the node's own address (measured
|
||||
# 2026-10-08: all 16 Gateway hostnames dead over the VPN while DNS still resolved). It is
|
||||
# allowlisted HERE because that is the only way the invariant can be WATCHED:
|
||||
# `VpnIngressNodeAnnouncesL2VIP` joins `kube_lease_owner` against this label, and the alert
|
||||
# unit-test harness refuses a dollar-brace placeholder in an alert expr (written out in words:
|
||||
# a literal one here is unparseable to envsubst and fails the whole Kustomization), so a knob
|
||||
# could not have been used. Without this line BOTH that alert and its
|
||||
# `VpnIngressNodeLabelMissing` watchdog are structurally inert. Same reasoning, and the same
|
||||
# 3-extra-series cost, as the two egress labels below.
|
||||
metricLabelsAllowlist:
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink,homelab.keithjackson.dev/vpn-ingress]"
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
|
||||
# --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
|
||||
# Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
|
||||
# `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
|
||||
|
||||
Reference in new issue
Block a user