revert(vpn-ingress): withdraw the L2 lease-collision fix — a node CAN reach a VIP it announces
PR #69 claimed a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
lease it holds, and excluded the VPN ingress node from announcing on that basis.
The claim makes a sharp prediction; today's power-cycle put ALL EIGHT
cilium-l2announce-* leases on k3s-prod-1, and the prediction failed.
Measured 2026-10-17 from the wireguard pod's host netns on each node: the holder
reaches 192.168.2.20/.22/.23/.24 normally (curl 200 in 18ms on .20) while its
ip neigh for every one of them sits at FAILED. The asserted cause is present
without the asserted effect, which falsifies it. .21 (unifi) fails from all three
nodes and is a separate pre-existing fault.
It is inert because no path resolves a VIP by ARP: cilium-dbg reports
KubeProxyReplacement on [eno1, wg0] and programs 192.168.2.20:443 =>
127.0.0.1:13606, so VIPs are DNAT'd in BPF before any neighbour lookup, on the
forwarded path as well as the host one.
Both probes behind the original diagnosis were bad. `ip neigh` FAILED is
permanent and harmless, and a ping artefact. `curl --interface wg0` — the control
used to rule out a host-netns artefact — fails on all three nodes, including ones
whose ARP resolves, so it discriminates nothing. The deeper error: a MASQUERADEd
client is FORWARDED while every pod-runnable probe is socket-originated. Sharing
a netns is not sharing a hook.
Reverted the policy nodeSelector, the Ansible node label, both alerts, their
promtool tests and the kube-state-metrics allowlist entry. Kept documentation
only: the measurements, which probes lie, and the fact that the forwarded path
remains unmeasured without a connected client.
Also recorded, for the reported "LAN and cluster work, internet does not":
new-peer.sh has emitted a FULL tunnel (0.0.0.0/0) by default since f92707f, so
there is no full-tunnel option to add and no code change is made. The server side
is capable — ip_forward=1, the 10.90.0.0/24 MASQUERADE present with an empty
CILIUM_POST_nat ahead of it, FORWARD accepts, default route intact, nodes reach
1.1.1.1. A full tunnel egresses the HOUSE WAN, not PIA: the egress policy selects
pods by egress-class=vpn, this pod is labelled internet, and tun0 exists on
k3s-prod-3 only. Root cause NOT found; no client has connected since the
power-cycle (0 forwarded packets, no handshake), so it is not reproducible here.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
4656b07178
commit
e82dab5df5
13 files changed
+338
-871
No files matched your search
@@ -1147,97 +1147,6 @@ spec:
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# REMOTE-ACCESS VPN (platform/vpn-ingress) -- NOT the egress tunnel above. Different
|
||||
# direction, different workload: this is inbound WireGuard for remote devices.
|
||||
# -----------------------------------------------------------------------------------------
|
||||
- alert: VpnIngressNodeAnnouncesL2VIP
|
||||
# THE FAILURE THIS CATCHES. MEASURED on prod 2026-10-08: a node CANNOT reach a
|
||||
# LoadBalancer VIP whose Cilium L2 announcement lease it holds itself -- the announcer
|
||||
# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and
|
||||
# packets die before leaving the node (`ip neigh show 192.168.2.20` -> FAILED on the
|
||||
# lease holder, resolved on both others). platform/vpn-ingress runs WireGuard on
|
||||
# `hostNetwork` and MASQUERADEs remote clients to the node's own address, so if the node
|
||||
# the router forwards udp/51820 to is also a lease holder, VPN clients cannot reach the
|
||||
# VIPs it announces. On 2026-10-08 that was the `shared` Gateway: all 16 app hostnames
|
||||
# dead over the VPN, while PiHole DNS kept answering (its VIP has two Services and the
|
||||
# other lease sat on prod-2) -- so the tunnel handshook, names resolved, and every app
|
||||
# hung. Nothing reported it for ~2 days.
|
||||
#
|
||||
# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
|
||||
# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so this should be
|
||||
# permanently unsatisfiable. The alert exists because the exclusion has ways to stop
|
||||
# working that no static check on that file's TEXT can see: the label being removed or
|
||||
# moved to the wrong node, a Cilium upgrade changing `NotIn` semantics or the
|
||||
# nodeSelector field, and the selector silently failing to match. This watches the
|
||||
# OUTCOME -- who actually holds the lease -- so all of them surface the same way.
|
||||
#
|
||||
# WHY THE LABEL JOIN RATHER THAN A NODE NAME. The node's identity is NOT written into
|
||||
# this expr, because the harness (tests/extract_rules.py) refuses any alert whose expr
|
||||
# contains a dollar-brace placeholder -- spelled in words because a literal one here is
|
||||
# unparseable to envsubst and would fail this whole Kustomization. Rightly so: the rule it
|
||||
# unit-tests would not be the
|
||||
# rule deployed. Joining `kube_lease_owner` against `kube_node_labels` keeps ONE source
|
||||
# of truth (the label itself, which is also what the policy selects on), so this alert
|
||||
# cannot drift out of agreement with the exclusion it guards. `label_replace` is needed
|
||||
# because kube_lease_owner names the holder `lease_holder` while kube_node_labels names
|
||||
# it `node`; the join is then on a shared `node` label. The label is in
|
||||
# kube-state-metrics' metricLabelsAllowlist (kube-prometheus-stack.yaml) -- without that
|
||||
# line this rule is inert, which `VpnIngressNodeLabelMissing` below exists to catch.
|
||||
#
|
||||
# `for: 15m` because leases legitimately move during a node reboot or a Cilium roll and
|
||||
# Cilium re-elects within seconds; a real collision persists, since the lease holder
|
||||
# keeps renewing and it never self-heals.
|
||||
expr: |
|
||||
max by (lease) (
|
||||
label_replace(
|
||||
kube_lease_owner{namespace="kube-system", lease=~"cilium-l2announce-.*"},
|
||||
"node", "$1", "lease_holder", "(.+)"
|
||||
)
|
||||
* on (node) group_left ()
|
||||
kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"}
|
||||
) == 1
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "VPN ingress node is announcing a LoadBalancer VIP ({{ $labels.lease }})"
|
||||
description: >-
|
||||
The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
|
||||
WireGuard, and it now holds the Cilium L2 announcement lease `{{ $labels.lease }}`. A
|
||||
node cannot reach a VIP it announces itself, so every remote VPN client has just lost
|
||||
all access to that VIP's Service -- silently, with DNS still resolving. Check that
|
||||
platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
|
||||
the label is still on the node the router forwards udp/51820 to.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
- alert: VpnIngressNodeLabelMissing
|
||||
# THE WATCHDOG for the rule above, and the reason it is not optional. The exclusion in
|
||||
# l2-announcement-policy.yaml is a `NotIn` on this label, and `NotIn` with an ABSENT key
|
||||
# MATCHES -- so if the label is on NO node, the policy excludes nothing, every node is an
|
||||
# announcement candidate again, and the 2026-10-08 failure is fully re-armed. The alert
|
||||
# above would not fire either: its join would match nothing and return no series. Present,
|
||||
# evaluating, and incapable of firing -- the exact "inert critical alert" shape that
|
||||
# VPNEgressUplinkLabelMissing exists for on the egress side, and that this repo has been
|
||||
# burned by before.
|
||||
#
|
||||
# The label is applied by infra/ansible/roles/node_baseline
|
||||
# (node_baseline_vpn_ingress_node); the most likely cause of it going absent is a node
|
||||
# rebuild where that role has not been re-run.
|
||||
expr: absent(kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"})
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
|
||||
description: >-
|
||||
No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
|
||||
l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
|
||||
any node may now announce a LoadBalancer VIP, including the one terminating
|
||||
remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
|
||||
the warning matters more than its severity suggests. Re-run the node_baseline role on
|
||||
the forwarded node, and confirm the label is in kube-state-metrics'
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
||||
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
||||
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
||||
|
||||
@@ -281,20 +281,8 @@ spec:
|
||||
# without anyone having to remember to re-enable it later. Scoped to one label on one resource:
|
||||
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
||||
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
||||
# `vpn-ingress` (added 2026-10-17) marks the node the SOHO router forwards WAN udp/51820 to.
|
||||
# platform-config/cilium/l2-announcement-policy.yaml excludes that node from announcing any
|
||||
# LoadBalancer VIP, because a node cannot reach a VIP whose L2 announcement lease it holds
|
||||
# itself and vpn-ingress MASQUERADEs remote clients to the node's own address (measured
|
||||
# 2026-10-08: all 16 Gateway hostnames dead over the VPN while DNS still resolved). It is
|
||||
# allowlisted HERE because that is the only way the invariant can be WATCHED:
|
||||
# `VpnIngressNodeAnnouncesL2VIP` joins `kube_lease_owner` against this label, and the alert
|
||||
# unit-test harness refuses a dollar-brace placeholder in an alert expr (written out in words:
|
||||
# a literal one here is unparseable to envsubst and fails the whole Kustomization), so a knob
|
||||
# could not have been used. Without this line BOTH that alert and its
|
||||
# `VpnIngressNodeLabelMissing` watchdog are structurally inert. Same reasoning, and the same
|
||||
# 3-extra-series cost, as the two egress labels below.
|
||||
metricLabelsAllowlist:
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink,homelab.keithjackson.dev/vpn-ingress]"
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
|
||||
# --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
|
||||
# Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
|
||||
# `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
|
||||
|
||||
@@ -1,203 +0,0 @@
|
||||
# promtool rule-unit tests for VpnIngressNodeAnnouncesL2VIP + VpnIngressNodeLabelMissing
|
||||
# (platform/monitoring/alerts.yaml)
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh
|
||||
# (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
|
||||
# MEASURED on prod 2026-10-08: a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
|
||||
# lease it holds ITSELF -- the announcer does not answer its own ARP request, so on the lease holder
|
||||
# the neighbour entry settles at `FAILED` and packets to that VIP die before leaving the node.
|
||||
# platform/vpn-ingress runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
|
||||
# own address, so a VPN client takes that node's LAN path and inherits the failure. With the
|
||||
# router's udp/51820 forward on k3s-prod-1, which held the `shared` Gateway lease, all 16 Gateway
|
||||
# hostnames were dead over the VPN -- while PiHole DNS kept answering, because its VIP has two
|
||||
# Services and k3s-prod-2 held the other lease. The tunnel handshook, names resolved, every app
|
||||
# hung, and nothing reported it for ~2 days.
|
||||
#
|
||||
# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
|
||||
# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so VpnIngressNodeAnnouncesL2VIP
|
||||
# should be permanently unsatisfiable. THAT IS EXACTLY WHY IT NEEDS A TEST: an alert expected never
|
||||
# to fire is indistinguishable, in production, from an alert that CANNOT fire. Nothing would ever
|
||||
# reveal a mistyped label, a broken `label_replace`, or a join on the wrong label. Test 1 proves it
|
||||
# fires on the real failure; test 2 proves it is quiet in the fixed state, so it is not a rule that
|
||||
# fires on anything; test 3 pins the `for: 15m` against routine lease churn.
|
||||
#
|
||||
# THE JOIN IS THE FRAGILE PART AND IS TESTED DELIBERATELY. `kube_lease_owner` names the holder in
|
||||
# `lease_holder`; `kube_node_labels` names it `node`. The rule bridges them with `label_replace`
|
||||
# and joins `on (node) group_left ()`. Fixtures therefore carry BOTH series shapes with real label
|
||||
# sets, because a join that silently matches nothing returns no series -- which looks exactly like
|
||||
# "healthy" and is the failure mode this whole group exists to prevent.
|
||||
#
|
||||
# Fixture values are prod's real shape (measured 2026-10-08): leases
|
||||
# `cilium-l2announce-gateway-cilium-gateway-shared` (k3s-prod-1) and
|
||||
# `cilium-l2announce-pihole-pihole-dns-tcp` (k3s-prod-2), in namespace kube-system, with
|
||||
# k3s-prod-1 the vpn-ingress node.
|
||||
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# =============================================================================================
|
||||
# 1. THE BUG, EXACTLY AS IT HAPPENED -- the vpn-ingress node holds the Gateway lease: FIRES.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: vpn-ingress node holds the shared Gateway L2 lease - VpnIngressNodeAnnouncesL2VIP fires
|
||||
input_series:
|
||||
# The collision. A lease holder keeps renewing, so this series does not lapse on its own --
|
||||
# which is why the condition never self-heals and why a 15m `for:` is safe.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x40'
|
||||
# A second lease held by a DIFFERENT node, present throughout. This is the discriminator that
|
||||
# stops the rule from being "any l2announce lease exists anywhere": if the join were dropped,
|
||||
# this series alone would fire the alert, and test 2 would fail.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-pihole-pihole-dns-tcp",lease_holder="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
# k3s-prod-1 is the vpn-ingress node. Only this node carries the label.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
|
||||
values: '1+0x40'
|
||||
# The other two nodes publish kube_node_labels WITHOUT the vpn-ingress label. Present so the
|
||||
# join is exercised against non-matching nodes rather than against nothing at all.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
|
||||
values: '1+0x40'
|
||||
alert_rule_test:
|
||||
# Held 15m, so nothing before then.
|
||||
- eval_time: 14m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
- eval_time: 20m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts:
|
||||
# EXACTLY ONE alert: the pihole lease on prod-2 must NOT produce a second one.
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
lease: cilium-l2announce-gateway-cilium-gateway-shared
|
||||
exp_annotations:
|
||||
summary: "VPN ingress node is announcing a LoadBalancer VIP (cilium-l2announce-gateway-cilium-gateway-shared)"
|
||||
description: >-
|
||||
The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
|
||||
WireGuard, and it now holds the Cilium L2 announcement lease `cilium-l2announce-gateway-cilium-gateway-shared`. A
|
||||
node cannot reach a VIP it announces itself, so every remote VPN client has just lost
|
||||
all access to that VIP's Service -- silently, with DNS still resolving. Check that
|
||||
platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
|
||||
the label is still on the node the router forwards udp/51820 to.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
# The label IS present, so the watchdog must stay quiet here.
|
||||
- eval_time: 40m
|
||||
alertname: VpnIngressNodeLabelMissing
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 2. THE FIXED STATE -- every lease is held by a node that is NOT the vpn-ingress node: QUIET.
|
||||
# This is what l2-announcement-policy.yaml's exclusion produces, and the assertion that keeps
|
||||
# test 1 meaningful rather than an alert that fires on anything.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: all l2 leases held by other nodes - VpnIngressNodeAnnouncesL2VIP stays quiet
|
||||
input_series:
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-plex-plex-l4-tcp",lease_holder="k3s-prod-3"}'
|
||||
values: '1+0x40'
|
||||
# A lease the vpn-ingress node DOES legitimately hold, in a different family -- proves the
|
||||
# `cilium-l2announce-.*` matcher is doing work. Every node holds operator/controller leases;
|
||||
# only the ANNOUNCEMENT ones matter here, and a matcher broad enough to catch this would page
|
||||
# on a healthy cluster forever.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-operator-resource-lock",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
|
||||
values: '1+0x40'
|
||||
alert_rule_test:
|
||||
- eval_time: 40m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
- eval_time: 40m
|
||||
alertname: VpnIngressNodeLabelMissing
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 3. A LEGITIMATE, TRANSIENT LEASE MOVE -- the vpn-ingress node holds it for 11m during a node
|
||||
# reboot or a Cilium roll, then Cilium re-elects away: QUIET. This pins the `for: 15m` as
|
||||
# deliberate. Without it, every rolling restart of the cilium DaemonSet would page, and a paging
|
||||
# alert that cries wolf on routine maintenance gets muted -- which is how the real collision
|
||||
# would go unseen again.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: transient lease move shorter than for 15m - VpnIngressNodeAnnouncesL2VIP stays quiet
|
||||
input_series:
|
||||
# Held by the vpn-ingress node for 11 samples, then the series goes absent (the lease moved,
|
||||
# so kube-state-metrics retires this label set and starts a new one).
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x10 _x30'
|
||||
# Re-elected onto prod-3 from t=11m. A label VALUE change retires one series and starts
|
||||
# another, which is why this is a second series and not a changing value.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-3"}'
|
||||
values: '_x11 1+0x29'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
|
||||
values: '1+0x40'
|
||||
alert_rule_test:
|
||||
# Inside the transient window: pending at most, never firing.
|
||||
- eval_time: 10m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
# Well past it, with the lease settled on another node.
|
||||
- eval_time: 30m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 4. THE WATCHDOG -- no node carries the label at all. The policy's `NotIn` then excludes
|
||||
# NOTHING (a missing key matches), so the 2026-10-08 failure is fully re-armed, AND
|
||||
# VpnIngressNodeAnnouncesL2VIP is structurally INERT: its join matches nothing and returns no
|
||||
# series, which is indistinguishable from healthy. Both halves are asserted here, because an
|
||||
# inert critical alert is the exact shape this repo has been burned by before.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: vpn-ingress label absent everywhere - watchdog fires and the critical alert is inert
|
||||
input_series:
|
||||
# A lease held by the node that IS (in reality) the forwarded one -- but with no label
|
||||
# anywhere, nothing can tell. This is the dangerous state.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x60'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
|
||||
values: '1+0x60'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
|
||||
values: '1+0x60'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
|
||||
values: '1+0x60'
|
||||
alert_rule_test:
|
||||
# The critical alert CANNOT fire -- this is the inertness being pinned, not an aside.
|
||||
- eval_time: 45m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
# ... which is precisely why the watchdog must.
|
||||
- eval_time: 45m
|
||||
alertname: VpnIngressNodeLabelMissing
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
# absent() propagates the matcher's own labels onto the series it returns, so the
|
||||
# alert inherits this one. Asserted rather than worked around: it is harmless, and
|
||||
# pinning it means a future change to the matcher cannot alter the alert's identity
|
||||
# (and therefore its Alertmanager grouping) unnoticed.
|
||||
label_homelab_keithjackson_dev_vpn_ingress: "true"
|
||||
exp_annotations:
|
||||
summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
|
||||
description: >-
|
||||
No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
|
||||
l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
|
||||
any node may now announce a LoadBalancer VIP, including the one terminating
|
||||
remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
|
||||
the warning matters more than its severity suggests. Re-run the node_baseline role on
|
||||
the forwarded node, and confirm the label is in kube-state-metrics'
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
@@ -117,19 +117,8 @@ boundary around this port is WireGuard's peer authentication, full stop.
|
||||
The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
|
||||
forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
|
||||
A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
|
||||
keys — so losing the forwarded node never costs a re-issued client profile. The two idle listeners
|
||||
cost ~15 MiB each.
|
||||
|
||||
> **This section used to say "losing the forwarded node is a one-line router change, with no cluster
|
||||
> work". It was wrong, and it cost two days of silently broken remote access on 2026-10-08.** The
|
||||
> nodes are warm, but they are **not interchangeable**: the node the forward points at must be the
|
||||
> one labelled `homelab.keithjackson.dev/vpn-ingress=true`, because that is the node
|
||||
> `platform-config/cilium/l2-announcement-policy.yaml` excludes from announcing LoadBalancer VIPs.
|
||||
> **A node cannot reach a VIP whose Cilium L2 announcement lease it holds itself** (see "Measured on
|
||||
> prod, 2026-10-08" below), and since this pod MASQUERADEs clients to the node's own address, a VPN
|
||||
> client inherits that. Failover is therefore **two** changes: move
|
||||
> `node_baseline_vpn_ingress_node` and re-run that role, then re-point the router —
|
||||
> `runbooks/vpn-remote-access.md` step 3.
|
||||
keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
|
||||
re-issued client profiles. The two idle listeners cost ~15 MiB each.
|
||||
|
||||
A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
|
||||
replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
|
||||
@@ -197,44 +186,85 @@ right way; one was a defect in my own manifest.
|
||||
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
||||
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
||||
|
||||
## Measured on prod, 2026-10-08 — why the forwarded node cannot be any node
|
||||
## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach
|
||||
|
||||
The first device enrolled, handshook, resolved names, and could not open a single connection to any
|
||||
cluster web app. The tunnel was not at fault — every part of it measured healthy:
|
||||
`net.ipv4.ip_forward=1`, `wg0` up at `10.90.0.1/24` MTU 1420, the peer authorised with real
|
||||
bidirectional transfer (346 KiB in / 200 KiB out), `MASQUERADE` present and counting 26 flows,
|
||||
`FORWARD -i/-o wg0 ACCEPT` counting 108 / 171 packets, and PiHole DNS answering from the node.
|
||||
A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose
|
||||
Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from
|
||||
announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is
|
||||
real but inert, and the distinction matters enough to record, because the wrong version is very
|
||||
easy to re-derive from `ip neigh` alone.
|
||||
|
||||
**The fault was that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
|
||||
holds itself.** Cilium's announcer answers ARP for the VIP on behalf of other LAN hosts, but not for
|
||||
the announcing node's own request, so the neighbour entry settles at `FAILED`:
|
||||
Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case,
|
||||
after the UPS power-cycle — from the host netns of the `wireguard` pod on each node:
|
||||
|
||||
| from | `192.168.2.20` (Gateway) | `192.168.2.22` (crafty) | `192.168.2.24` (plex) | announces |
|
||||
|---|---|---|---|---|
|
||||
| `k3s-prod-1` | **FAILED** | — | — | `.20` |
|
||||
| `k3s-prod-2` | ok | **INCOMPLETE** | ok | `.21`, `.22`, `.23-tcp` |
|
||||
| `k3s-prod-3` | ok | ok | **INCOMPLETE** | `.24` |
|
||||
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|
||||
|---|---|---|
|
||||
| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
|
||||
| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms |
|
||||
| `nc -z 192.168.2.20 443` | **OK** | OK |
|
||||
| plex `192.168.2.24:32400` | **`200`** | `200` |
|
||||
|
||||
Symmetric across 3 nodes × 5 VIPs: each node reached every VIP **except** the ones it alone
|
||||
announced. It is a property of L2 announcement, not of any one Service, and it is harmless for
|
||||
ordinary pods — they reach Services through the cluster datapath and never ARP for a VIP. This
|
||||
component is the exception, because `hostNetwork` + `MASQUERADE` put remote clients on the node's
|
||||
own LAN path.
|
||||
So the announcer really does **not** answer its own ARP request — that part of the observation is
|
||||
reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**:
|
||||
|
||||
The forward pointed at `k3s-prod-1`, which held the `shared` Gateway lease, so **all 16 Gateway
|
||||
hostnames were dead over the VPN**. Two things hid it:
|
||||
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
|
||||
`Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**,
|
||||
so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before
|
||||
any route or neighbour lookup for the VIP happens**.
|
||||
* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the
|
||||
shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without
|
||||
a packet crossing the LAN at all.
|
||||
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
|
||||
which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real**
|
||||
service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes —
|
||||
that is service-aware DNAT, not wire traffic.
|
||||
|
||||
* **PiHole survived by accident.** `192.168.2.23` is announced by *two* Services
|
||||
(`pihole-dns-udp` on prod-1, `pihole-dns-tcp` on prod-2), so prod-2 answered the ARP prod-1 could
|
||||
not. Names resolved perfectly and every app hung — the hardest shape to diagnose from a client.
|
||||
* **Nothing reported it.** No probe in this repo compares the forwarded node against the lease
|
||||
holders, and from outside WireGuard is silent to unauthenticated traffic anyway.
|
||||
**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis:
|
||||
|
||||
Closed by labelling the forwarded node `homelab.keithjackson.dev/vpn-ingress=true` and excluding
|
||||
that label in `platform-config/cilium/l2-announcement-policy.yaml` (the node then ARP-resolves every
|
||||
VIP), with `VpnIngressNodeAnnouncesL2VIP` watching the lease holders so a recurrence is loud and
|
||||
`VpnIngressNodeLabelMissing` watching for the label going absent, which would silently re-arm the
|
||||
whole failure. Cost: VIP announcement candidacy drops from three nodes to two.
|
||||
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an
|
||||
artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not
|
||||
anything is wrong.
|
||||
* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the
|
||||
two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it
|
||||
discriminates nothing — and it is not the client path either (see below).
|
||||
|
||||
**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are
|
||||
**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is
|
||||
**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the
|
||||
forwarded path in either direction: it reports success where the forwarded path might fail, and
|
||||
`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with
|
||||
a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE`
|
||||
counters move.
|
||||
|
||||
## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses
|
||||
|
||||
`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the
|
||||
component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature
|
||||
to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel
|
||||
client cannot reach the internet, nothing in the list below is the reason:
|
||||
|
||||
| checked | result on all three nodes |
|
||||
|---|---|
|
||||
| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` |
|
||||
| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change |
|
||||
| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE |
|
||||
| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present |
|
||||
| default route | `default via 192.168.2.1 dev eno1 onlink`, intact |
|
||||
| node's own internet reachability | `http://1.1.1.1/` returns `301` |
|
||||
|
||||
**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is
|
||||
closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by
|
||||
`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled
|
||||
`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity
|
||||
traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway
|
||||
node. A remote client's apparent public address is therefore the **home WAN IP**, which is the
|
||||
intended behaviour and the thing to state when enrolling a device.
|
||||
|
||||
**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one
|
||||
enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and
|
||||
the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the
|
||||
power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the
|
||||
forwarded node shows a peer with an `endpoint` and a recent `latest handshake`.
|
||||
|
||||
## Prerequisites outside this directory
|
||||
|
||||
|
||||
@@ -62,39 +62,9 @@
|
||||
# as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
|
||||
# whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
|
||||
# DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
|
||||
# them, and if that node is lost every other node is already warm and answers the same keys, so no
|
||||
# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict,
|
||||
# and the two idle listeners cost ~15 MiB each.
|
||||
#
|
||||
# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE.
|
||||
# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the
|
||||
# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node
|
||||
# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer
|
||||
# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to
|
||||
# that VIP die before leaving the node:
|
||||
#
|
||||
# k3s-prod-1, holding the shared Gateway lease: ip neigh show 192.168.2.20 -> FAILED
|
||||
# k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr …(prod-1)
|
||||
#
|
||||
# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client
|
||||
# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway
|
||||
# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that
|
||||
# handshakes, resolves names, and then hangs on every single app.
|
||||
#
|
||||
# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled
|
||||
# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from
|
||||
# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes
|
||||
# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a
|
||||
# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test
|
||||
# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move
|
||||
# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router
|
||||
# (runbooks/vpn-remote-access.md step 3).
|
||||
# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again.
|
||||
#
|
||||
# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge
|
||||
# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness
|
||||
# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a
|
||||
# property of the CLUSTER's announcement layout, so it is watched there.
|
||||
# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
|
||||
# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
|
||||
# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
|
||||
#
|
||||
# All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
|
||||
# DaemonSet to cover the cluster; add them if a node is ever tainted.
|
||||
|
||||
Reference in new issue
Block a user