home-network-k3s-impl

1,078 Commits 140 Branches 0 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	37a6c94168	fix(vpn): two gateways on one pinned PIA server evict each other every 2 minutes Standing up prod broke dev, and the mechanism is worth spelling out because "same PIA account, well inside its connection limit" is true and still not enough. The vaulted .ovpn pins ONE server. With both gateways using it, dev and prod landed on the SAME server AND were handed the SAME tunnel address — both journals read `[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198` and `ifconfig 10.72.192.7 255.255.192.0`. Each reconnect steals that address on that server; the loser then receives nothing, trips `--ping-restart` after 60s, reconnects and steals it back. Measured cycle: dev restarting at 22:57:05, 22:59:03, 23:01:05 UTC with prod interleaved about a minute off, each ~2 minutes apart, indefinitely. Neither environment had reliable vpn-class egress — prod's probes looked green only because they happened to land inside an up-window. Fix: `vpn_egress_gateway_pia_remote` ("<host> <port>", empty = honour the .ovpn) rewrites just the `remote` line of the deployed .ovpn, and prod's group_vars sets it to 158.173.241.135. That is another member of the same PIA Stockholm LB pool — `sweden.privacy.network` resolved from k3s-prod-3 to 158.173.241.{34,55,112,135,239} — and every server in the pool takes the same credentials and the same inline CA, so only the address differs. Dev keeps the .ovpn's .112 untouched, which is why the override is per-group rather than in the role or a second vaulted .ovpn. Two details that matter for whoever touches this next: * The regex is anchored to a line STARTING with `remote` plus whitespace, so `remote-cert-tls` and `remote-random` are not collateral damage. The task keeps `no_log: true` — the transformation never widens what is logged. * A PIA server address is NOT a credential, so it lives in group_vars/vpn_egress_gateway_prod/vars.yml in the clear, next to the vault rather than inside it. Keeping it out of the vault is what makes it reviewable. If prod ever gets its own PIA account, this override can be deleted again. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 18:05:49 -05:00
jakman85andClaude Opus 5	6f698f59af	fix(vpn): Cilium never re-resolves tun0 — the gateway dies on every reconnect Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the dev gateway silently dead for ~80 minutes before this change: 1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's `interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the map to 1987 and egress recovered instantly. `egress-gateway-reconciliation- trigger-interval: 1s` does NOT cover netlink link events — dev has that same 1s setting and still did not self-heal, 20s+ after recovery. Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label. A node update makes the gateway re-process the policy (verified: cached ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the policy, because the policy is Flux-managed GitOps and the node's labels are already this role's territory. Fired by the OpenVPN up-script on every connect, plus a 2-minute convergence timer for the boot case where the kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the identical sequence stayed broken indefinitely. 2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only diagnostic table on the gateway node recorded, while a vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA range, against 45166 packets on the same postrouting hook for WAN traffic and 16 packets for a node-originated curl out tun0. Cilium's eBPF datapath bpf_redirect()s these skbs to the egress device without invoking netfilter. So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods. What actually fails closed is Cilium's own drop when the cached egress interface is unusable — tcpdump during a tunnel stop AND during a restart showed 0 packets leaving either the WAN or tun0. Every rule now carries a `counter` so the claim stays falsifiable, and every comment that called this table the authoritative vpn-class backstop is corrected rather than left to mislead the next reader. Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable: nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is a hazard to whatever picks it next. 3. The up-script flushed the nft source set on every connect, so a packet still sourced from the previous tunnel address matched neither the ip-rule nor the drop rule. The set is now additive (`add element` is idempotent; the down-script already retained it): an element can only ever tighten the guard. Finding 2 means that window could not have leaked POD traffic, but node-originated tunnel traffic does traverse netfilter, and additive is free. Kill-switch reloads no longer disarm the guard either: the .nft file uses `table` + `flush table`, which clears rules but leaves the named set's elements intact, so an Ansible run cannot empty the set mid-flight. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 17:38:27 -05:00
jakman85andClaude Opus 4.8	2c109e2a31	feat(vpn): node-level split-tunnel egress gateway Ansible role (ADR-0015) Author the vpn_egress_gateway role + playbook that stands up the NODE-level PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the same contract (group vpn_egress_gateway, node label homelab.keithjackson.dev/egress-gateway=true, host iface tun0). Honors all four ADR-0015 invariants: - Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore redirect-gateway => installs no routes; policy routing in a dedicated table (51820) with `ip rule from <tun0-addr>` steers ONLY Cilium-SNAT'd vpn-class traffic. Node default route stays on the WAN, tunnel up or down. - Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 => vpn-class fails closed (blackhole), node/internet-class fail open (WAN), never matched. PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault (host-side per ADR-0015/docs-07); vault keys documented, no secret values committed. ansible-lint (production) + yamllint clean. Not deployed: green conformance is gated on the live tunnel (operator host creds) — see role README. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-24 01:09:19 -05:00

Author

SHA1

Message

Date

jakman85andClaude Opus 5

37a6c94168

fix(vpn): two gateways on one pinned PIA server evict each other every 2 minutes

Standing up prod broke dev, and the mechanism is worth spelling out because
"same PIA account, well inside its connection limit" is true and still not enough.

The vaulted .ovpn pins ONE server. With both gateways using it, dev and prod
landed on the SAME server AND were handed the SAME tunnel address — both journals
read `[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198`
and `ifconfig 10.72.192.7 255.255.192.0`. Each reconnect steals that address on
that server; the loser then receives nothing, trips `--ping-restart` after 60s,
reconnects and steals it back. Measured cycle: dev restarting at 22:57:05,
22:59:03, 23:01:05 UTC with prod interleaved about a minute off, each ~2 minutes
apart, indefinitely. Neither environment had reliable vpn-class egress — prod's
probes looked green only because they happened to land inside an up-window.

Fix: `vpn_egress_gateway_pia_remote` ("<host> <port>", empty = honour the .ovpn)
rewrites just the `remote` line of the deployed .ovpn, and prod's group_vars sets
it to 158.173.241.135. That is another member of the same PIA Stockholm LB pool —
`sweden.privacy.network` resolved from k3s-prod-3 to
158.173.241.{34,55,112,135,239} — and every server in the pool takes the same
credentials and the same inline CA, so only the address differs. Dev keeps the
.ovpn's .112 untouched, which is why the override is per-group rather than in the
role or a second vaulted .ovpn.

Two details that matter for whoever touches this next:

* The regex is anchored to a line STARTING with `remote` plus whitespace, so
  `remote-cert-tls` and `remote-random` are not collateral damage. The task keeps
  `no_log: true` — the transformation never widens what is logged.
* A PIA server address is NOT a credential, so it lives in
  group_vars/vpn_egress_gateway_prod/vars.yml in the clear, next to the vault
  rather than inside it. Keeping it out of the vault is what makes it reviewable.

If prod ever gets its own PIA account, this override can be deleted again.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-12 18:05:49 -05:00

jakman85andClaude Opus 5

6f698f59af

fix(vpn): Cilium never re-resolves tun0 — the gateway dies on every reconnect

Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the
dev gateway silently dead for ~80 minutes before this change:

1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's
   `interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel
   restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed
   from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf
   egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the
   map to 1987 and egress recovered instantly. `egress-gateway-reconciliation-
   trigger-interval: 1s` does NOT cover netlink link events — dev has that same
   1s setting and still did not self-heal, 20s+ after recovery.

   Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label.
   A node update makes the gateway re-process the policy (verified: cached
   ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the
   policy, because the policy is Flux-managed GitOps and the node's labels are
   already this role's territory. Fired by the OpenVPN up-script on every
   connect, plus a 2-minute convergence timer for the boot case where the
   kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now
   recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the
   identical sequence stayed broken indefinitely.

2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only
   diagnostic table on the gateway node recorded, while a vpn-class pod egressed:
   0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA
   range, against 45166 packets on the same postrouting hook for WAN traffic and
   16 packets for a node-originated curl out tun0. Cilium's eBPF datapath
   bpf_redirect()s these skbs to the egress device without invoking netfilter.

   So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods.
   What actually fails closed is Cilium's own drop when the cached egress
   interface is unusable — tcpdump during a tunnel stop AND during a restart
   showed 0 packets leaving either the WAN or tun0. Every rule now carries a
   `counter` so the claim stays falsifiable, and every comment that called this
   table the authoritative vpn-class backstop is corrected rather than left to
   mislead the next reader.

   Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable:
   nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the
   L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and
   a live rule matching an arbitrary mark is a hazard to whatever picks it next.

3. The up-script flushed the nft source set on every connect, so a packet still
   sourced from the previous tunnel address matched neither the ip-rule nor the
   drop rule. The set is now additive (`add element` is idempotent; the
   down-script already retained it): an element can only ever tighten the guard.
   Finding 2 means that window could not have leaked POD traffic, but
   node-originated tunnel traffic does traverse netfilter, and additive is free.

Kill-switch reloads no longer disarm the guard either: the .nft file uses
`table` + `flush table`, which clears rules but leaves the named set's elements
intact, so an Ansible run cannot empty the set mid-flight.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-12 17:38:27 -05:00

jakman85andClaude Opus 4.8

2c109e2a31

feat(vpn): node-level split-tunnel egress gateway Ansible role (ADR-0015)

Author the vpn_egress_gateway role + playbook that stands up the NODE-level
PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The
CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the
same contract (group vpn_egress_gateway, node label
homelab.keithjackson.dev/egress-gateway=true, host iface tun0).

Honors all four ADR-0015 invariants:
- Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore
  redirect-gateway => installs no routes; policy routing in a dedicated table
  (51820) with `ip rule from <tun0-addr>` steers ONLY Cilium-SNAT'd vpn-class
  traffic. Node default route stays on the WAN, tunnel up or down.
- Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops
  only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 =>
  vpn-class fails closed (blackhole), node/internet-class fail open (WAN),
  never matched.

PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault
(host-side per ADR-0015/docs-07); vault keys documented, no secret values
committed. ansible-lint (production) + yamllint clean. Not deployed: green
conformance is gated on the live tunnel (operator host creds) — see role README.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

2026-08-24 01:09:19 -05:00