Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
69a25cff827d6cab2cab3ed4287c59efabb587d2
home-network-k3s-impl/platform/vpn-egress/README.md
T
jakman85andClaude Opus 5 6a8953e288 feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing 
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 00:59:22 -05:00

95 lines
6.3 KiB
Markdown
Raw Blame History

# platform/vpn-egress — VPN egress steering (Phase 3, ADR-0015)
`vpn`-class services (sonarr, radarr, jackett, qbittorrent) egress **only** through a PIA VPN
tunnel; `internet`-class services go out the normal WAN. Serves **R-NET-05/06/07**.
Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md)**
(spec `docs/03`): a **node-level host OpenVPN split-tunnel** on a labeled egress-gateway node, steered
by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster
gateway pod** — the tunnel is host infrastructure, not a Flux workload.
> **STATUS: live + conformance-green.** The node tunnel was validated end-to-end; the `egress-class`
> (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests pass against the live cluster.
## How it works (ADR-0015)
Three layers, two owners:
- **Tunnel — host OpenVPN (Ansible).** A labeled egress-gateway node runs a **host OpenVPN systemd
service** (PIA over OpenVPN-UDP — provider/protocol still per ADR-0013). It is **split-tunnel**
(`route-noexec` + policy routing): it creates `tun0` but never becomes the node default route, so the
node's own traffic (control plane, image pulls, other pods) stays on the WAN. Managed by the Ansible
role **`infra/ansible/roles/vpn_egress_gateway/`**, like k3s itself — not a cluster workload.
- **Steering — Cilium Egress Gateway (GitOps).** `egress-steering.yaml` holds a
`CiliumEgressGatewayPolicy` that selects pods labeled `homelab.keithjackson.dev/egress-class=vpn`
(Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod
changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local
datapath; only public destinations leave via the tunnel.
- **Kill-switch — host nftables (Ansible).** A host nftables rule on the gateway node DROPS the steered
`vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so
`internet`-class and node traffic are unaffected.
## Resilience — there is NO gateway failover (spec correction, escalated)
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
provide the active/standby the earlier ADRs asked for", and spec `docs/03` repeats it. That feature
does not exist in Cilium OSS.** The supported topology is **exactly one gateway node**.
Verified against the CRD actually deployed on this cluster
(`kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`, apiVersion `cilium.io/v2`,
Cilium **v1.20.1**) — the field's own description:
> **`egressGateway`** — "EgressGateway is the gateway node responsible for SNATing traffic.
> **In case multiple nodes are a match for the given set of labels, the first node in lexical
> ordering based on their name will be selected.**"
Consequences, stated plainly:
- **One gateway node is the supported topology.** `infra/ansible/inventory/hosts.ini`'s
`[vpn_egress_gateway]` group is singleton by design.
- **A second labelled node does NOT provide failover.** It is not a standby; it carries no traffic
and takes over nothing when the first node dies. Gateway-node loss = `vpn`-class egress down
(fail-closed, per R-NET-06 — no leak, but no service).
- **Lexical ordering decides who wins if two are ever labelled.** Labelling `k3s-prod-1` while
`k3s-prod-3` is the live gateway would **silently** move the role to `k3s-prod-1` — because `1`
sorts before `3` — taking the tunnel and the public egress IP with it, with no event and no alert.
- **The plural `egressGateways` field (Cilium 1.20) is not a workaround.** It *shards* endpoints
across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node
blackholes that entry's share and re-shards the remainder (breaking live connections), and each
shard leaves from a *different* public IP. Upstream confirmation:
[cilium/cilium#30157](https://github.com/cilium/cilium/issues/30157) ("egress gateway doesn't
support failover between nodes") and
[cilium/cilium#18230](https://github.com/cilium/cilium/issues/18230) ("egress gateway does not
tolerate single node failure").
**Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents
(spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited
here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in
`DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file
and `egress-steering.yaml` are the accurate statement of what the implementation can actually do.
## Secrets — host-side now
PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not**
a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one.
The `pia-credentials.sops.yaml` and `pia-custom-ovpn.sops.yaml` secrets in this directory are **now
UNUSED** — leftovers from the retired Gluetun pod, **retained pending a separate cleanup change**.
## Pieces in this directory
- `namespace.yaml` — the `vpn-egress` namespace (now holds only the Cilium policy + the retained,
unused secrets; no privileged gateway pod under ADR-0015).
- `egress-steering.yaml` — the `CiliumEgressGatewayPolicy` (the steering datapath).
- `pia-credentials.sops.yaml`, `pia-custom-ovpn.sops.yaml` — **unused**, retained pending cleanup.
## History (brief)
ADR-0013 originally placed the VPN in a single shared **Gluetun pod** and deferred the steering
mechanism. Implementation proved that a Gluetun-in-a-pod tunnel **cannot be a Cilium Egress Gateway
target**: the policy SNATs out a *node* interface, but the pod's `tun0` lives in the pod netns, and
running Gluetun `hostNetwork` to expose `tun0` on the node hijacks the node's default route (breaking
k3s/etcd/kubelet). ADR-0015 resolved this by moving the tunnel to the **node** (host OpenVPN
split-tunnel) and steering with Cilium's native egress gateway. See ADR-0015 for the full rationale.
## Port forwarding (qBittorrent) — follow-up
PIA assigns a forwarded port that can change on reconnect. Under ADR-0015 this is re-solved **host-side**
(a small pf/natpmp helper on the gateway node that publishes the assigned port to the app) — Phase 6,
alongside the qBittorrent app.
Reference in new issue View Git Blame Copy Permalink