0 Fork 0

Files

T

jakman85andClaude Opus 5 6a8953e288 feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-07 00:59:22 -05:00

6.3 KiB

Raw Blame History

platform/vpn-egress — VPN egress steering (Phase 3, ADR-0015)

vpn-class services (sonarr, radarr, jackett, qbittorrent) egress only through a PIA VPN tunnel; internet-class services go out the normal WAN. Serves R-NET-05/06/07.

Egress is realized per ADR-0015 (spec docs/03): a node-level host OpenVPN split-tunnel on a labeled egress-gateway node, steered by a Cilium Egress Gateway policy, with a host nftables kill-switch. There is no in-cluster gateway pod — the tunnel is host infrastructure, not a Flux workload.

STATUS: live + conformance-green. The node tunnel was validated end-to-end; the egress-class (R-NET-05/07) and vpn-killswitch (R-NET-06) conformance tests pass against the live cluster.

How it works (ADR-0015)

Three layers, two owners:

Tunnel — host OpenVPN (Ansible). A labeled egress-gateway node runs a host OpenVPN systemd service (PIA over OpenVPN-UDP — provider/protocol still per ADR-0013). It is split-tunnel (route-noexec + policy routing): it creates tun0 but never becomes the node default route, so the node's own traffic (control plane, image pulls, other pods) stays on the WAN. Managed by the Ansible role infra/ansible/roles/vpn_egress_gateway/, like k3s itself — not a cluster workload.
Steering — Cilium Egress Gateway (GitOps). egress-steering.yaml holds a CiliumEgressGatewayPolicy that selects pods labeled homelab.keithjackson.dev/egress-class=vpn (Kyverno-enforced) and SNATs them out the gateway node's tun0. Native Cilium eBPF — no client-pod changes, no sidecar, no overlay. excludedCIDRs keeps RFC1918 / cluster / LAN traffic on the local datapath; only public destinations leave via the tunnel.
Kill-switch — host nftables (Ansible). A host nftables rule on the gateway node DROPS the steered vpn-class traffic whenever tun0 is down (fail-closed, R-NET-06), scoped to vpn-class only so internet-class and node traffic are unaffected.

Resilience — there is NO gateway failover (spec correction, escalated)

ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover) provide the active/standby the earlier ADRs asked for", and spec docs/03 repeats it. That feature does not exist in Cilium OSS. The supported topology is exactly one gateway node.

Verified against the CRD actually deployed on this cluster (kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json, apiVersion cilium.io/v2, Cilium v1.20.1) — the field's own description:

egressGateway — "EgressGateway is the gateway node responsible for SNATing traffic. In case multiple nodes are a match for the given set of labels, the first node in lexical ordering based on their name will be selected."

Consequences, stated plainly:

One gateway node is the supported topology. infra/ansible/inventory/hosts.ini's [vpn_egress_gateway] group is singleton by design.
A second labelled node does NOT provide failover. It is not a standby; it carries no traffic and takes over nothing when the first node dies. Gateway-node loss = vpn-class egress down (fail-closed, per R-NET-06 — no leak, but no service).
Lexical ordering decides who wins if two are ever labelled. Labelling k3s-prod-1 while k3s-prod-3 is the live gateway would silently move the role to k3s-prod-1 — because 1 sorts before 3 — taking the tunnel and the public egress IP with it, with no event and no alert.
The plural egressGateways field (Cilium 1.20) is not a workaround. It shards endpoints across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node blackholes that entry's share and re-shards the remainder (breaking live connections), and each shard leaves from a different public IP. Upstream confirmation: cilium/cilium#30157 ("egress gateway doesn't support failover between nodes") and cilium/cilium#18230 ("egress gateway does not tolerate single node failure").

Governance: ADR-0015 and docs/03 are specifications and are immutable to agents (spec AGENTS.md §11, R-TEST-06, enforced by ci/spec-immutability.sh) — they are not edited here. The correction is raised as an escalation with a drafted ADR-0016 supersede in DECISIONS-LOG.md (entry of 2026-09-06) for the operator/architect to accept. Until then this file and egress-steering.yaml are the accurate statement of what the implementation can actually do.

Secrets — host-side now

PIA credentials + the .ovpn live on the gateway node via Ansible vault (see spec docs/07), not a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one.

The pia-credentials.sops.yaml and pia-custom-ovpn.sops.yaml secrets in this directory are now UNUSED — leftovers from the retired Gluetun pod, retained pending a separate cleanup change.

Pieces in this directory

namespace.yaml — the vpn-egress namespace (now holds only the Cilium policy + the retained, unused secrets; no privileged gateway pod under ADR-0015).
egress-steering.yaml — the CiliumEgressGatewayPolicy (the steering datapath).
pia-credentials.sops.yaml, pia-custom-ovpn.sops.yaml — unused, retained pending cleanup.

History (brief)

ADR-0013 originally placed the VPN in a single shared Gluetun pod and deferred the steering mechanism. Implementation proved that a Gluetun-in-a-pod tunnel cannot be a Cilium Egress Gateway target: the policy SNATs out a node interface, but the pod's tun0 lives in the pod netns, and running Gluetun hostNetwork to expose tun0 on the node hijacks the node's default route (breaking k3s/etcd/kubelet). ADR-0015 resolved this by moving the tunnel to the node (host OpenVPN split-tunnel) and steering with Cilium's native egress gateway. See ADR-0015 for the full rationale.

Port forwarding (qBittorrent) — follow-up

PIA assigns a forwarded port that can change on reconnect. Under ADR-0015 this is re-solved host-side (a small pf/natpmp helper on the gateway node that publishes the assigned port to the app) — Phase 6, alongside the qBittorrent app.

6.3 KiB Raw Blame History