jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
104 lines
6.8 KiB
YAML
104 lines
6.8 KiB
YAML
---
|
|
# ============================================================================
|
|
# Cilium Egress Gateway steering for `egress-class=vpn` pods — ADR-0015.
|
|
# Serves: R-NET-04, R-NET-05, R-NET-06, R-NET-07.
|
|
# R-NET-04 added 2026-09-06: this file IS the R-NET-04 datapath — the rule that selects between the
|
|
# two egress paths. Omitting it made `run.sh --status` report R-NET-04 as "TEST-ONLY / impl none"
|
|
# even though the implementation is right here.
|
|
#
|
|
# ##########################################################################
|
|
# ## LIVE (2026-08-24): wired into kustomization.yaml, reconciled by Flux ##
|
|
# ## and conformance-green (egress-class + vpn-killswitch pass on dev). ##
|
|
# ##########################################################################
|
|
#
|
|
# This policy steers vpn-class pods out the gateway node's host `tun0` interface. It REQUIRES the
|
|
# Ansible-managed host OpenVPN split-tunnel service on a node labeled
|
|
# `homelab.keithjackson.dev/egress-gateway=true` — deploy that role BEFORE this policy reconciles, or
|
|
# vpn-class pods are SNAT'd to a non-existent interface. The tunnel is live and validated end-to-end;
|
|
# if tun0 is ever down, the host nftables kill-switch fails vpn-class CLOSED (see below). ADR-0015
|
|
# chose this mechanism (node tunnel + Cilium Egress Gateway); this file IS that datapath.
|
|
#
|
|
# ---------------------------------------------------------------------------
|
|
# PREREQUISITES CONFIRMED (2026-08-24):
|
|
# * Cilium Egress Gateway is ENABLED per ADR-0002 — platform/cilium/helmrelease.yaml and
|
|
# platform/cilium/values.yaml both set (all [parity] with the Ansible bootstrap render):
|
|
# egressGateway.enabled: true
|
|
# bpf.masquerade: true (Egress Gateway REQUIRES eBPF masquerade)
|
|
# kubeProxyReplacement: true
|
|
# No Cilium HelmRelease change is needed for this policy. (If any of the three were false this
|
|
# would be an ESCALATION, not a silent HelmRelease edit — they are all true, so we proceed.)
|
|
#
|
|
# INTERFACE CONTRACT (shared verbatim with the parallel Ansible host-tunnel role — ADR-0015):
|
|
# * Gateway node label ...... homelab.keithjackson.dev/egress-gateway=true (applied by Ansible)
|
|
# * Host tunnel interface .... tun0 (split-tunnel OpenVPN)
|
|
# * Client selection ......... homelab.keithjackson.dev/egress-class=vpn (Kyverno-enforced)
|
|
# ---------------------------------------------------------------------------
|
|
apiVersion: cilium.io/v2
|
|
kind: CiliumEgressGatewayPolicy
|
|
metadata:
|
|
name: vpn-egress-steering
|
|
annotations:
|
|
homelab.keithjackson.dev/requirements: R-NET-04,R-NET-05,R-NET-06,R-NET-07
|
|
homelab.keithjackson.dev/adr: ADR-0015
|
|
spec:
|
|
# --- Client selection (INVARIANTS 1 & 2) --------------------------------
|
|
# ONLY pods explicitly opted into the vpn class are matched. Default/internet-class pods and node
|
|
# traffic are never selected, so they stay on the local/private-WAN path (invariant 1: default is
|
|
# the local path). Because ONLY these flows are steered, tun0 never becomes a default route for the
|
|
# node or for any other pod (invariant 2). The selector is intentionally cluster-wide (no namespace
|
|
# scope) so the class label alone is the contract — every namespace's vpn-class pods route here.
|
|
selectors:
|
|
- podSelector:
|
|
matchLabels:
|
|
homelab.keithjackson.dev/egress-class: vpn
|
|
# --- What gets tunneled (INVARIANT 1: only PUBLIC internet leaves via the VPN) ------------------
|
|
# destinationCIDRs is the full internet; excludedCIDRs carves out everything that must stay LOCAL:
|
|
# all RFC1918 private ranges (which also cover the Cilium cluster-pool pod CIDR 10.0.0.0/8, the k3s
|
|
# service/pod ranges, and the LAN 192.168.x/10.73.x node network). So intra-cluster east-west,
|
|
# pod->service, and LAN traffic are NOT SNAT'd out tun0 — only genuinely public destinations are.
|
|
# Excluded flows fall through to Cilium's normal local datapath (invariant 1).
|
|
destinationCIDRs:
|
|
- "0.0.0.0/0"
|
|
excludedCIDRs:
|
|
- "10.0.0.0/8" # RFC1918 + Cilium cluster-pool pod CIDR (CLUSTER_POD_CIDR) + k3s service net
|
|
- "172.16.0.0/12" # RFC1918
|
|
- "192.168.0.0/16" # RFC1918 (home LAN / node network family)
|
|
# --- Where it egresses --------------------------------------------------
|
|
# The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. When
|
|
# tun0 is absent/down, Cilium has no egress interface here; the host nftables kill-switch (Ansible,
|
|
# ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to
|
|
# vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED,
|
|
# internet fails OPEN).
|
|
#
|
|
# SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby.
|
|
# `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD
|
|
# deployed on this cluster (`ciliumegressgatewaypolicies.cilium.io`, v2, Cilium 1.20.1 —
|
|
# `kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`):
|
|
#
|
|
# "EgressGateway is the gateway node responsible for SNATing traffic.
|
|
# In case multiple nodes are a match for the given set of labels, the first node
|
|
# in lexical ordering based on their name will be selected."
|
|
#
|
|
# So labelling a second node does not create an active/standby pair — it just hands the role to
|
|
# whichever node sorts FIRST BY NAME. A labelled `k3s-prod-1` would silently take the role from a
|
|
# labelled `k3s-prod-3`, moving the tunnel (and the public egress IP) with no event and no alert.
|
|
# LABEL EXACTLY ONE NODE. The plural `egressGateways` field (also present in 1.20) is NOT a fix:
|
|
# it SHARDS endpoints across entries by CiliumEndpoint UID, so losing one entry's node blackholes
|
|
# that shard and re-shards the rest (breaking live connections), and each shard exits from a
|
|
# DIFFERENT public IP. Upstream: cilium/cilium#30157, cilium/cilium#18230.
|
|
# ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
|
|
# is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
|
|
# platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
|
|
egressGateway:
|
|
nodeSelector:
|
|
matchLabels:
|
|
homelab.keithjackson.dev/egress-gateway: "true"
|
|
interface: tun0
|
|
# ---------------------------------------------------------------------------
|
|
# DONE (2026-08-24): enabled + validated + Gluetun retired.
|
|
# * Host tun0 split-tunnel is up on the egress-gateway node; host nftables kill-switch in place.
|
|
# * This file is wired into platform/vpn-egress/kustomization.yaml and reconciled by Flux.
|
|
# * `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests PASS live.
|
|
# * The Gluetun Deployment was retired per ADR-0015 ("Gluetun is dropped as the gateway").
|
|
# ---------------------------------------------------------------------------
|