`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as
vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first
resolution, so after a reconnect the BPF egress map points at a destroyed
interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat
in exactly that state for ~80 minutes with nothing alerting.
Two rules close it:
* VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was
re-created more times in the last hour than the gateway node's `egress-uplink`
label changed, i.e. Cilium was never told about a reconnect.
* VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the
watchdog: a gateway node whose uplink label never reaches kube-state-metrics is
a cluster where the rule above is inert.
Both gated on a gateway node existing, reusing `VPNTunnelDown`'s
`kube_node_labels` guard — a cluster with no gateway cannot be made to fire them.
The direct label-vs-map comparison is NOT available and nothing was invented to
pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the
cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS
publishes no series carrying egress-map contents at all — the map is readable only
via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the
live ifindex) against a number inside a label value, so the level-triggered
equality test is not expressible; the same invariant is counted over a window
instead. Both limits, and the textfile-collector/black-box-probe routes that would
lift them, are written down in the rule comments and the DECISIONS-LOG.
`egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same
kustomization, because without it the first rule has nothing to compare.
Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr
extracted from `kubectl kustomize platform-config`: empty when healthy; empty when
evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at
a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier
today. Nothing on prod was disturbed to produce that.
Co-Authored-By: Claude Opus 5 <[email protected]>
Closes the loop on the collision entry with the recovery evidence rather than
leaving it as an open question for the operator: dev's last Inactivity timeout is
23:05:06 UTC — the minute prod switched to .135 — and at 23:12:03 that same
session was still the live one, seven minutes unbroken against a two-minute
cycle, 4/4 ping replies at 146 ms, 0% loss, and vpn-class pod egress back to
158.173.241.36 on three consecutive probes.
Co-Authored-By: Claude Opus 5 <[email protected]>
The stand-up broke dev, so the "one gateway per PIA server" rule and the
measurement behind it are now written down where the next person will look:
platform/vpn-egress/README.md and the session's DECISIONS-LOG entry. Both carry
the verbatim journal lines — same server, same pushed `ifconfig 10.72.192.7` on
both gateways — and the flap timestamps, because "same account, inside the
connection limit" is true and is exactly what makes this non-obvious.
Also corrected: the dev/prod table now shows the two environments on different
servers with different exit IPs (dev 158.173.241.36, prod 158.173.241.11), and the
open item about them sharing one public IP is replaced with the real remaining
decision — whether prod gets its own PIA account, which would remove the
hand-maintained per-environment server value entirely.
The unplanned bonus, recorded in both places: moving prod to .135 gave it a
genuinely NEW tunnel address on a real reconnect (10.72.192.7 ifindex 128 ->
10.150.192.86 ifindex 129) — the precise case Cilium does not follow and the one
defect 1 was about. The egress-uplink label moved on its own, Cilium re-resolved,
and all four vpn-class workloads kept egressing with no manual step. Better
evidence than the synthetic address swap used to characterise it, and it cost
nothing to capture.
Co-Authored-By: Claude Opus 5 <[email protected]>
Four places in the tree told the reader that the host nftables table is the
authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress
README, egress-steering.yaml, the role README, and the vpn-killswitch test's own
GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway
datapath bpf_redirect()s those skbs to the egress device without invoking
netfilter POSTROUTING. Corrected with the measurement in each place rather than
softened, because a guard the repo trusts and that does not fire is worse than no
guard.
What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops
the traffic when it cannot use its cached egress interface, verified by tcpdump on
both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out
either). The nft table keeps a narrower, now-proven job: node-originated traffic
sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on
k3s-prod-3 when such a packet was routed at the WAN).
Also recorded:
* egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the
gateway node is what tells Cilium it changed (the egress-uplink node label). A
future reader's instinct will be to have the node write to that object; the
comment says not to, because it is Flux-managed, and points at the label and
`cilium-dbg bpf egress list` as the debugging entry points instead.
* egress-steering.yaml — why the new NFS media mounts are not affected by
pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns,
so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server
too, but that is the second reason, not the first.
* vpn-killswitch test — its recovery step depends on the re-steer path, and could
not have passed before it existed. Its no-leak assertion was always right; only
the explanation was wrong. Also states the prerequisite plainly: the test cannot
run at all without a labelled gateway node.
* platform README — prod is live on k3s-prod-3 with the before/after egress IPs,
and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no
matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway
restored egress rather than moving it off the WAN.
* role README — the standing "add the shared fwmark to the interface contract"
escalation is closed as not implementable on Cilium OSS, rather than left open
for someone to attempt.
DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 /
docs-03 amendment for the architect (specs are immutable to agents) and the five
things left for the operator — chief among them that a silent gateway outage still
has no alert, which is what let the dev gateway sit dead for 80 minutes.
Co-Authored-By: Claude Opus 5 <[email protected]>
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host
OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium
Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials
moved host-side to the Ansible vault. Update docs/comments only:
- platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green);
drop the resolved "option 1 not viable with Gluetun" finding to a brief history note.
- platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering.
- platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering.
- schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium.
- clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments
to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented).
- platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015.
- pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note
they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched.
No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads
changed. No spec-repo edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author the internet/vpn egress-class fixture pods for the egress-class + vpn-killswitch
conformance tests. Record the datapath finding: ADR-0013's preferred steering (Cilium
egress gateway -> gluetun tun0) is NOT viable because gluetun must run hostNetwork to
put tun0 on the node, and gluetun then redirects the NODE's default route through the
VPN (breaks k3s/etcd). Escalated: remaining viable path is option 2 (gateway-pod
routing). No steering datapath committed until the mechanism is chosen + validated.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gluetun cannot do PIA over WireGuard (upstream unsupported; live "provider name
is not valid for Wireguard"). Operator approved OpenVPN-UDP and ADR-0013 was
amended in the spec. This aligns the impl:
- platform/kustomization.yaml: re-enable `- vpn-egress` (Phase 3, OpenVPN-UDP)
- gluetun-gateway.yaml: clean stale WireGuard comments (header, image note,
VPN_TYPE/creds comments); config already VPN_TYPE=openvpn
- README.md: PIA-over-OpenVPN, status=enabled, steering iface wg0 -> tun0
- DECISIONS-LOG.md: disk/drive incident (k3s -> NVMe; failing Seagate) + Phase-3
OpenVPN resolution
Stale WireGuard comments inside the operator's encrypted pia-credentials.sops.yaml
header left untouched (editing breaks the SOPS MAC; cosmetic).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
gluetun-gateway.yaml referenced ${PIA_REGION}/${CLUSTER_POD_CIDR} but they were never defined
in cluster-vars, and the README didn't say where to set them. Add both to
clusters/base/cluster-vars.yaml (ConfigMap data:, string values, Flux postBuild substitution).
CLUSTER_POD_CIDR is 10.0.0.0/8 (Cilium cluster-pool — pods get 10.x, confirmed live), NOT the
k3s 10.42.0.0/16 I'd assumed. README now names the file + format + per-env override + PF-region note.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per ADR-0013: platform/vpn-egress/ = a single shared Gluetun gateway (PIA over WireGuard,
built-in kill-switch FIREWALL=on, native port forwarding), namespace (non-app, NET_ADMIN +
/dev/net/tun), control-API Service, and a pre-wired (commented) pia-credentials SOPS line.
Defined the PIA secret location (platform/vpn-egress, keys username/password → OPENVPN_USER/
OPENVPN_PASSWORD) and updated the SOPS template accordingly.
NOT wired into platform/kustomization.yaml (Phase 3, author-first) — deploy needs PIA creds +
validation of the egress-steering mechanism (Cilium Egress Gateway → the Gluetun node's wg0;
documented in README with options + a fail-closed rationale, deliberately NOT committed as an
untested CiliumEgressGatewayPolicy). WireGuard chosen (UDP, protocol != TCP); PIA supports it on
all regions and the secret stays username/password.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 5