feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628) rather than by merging work/prod-standup-campaign. That branch carries the DNS apex inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun rewriting apex records dev owns while the non-k3s prod still serves 443 there. Operator sequencing is "dev migrates first". Nothing under clusters/ is touched here at all, which is what makes this safe to land now. Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of platform/velero/ and the CNPG destinationPath changes. Those land as feat/r2-backup-prod together with the endpoint and bucket knobs. WHAT THIS FIXES 1. Nothing was watching CNPG. All three prod databases have run since bootstrap with ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed anywhere. Because archive_mode=on and the archive command fails, Postgres refuses to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within the hour, and evaluates to 3 firing series on prod today. Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because nextcloud-db's exporter cannot log in to its own database (permission denied for database "nextcloud", both clusters) so the pg_stat_archiver family does not exist for it — the obvious metric would have been blind to one database in three. 2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to job="kubelet" (the apiserver copy of the same series reports namespace=default), and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report available=0 forever and would otherwise have fired critical permanently. 3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment a node carries the egress-gateway label, with dev's behaviour unchanged today. 4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored LITERALLY and the import still reports success, then sits at Done=True and never re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was green. The CronJob is the level-triggered correction; two alerts make its silence observable. Repaired live already — this stops it recurring. Also: the R-NET-04 conformance scan descended into nested agent worktrees under .claude/, counting 4 egress policies where 1 is authored. Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 00:59:22 -05:00
1 parent d1a6628a97
commit 6a8953e288
@@ -28,6 +28,45 @@ Three layers, two owners:
  `vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so
  `internet`-class and node traffic are unaffected.

+## Resilience — there is NO gateway failover (spec correction, escalated)
+
+**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
+provide the active/standby the earlier ADRs asked for", and spec `docs/03` repeats it. That feature
+does not exist in Cilium OSS.** The supported topology is **exactly one gateway node**.
+
+Verified against the CRD actually deployed on this cluster
+(`kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`, apiVersion `cilium.io/v2`,
+Cilium **v1.20.1**) — the field's own description:
+
+> **`egressGateway`** — "EgressGateway is the gateway node responsible for SNATing traffic.
+> **In case multiple nodes are a match for the given set of labels, the first node in lexical
+> ordering based on their name will be selected.**"
+
+Consequences, stated plainly:
+
+- **One gateway node is the supported topology.** `infra/ansible/inventory/hosts.ini`'s
+  `[vpn_egress_gateway]` group is singleton by design.
+- **A second labelled node does NOT provide failover.** It is not a standby; it carries no traffic
+  and takes over nothing when the first node dies. Gateway-node loss = `vpn`-class egress down
+  (fail-closed, per R-NET-06 — no leak, but no service).
+- **Lexical ordering decides who wins if two are ever labelled.** Labelling `k3s-prod-1` while
+  `k3s-prod-3` is the live gateway would **silently** move the role to `k3s-prod-1` — because `1`
+  sorts before `3` — taking the tunnel and the public egress IP with it, with no event and no alert.
+- **The plural `egressGateways` field (Cilium 1.20) is not a workaround.** It *shards* endpoints
+  across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node
+  blackholes that entry's share and re-shards the remainder (breaking live connections), and each
+  shard leaves from a *different* public IP. Upstream confirmation:
+  [cilium/cilium#30157](https://github.com/cilium/cilium/issues/30157) ("egress gateway doesn't
+  support failover between nodes") and
+  [cilium/cilium#18230](https://github.com/cilium/cilium/issues/18230) ("egress gateway does not
+  tolerate single node failure").
+
+**Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents
+(spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited
+here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in
+`DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file
+and `egress-steering.yaml` are the accurate statement of what the implementation can actually do.
+
 ## Secrets — host-side now
 PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not**
 a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one.
@@ -69,6 +69,26 @@ spec:
  # ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to
  # vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED,
  # internet fails OPEN).
+  #
+  # SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby.
+  # `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD
+  # deployed on this cluster (`ciliumegressgatewaypolicies.cilium.io`, v2, Cilium 1.20.1 —
+  # `kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`):
+  #
+  #     "EgressGateway is the gateway node responsible for SNATing traffic.
+  #      In case multiple nodes are a match for the given set of labels, the first node
+  #      in lexical ordering based on their name will be selected."
+  #
+  # So labelling a second node does not create an active/standby pair — it just hands the role to
+  # whichever node sorts FIRST BY NAME. A labelled `k3s-prod-1` would silently take the role from a
+  # labelled `k3s-prod-3`, moving the tunnel (and the public egress IP) with no event and no alert.
+  # LABEL EXACTLY ONE NODE. The plural `egressGateways` field (also present in 1.20) is NOT a fix:
+  # it SHARDS endpoints across entries by CiliumEndpoint UID, so losing one entry's node blackholes
+  # that shard and re-shards the rest (breaking live connections), and each shard exits from a
+  # DIFFERENT public IP. Upstream: cilium/cilium#30157, cilium/cilium#18230.
+  # ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
+  # is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
+  # platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
  egressGateway:
    nodeSelector:
      matchLabels: