feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-07 00:59:22 -05:00
1 parent d1a6628a97
commit 6a8953e288
46 files changed
+7898 -71

No files matched your search

@@ -68,6 +68,15 @@ spec:
timeouts:
apply: 60s
assert: 180s
# -- R-ENV-02: the fixture's route hostnames MUST fall under the Gateway's wildcard listener
# (*.${BASE_DOMAIN}) or they never attach — so the suffix is READ FROM THE CLUSTER UNDER TEST
# (flux-system/cluster-vars, the ConfigMap Flux substitutes ${BASE_DOMAIN} from) rather than
# authored here. Test-level => in scope for the fixture apply AND every probe step below
# (step `outputs`, by contrast, are step-scoped — see the gwvip re-capture comments).
# fixtures/app-and-probe.yaml consumes it via chainsaw resource templating.
bindings:
- name: baseDomain
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
steps:
# -- GIVEN: the exposed-app fixture (default-deny + gateway-ingress allow + backend +
# Service + declared/undeclared HTTPRoutes) and a probe namespace. Assert the shared
@@ -139,12 +148,14 @@ spec:
env:
- name: GWVIP
value: ($gwvip)
- name: BASE_DOMAIN
value: ($baseDomain)
command:
- sh
- -c
- |
set -u
host="gw-declared-conformance.home.keithjackson.dev"
host="gw-declared-conformance.${BASE_DOMAIN}"
last=000
i=0
while [ "$i" -lt 30 ]; do
@@ -207,12 +218,14 @@ spec:
env:
- name: GWVIP
value: ($gwvip)
- name: BASE_DOMAIN
value: ($baseDomain)
command:
- sh
- -c
- |
set -u
host="gw-undeclared-conformance.home.keithjackson.dev"
host="gw-undeclared-conformance.${BASE_DOMAIN}"
reached_gw=no
i=0
while [ "$i" -lt 8 ]; do
@@ -300,3 +313,20 @@ spec:
apiVersion: v1
kind: Namespace
name: conformance-gw-probe
# -----------------------------------------------------------------------------------
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
# chainsaw already fails the test before reaching this step if any of them fails — but a
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
# forever and NEVER become a recorded proof of R-NET-01/02/03 — north-south reachability under default-deny, part of the isolation proof
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
# is reached only when everything above has passed. Asserted by
# tests/l0-static/auditability-proofs-producible.
- name: then-emit-the-proof-of-pass-sentinel
try:
- script:
content: echo GATEWAY_INGRESS_ISOLATION_PROVEN_OK
check:
(contains($stdout, 'GATEWAY_INGRESS_ISOLATION_PROVEN_OK')): true
@@ -10,6 +10,13 @@
# The namespace opts into the shared Gateway (gateway-access: shared) exactly like a
# rendered app namespace, so its HTTPRoutes attach to the real platform Gateway and the
# probe exercises the true north-south path (Gateway Envoy -> backend), not a stand-in.
#
# ENVIRONMENT-AGNOSTIC HOSTNAMES (R-ENV-02): the two HTTPRoute hostnames are NOT literals. They
# are chainsaw-templated from the `baseDomain` binding declared in ../chainsaw-test.yaml, which
# reads BASE_DOMAIN out of the flux-system/cluster-vars ConfigMap of the cluster under test — the
# same ConfigMap Flux substitutes ${BASE_DOMAIN} from. They MUST fall under the shared Gateway's
# wildcard listener (*.${BASE_DOMAIN}) or the routes never attach, which is precisely why a
# hardcoded suffix silently broke this test on any cluster that did not hold the apex.
---
apiVersion: v1
kind: Namespace
@@ -134,7 +141,8 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- gw-declared-conformance.home.keithjackson.dev
# gw-declared-conformance.<this cluster's BASE_DOMAIN> — see the header.
- (join('.', ['gw-declared-conformance', $baseDomain]))
rules:
- matches:
- path:
@@ -158,7 +166,8 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- gw-undeclared-conformance.home.keithjackson.dev
# gw-undeclared-conformance.<this cluster's BASE_DOMAIN> — see the header.
- (join('.', ['gw-undeclared-conformance', $baseDomain]))
rules:
- matches:
- path: