feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
d1a6628a97
commit
6a8953e288
46 files changed
+7898
-71
No files matched your search
@@ -68,6 +68,15 @@ spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 180s
|
||||
# -- R-ENV-02: the fixture's route hostnames MUST fall under the Gateway's wildcard listener
|
||||
# (*.${BASE_DOMAIN}) or they never attach — so the suffix is READ FROM THE CLUSTER UNDER TEST
|
||||
# (flux-system/cluster-vars, the ConfigMap Flux substitutes ${BASE_DOMAIN} from) rather than
|
||||
# authored here. Test-level => in scope for the fixture apply AND every probe step below
|
||||
# (step `outputs`, by contrast, are step-scoped — see the gwvip re-capture comments).
|
||||
# fixtures/app-and-probe.yaml consumes it via chainsaw resource templating.
|
||||
bindings:
|
||||
- name: baseDomain
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
||||
steps:
|
||||
# -- GIVEN: the exposed-app fixture (default-deny + gateway-ingress allow + backend +
|
||||
# Service + declared/undeclared HTTPRoutes) and a probe namespace. Assert the shared
|
||||
@@ -139,12 +148,14 @@ spec:
|
||||
env:
|
||||
- name: GWVIP
|
||||
value: ($gwvip)
|
||||
- name: BASE_DOMAIN
|
||||
value: ($baseDomain)
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -u
|
||||
host="gw-declared-conformance.home.keithjackson.dev"
|
||||
host="gw-declared-conformance.${BASE_DOMAIN}"
|
||||
last=000
|
||||
i=0
|
||||
while [ "$i" -lt 30 ]; do
|
||||
@@ -207,12 +218,14 @@ spec:
|
||||
env:
|
||||
- name: GWVIP
|
||||
value: ($gwvip)
|
||||
- name: BASE_DOMAIN
|
||||
value: ($baseDomain)
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -u
|
||||
host="gw-undeclared-conformance.home.keithjackson.dev"
|
||||
host="gw-undeclared-conformance.${BASE_DOMAIN}"
|
||||
reached_gw=no
|
||||
i=0
|
||||
while [ "$i" -lt 8 ]; do
|
||||
@@ -300,3 +313,20 @@ spec:
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
name: conformance-gw-probe
|
||||
# -----------------------------------------------------------------------------------
|
||||
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
|
||||
# chainsaw already fails the test before reaching this step if any of them fails — but a
|
||||
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
|
||||
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
|
||||
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
|
||||
# forever and NEVER become a recorded proof of R-NET-01/02/03 — north-south reachability under default-deny, part of the isolation proof
|
||||
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
|
||||
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
|
||||
# is reached only when everything above has passed. Asserted by
|
||||
# tests/l0-static/auditability-proofs-producible.
|
||||
- name: then-emit-the-proof-of-pass-sentinel
|
||||
try:
|
||||
- script:
|
||||
content: echo GATEWAY_INGRESS_ISOLATION_PROVEN_OK
|
||||
check:
|
||||
(contains($stdout, 'GATEWAY_INGRESS_ISOLATION_PROVEN_OK')): true
|
||||
+11
-2
@@ -10,6 +10,13 @@
|
||||
# The namespace opts into the shared Gateway (gateway-access: shared) exactly like a
|
||||
# rendered app namespace, so its HTTPRoutes attach to the real platform Gateway and the
|
||||
# probe exercises the true north-south path (Gateway Envoy -> backend), not a stand-in.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC HOSTNAMES (R-ENV-02): the two HTTPRoute hostnames are NOT literals. They
|
||||
# are chainsaw-templated from the `baseDomain` binding declared in ../chainsaw-test.yaml, which
|
||||
# reads BASE_DOMAIN out of the flux-system/cluster-vars ConfigMap of the cluster under test — the
|
||||
# same ConfigMap Flux substitutes ${BASE_DOMAIN} from. They MUST fall under the shared Gateway's
|
||||
# wildcard listener (*.${BASE_DOMAIN}) or the routes never attach, which is precisely why a
|
||||
# hardcoded suffix silently broke this test on any cluster that did not hold the apex.
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
@@ -134,7 +141,8 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- gw-declared-conformance.home.keithjackson.dev
|
||||
# gw-declared-conformance.<this cluster's BASE_DOMAIN> — see the header.
|
||||
- (join('.', ['gw-declared-conformance', $baseDomain]))
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -158,7 +166,8 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- gw-undeclared-conformance.home.keithjackson.dev
|
||||
# gw-undeclared-conformance.<this cluster's BASE_DOMAIN> — see the header.
|
||||
- (join('.', ['gw-undeclared-conformance', $baseDomain]))
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
Reference in new issue
Block a user