feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-07 00:59:22 -05:00
1 parent d1a6628a97
commit 6a8953e288
46 files changed
+7898 -71

No files matched your search

@@ -0,0 +1,77 @@
# =============================================================================
# L0 STATIC — prod is bare metal, and is not a development surface (R-ENV-03) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
# Spec: docs/00 R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local
# storage; it MUST NOT be an active development surface until the roadmap
# declares it ready"); docs/02 (prod = 3 physical nodes, local SSD/NVMe to
# Longhorn, "the *same* Ansible bootstrap runs; only the machine-creation step
# differs", "Never develop against prod"); roadmap/phases.md Phase 7.
#
# HOW A "MUST NOT" IS MADE TESTABLE — the same discipline as R-TEST-06's test: a
# governance rule is verified through the property that ENFORCES it, never by observing
# that nobody misbehaved today. Here that property is structural: prod receives the base
# configuration plus environment-scoped overrides and nothing else, from the same branch
# and the same Ansible code as dev, so there is no place to put prod-only work at all.
#
# ENVIRONMENT-AGNOSTIC: the prod inventory group is DERIVED (it is the group whose
# group_vars retargets the k3s roles away from the global default), the prod overlay is
# derived from CLUSTER_ENV, and the storage/replica expectations are compared against the
# node count that inventory resolves to. No hostname, address or environment name is
# written into the test.
#
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
# WHEN the prod inventory, overlay, IaC module and readiness record are inspected — and
# the shared provisioning chain is actually retargeted at prod with --list-hosts,
# THEN prod resolves to exactly three nodes reached directly on their own sshd; no IaC
# resource creates any of them (bare metal is imaged, not instantiated); prod's
# Longhorn path is node-local with one replica per node and the default
# StorageClass is the local driver; prod runs the same playbooks/roles as dev,
# retargeted only by variables the shared code already defines; the prod overlay
# carries only the base plus environment-scoped patches on the same Git ref dev
# tracks, with no prod-only manifest anywhere in the shared configuration; and prod
# artifacts exist only alongside the operator's recorded opening of the prod phase.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
# where present — ansible-playbook in --list-hosts mode, which opens no connection);
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
# wraps the SAME script.
#
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
# * THE HARDWARE. That the three machines are mini-PCs, are bare metal, and hold local
# SSD/NVMe is operator-attested (infra/PROD-STANDUP.md, PROD-PROVISIONING.md); the
# node-health/BIOS surveys that establish it write to gitignored artifacts, and the
# Kubernetes API exposes no "am I virtual" fact, so nothing here asserts it.
# * The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
# disks at the local path) — cluster state, belonging to R-PLAT-01's still-uncovered
# HA test; asserting it here would also make this test environment-specific, since
# the same suite must pass against dev.
# * Git HISTORY: this is the working tree at HEAD.
# * Whether the roadmap SHOULD have opened prod — only that prod work is accompanied by
# the recorded declaration.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: prod-baremetal-not-dev-surface
labels:
conformance.home.k3s/R-ENV-03: "true"
annotations:
conformance.home.k3s/requirements: "R-ENV-03"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given the tracked tree, when prod's inventory, overlay, IaC and readiness record are
inspected, then prod is authored as three bare-metal nodes with node-local storage,
bootstrapped by dev's own code retargeted by variables, receiving only
environment-scoped overrides on the same branch — so there is nowhere to develop
against prod — and it exists only because the roadmap phase was opened.
spec:
steps:
- name: then-prod-is-bare-metal-and-carries-no-development-of-its-own
description: >-
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
conformance/run.sh --layer l0 execute).
try:
- script:
content: bash ./check.sh