feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
d1a6628a97
commit
6a8953e288
46 files changed
+7898
-71
No files matched your search
@@ -0,0 +1,106 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
|
||||
# operator can prove isolation, egress path, cert validity, and backup integrity)"),
|
||||
# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
|
||||
# must be able to prove": Isolation (denied-probe test), Egress path (egress IP
|
||||
# check), Kill-switch (fault-injection test), TLS (cert status + external check),
|
||||
# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
|
||||
# one-time checks", with the acceptance gate "The five auditability proofs above can
|
||||
# be produced on demand."
|
||||
#
|
||||
# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
|
||||
# -----------------------------------------------------------------------------------
|
||||
# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
|
||||
# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
|
||||
# failure modes are asserted instead, each by DRIVING the real machinery:
|
||||
# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
|
||||
# the spec's own words require (fault injection / a DR drill => L3), taken from
|
||||
# lib/status.py's own verdict so this check cannot disagree with the status report;
|
||||
# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
|
||||
# must resolve to real test dirs AND pass the merge gate's coverage assertion;
|
||||
# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
|
||||
# attribute — a proof that passes but records nothing leaves the operator with a green
|
||||
# terminal and no evidence;
|
||||
# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
|
||||
# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
|
||||
# token, and a ledger row claiming `proven` at the repository's ROOT commit — and
|
||||
# requires record_run.py to report not-proven and status.py to report `stale-proof`. It
|
||||
# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
|
||||
# ci/ or run.sh ran before this test existed.
|
||||
# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
|
||||
# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
|
||||
# so a sixth proof added to the spec fails here instead of being silently unverified.
|
||||
#
|
||||
# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
|
||||
# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
|
||||
# anyone needs it.
|
||||
#
|
||||
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
||||
# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
|
||||
# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
|
||||
# and two synthetic forgeries are presented to them,
|
||||
# THEN each of docs/07's five proofs maps to an executable test at the required layer, the
|
||||
# documented one-command invocation selects and asserts them, each is capturable as a
|
||||
# ledger row, and neither a self-skip nor a stale row can be recorded as proof.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
|
||||
# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
|
||||
# it directly, and this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
|
||||
# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
|
||||
# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
|
||||
# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
|
||||
# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
|
||||
# assertion here was NOT relaxed to accommodate them (R-TEST-06).
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * That the five proofs PASS. This asserts they are present, strong, runnable by one
|
||||
# command, capturable, and that the chain refuses forgeries — not that isolation holds
|
||||
# today (R-NET-01's test) or that the restore works (R-STATE-03's).
|
||||
# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
|
||||
# suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
|
||||
# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
|
||||
# would need signed history.
|
||||
# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
|
||||
# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
|
||||
# not re-asserted here.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: auditability-proofs-producible
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree and the harness's own tooling, when status.py, the requirement
|
||||
resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
|
||||
tools' self-tests are driven and two synthetic forgeries are presented to them, then each
|
||||
of docs/07's five auditability proofs has an executable test at the layer the spec
|
||||
requires, is selectable and asserted by the documented one-command invocation, is
|
||||
capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
|
||||
evidence.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
|
||||
# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
|
||||
# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
|
||||
# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
|
||||
# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
|
||||
# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
|
||||
# the wrong trade (R-TEST-06).
|
||||
Reference in new issue
Block a user