jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
107 lines
7.2 KiB
YAML
107 lines
7.2 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
|
|
# operator can prove isolation, egress path, cert validity, and backup integrity)"),
|
|
# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
|
|
# must be able to prove": Isolation (denied-probe test), Egress path (egress IP
|
|
# check), Kill-switch (fault-injection test), TLS (cert status + external check),
|
|
# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
|
|
# one-time checks", with the acceptance gate "The five auditability proofs above can
|
|
# be produced on demand."
|
|
#
|
|
# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
|
|
# -----------------------------------------------------------------------------------
|
|
# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
|
|
# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
|
|
# failure modes are asserted instead, each by DRIVING the real machinery:
|
|
# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
|
|
# the spec's own words require (fault injection / a DR drill => L3), taken from
|
|
# lib/status.py's own verdict so this check cannot disagree with the status report;
|
|
# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
|
|
# must resolve to real test dirs AND pass the merge gate's coverage assertion;
|
|
# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
|
|
# attribute — a proof that passes but records nothing leaves the operator with a green
|
|
# terminal and no evidence;
|
|
# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
|
|
# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
|
|
# token, and a ledger row claiming `proven` at the repository's ROOT commit — and
|
|
# requires record_run.py to report not-proven and status.py to report `stale-proof`. It
|
|
# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
|
|
# ci/ or run.sh ran before this test existed.
|
|
# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
|
|
# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
|
|
# so a sixth proof added to the spec fails here instead of being silently unverified.
|
|
#
|
|
# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
|
|
# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
|
|
# anyone needs it.
|
|
#
|
|
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
|
# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
|
|
# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
|
|
# and two synthetic forgeries are presented to them,
|
|
# THEN each of docs/07's five proofs maps to an executable test at the required layer, the
|
|
# documented one-command invocation selects and asserts them, each is capturable as a
|
|
# ledger row, and neither a self-skip nor a stale row can be recorded as proof.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
|
|
# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
|
|
# it directly, and this Chainsaw Test wraps the SAME script.
|
|
#
|
|
# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
|
|
# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
|
|
# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
|
|
# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
|
|
# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
|
|
# assertion here was NOT relaxed to accommodate them (R-TEST-06).
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * That the five proofs PASS. This asserts they are present, strong, runnable by one
|
|
# command, capturable, and that the chain refuses forgeries — not that isolation holds
|
|
# today (R-NET-01's test) or that the restore works (R-STATE-03's).
|
|
# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
|
|
# suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
|
|
# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
|
|
# would need signed history.
|
|
# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
|
|
# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
|
|
# not re-asserted here.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: auditability-proofs-producible
|
|
labels:
|
|
conformance.home.k3s/R-SEC-03: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-SEC-03"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree and the harness's own tooling, when status.py, the requirement
|
|
resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
|
|
tools' self-tests are driven and two synthetic forgeries are presented to them, then each
|
|
of docs/07's five auditability proofs has an executable test at the layer the spec
|
|
requires, is selectable and asserted by the documented one-command invocation, is
|
|
capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
|
|
evidence.
|
|
spec:
|
|
steps:
|
|
- name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|
|
# ---------------------------------------------------------------------------------------------
|
|
# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
|
|
# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
|
|
# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
|
|
# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
|
|
# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
|
|
# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
|
|
# the wrong trade (R-TEST-06).
|