Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
Closes the LAST scaffold-only safety-critical requirement. The operator has attested DR
works (Velero -> Proxmox versitygw S3, restore demonstrated), but the L3 test carried the
right tag and layer label over a 5-line TODO body — so coverage counted it and reported
"0 safety-critical gaps" while the suite asserted nothing. That was test debt, not
capability debt. This makes the suite assert what the operator already demonstrated.
Three steps:
1. assert-backup-target-is-off-cluster (read-only): BSL must be phase Available (Velero
itself reached the store), s3Url host is not localhost/*.svc/*.cluster.local, and the
resolved IPv4 is none of — loopback, any Node Internal/ExternalIP, any Service
ClusterIP, any Pod IP, inside any node podCIDR, inside any ServiceCIDR. A BSL that
exists but never reaches Available FAILS (a live DR defect), it does not skip.
2. assert-backups-are-scheduled: R-STATE-03 says SCHEDULED off-cluster backups — Velero
+ BSL but zero Schedule in phase Enabled is a fail.
3. The drill: seed a nanosecond-unique canary onto a longhorn PVC in its own scratch
namespace; Backup to Completed; assert a PodVolumeBackup for that volume is Completed
WITH a non-empty status.snapshotID (the kopia snapshot of volume DATA reached the
remote repo, not just resource JSON); ANNIHILATE the source (delete ns, poll until
gone, force-delete a lingering PV, FAIL if the original PV survived — a survivor makes
the restore prove nothing); Restore with namespaceMapping into a clean namespace;
assert the restored PVC bound a DIFFERENT PV and that the canary bytes are identical.
Off-cluster proof = endpoint outside every cluster address + BSL Available + kopia
snapshotID present + every in-cluster copy destroyed + restore landing on a new PV. The
bytes could only have come from outside.
Skips GREEN on: no longhorn SC, Velero API absent, no BSL, Velero/node-agent not ready —
all three steps re-derive the same gates so it can never half-run. An RBAC denial is
deliberately NOT a skip: it fails loudly as a harness defect. Teardown is a step-level
`finally` that ALWAYS runs — deletes the Restore, issues a DeleteBackupRequest so objects
leave the bucket (2h ttl as backstop), deletes both scratch namespaces by name and by
label, releases leftover PVs whose claimRef is one of its own. Everything it destroys is
something it created.
conformance/incluster/rbac.yaml: authored the R-STATE-03 grants (velero.io backups/
restores/deletebackuprequests create+delete+read; BSL/schedules/podvolume* read-only;
persistentvolumes get/list/DELETE — the off-cluster proof requires destroying the PV).
Also two pre-existing gaps other tests already depended on: storageclasses get/list (the
skip gate of five tests) and apps deployments/daemonsets read. No write verbs on platform
controllers.
NOT ASSERTED, stated in the file header rather than implied: off-HOST (on dev the nodes
are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor
loss); no direct S3 client read (that would hand a privileged test identity the backup
credential for weaker evidence); real service data (Nextcloud/Gitea/CNPG drills stay
operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC.
status: scaffold-only 5 -> 4; safety-critical w/o executable L3 1 -> 0 (of 6); executable
35 -> 36. coverage unchanged at 40/51 (correct — coverage only asks whether a tagged test
exists). PROVEN stayed 0 / not-derivable: it did NOT flip to proven just because the body
is non-empty.
AUTHORED, NEVER EXECUTED — carries a conformance.home.k3s/never-executed annotation. No
cluster access from this workstation. To convert authored -> proven the operator must, IN
A MAINTENANCE WINDOW (it deletes namespaces and PVs and writes/deletes bucket objects):
apply the updated rbac.yaml, run `./conformance/run.sh --requirements R-STATE-03 --layer
l3`, confirm the log shows DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK and NOT
DR_RESTORE_SKIPPED_OK (green but proves nothing), confirm no leaked *-dr-src/*-dr-dst
namespaces or orphan PVs or *-dr-backup objects, and record the green run.
Serves: R-STATE-03, R-TEST-05.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
My previous commit converted =~ to contains() but kept flow style
'check: { (contains($stdout, 'X')): true }' — the comma inside contains() is a
YAML flow-map separator, so 9 tests failed yamllint ('expected , or }'). Convert
those checks to block style (matching data-survival), where a comma in the key
scalar is fine. Gate PASS; data-survival still green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Every chainsaw `check` used the regex operator `=~` (`($stdout =~ 'X'): true`),
which chainsaw's expression engine rejects ('SyntaxError: Unknown char: ~') — so
EVERY such test failed on the assertion even when the behavior passed. Proven with
data-survival (R-STATE-01): the Longhorn canary genuinely SURVIVED pod destruction
(DATA_SURVIVED_OK) but the test FAILED on the `=~` syntax. Converted all 10 tests to
JMESPath `contains($stdout, 'X')` (data-survival's alternation → contains||contains).
data-survival now PASSES; the other 9 (later-phase components) are syntactically fixed
for when their phases deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 5