jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
69 lines
4.0 KiB
YAML
69 lines
4.0 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — a placeholder value in cluster-vars is REFUSED by the guard that owns it
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# THE BUG CLASS THIS CLOSES (it has now bitten three times in one file). A guard in
|
|
# `bootstrap/standup.sh` keys on the SHAPE of a value that lives in
|
|
# `clusters/base/cluster-vars.yaml`, and nothing tied the two together. On 2026-09-05 the
|
|
# S3 placeholder changed from `TBD-PROD-S3-BACKUP-ENDPOINT` to
|
|
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`; the three guards matched the glob
|
|
# `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is False. All three
|
|
# guards silently stopped matching, and `standup.sh resume` would have resumed `velero` on
|
|
# prod against a nonexistent hostname: installs clean, reports healthy, NO BACKUPS EXIST.
|
|
# It was found by accident. The guard is fixed (`s3_is_placeholder`, substring); the
|
|
# COUPLING was untested, which is what makes the class reintroducible.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# WHEN the REAL predicate is extracted from bootstrap/standup.sh and executed by bash
|
|
# (never reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"),
|
|
# THEN (1) every authored value of a guarded knob that is placeholder-shaped by an
|
|
# INDEPENDENT detector is actually refused by that guard, in base and in every
|
|
# environment overlay;
|
|
# (2) no environment overlay carries a placeholder-shaped value for a knob no guard
|
|
# refuses;
|
|
# (3) every function in standup.sh that reads a guarded knob applies the guard to it
|
|
# (the "three guards died at once" regression);
|
|
# (4) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix
|
|
# — the exact anchoring defect above — and still accepts a real endpoint;
|
|
# AND (5) the extraction is faithful, and the check fails LOUDLY rather than silently
|
|
# testing a copy if the guard stops being a self-contained predicate.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
|
|
# PyYAML + bash); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test
|
|
# wraps the SAME script.
|
|
#
|
|
# NOT proven: that the endpoint is REACHABLE or that backups exist (runtime facts —
|
|
# `standup.sh s3check` probes the former from a pod, l3-fault-injection/dr-restore proves
|
|
# the latter); that guards for knobs OTHER than the placeholder-refusing ones are correct;
|
|
# and that the guard runs at all in a real standup (the dispatch path is shell, executed
|
|
# only against a live cluster). R-STATE-03 is tagged for the narrow static slice only: the
|
|
# authored backup target cannot be a placeholder that the guard fails to recognise.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: placeholder-guard-coupling
|
|
labels:
|
|
conformance.home.k3s/R-ENV-02: "true"
|
|
conformance.home.k3s/R-STATE-03: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Every placeholder-shaped value authored in cluster-vars, for any environment
|
|
overlay, is actually recognised as a placeholder by the real bootstrap guard that
|
|
is supposed to refuse it — so a value change in one file cannot silently disarm a
|
|
guard in another and enable velero against an endpoint that does not exist.
|
|
spec:
|
|
steps:
|
|
- name: couple-authored-placeholders-to-the-real-guard
|
|
description: >-
|
|
Extract the real placeholder predicate from bootstrap/standup.sh, run it against
|
|
every authored cluster-vars value the repo actually carries, and assert the guard
|
|
refuses each one an independent detector calls a placeholder.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|