jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
ci/ — portable merge gate (runnable today, without Gitea)
Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).
The same merge gate the Gitea Actions workflow runs
(.gitea/workflows/), as plain scripts + a Makefile so the gates are
runnable now, before any Gitea host or cluster exists. CI and a laptop call the identical
scripts, so make -C ci gate == the CI merge gate.
Quick start
make -C ci gate # the single command: run every stage + the R-TEST-03 decision
make -C ci help # list targets
make -C ci tools # which pinned tools are installed locally
make -C ci touched # print the touched-requirement subset for this branch vs main
Individual stages:
make -C ci lint # yaml/lint (no cluster; live now)
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
make -C ci l1 # L1 rendering checks (no cluster; live now)
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
The l0 stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
(kyverno test), and conformance/run.sh --layer l0 — the whole-repo L0 invariants a
per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
spec-immutability enforcement). See conformance/tests/README.md.
Exit-code contract
Every stage script uses: 0 = pass · 1 = fail (blocks merge) · 3 = SKIPPED (cluster-gated
with no cluster, or an unwired seam / missing tool). A 3 is never counted as green — that is how
the gate stays honest in Phase 0 (no fake green).
The touched-requirement subset can also be undeterminable (exit 3)
touched-requirements.sh is the input to the R-TEST-03 decision. It resolves a diff base in this
order, verifying each candidate exists before using it:
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
If no candidate resolves, if there is no merge base, if the diff command fails, or if HEAD is not
ahead of the base (you are sitting on main, or the branch is already merged), it prints a loud
banner, writes nothing to stdout, and exits 3 — plus a machine-readable reason code
(on-base / no-base / diff-failed) to $TOUCHED_REASON_FILE when that variable names a path.
Why this is not cosmetic. Every one of those conditions used to degrade silently to an empty
touched set and exit 0. An empty set means conformance/run.sh is invoked with no --requirements,
which means its coverage --require gate never runs — so on 2026-08-31 the local gate printed
MERGE GATE: PASS having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
on real coverage debt. There is no silent narrowing left: the HEAD~1 last resort and the two-dot
diff fallback are both gone. Locked down by ci/tests/touched-requirements-base.test.sh.
gate.sh turns exit 3 into an UNDECIDED merge decision, never a pass:
| reason | headline | exit |
|---|---|---|
on-base |
MERGE GATE: NOT APPLICABLE — no branch under gate |
0 (a push-to-main CI run legitimately lands here; it just must not read as "my branch is green") |
no-base, diff-failed |
MERGE GATE: BLOCKED |
1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
A stage that needs the subset (l0.sh, l1.sh) still runs its FULL layer as a superset sanity net
in that case — nothing goes untested — and says plainly that the scoped check did not happen.
Reproducibility (R-TEST-02)
- Version-pinned: all tool versions live in
versions.env— the single source of truth shared by the local scripts and the CI installer. Bumping a version is a normal feature-branch → main change. - Single command, environment-agnostic:
make -C ci gateon any machine with the pinned tools. - Deterministic: no cluster state is assumed; L2/L3 SKIP unless a cluster is explicitly declared.
Files
ci/
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
versions.env PINNED tool versions (R-TEST-02) — single source of truth
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
yamllint.yaml lint config for the yaml/lint stage
lint.sh Stage: yaml/lint
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
exits 3 LOUDLY rather than guessing when the diff base is unreliable
gate.sh run all stages + the merge decision (the single command)
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
throwaway git repos via the REPO_ROOT override — no network, no cluster
README.md this file
Integration seams — entrypoints owned by OTHER agents
This gate is an orchestrator. It does not implement schema validation, Kyverno policies, or the conformance runner — those are other workstreams' deliverables. Each seam is env-overridable so the owning agent / PM can reconcile the real entrypoint without editing these scripts. Defaults below are this workstream's assumptions — flagged for the PM to verify at integration.
| Seam | Owner / task | Env override | Assumed default | Assumed interface |
|---|---|---|---|---|
| ServiceDefinition schema validator | WS-GITOPS · T-0.3 (schemas/) |
SCHEMA_VALIDATE_CMD |
schemas/validate.sh |
validate.sh catalog/*.yaml → exit 0/1; --version supported |
| Kyverno L0 policies + tests | WS-TEST · T-0.5 (platform/kyverno/) |
KYVERNO_POLICY_DIR |
platform/kyverno |
contains kyverno-test.yaml files discoverable by kyverno test <dir> |
| Conformance runner (Chainsaw) | WS-TEST · T-0.4 (conformance/) |
CONFORMANCE_CMD |
conformance/run.sh |
run.sh --layer <l0|l1|l2,l3> [--requirements R-A,R-B]; artifacts land in conformance/_artifacts/ |
| Catalog (ServiceDefinitions) | WS-APPS | CATALOG_DIR |
catalog |
*.yaml ServiceDefinition files |
Artifact locations (git-ignored): JUnit → results/junit/*.xml; coverage → reports/coverage/**.
Override with JUNIT_DIR / COVERAGE_DIR if the runner writes elsewhere.
A seam that is missing or not executable reports SKIPPED (exit 3), not a false pass — so during parallel Phase 0 development the gate is green-honest, and each seam goes live automatically the moment its owner delivers a runnable entrypoint.
Cluster gating
CLUSTER_AVAILABLE=true (env locally, vars.CLUSTER_AVAILABLE in Gitea) makes the L2/L3 stage run and
be required; anything else makes it SKIP (Phase 0). REQUIRE_CLUSTER_TESTS=true (Phase 1+)
turns any unverified cluster-needing touched requirement into a hard block.