jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
78 lines
4.7 KiB
YAML
78 lines
4.7 KiB
YAML
# =============================================================================
|
||
# L0 STATIC — prod is bare metal, and is not a development surface (R-ENV-03) [EXECUTABLE]
|
||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||
# Spec: docs/00 R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local
|
||
# storage; it MUST NOT be an active development surface until the roadmap
|
||
# declares it ready"); docs/02 (prod = 3 physical nodes, local SSD/NVMe to
|
||
# Longhorn, "the *same* Ansible bootstrap runs; only the machine-creation step
|
||
# differs", "Never develop against prod"); roadmap/phases.md Phase 7.
|
||
#
|
||
# HOW A "MUST NOT" IS MADE TESTABLE — the same discipline as R-TEST-06's test: a
|
||
# governance rule is verified through the property that ENFORCES it, never by observing
|
||
# that nobody misbehaved today. Here that property is structural: prod receives the base
|
||
# configuration plus environment-scoped overrides and nothing else, from the same branch
|
||
# and the same Ansible code as dev, so there is no place to put prod-only work at all.
|
||
#
|
||
# ENVIRONMENT-AGNOSTIC: the prod inventory group is DERIVED (it is the group whose
|
||
# group_vars retargets the k3s roles away from the global default), the prod overlay is
|
||
# derived from CLUSTER_ENV, and the storage/replica expectations are compared against the
|
||
# node count that inventory resolves to. No hostname, address or environment name is
|
||
# written into the test.
|
||
#
|
||
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
||
# WHEN the prod inventory, overlay, IaC module and readiness record are inspected — and
|
||
# the shared provisioning chain is actually retargeted at prod with --list-hosts,
|
||
# THEN prod resolves to exactly three nodes reached directly on their own sshd; no IaC
|
||
# resource creates any of them (bare metal is imaged, not instantiated); prod's
|
||
# Longhorn path is node-local with one replica per node and the default
|
||
# StorageClass is the local driver; prod runs the same playbooks/roles as dev,
|
||
# retargeted only by variables the shared code already defines; the prod overlay
|
||
# carries only the base plus environment-scoped patches on the same Git ref dev
|
||
# tracks, with no prod-only manifest anywhere in the shared configuration; and prod
|
||
# artifacts exist only alongside the operator's recorded opening of the prod phase.
|
||
#
|
||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
|
||
# where present — ansible-playbook in --list-hosts mode, which opens no connection);
|
||
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
|
||
# wraps the SAME script.
|
||
#
|
||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||
# * THE HARDWARE. That the three machines are mini-PCs, are bare metal, and hold local
|
||
# SSD/NVMe is operator-attested (infra/PROD-STANDUP.md, PROD-PROVISIONING.md); the
|
||
# node-health/BIOS surveys that establish it write to gitignored artifacts, and the
|
||
# Kubernetes API exposes no "am I virtual" fact, so nothing here asserts it.
|
||
# * The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
|
||
# disks at the local path) — cluster state, belonging to R-PLAT-01's still-uncovered
|
||
# HA test; asserting it here would also make this test environment-specific, since
|
||
# the same suite must pass against dev.
|
||
# * Git HISTORY: this is the working tree at HEAD.
|
||
# * Whether the roadmap SHOULD have opened prod — only that prod work is accompanied by
|
||
# the recorded declaration.
|
||
# =============================================================================
|
||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||
kind: Test
|
||
metadata:
|
||
name: prod-baremetal-not-dev-surface
|
||
labels:
|
||
conformance.home.k3s/R-ENV-03: "true"
|
||
annotations:
|
||
conformance.home.k3s/requirements: "R-ENV-03"
|
||
conformance.home.k3s/layer: "L0"
|
||
conformance.home.k3s/safety-critical: "false"
|
||
conformance.home.k3s/executable: "true"
|
||
conformance.home.k3s/intent: >-
|
||
Given the tracked tree, when prod's inventory, overlay, IaC and readiness record are
|
||
inspected, then prod is authored as three bare-metal nodes with node-local storage,
|
||
bootstrapped by dev's own code retargeted by variables, receiving only
|
||
environment-scoped overrides on the same branch — so there is nowhere to develop
|
||
against prod — and it exists only because the roadmap phase was opened.
|
||
spec:
|
||
steps:
|
||
- name: then-prod-is-bare-metal-and-carries-no-development-of-its-own
|
||
description: >-
|
||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||
conformance/run.sh --layer l0 execute).
|
||
try:
|
||
- script:
|
||
content: bash ./check.sh
|