jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
76 lines
4.5 KiB
YAML
76 lines
4.5 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — dev/test nodes are provisioned as code, end to end (R-ENV-01) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC:
|
|
# created, baselined, cluster-joined, and managed as code"); the four stages
|
|
# are enumerated in docs/02 §"Node lifecycle stages".
|
|
#
|
|
# WHY L0: R-ENV-01 is a requirement about HOW the dev nodes came to exist, and a live
|
|
# Node object cannot answer it — a node built by OpenTofu and one built by hand in a
|
|
# hypervisor UI are indistinguishable through the Kubernetes API. What is provable, with
|
|
# no cluster, is that a COMPLETE and EXECUTABLE provisioning path exists for all four
|
|
# stages and covers EVERY dev node. For the Ansible half that is proven by DRIVING the
|
|
# real toolchain offline (`--syntax-check`, `--list-hosts` — neither opens a connection
|
|
# to any node), not by reading files: a chain that does not resolve fails the check.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count appears in the test.
|
|
# The dev group names are read from the playbooks' own default host patterns, the node
|
|
# names are generated from the IaC module's prefix/count variables, and the two sets are
|
|
# compared against each other — so the assertion survives a rename or a resize.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN the provisioning chain is parsed and (where offline execution is possible) driven,
|
|
# THEN the IaC module creates the dev nodes on a LOCAL KVM hypervisor (allowlisted
|
|
# provider — a cloud provider fails), from a for_each map rather than a count index,
|
|
# each with CPU/RAM/OS disk/dedicated Longhorn data disk/NIC/cloud-init seed; the
|
|
# node set it creates is EXACTLY the set Ansible manages; the baseline stage applies
|
|
# the package floor, kernel modules, sysctls, data-disk mount, swap-off and iscsid
|
|
# declaratively; one k3s role founds the cluster with --cluster-init and joins the
|
|
# rest with --server while disabling flannel/kube-proxy/traefik/servicelb at
|
|
# install; and site.yml's chain (baseline -> k3s -> Cilium -> verify -> Flux)
|
|
# resolves and targets every dev node at every stage.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
|
|
# where present — ansible-playbook in no-connection modes; no cluster, no network);
|
|
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
|
|
# wraps the SAME script.
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * That the RUNNING dev VMs were created by this module — needs the OpenTofu state or
|
|
# hypervisor access, neither available offline.
|
|
# * `tofu validate` — needs `tofu init` (provider registry), i.e. network. The module is
|
|
# parsed, not evaluated; a network-allowed CI stage would close this.
|
|
# * "Never hand-edited on the box" (docs/02 stage 4's second half) — a live drift check.
|
|
# * REBUILD/re-converge (docs/02's own proof line) — exercised at L3 by
|
|
# tests/l3-fault-injection/node-loss (R-STATE-02).
|
|
# * docs/02 prose puts the baseline in cloud-init; the implementation moved it into the
|
|
# Ansible node_baseline role. R-ENV-01 requires the node to be "baselined ... as code",
|
|
# not that cloud-init be the tool, so the check asserts the STAGE and RECORDS the
|
|
# divergence rather than hiding it.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: dev-nodes-provisioned-as-code
|
|
labels:
|
|
conformance.home.k3s/R-ENV-01: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-ENV-01"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree, when the dev provisioning chain is parsed and driven
|
|
offline, then every dev node is created by local-KVM IaC, baselined, cluster-joined
|
|
and handed to Flux by one code path that covers all of them — with no node outside
|
|
it and no stage missing.
|
|
spec:
|
|
steps:
|
|
- name: then-every-dev-node-is-created-baselined-joined-and-managed-as-code
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|