jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
89 lines
5.6 KiB
YAML
89 lines
5.6 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — the HA control plane is authored, for every environment (R-PLAT-01) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
|
|
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
|
|
# exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA,
|
|
# embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)",
|
|
# "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs").
|
|
#
|
|
# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the
|
|
# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments
|
|
# from one code path, with exactly one --cluster-init founder, no external datastore, and a
|
|
# floating API endpoint that is SAN'd into the serving cert and cannot collide with the
|
|
# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live
|
|
# cluster cannot tell you what the NEXT environment brought up from this repo would look
|
|
# like. The observed half is asserted separately at L2 by
|
|
# tests/l2-behavioral/ha-control-plane, which carries the same requirement label.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down.
|
|
# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group /
|
|
# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay
|
|
# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN the inventory, the k3s_server and kube_vip roles, their plays and each
|
|
# environment's Flux overlay are parsed and cross-checked against each other,
|
|
# THEN k3s is the pinned distro and nothing is installed as an agent; every environment
|
|
# resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY
|
|
# ONE --cluster-init founder and every other server joining it with --server;
|
|
# the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd);
|
|
# the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip
|
|
# DaemonSet on every server whose address is the environment-scoped VIP, that VIP is
|
|
# in the apiserver serving cert's SANs together with every server's own address, and
|
|
# it is neither a node address nor the ingress Gateway VIP nor inside that
|
|
# environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME
|
|
# roles and plays retargeted by variables only.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
|
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
|
# Chainsaw Test wraps the SAME script.
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating,
|
|
# API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane.
|
|
# * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves)
|
|
# is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss
|
|
# drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
|
|
# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
|
|
# infra/PROD-PROVISIONING.md).
|
|
#
|
|
# TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible):
|
|
# * `K3S_API_HOST` (Cilium's k8sServiceHost) was the PRIMARY NODE's address in both
|
|
# environments rather than the floating VIP, contradicting the prod overlay's own comment.
|
|
# It was ESCALATED, not asserted, because the workstream could not decide it (R-TEST-06).
|
|
# The operator decided the VIP, so it is now ASSERTED — on both sides of the [parity]
|
|
# contract and per environment.
|
|
# * kube-vip was NOT imported by playbooks/site.yml. It is now, BETWEEN k3s and cilium, and
|
|
# the order is asserted — not as a tidiness preference but because the CNI's endpoint is
|
|
# the VIP, so an unfloated VIP means a cold build brings up a CNI that cannot reach its
|
|
# API server.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: ha-control-plane-as-code
|
|
labels:
|
|
conformance.home.k3s/R-PLAT-01: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-PLAT-01"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each
|
|
environment's Flux overlay are parsed and cross-checked, then every environment is
|
|
authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one
|
|
embedded-etcd founder, no external datastore, and a leader-elected control-plane-only
|
|
VIP that is in the serving cert's SANs and collides with no other allocator — built by
|
|
one code path for both environments.
|
|
spec:
|
|
steps:
|
|
- name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|