jakman85andClaude Opus 5
78d0f1b0f6
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the
declarative replacement gives what a hand-written stage cannot — selection by
`conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a
requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence,
and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3.
The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing
and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule
would otherwise be gated by nothing.
The check asserts the pins and their lockstep, that the image bakes and records promtool,
that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine
still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that
every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting
a synthetic rule through it twice with one expectation mutated.
MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of
58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests
exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG
tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only
register, printed with counts every run (13 alert files nothing tests at all; 35 named
alerts inside covered files that nothing asserts on), and it cannot grow: a new
PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale
entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one
genuine environment gap, promtool being absent.
PROVEN, in the rebuilt image (promtool 3.14.0):
* both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS.
* the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files.
* NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it
red (pending expected at 25m/29m, got firing), reverted. Adding an untested
PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet,
removed. The harness's own mutation self-test runs on every invocation.
* the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99
--include-scope whole-repo --print-selection` lists alert-rules-unit-tested.
Co-Authored-By: Claude Opus 5 <[email protected]>
Dev toolbox (containerized baseline)
The toolchain for this repo is not all Python (OpenTofu, age, sops, flux, chainsaw,
kyverno are Go binaries; only Ansible and a couple of linters are Python). Per the operator's
rule it is therefore delivered as a pinned dev container — no host pollution, one reproducible
baseline, and the same binaries CI uses.
This is the supported way to run anything in this repo. Don't install the toolchain on the host.
What's inside (pinned in versions.env)
| Group | Tools |
|---|---|
| Provisioning | OpenTofu (bpg/proxmox provider), ansible-core, openssh-clients |
| GitOps / cluster | flux, kubectl, helm |
| Secrets | sops, age + age-keygen |
| CI gate | yamllint, check-jsonschema, kyverno CLI, kyverno-json, chainsaw, yq, diffutils, make |
| Misc | qemu-img, cloud-utils/genisoimage, libvirt-client (disk-image utilities only — leftovers from the libvirt era, harmless) |
The conformance-tool versions are kept in lockstep with ci/versions.env so
make -C ci gate in here is byte-identical to the CI runner (R-TEST-02). The image records
the resolved set at /etc/dev-toolchain-versions.
Use
cd infra/devcontainer
./dev.sh build # build the image (needs network)
./dev.sh # build if needed + shell in /work (the repo)
./dev.sh run make -C ci gate # the merge gate, exactly as CI runs it
./dev.sh run sops -d path/to.sops.yaml
./dev.sh run tofu -chdir=/work/infra/tofu plan
ENGINE=docker ./dev.sh to use docker instead of rootless podman.
Choosing a cluster — CLUSTER=dev|prod
./dev.sh run kubectl get nodes # DEV — the default, unchanged
CLUSTER=prod ./dev.sh run kubectl get nodes # PROD
CLUSTER=prod ./dev.sh # interactive shell pointed at prod
An env-var prefix, so the target sits on the same line as the command. Every invocation prints
a banner naming the cluster before the container starts, because CLUSTER can also be inherited
from an exported shell variable.
Inside the container KUBECONFIG is a two-file chain,
<per-invocation selector>:<shared merged config>:
| file | shared? | holds |
|---|---|---|
selector, in a private mktemp dir |
no — one per dev.sh invocation |
only current-context: k3s-$CLUSTER; no credentials |
infra/ansible/.cache/clusters.kubeconfig |
yes — bind-mounted, read by every container | both clusters under unambiguous names; no current-context |
kubectl merges the chain first-wins, so the selector supplies this invocation's default:
CURRENT NAME CLUSTER AUTHINFO
* k3s-dev k3s-dev k3s-dev-admin
k3s-prod k3s-prod k3s-prod-admin
So kubectl config current-context answers "which cluster am I on?" honestly, and
kubectl --context k3s-prod … works from a dev-defaulted shell.
Why the default context is not stored in the merged file. It used to be, and that was a
cross-cluster data-corruption bug. The merged file lives in the repo's .cache, which every
container bind-mounts, so current-context was shared mutable state: two concurrent dev.sh
runs with different CLUSTER values overwrote each other's default mid-command. A
CLUSTER=prod script read prod for its first half and dev for its second; it was caught only
because a dev-only PVC name showed up in a prod query. Moving selection into a per-invocation file
removes the shared field rather than locking it — there is no window left to lose. Read on its own
the merged file now resolves to nothing (error: current-context is not set), which is loud and
safe; --context k3s-dev / --context k3s-prod against it are unaffected and remain the pattern
to prefer in scripts. Concurrent CLUSTER=dev and CLUSTER=prod invocations are now safe.
Consequences worth knowing:
- Export what
kubeconfig.shprints verbatim — it is a colon-joined chain, not one path. kubectl config use-context …in an interactive toolbox shell writes to the first file in the chain (the selector), so a manual switch stays local to that container.
Why a merge script and not KUBECONFIG=dev.kubeconfig:prod.kubeconfig. k3s names the context,
the cluster and the user in both files default. kubectl's merge keeps the first
occurrence of a duplicated name, so a colon-joined KUBECONFIG silently resolves default to
dev and --context default can never reach prod — you would believe you were on prod while
issuing commands against the live dev cluster. kubeconfig.sh therefore renames on merge
(k3s-dev / k3s-prod, *-admin users) with yq, structurally — not with sed, which corrupts
these files because the cluster and the user entry are both - name: default at identical indent.
- Generator:
kubeconfig.sh. Output:infra/ansible/.cache/clusters.kubeconfig, mode 0600, regenerated on everydev.sh run— a build artifact, never hand-edited, never committed (.cache/and*.kubeconfigare both gitignored; it embeds cluster-admin certs). Its content no longer depends on which cluster was asked for, so concurrent regenerations write identical bytes; the write is published by atomic rename so a concurrent reader can never load a half-written kubeconfig. - A cluster whose source kubeconfig is missing is simply omitted from the merge, so a machine with
only
dev.kubeconfigstill works. - Asking for a cluster whose kubeconfig is absent is a hard error.
dev.shrefuses on the host before starting a container and never falls back to the other cluster — that fallback is exactly how a prod command lands on dev. bootstrap/standup.shis unaffected: it exports its ownKUBECONFIGfrom the mandatory--kubeconfigflag, which overrides whatdev.shsets. Its--expect-serverguard remains the thing that makes it safe, notCLUSTER.
DEV_KUBECONFIG — superseded, still honoured
DEV_KUBECONFIG=<path> ./dev.sh run … still pins KUBECONFIG verbatim and takes precedence over
CLUSTER (nothing depending on it breaks), but it is deprecated: the name says "dev" while the
established workaround set it to a prod path, and it produces a single-cluster config whose
context is the ambiguous default. Use CLUSTER=dev|prod. Invocations that use it print a warning.
What dev.sh mounts, and why each one matters
--network host— Ansible SSHes to the nodes through the pfSense NAT (192.168.2.102:225x) andkubectl/flux/tofureach the kube-API (192.168.2.50:6443) and the Proxmox API (192.168.2.102:8006) on the host's network.-v $REPO:/work(workdir) — the repo.-v ~/.config/sops:/root/.config/sops— the age keyring, read-write. This is deliberate: the private key generated bybootstrap/sops/age-keygen.shmust land on the host, or it dies with the--rmcontainer and every SOPS-encrypted secret in the repo becomes undecryptable.-v ~/.ssh:/root/.ssh:ro— node SSH key (home_network).-v ~/.config/home-k3s/ansible-vault-pass:...:ro— the Ansible vault passphrase, so vaultedgroup_varsdecrypt unattended.-v ../home-network-k3s:/home-network-k3s:ro+SPEC_REPO— the spec repo, read-only, when it exists beside this one. Without itci/spec-immutability.shhonestly skips rather than passing — mount it, or that gate silently verifies nothing.--security-opt label=disable— SELinux hosts, so the bind mounts work without relabeling.
tofu and ansible now run inside the container
This reverses the old Phase-1 limitation. When the dev cluster was on host libvirt, OpenTofu's
dmacvicar/libvirt provider could not do a read-write qemu:///system connect from inside a
container (polkit/PID-identity: Cannot find start time for pid), so tofu had to be extracted and
run on the host. The cluster moved to Proxmox VE and the provider is now bpg/proxmox, which
talks to the Proxmox HTTP API — nothing host-local is required. Ansible likewise only needs
SSH, which --network host provides.
So: run everything in here. The host needs only podman (or docker).
kubectl inside the container — was a known gap, now fixed
dev.sh used to hard-wire KUBECONFIG to dev.kubeconfig with no supported way to reach prod, and
it does not mount ~/.kube. It now selects between infra/ansible/.cache/dev.kubeconfig and
prod.kubeconfig via CLUSTER=dev|prod (see above) and merges both under distinct context names.
The container still needs a break-glass cert kubeconfig, not the k3s-oidc context — OIDC login
needs a browser, which a container shell does not have. See
../PHASE1-STATUS.md for both access paths and for how to regenerate these
two files from /etc/rancher/k3s/k3s.yaml.
Notes
- Versions are best-effort pins; if a download 404s at build, bump the value in
versions.envand rebuild. Keep the conformance-tool block aligned withci/versions.env. - Secrets never enter the image — they arrive only as the bind mounts listed above.
- This container is the toolchain, not the cluster. The cluster runs on Proxmox VE; see
../PHASE1-STATUS.md.