Compare commits
146
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
37eea51f24 | ||
|
|
2f579f103b | ||
|
|
830983b791 | ||
|
|
6d13822941 | ||
|
|
a92fa11701 | ||
|
|
cd64b4b672 | ||
|
|
377fa205ea | ||
|
|
63bc890261 | ||
|
|
f4f6e5f7a3 | ||
|
|
a79693d3bd | ||
|
|
16b7fcd677 | ||
|
|
5fa893095b | ||
|
|
e49b86eb41 | ||
|
|
d616fd4025 | ||
|
|
87c3d83dff | ||
|
|
bce8778096 | ||
|
|
78d0f1b0f6 | ||
|
|
55b90849d1 | ||
|
|
b7341b329b | ||
|
|
2f38b8156b | ||
|
|
44983095e1 | ||
|
|
4865734c5f | ||
|
|
b9f8807aa6 | ||
|
|
7057bbb711 | ||
|
|
5fa278f61c | ||
|
|
78d76fa089 | ||
|
|
9e7d919d12 | ||
|
|
9b4744d450 | ||
|
|
78bc53afc7 | ||
|
|
505083fded | ||
|
|
df77affc23 | ||
|
|
501d587541 | ||
|
|
bacb4c61ee | ||
|
|
7880d84bce | ||
|
|
5e7ab65c65 | ||
|
|
21491f901b | ||
|
|
ebce6dfabb | ||
|
|
eda4a9f674 | ||
|
|
d617983e83 | ||
|
|
1f20506829 | ||
|
|
a863542bf7 | ||
|
|
f252ecec25 | ||
|
|
ae37e27f14 | ||
|
|
9161cab9cd | ||
|
|
b9c0970232 | ||
|
|
f7d4648851 | ||
|
|
6202d92c27 | ||
|
|
23ea72d01e | ||
|
|
51d8676f90 | ||
|
|
fa4deac877 | ||
|
|
a317493bca | ||
|
|
c66a7e29e9 | ||
|
|
155df8ee3f | ||
|
|
9d0814ae61 | ||
|
|
19fc8bcb27 | ||
|
|
53e09a2610 | ||
|
|
9b97e073e2 | ||
|
|
38e4bf1459 | ||
|
|
37a6c94168 | ||
|
|
04606d48b2 | ||
|
|
60e3f09c67 | ||
|
|
6f698f59af | ||
|
|
839f7fed4a | ||
|
|
2d5171e064 | ||
|
|
2e259a808f | ||
|
|
3ec67bafdd | ||
|
|
5da116b34d | ||
|
|
6b953cc56b | ||
|
|
f5e62d9169 | ||
|
|
daac05e7e8 | ||
|
|
da8a0eac04 | ||
|
|
cac413e87a | ||
|
|
82aff4d053 | ||
|
|
e8d8601ca6 | ||
|
|
2590bc4317 | ||
|
|
cec550045b | ||
|
|
f338e91770 | ||
|
|
a624c638ef | ||
|
|
90b39de527 | ||
|
|
0247d796a3 | ||
|
|
c9e9dcf960 | ||
|
|
89cf59ffb7 | ||
|
|
e319777849 | ||
|
|
730d12ddc4 | ||
|
|
df1d2ab44d | ||
|
|
69a25cff82 | ||
|
|
c1e36d9353 | ||
|
|
9d00a3fcd3 | ||
|
|
0e92d258dd | ||
|
|
08fbe85638 | ||
|
|
863af01062 | ||
|
|
f7a9229d81 | ||
|
|
a819c50ffe | ||
|
|
efc6db6c1f | ||
|
|
09265e0381 | ||
|
|
e65a76dc51 | ||
|
|
bc9c93d2ab | ||
|
|
6548f9a89c | ||
|
|
bfa39bc5d0 | ||
|
|
3ca4ad6011 | ||
|
|
708a05dfc1 | ||
|
|
161430d915 | ||
|
|
e7c4a95c2d | ||
|
|
0a0527e7db | ||
|
|
dc2d1ce17e | ||
|
|
6a8953e288 | ||
|
|
9739e9be91 | ||
|
|
d1a6628a97 | ||
|
|
f178d85c71 | ||
|
|
f21e89d9d2 | ||
|
|
2d7d302c7e | ||
|
|
c8c1d2935d | ||
|
|
9b8979a427 | ||
|
|
05baae90ec | ||
|
|
8e913fdd7b | ||
|
|
b19c5db9ae | ||
|
|
6e4d311545 | ||
|
|
4809199da6 | ||
|
|
0a733294ff | ||
|
|
cf33338448 | ||
|
|
47f02c6e06 | ||
|
|
49d82491dd | ||
|
|
5ea9c93f58 | ||
|
|
3a88abb958 | ||
|
|
086e9091c9 | ||
|
|
9f7052ab2b | ||
|
|
3c8f809aaf | ||
|
|
11b21f7347 | ||
|
|
cb69a53825 | ||
|
|
f1daa8cae0 | ||
|
|
380ea98a38 | ||
|
|
acae2ad36c | ||
|
|
fdc3265585 | ||
|
|
4cb5e7a8d4 | ||
|
|
57d4df8c4f | ||
|
|
14494e1947 | ||
|
|
e9a4b669d4 | ||
|
|
aa0a50a143 | ||
|
|
22174186ff | ||
|
|
23aa2854f1 | ||
|
|
63f98aab26 | ||
|
|
62968c9ee9 | ||
|
|
645994e3a3 | ||
|
|
ac9f20034c | ||
|
|
4c4f5355d9 | ||
|
|
63002d19c8 |
No files matched your search
+43
-20
@@ -1,25 +1,44 @@
|
||||
# .gitea/workflows/gate.yml — the ENFORCED PR merge gate (Phase 6, WS-GITOSS).
|
||||
#
|
||||
# This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
|
||||
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `make -C ci gate` — so CI
|
||||
# runs BYTE-FOR-BYTE the same command a human runs in the devcontainer (R-TEST-02): the
|
||||
# runner just installs the pinned tools from ci/versions.env first. `make -C ci gate`
|
||||
# (ci/gate.sh) runs every stage in order — spec-immutability, lint, Kyverno L0,
|
||||
# render==golden, L1 offline render — and applies the R-TEST-03 decision, printing
|
||||
# "MERGE GATE: PASS" / "MERGE GATE: BLOCKED". It is the single required status check on
|
||||
# `main` (see branch protection). A non-zero exit fails the job, so a BLOCKED gate blocks
|
||||
# the merge.
|
||||
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
|
||||
# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
|
||||
# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
|
||||
# below for why the script, not `make`, is invoked here). It runs every stage in order —
|
||||
# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
|
||||
# R-TEST-03 decision, printing
|
||||
# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
|
||||
# single required status check on `main` (see branch protection). A non-zero exit fails the job,
|
||||
# so a BLOCKED gate blocks the merge.
|
||||
#
|
||||
# WHY THIS STEP DECLARES GATE_STRICT_SKIPS AND TOLERATES EXIT 3 (2026-09-13). A SKIPPED stage is
|
||||
# never green, and since 2026-09-13 gate.sh refuses to print PASS when one did not run (it used to
|
||||
# print "— not green" and then "MERGE GATE: PASS", exit 0 — fail-open in the gate itself). Two
|
||||
# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
|
||||
# environment it is:
|
||||
# * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
|
||||
# installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
|
||||
# "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
|
||||
# * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
|
||||
# not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
|
||||
# is the ONLY skip that can still reach exit 3 under strict skips, which is what makes
|
||||
# `|| [ "$?" -eq 3 ]` a narrow, reviewed tolerance rather than a blanket one. The log still
|
||||
# says UNDETERMINED and names R-TEST-06 as unverified. Wire the spec checkout (or set
|
||||
# REQUIRE_SPEC_VERIFICATION=true) and this tolerance should be deleted.
|
||||
# Called DIRECTLY, not through `make`: GNU make reports any recipe failure as its own exit 2, so
|
||||
# `make -C ci gate || [ "$?" -eq 3 ]` would see 2 and fail the job. Same reason the
|
||||
# spec-immutability pre-step below calls the script directly.
|
||||
#
|
||||
# The cluster-gated L2/L3 conformance stage SKIPS honestly here (exit 3, surfaced not faked
|
||||
# — HARD RULE 4) because this runner declares no cluster: `make -C ci gate` does not set
|
||||
# CLUSTER_AVAILABLE, so gate.sh reports the touched requirements UNVERIFIED without blocking
|
||||
# Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
|
||||
# — HARD RULE 4) because this runner declares no cluster: nothing here sets CLUSTER_AVAILABLE,
|
||||
# so gate.sh reports the touched requirements UNVERIFIED — and NAMES that declared gap in its
|
||||
# PASS headline — without blocking Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
|
||||
# no broad kubeconfig) driven by the separate `conformance` job below.
|
||||
#
|
||||
# NOTE (R-TEST-06 / gate-semantics): spec-immutability is ALSO surfaced as its own explicit
|
||||
# early step (below) so its result is a distinct, tolerant status in the Gitea UI; gate.sh
|
||||
# re-runs it as its first stage, which is the authoritative one. `make -C ci gate` here is
|
||||
# the byte-identical laptop gate; the extra pre-step is a read-only git-diff check and is
|
||||
# re-runs it as its first stage, which is the authoritative one. The gate step below is the
|
||||
# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
|
||||
# purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
|
||||
# is retained dormant in merge-gate.yaml.
|
||||
---
|
||||
@@ -69,13 +88,17 @@ jobs:
|
||||
- name: Spec immutability (R-TEST-06, skips if spec repo absent)
|
||||
run: ./ci/spec-immutability.sh || [ "$?" -eq 3 ]
|
||||
|
||||
# Aggregate merge gate: byte-for-byte the command a human runs in the devcontainer
|
||||
# (`make -C ci gate` -> ci/gate.sh). Runs spec-immutability, lint, Kyverno L0,
|
||||
# render==golden and L1 render, then the R-TEST-03 decision. Exits non-zero on a
|
||||
# BLOCKED gate, which fails this job. The cluster-gated L2/L3 stage SKIPS here (no
|
||||
# CLUSTER_AVAILABLE) — surfaced as UNVERIFIED, never faked green (see header).
|
||||
- name: Merge gate — make -C ci gate
|
||||
run: make -C ci gate
|
||||
# Aggregate merge gate: the same script a human runs in the devcontainer via
|
||||
# `make -C ci gate`. Runs spec-immutability, lint, Kyverno L0, render==golden and L1
|
||||
# render, then the R-TEST-03 decision. Exits non-zero on a BLOCKED gate, which fails this
|
||||
# job. The cluster-gated L2/L3 stage SKIPS here (no CLUSTER_AVAILABLE) — surfaced as
|
||||
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
|
||||
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
|
||||
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
|
||||
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
|
||||
env:
|
||||
GATE_STRICT_SKIPS: "true"
|
||||
run: ./ci/gate.sh || [ "$?" -eq 3 ]
|
||||
|
||||
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
||||
# ("not supported on GHES" — Gitea does not implement the artifact-v2 backend), which
|
||||
|
||||
@@ -39,6 +39,21 @@ install_chainsaw() {
|
||||
curl -fsSL "$url" | tar -xz -C "$bindir" chainsaw
|
||||
}
|
||||
|
||||
# promtool — the alert-rule UNIT-TEST engine the L0 check
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested runs. Installed for the l0 stage because
|
||||
# without it that check exits 3 (SKIPPED, never green) on the runner and the alert unit tests
|
||||
# would be enforced nowhere — the exact "coverage that looks present" failure it exists to end.
|
||||
# Pinned from ci/versions.env to the Prometheus version prod runs (promtool IS the rule engine);
|
||||
# the check asserts `promtool --version` equals that pin, so a drifted install FAILS, not skips.
|
||||
install_promtool() {
|
||||
have promtool && return 0
|
||||
log "promtool ${PROMTOOL_VERSION} (from the prometheus release tarball)"
|
||||
local tarball="prometheus-${PROMTOOL_VERSION}.${os}-${arch}"
|
||||
curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/${tarball}.tar.gz" \
|
||||
| tar -xz -C "$bindir" --strip-components=1 "${tarball}/promtool"
|
||||
chmod +x "${bindir}/promtool"
|
||||
}
|
||||
|
||||
install_kubectl() {
|
||||
have kubectl && return 0
|
||||
log "kubectl ${KUBECTL_VERSION}"
|
||||
@@ -50,10 +65,10 @@ have() { command -v "$1" >/dev/null 2>&1; }
|
||||
|
||||
case "$stage" in
|
||||
lint) install_yamllint ;;
|
||||
l0) install_yamllint; install_check_jsonschema; install_kyverno ;; # schema (check-jsonschema) + Kyverno L0
|
||||
l0) install_yamllint; install_check_jsonschema; install_kyverno; install_promtool ;; # schema (check-jsonschema) + Kyverno L0 + alert unit tests
|
||||
l1) install_chainsaw ;; # L1 render via conformance runner (Chainsaw)
|
||||
conformance) install_chainsaw; install_kubectl ;; # trigger the in-cluster Job
|
||||
all) install_yamllint; install_check_jsonschema; install_kyverno; install_chainsaw; install_kubectl ;;
|
||||
all) install_yamllint; install_check_jsonschema; install_kyverno; install_promtool; install_chainsaw; install_kubectl ;;
|
||||
*) echo "unknown stage: $stage" >&2; exit 2 ;;
|
||||
esac
|
||||
log "stage '${stage}' tools ready"
|
||||
@@ -0,0 +1,57 @@
|
||||
A|*.keithjackson.dev|136.51.115.252
|
||||
A|auth.home.keithjackson.dev|192.168.1.53
|
||||
A|brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|crafty-mc.home.keithjackson.dev|192.168.1.55
|
||||
A|crafty.home.keithjackson.dev|192.168.1.53
|
||||
A|dev.brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|gitea.home.keithjackson.dev|192.168.1.53
|
||||
A|grafana.home.keithjackson.dev|192.168.1.53
|
||||
A|headlamp.home.keithjackson.dev|192.168.1.53
|
||||
A|home.keithjackson.dev|192.168.1.53
|
||||
A|jackett.home.keithjackson.dev|192.168.1.53
|
||||
A|keycloak.home.keithjackson.dev|192.168.1.53
|
||||
A|nextcloud.home.keithjackson.dev|192.168.1.53
|
||||
A|pihole.home.keithjackson.dev|192.168.1.53
|
||||
A|plex.home.keithjackson.dev|192.168.1.53
|
||||
A|qbittorrent.home.keithjackson.dev|192.168.1.53
|
||||
A|radarr.home.keithjackson.dev|192.168.1.53
|
||||
A|seerr.home.keithjackson.dev|192.168.1.53
|
||||
A|semaphore.home.keithjackson.dev|192.168.1.53
|
||||
A|sonarr.home.keithjackson.dev|192.168.1.53
|
||||
A|unifi.home.keithjackson.dev|192.168.1.54
|
||||
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
@@ -0,0 +1,95 @@
|
||||
A|*.keithjackson.dev|136.51.115.252
|
||||
A|auth.home.keithjackson.dev|192.168.1.53
|
||||
A|auth.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|conformance-dns.home.keithjackson.dev|192.168.1.56
|
||||
A|crafty-mc.home.keithjackson.dev|192.168.1.55
|
||||
A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
|
||||
A|crafty.home.keithjackson.dev|192.168.1.53
|
||||
A|crafty.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|dev.brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|gitea.home.keithjackson.dev|192.168.1.53
|
||||
A|gitea.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|grafana.home.keithjackson.dev|192.168.1.53
|
||||
A|grafana.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|headlamp.home.keithjackson.dev|192.168.1.53
|
||||
A|headlamp.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|home.keithjackson.dev|192.168.1.53
|
||||
A|jackett.home.keithjackson.dev|192.168.1.53
|
||||
A|jackett.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|keycloak.home.keithjackson.dev|192.168.1.53
|
||||
A|keycloak.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|nextcloud.home.keithjackson.dev|192.168.1.53
|
||||
A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|pihole.home.keithjackson.dev|192.168.1.53
|
||||
A|pihole.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|plex.home.keithjackson.dev|192.168.1.53
|
||||
A|plex.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|prod.home.keithjackson.dev|192.168.2.20
|
||||
A|qbittorrent.home.keithjackson.dev|192.168.1.53
|
||||
A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|radarr.home.keithjackson.dev|192.168.1.53
|
||||
A|radarr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|seerr.home.keithjackson.dev|192.168.1.53
|
||||
A|seerr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|semaphore.home.keithjackson.dev|192.168.1.53
|
||||
A|semaphore.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|sonarr.home.keithjackson.dev|192.168.1.53
|
||||
A|sonarr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|unifi.home.keithjackson.dev|192.168.1.54
|
||||
A|unifi.prod.home.keithjackson.dev|192.168.2.21
|
||||
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|a-conformance-dns.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/chainsaw-growing-grouse/dns-probe-svc
|
||||
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
@@ -0,0 +1,93 @@
|
||||
A|*.keithjackson.dev|136.51.115.252
|
||||
A|auth.home.keithjackson.dev|192.168.1.53
|
||||
A|auth.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|crafty-mc.home.keithjackson.dev|192.168.1.55
|
||||
A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
|
||||
A|crafty.home.keithjackson.dev|192.168.1.53
|
||||
A|crafty.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|dev.brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|gitea.home.keithjackson.dev|192.168.1.53
|
||||
A|gitea.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|grafana.home.keithjackson.dev|192.168.1.53
|
||||
A|grafana.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|headlamp.home.keithjackson.dev|192.168.1.53
|
||||
A|headlamp.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|home.keithjackson.dev|192.168.1.53
|
||||
A|jackett.home.keithjackson.dev|192.168.1.53
|
||||
A|jackett.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|keycloak.home.keithjackson.dev|192.168.1.53
|
||||
A|keycloak.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|nextcloud.home.keithjackson.dev|192.168.1.53
|
||||
A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|pihole.home.keithjackson.dev|192.168.1.53
|
||||
A|pihole.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|plex.home.keithjackson.dev|192.168.1.53
|
||||
A|plex.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|prod.home.keithjackson.dev|192.168.2.20
|
||||
A|qbittorrent.home.keithjackson.dev|192.168.1.53
|
||||
A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|radarr.home.keithjackson.dev|192.168.1.53
|
||||
A|radarr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|seerr.home.keithjackson.dev|192.168.1.53
|
||||
A|seerr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|semaphore.home.keithjackson.dev|192.168.1.53
|
||||
A|semaphore.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|sonarr.home.keithjackson.dev|192.168.1.53
|
||||
A|sonarr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|unifi.home.keithjackson.dev|192.168.1.54
|
||||
A|unifi.prod.home.keithjackson.dev|192.168.2.21
|
||||
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
+1348
File diff suppressed because it is too large.
Load diff
@@ -102,7 +102,7 @@ spec:
|
||||
userData: |-
|
||||
#cloud-config
|
||||
hostname: unifi
|
||||
fqdn: unifi.home.keithjackson.dev
|
||||
fqdn: unifi.${BASE_DOMAIN}
|
||||
package_update: true
|
||||
package_upgrade: false
|
||||
packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
|
||||
|
||||
@@ -46,7 +46,7 @@ metadata:
|
||||
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
||||
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
|
||||
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
|
||||
external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
|
||||
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
+26
-5
@@ -90,9 +90,9 @@ spec:
|
||||
name: gitea-bootstrap
|
||||
key: secret-key
|
||||
- name: GITEA__server__DOMAIN
|
||||
value: gitea.home.keithjackson.dev
|
||||
value: gitea.${BASE_DOMAIN}
|
||||
- name: GITEA__server__ROOT_URL
|
||||
value: https://gitea.home.keithjackson.dev/
|
||||
value: https://gitea.${BASE_DOMAIN}/
|
||||
- name: GITEA__actions__ENABLED
|
||||
value: 'true'
|
||||
- name: GITEA__actions__DEFAULT_ACTIONS_URL
|
||||
@@ -127,8 +127,8 @@ spec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1;\
|
||||
\ break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n"
|
||||
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1;\
|
||||
\ break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
@@ -170,6 +170,27 @@ spec:
|
||||
- port: '3000'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): gitea-allow-flux-ingress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: gitea-allow-flux-ingress
|
||||
namespace: gitea
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: flux-system
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '3000'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
@@ -207,7 +228,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- gitea.home.keithjackson.dev
|
||||
- gitea.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -74,7 +74,7 @@ spec:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
spec:
|
||||
containers:
|
||||
- name: jackett
|
||||
@@ -83,7 +83,7 @@ spec:
|
||||
- containerPort: 9117
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
mountPath: /config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
@@ -91,16 +91,17 @@ spec:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:9117
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://jackett.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
@@ -266,14 +267,14 @@ metadata:
|
||||
app.kubernetes.io/name: jackett
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- jackett.home.keithjackson.dev
|
||||
- jackett.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -11,6 +11,7 @@ resources:
|
||||
- jackett-oauth2-proxy-credentials.sops.yaml
|
||||
- jackett.yaml
|
||||
- nextcloud-db-app.sops.yaml
|
||||
- nextcloud-db-backup-credentials-prod.sops.yaml
|
||||
- nextcloud-db-backup-credentials.sops.yaml
|
||||
- nextcloud-db.yaml
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
@@ -19,5 +20,6 @@ resources:
|
||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||
- qbittorrent.yaml
|
||||
- semaphore-db-app.sops.yaml
|
||||
- semaphore-db-backup-credentials-prod.sops.yaml
|
||||
- semaphore-db-backup-credentials.sops.yaml
|
||||
- semaphore-db.yaml
|
||||
@@ -0,0 +1,29 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: nextcloud-db-backup-credentials-prod
|
||||
namespace: nextcloud-db
|
||||
type: Opaque
|
||||
stringData:
|
||||
ACCESS_KEY_ID: ENC[AES256_GCM,data:4lx6et4HcXDegulyIoa7jZR/lVgapGPEkukajfCKjvk=,iv:p6V7tizbikk55tOvpPOsPHTe/p52QLX5A5xrwuB14rA=,tag:oNJuipT/Lb6zynJ2K2Q71A==,type:str]
|
||||
ACCESS_SECRET_KEY: ENC[AES256_GCM,data:V5bHPoPeBOVyLp1n1Jilyso4i5P2aPSH0hrryfweg2q/SctWZxRbLBQ/FLzOigbJsAhlwOGu069mQPma6xjfdw==,iv:8u4UOvI8dC/cm6mEWDlyN5E3SfRZmwKY9uZY/DjPEsQ=,tag:tw7tGs29TmUvMPBCtCszAA==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiSUt5N2k2N2lDU2ZsMy9m
|
||||
M1NFS0M2QjFsNFhnR0hCMzFyaTdiQWFZRFJZCmhBTWRXNmxLMGtsc3Rxd2FSMDZp
|
||||
aktjcTE2c1hmU1EzM3RWNksyb3Q1YVUKLS0tIDBFS1I5aDBER2cvcWtLZ3BXTEp5
|
||||
ZGI5ZmZIcStheUNQamxIVVVpc0ZMOHMKlE5aHewtHJpeU24F5sodc9CEm9KTjHQQ
|
||||
8ThEQ/8vyazL9V4ojlICxSHU/SHkHpD1eBaTKcAQd/LZzujIcJ+QfA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-07T06:36:33Z"
|
||||
mac: ENC[AES256_GCM,data:8awz1Jh8t4zjDMyd93/v+6on9iiIaSi01293A7uYhxj9L1+g63zeO+DPmSn6+KbZvHdqU2A7up09H1NzfXAEdGuvVbB36cdGq0zKank3PUi8c7IT6qNuWydxRAcqLnbbD+c30Z6GB+gAuZHstNOe4Rgc4aJvMN6FuP6x6Z0870c=,iv:KGGKJq2QA5xztvoHs/SViXW72DYwT16i+W+dJBUeIJQ=,tag:3UpYqx2+RIK0QdMH/BSIpA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+304
-14
@@ -37,16 +37,53 @@ metadata:
|
||||
name: nextcloud-db
|
||||
namespace: nextcloud-db
|
||||
spec:
|
||||
# Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
|
||||
# keycloak-db). The storage is already 3-way replicated by Longhorn; raise to 3 for Postgres-level
|
||||
# HA as a future hardening.
|
||||
instances: 1
|
||||
# THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db; see
|
||||
# platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: Longhorn's 3-way
|
||||
# replication saves the DATA on node loss, not the SERVICE, and with `instances: 1` all three CNPG
|
||||
# primaries had drifted onto k3s-prod-2, so one node took Nextcloud down alongside SSO and
|
||||
# Semaphore. A streaming standby lets CNPG promote in seconds and repoint the -rw Service.
|
||||
# Gated on the Barman archive below actually working (ContinuousArchiving=True with a real
|
||||
# firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
|
||||
#
|
||||
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
|
||||
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
|
||||
instances: 3
|
||||
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
|
||||
# Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
|
||||
# nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
|
||||
inheritedMetadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud-db
|
||||
# HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
|
||||
# primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
|
||||
# recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
|
||||
# version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
|
||||
# after a node loss it may stack two instances on one node — which is how `instances: 3` decays
|
||||
# back into the single-node failure domain this change removes. `required` makes it a constraint;
|
||||
# the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
|
||||
# serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
|
||||
#
|
||||
# APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
|
||||
# switchover. Expect a single brief connection reset for Nextcloud.
|
||||
affinity:
|
||||
enablePodAntiAffinity: true
|
||||
podAntiAffinityType: required
|
||||
primaryUpdateMethod: switchover
|
||||
# SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
|
||||
# rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
|
||||
# last few transactions when a primary dies before its standbys catch up, and `method: any` +
|
||||
# `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
|
||||
# never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
|
||||
# are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
|
||||
# standby into a total write outage. `preferred` relaxes to async instead.
|
||||
#
|
||||
# NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
|
||||
# no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
|
||||
postgresql:
|
||||
synchronous:
|
||||
method: any
|
||||
number: 1
|
||||
dataDurability: preferred
|
||||
storage:
|
||||
storageClass: longhorn
|
||||
size: 5Gi
|
||||
@@ -68,25 +105,38 @@ spec:
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
|
||||
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
|
||||
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
|
||||
# backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
|
||||
# (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP.
|
||||
# Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials
|
||||
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
|
||||
# endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
|
||||
# substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
|
||||
# backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
|
||||
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/nextcloud-db/ — dev
|
||||
# s3://cnpg/nextcloud-db/ on the on-Proxmox MinIO, prod
|
||||
# s3://k3s-prod-backup/barman/nextcloud-db/ on Cloudflare R2. Both endpointURL and
|
||||
# destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
|
||||
# Credentials: PER-ENVIRONMENT (2026-09-07). dev reads nextcloud-db-backup-credentials (the
|
||||
# scoped MinIO key), prod reads nextcloud-db-backup-credentials-prod (the Cloudflare R2
|
||||
# token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
|
||||
# generated golden — render.py places the -prod sidecar in this DB's wave); the
|
||||
# cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
|
||||
# one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
|
||||
# ONE shared Secret until the split, so giving prod its R2 token would have left dev
|
||||
# signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
|
||||
# disk-full clock below rather than a merely missing backup.
|
||||
#
|
||||
# archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
|
||||
# Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
|
||||
# the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
|
||||
# endpoint. Treat ContinuousArchiving=False here as an availability incident.
|
||||
backup:
|
||||
retentionPolicy: "30d"
|
||||
barmanObjectStore:
|
||||
destinationPath: s3://cnpg
|
||||
destinationPath: ${CNPG_BACKUP_PATH}
|
||||
endpointURL: ${S3_BACKUP_ENDPOINT}
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: nextcloud-db-backup-credentials
|
||||
name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
key: ACCESS_KEY_ID
|
||||
secretAccessKey:
|
||||
name: nextcloud-db-backup-credentials
|
||||
name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
key: ACCESS_SECRET_KEY
|
||||
wal:
|
||||
compression: gzip
|
||||
@@ -108,3 +158,243 @@ spec:
|
||||
cluster:
|
||||
name: nextcloud-db
|
||||
method: barmanObjectStore
|
||||
---
|
||||
# =============================================================================
|
||||
# CNPG METRICS-EXPORTER CONNECT RECONCILER (R-OBS-01)
|
||||
#
|
||||
# THE BUG THIS CLOSES (true positive, verified live on prod 2026-09-12, and the
|
||||
# condition platform/cnpg/alerts.yaml already recorded on BOTH clusters 2026-09-06):
|
||||
# the `nextcloud` database's ACL had been narrowed out-of-band so that PUBLIC held
|
||||
# nothing, and CNPG's metrics exporter role therefore could not even CONNECT:
|
||||
#
|
||||
# nextcloud-db-1/-2/-3, every scrape:
|
||||
# FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
|
||||
# DETAIL: User does not have CONNECT privilege.
|
||||
# msg="Error collecting user query" targetDatabase=nextcloud
|
||||
#
|
||||
# CONSEQUENCE: `cnpg_last_error` = 1 on all three instances and the ENTIRE
|
||||
# cnpg_pg_stat_* / pg_settings / pg_postmaster family MISSING for this database,
|
||||
# while the scrape target still reported up=1 — a blind spot that looks like health.
|
||||
# Measured before the fix: cnpg_pg_settings_setting had 765 series for keycloak and
|
||||
# 765 for semaphore-db and ZERO for nextcloud-db. Nextcloud's database had never
|
||||
# been monitored.
|
||||
#
|
||||
# ROOT CAUSE, AND WHY ONLY THIS DATABASE. All three Cluster specs are identical in
|
||||
# their bootstrap; the difference is entirely in the live ACL:
|
||||
#
|
||||
# nextcloud : datacl = {nextcloud=CTc/nextcloud} <-- explicit, no PUBLIC entry
|
||||
# keycloak : datacl = NULL
|
||||
# semaphore : datacl = NULL
|
||||
#
|
||||
# A NULL datacl means "PostgreSQL defaults", which include CONNECT and TEMP for
|
||||
# PUBLIC — so keycloak and semaphore let the exporter in only by inheriting PUBLIC's
|
||||
# implicit CONNECT. On `nextcloud` something ran the equivalent of
|
||||
# `REVOKE ALL ON DATABASE nextcloud FROM PUBLIC`, which MATERIALISES datacl and
|
||||
# drops the implicit PUBLIC entry with it. That REVOKE is not in this repo and not
|
||||
# in DECISIONS-LOG.md — it was applied out of band, which is exactly why a
|
||||
# git-declared fix is needed rather than another hand-run psql.
|
||||
#
|
||||
# WHY THE GRANT IS EXPLICIT RATHER THAN "RE-GRANT CONNECT TO PUBLIC": restoring
|
||||
# PUBLIC's CONNECT would undo whatever hardening that REVOKE intended and would
|
||||
# leave the exporter depending on an implicit default again — the precise fragility
|
||||
# that broke here and that keycloak/semaphore are still relying on. Naming the
|
||||
# exporter role makes the dependency explicit and survives a future REVOKE FROM
|
||||
# PUBLIC on any of the three databases.
|
||||
#
|
||||
# WHY A CronJob AND NOT A CNPG FIELD — CNPG 1.30.0's API GENUINELY CANNOT EXPRESS
|
||||
# THIS. Checked against the live CRDs on prod:
|
||||
# * `Database` (databases.postgresql.cnpg.io) has no database-ACL field at all.
|
||||
# `allowConnections` is the pg_database.datallowconn FLAG (open/closed to
|
||||
# everyone), not a per-role GRANT; its only grant/revoke verbs are the `usage`
|
||||
# lists on `fdws` and `servers`.
|
||||
# * `spec.managed.roles` and `DatabaseRole` manage ROLE ATTRIBUTES (login,
|
||||
# superuser, connectionLimit, ...) plus `inRoles` MEMBERSHIP. Membership is the
|
||||
# wrong tool: the only local role holding CONNECT here is the OWNER, so
|
||||
# `inRoles: [nextcloud]` would hand the exporter the owner's CREATE and TEMP on
|
||||
# the database and read/write on every application table — the opposite of
|
||||
# least privilege, to buy one CONNECT.
|
||||
# * `bootstrap.initdb.postInitApplicationSQL` runs ONCE, at bootstrap. It cannot
|
||||
# repair these three already-bootstrapped instances (the actual bug), and on a
|
||||
# FRESH cluster the grant is unnecessary anyway (datacl starts NULL). Adding it
|
||||
# would be a second mechanism that fixes nothing this one does not.
|
||||
# So the SQL has to be applied by something, and this repo already has the pattern
|
||||
# for state Flux cannot reach: the level-triggered CronJob reconciler in
|
||||
# platform/keycloak/client-secret-reconciler.yaml, whose reasoning transfers
|
||||
# verbatim — the drifted state lives inside PostgreSQL, not in a Kubernetes object,
|
||||
# so no admission or mutation policy can see it. A one-shot Job is deliberately NOT
|
||||
# used: a completed Job is never re-run, so a DB rebuild or a PITR restore would
|
||||
# silently reopen the blind spot. A loop converges instead.
|
||||
#
|
||||
# PRIVILEGE GRANTED — EXACTLY ONE, AND NOTHING ELSE:
|
||||
# GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
|
||||
# Resulting ACL entry is `cnpg_metrics_exporter=c/nextcloud` — the `c` is CONNECT;
|
||||
# there is no `C` (CREATE) and no `T` (TEMPORARY). No ownership, no superuser, no
|
||||
# write, no table or schema grants. Nothing further is needed because CNPG already
|
||||
# creates this role as a member of the built-in `pg_monitor` role (verified on all
|
||||
# three prod clusters), and pg_monitor already carries the read access every
|
||||
# collector query wants — they read pg_catalog views (pg_stat_database,
|
||||
# pg_stat_archiver, pg_settings, pg_postmaster) and no application tables. Proof
|
||||
# that CONNECT alone is sufficient: after this grant, `Error collecting user query`
|
||||
# went to ZERO on all three pods, i.e. every collector query now succeeds.
|
||||
#
|
||||
# IT RUNS AS THE DATABASE OWNER, NOT AS A SUPERUSER. `enableSuperuserAccess` is off
|
||||
# on this Cluster (there is no nextcloud-db-superuser Secret), and it stays off: the
|
||||
# owner of a database holds the grant option on that database's privileges, so
|
||||
# `nextcloud` can issue this GRANT itself. Verified live before committing, by
|
||||
# running the statement under `SET ROLE nextcloud`. Credentials come from the
|
||||
# in-namespace SOPS Secret nextcloud-db-app that already exists for the app.
|
||||
#
|
||||
# NETWORKING: nextcloud-db is intentionally NOT an app-namespace (see this file's
|
||||
# header), so Kyverno generates no default-deny baseline here and this same-namespace
|
||||
# pod -> nextcloud-db-rw:5432 connection needs no rendered allow.
|
||||
#
|
||||
# THE POD LABEL IS DELIBERATELY *NOT* app.kubernetes.io/name=nextcloud-db. That
|
||||
# label is what `inheritedMetadata` stamps on the CNPG instance pods and what the
|
||||
# rendered nextcloud egress CiliumNetworkPolicy selects; reusing it here would quietly
|
||||
# make this reconciler a permitted destination for the Nextcloud app. It gets its own
|
||||
# name so that selector keeps meaning "the database".
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: nextcloud-db-exporter-grant
|
||||
namespace: nextcloud-db
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud-db-exporter-grant
|
||||
app.kubernetes.io/part-of: nextcloud-db
|
||||
spec:
|
||||
# Every 15 minutes, matching the keycloak reconciler's cadence. A converged
|
||||
# database makes this one idempotent GRANT and exits; the cost is negligible, and
|
||||
# after a rebuild or PITR restore monitoring is blind for at most one interval
|
||||
# (well inside the alert's `for: 6h`).
|
||||
schedule: "*/15 * * * *"
|
||||
# A run still waiting on the primary must not be lapped by the next tick.
|
||||
concurrencyPolicy: Forbid
|
||||
startingDeadlineSeconds: 300
|
||||
successfulJobsHistoryLimit: 1
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
activeDeadlineSeconds: 300
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud-db-exporter-grant
|
||||
app.kubernetes.io/part-of: nextcloud-db
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
# Talks only to PostgreSQL; its one input is a mounted Secret, so it needs
|
||||
# no Kubernetes API access at all (R-SEC-02, least privilege).
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
# uid/gid 26 is `postgres` in the CNPG PostgreSQL image.
|
||||
runAsUser: 26
|
||||
runAsGroup: 26
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: grant
|
||||
# The SAME image the instances above run, pinned to the same tag. psql
|
||||
# is already in it and it is already present on every node, so this
|
||||
# adds no new image and no new supply-chain surface. Keep the tag in
|
||||
# step with spec.imageName above when Renovate bumps it.
|
||||
image: ghcr.io/cloudnative-pg/postgresql:16.4
|
||||
imagePullPolicy: IfNotPresent
|
||||
command:
|
||||
- psql
|
||||
# -w never prompts; -X skips ~/.psqlrc; ON_ERROR_STOP makes the Job
|
||||
# FAIL LOUDLY rather than reporting success on a refused grant.
|
||||
- "-w"
|
||||
- "-X"
|
||||
- "-v"
|
||||
- "ON_ERROR_STOP=1"
|
||||
- "-f"
|
||||
- /etc/grant/grant.sql
|
||||
env:
|
||||
# libpq connection, assembled from the owner Secret. Target the -rw
|
||||
# Service: GRANT is a write and must reach the primary.
|
||||
- name: PGHOST
|
||||
value: nextcloud-db-rw.nextcloud-db.svc.cluster.local
|
||||
- name: PGPORT
|
||||
value: "5432"
|
||||
# Connecting to the application database itself is deliberate: the
|
||||
# owner has CONNECT on it, and a successful connection is the same
|
||||
# check the exporter was failing.
|
||||
- name: PGDATABASE
|
||||
value: nextcloud
|
||||
- name: PGUSER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: nextcloud-db-app
|
||||
key: username
|
||||
# Read by libpq from the environment and never logged; psql echoes
|
||||
# no connection parameters.
|
||||
- name: PGPASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: nextcloud-db-app
|
||||
key: password
|
||||
- name: PGCONNECT_TIMEOUT
|
||||
value: "10"
|
||||
volumeMounts:
|
||||
- name: sql
|
||||
mountPath: /etc/grant
|
||||
readOnly: true
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
volumes:
|
||||
- name: sql
|
||||
configMap:
|
||||
name: nextcloud-db-exporter-grant
|
||||
defaultMode: 0444
|
||||
---
|
||||
# The desired state, as SQL. Idempotent: re-granting an existing privilege is a
|
||||
# no-op, so a converged run changes nothing. The DO block is the ASSERTION — it
|
||||
# turns "the grant did not take" into a failed Job instead of a silent success.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: nextcloud-db-exporter-grant
|
||||
namespace: nextcloud-db
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud-db-exporter-grant
|
||||
app.kubernetes.io/part-of: nextcloud-db
|
||||
data:
|
||||
grant.sql: |
|
||||
-- The ONE privilege the CNPG metrics exporter is missing on this database.
|
||||
-- CONNECT only: no CREATE, no TEMPORARY, no ownership, no table/schema grants.
|
||||
-- Read access for the collector's pg_catalog queries already comes from the
|
||||
-- built-in pg_monitor role, which CNPG grants this role when it creates it.
|
||||
GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
|
||||
|
||||
-- Fail the Job if the privilege is somehow still absent, so a broken
|
||||
-- reconciler surfaces as a failed Job rather than as a quiet blind spot.
|
||||
-- NO PL/pgSQL, AND NOT ONE DOLLAR SIGN IN THIS FILE — DELIBERATE, including in these
|
||||
-- comments. This assertion began life as a dollar-quoted DO block, which cannot survive
|
||||
-- being shipped this way: the ConfigMap is rendered through Flux's postBuild
|
||||
-- substitution, and envsubst collapses a doubled dollar into a single one. The block
|
||||
-- reached Postgres with its opening quote halved and every Flux-reconciled run died
|
||||
-- with a syntax error pointing at that character.
|
||||
-- It PASSED when hand-applied with kubectl, because that path runs no envsubst — so
|
||||
-- the defect was invisible until Flux owned the object, and three runs of Error pods
|
||||
-- were the first sign. Escaping by doubling again would work and would also be the
|
||||
-- next person's trap; plain SQL leaves substitution nothing to touch.
|
||||
--
|
||||
-- has_database_privilege() returns boolean; ::int is 1 when granted and 0 when not, so
|
||||
-- the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job. The divisor
|
||||
-- is not a constant, so it cannot be folded away at plan time.
|
||||
\echo 'asserting cnpg_metrics_exporter has CONNECT on database nextcloud'
|
||||
SELECT 1 / has_database_privilege('cnpg_metrics_exporter', 'nextcloud', 'CONNECT')::int
|
||||
AS exporter_can_connect;
|
||||
@@ -91,16 +91,16 @@ spec:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:80
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://pihole.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
@@ -229,7 +229,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- pihole.home.keithjackson.dev
|
||||
- pihole.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
+52
-1
@@ -35,6 +35,34 @@ spec:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PersistentVolume plex-media - static external NFS (bulk tier)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: plex-media
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
spec:
|
||||
capacity:
|
||||
storage: 9Ti
|
||||
accessModes:
|
||||
- ReadOnlyMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: ''
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
- hard
|
||||
- noatime
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
nfs:
|
||||
server: ${BULK_NFS_SERVER}
|
||||
path: /mnt/Storage/plex
|
||||
readOnly: true
|
||||
claimRef:
|
||||
namespace: plex
|
||||
name: plex-media
|
||||
---
|
||||
# PVC plex-config (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
@@ -51,6 +79,23 @@ spec:
|
||||
requests:
|
||||
storage: 8Gi
|
||||
---
|
||||
# PVC plex-media (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: plex-media
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadOnlyMany
|
||||
storageClassName: ''
|
||||
volumeName: plex-media
|
||||
resources:
|
||||
requests:
|
||||
storage: 9Ti
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -90,10 +135,16 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: media
|
||||
mountPath: /media
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
claimName: plex-config
|
||||
- name: media
|
||||
persistentVolumeClaim:
|
||||
claimName: plex-media
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
@@ -183,7 +234,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- plex.home.keithjackson.dev
|
||||
- plex.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -35,6 +35,33 @@ spec:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PersistentVolume qbittorrent-downloads - static external NFS (bulk tier)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: qbittorrent-downloads
|
||||
labels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
spec:
|
||||
capacity:
|
||||
storage: 9Ti
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: ''
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
- hard
|
||||
- noatime
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
nfs:
|
||||
server: ${BULK_NFS_SERVER}
|
||||
path: /mnt/Storage/plex
|
||||
claimRef:
|
||||
namespace: qbittorrent
|
||||
name: qbittorrent-downloads
|
||||
---
|
||||
# PVC qbittorrent-config (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
@@ -51,6 +78,23 @@ spec:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
# PVC qbittorrent-downloads (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: qbittorrent-downloads
|
||||
namespace: qbittorrent
|
||||
labels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: ''
|
||||
volumeName: qbittorrent-downloads
|
||||
resources:
|
||||
requests:
|
||||
storage: 9Ti
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -74,16 +118,24 @@ spec:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
spec:
|
||||
containers:
|
||||
- name: qbittorrent
|
||||
image: lscr.io/linuxserver/qbittorrent:5.1.2
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
env:
|
||||
- name: PUID
|
||||
value: '1000'
|
||||
- name: PGID
|
||||
value: '3000'
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
mountPath: /config
|
||||
- name: downloads
|
||||
mountPath: /downloads
|
||||
subPath: downloads
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
@@ -91,16 +143,17 @@ spec:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:8080
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://qbittorrent.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
@@ -148,10 +201,15 @@ spec:
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
securityContext:
|
||||
fsGroup: 3000
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
claimName: qbittorrent-config
|
||||
- name: downloads
|
||||
persistentVolumeClaim:
|
||||
claimName: qbittorrent-downloads
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
@@ -266,14 +324,14 @@ metadata:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- qbittorrent.home.keithjackson.dev
|
||||
- qbittorrent.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: semaphore-db-backup-credentials-prod
|
||||
namespace: semaphore-db
|
||||
type: Opaque
|
||||
stringData:
|
||||
ACCESS_KEY_ID: ENC[AES256_GCM,data:OhWEEYqrvmDB40NmiETgeUisBvaO9ocevCZAsurNijw=,iv:kEjOIg9NHbRYm+Ul1WrBAzb3gLbIv+ipiYCxFK0QoVc=,tag:KAB1JAIw0ATwbjDEKZVT9A==,type:str]
|
||||
ACCESS_SECRET_KEY: ENC[AES256_GCM,data:C0PNKv07LCYpFONLqd55sz+ioyU/i2YJUkbZ8OX/dCErM/DilqchyUAT5EqnT/r3YJmdaA1pX85S3fxlVVHkbg==,iv:4ha28b+xIebvX2JPITmtTovIe4vdmj2XfI3cgkUHH4c=,tag:xbMyflcHojytPzgaoOYcXQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZVTJ3cXR6cks3Y0RNS2hx
|
||||
K3pjZEduYUxmL0V3SmFUcU9XbGVHSVFITWh3Cjl1eHlTLytoOW5PaVNFVkJoeTBo
|
||||
T1FnRHNDVzJ6dWpuR09WaElvbWF4YU0KLS0tIEJoQ1VsM1cxRlg0R1dtNGdOenNn
|
||||
aVNjMjVBTytLdHlFdmZCS2tPelo1Y2MKYt1G5t+sDiHgcg+Q9ECBzJn2gS/paSEC
|
||||
7aoX60SBrPgHvnGls5lzc3Tin9nH4jWYl6Nu8DAvIE0YPXDFr5a0UA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-07T06:36:33Z"
|
||||
mac: ENC[AES256_GCM,data:mhVP7tjdZK2FR2imvpJbTyT7BigvcIKEpLSnBS1oSITUirB8xZC7aHBfyFuIoKOrpOJgEe40nHUdDYCHRJtb4dLeVja8+7meVaQWzxC8BWVHsksW+73hsqhjAzlHeobx6Irbk4PAMqIY0/jnf4MHryjnRn3dUuwu9U1CqL30JbE=,iv:57oCcp6FW9EAIVKnYGtkaTrnbijyv54T+vmZxMbMqnw=,tag:agrRP/JLypU60KOeN0s/5w==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -38,16 +38,53 @@ metadata:
|
||||
name: semaphore-db
|
||||
namespace: semaphore-db
|
||||
spec:
|
||||
# Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
|
||||
# keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for
|
||||
# Postgres-level HA as a future hardening.
|
||||
instances: 1
|
||||
# THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db /
|
||||
# nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version:
|
||||
# Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with
|
||||
# `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore
|
||||
# down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint
|
||||
# the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True
|
||||
# with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
|
||||
#
|
||||
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
|
||||
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
|
||||
instances: 3
|
||||
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
|
||||
# Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
|
||||
# semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
|
||||
inheritedMetadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: semaphore-db
|
||||
# HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
|
||||
# primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
|
||||
# recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
|
||||
# version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
|
||||
# after a node loss it may stack two instances on one node — which is how `instances: 3` decays
|
||||
# back into the single-node failure domain this change removes. `required` makes it a constraint;
|
||||
# the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
|
||||
# serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
|
||||
#
|
||||
# APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
|
||||
# switchover. Expect a single brief connection reset for Semaphore.
|
||||
affinity:
|
||||
enablePodAntiAffinity: true
|
||||
podAntiAffinityType: required
|
||||
primaryUpdateMethod: switchover
|
||||
# SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
|
||||
# rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
|
||||
# last few transactions when a primary dies before its standbys catch up, and `method: any` +
|
||||
# `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
|
||||
# never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
|
||||
# are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
|
||||
# standby into a total write outage. `preferred` relaxes to async instead.
|
||||
#
|
||||
# NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
|
||||
# no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
|
||||
postgresql:
|
||||
synchronous:
|
||||
method: any
|
||||
number: 1
|
||||
dataDurability: preferred
|
||||
storage:
|
||||
storageClass: longhorn
|
||||
size: 5Gi
|
||||
@@ -69,25 +106,38 @@ spec:
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
|
||||
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
|
||||
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
|
||||
# backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
|
||||
# (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
|
||||
# Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
|
||||
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
|
||||
# endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
|
||||
# substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
|
||||
# backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
|
||||
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/semaphore-db/ — dev
|
||||
# s3://cnpg/semaphore-db/ on the on-Proxmox MinIO, prod
|
||||
# s3://k3s-prod-backup/barman/semaphore-db/ on Cloudflare R2. Both endpointURL and
|
||||
# destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
|
||||
# Credentials: PER-ENVIRONMENT (2026-09-07). dev reads semaphore-db-backup-credentials (the
|
||||
# scoped MinIO key), prod reads semaphore-db-backup-credentials-prod (the Cloudflare R2
|
||||
# token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
|
||||
# generated golden — render.py places the -prod sidecar in this DB's wave); the
|
||||
# cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
|
||||
# one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
|
||||
# ONE shared Secret until the split, so giving prod its R2 token would have left dev
|
||||
# signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
|
||||
# disk-full clock below rather than a merely missing backup.
|
||||
#
|
||||
# archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
|
||||
# Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
|
||||
# the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
|
||||
# endpoint. Treat ContinuousArchiving=False here as an availability incident.
|
||||
backup:
|
||||
retentionPolicy: "30d"
|
||||
barmanObjectStore:
|
||||
destinationPath: s3://cnpg
|
||||
destinationPath: ${CNPG_BACKUP_PATH}
|
||||
endpointURL: ${S3_BACKUP_ENDPOINT}
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: semaphore-db-backup-credentials
|
||||
name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
key: ACCESS_KEY_ID
|
||||
secretAccessKey:
|
||||
name: semaphore-db-backup-credentials
|
||||
name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
key: ACCESS_SECRET_KEY
|
||||
wal:
|
||||
compression: gzip
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
# Namespace (trust zone; default-deny + policies apply)
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: vm-demo
|
||||
labels:
|
||||
homelab.keithjackson.dev/app-namespace: 'true'
|
||||
annotations:
|
||||
homelab.keithjackson.dev/backup-policy: daily-offcluster
|
||||
---
|
||||
# Default-deny baseline (R-NET-01): deny all, allow only DNS
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
namespace: vm-demo
|
||||
labels:
|
||||
homelab.keithjackson.dev/baseline: default-deny
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress: []
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# CDI DataVolume (standalone; survives rebuild, R-STATE-01)
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
kind: DataVolume
|
||||
metadata:
|
||||
name: vm-demo-rootdisk
|
||||
namespace: vm-demo
|
||||
spec:
|
||||
source:
|
||||
registry:
|
||||
url: docker://quay.io/containerdisks/fedora:41
|
||||
storage:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 6Gi
|
||||
---
|
||||
# KubeVirt VirtualMachine (R-PLAT-03)
|
||||
apiVersion: kubevirt.io/v1
|
||||
kind: VirtualMachine
|
||||
metadata:
|
||||
name: vm-demo
|
||||
namespace: vm-demo
|
||||
labels:
|
||||
app.kubernetes.io/name: vm-demo
|
||||
spec:
|
||||
running: false
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
kubevirt.io/vm: vm-demo
|
||||
homelab.keithjackson.dev/egress-class: internet
|
||||
homelab.keithjackson.dev/auth-mode: none
|
||||
spec:
|
||||
domain:
|
||||
cpu:
|
||||
cores: 1
|
||||
memory:
|
||||
guest: 1536Mi
|
||||
devices:
|
||||
disks:
|
||||
- name: rootdisk
|
||||
disk:
|
||||
bus: virtio
|
||||
- name: cloudinit
|
||||
disk:
|
||||
bus: virtio
|
||||
interfaces:
|
||||
- name: default
|
||||
masquerade: {}
|
||||
resources:
|
||||
requests:
|
||||
memory: 1536Mi
|
||||
networks:
|
||||
- name: default
|
||||
pod: {}
|
||||
volumes:
|
||||
- name: rootdisk
|
||||
dataVolume:
|
||||
name: vm-demo-rootdisk
|
||||
- name: cloudinit
|
||||
cloudInitNoCloud:
|
||||
userData: "#cloud-config\npackage_update: true\npackages:\n - dnf-automatic\nwrite_files:\n - path: /etc/dnf/automatic.conf\n content: |\n [commands]\n upgrade_type = default\n random_sleep = 0\n download_updates = yes\n apply_updates = yes\nruncmd:\n - [ systemctl, enable, --now, dnf-automatic.timer ]\n"
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): vm-demo-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: vm-demo-allow-internet-egress
|
||||
namespace: vm-demo
|
||||
labels:
|
||||
app.kubernetes.io/name: vm-demo
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: vm-demo
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: vm-demo
|
||||
namespace: vm-demo
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: vm-demo
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: vm-demo
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- vm-demo
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: vm-demo-slis
|
||||
namespace: vm-demo
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: vm-demo
|
||||
spec:
|
||||
groups:
|
||||
- name: vm-demo.slis
|
||||
rules:
|
||||
- alert: vm-demo-down
|
||||
expr: up{job="vm-demo"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: vm-demo
|
||||
annotations:
|
||||
summary: vm-demo is down
|
||||
description: 'vm-demo has been unreachable for 5m (SLI: up).'
|
||||
@@ -165,7 +165,7 @@ spec:
|
||||
exec /sbin/tini -- /opt/act/run.sh
|
||||
env:
|
||||
- name: GITEA_INSTANCE_URL
|
||||
value: https://gitea.home.keithjackson.dev
|
||||
value: https://gitea.${BASE_DOMAIN}
|
||||
- name: GITEA_RUNNER_REGISTRATION_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
---
|
||||
# Namespace (trust zone; default-deny + policies apply)
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: keycloak
|
||||
labels:
|
||||
homelab.keithjackson.dev/app-namespace: 'true'
|
||||
gateway-access: shared
|
||||
annotations:
|
||||
homelab.keithjackson.dev/backup-policy: daily-offcluster
|
||||
---
|
||||
# Default-deny baseline (R-NET-01): deny all, allow only DNS
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
namespace: keycloak
|
||||
labels:
|
||||
homelab.keithjackson.dev/baseline: default-deny
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress: []
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PVC keycloak-db (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: keycloak-db
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
homelab.keithjackson.dev/egress-class: internet
|
||||
homelab.keithjackson.dev/auth-mode: native
|
||||
spec:
|
||||
containers:
|
||||
- name: keycloak
|
||||
image: quay.io/keycloak/keycloak
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
volumeMounts:
|
||||
- name: db
|
||||
mountPath: /data/db
|
||||
volumes:
|
||||
- name: db
|
||||
persistentVolumeClaim:
|
||||
claimName: keycloak-db
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: keycloak
|
||||
ports:
|
||||
- name: port-8080
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
---
|
||||
# CiliumNetworkPolicy egress allow (R-NET-02): keycloak-allow-keycloak-db-5432
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: keycloak-allow-keycloak-db-5432
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
egress:
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
app.kubernetes.io/name: keycloak-db
|
||||
k8s:io.kubernetes.pod.namespace: keycloak-db
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '5432'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): keycloak-allow-gateway-ingress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: keycloak-allow-gateway-ingress
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
ingress:
|
||||
- fromEntities:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '8080'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): keycloak-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: keycloak-allow-internet-egress
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
---
|
||||
# HTTPRoute keycloak-8080 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: HTTPRoute
|
||||
metadata:
|
||||
name: keycloak-8080
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- keycloak.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
type: PathPrefix
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: keycloak
|
||||
port: 8080
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- keycloak
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: keycloak-slis
|
||||
namespace: keycloak
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
groups:
|
||||
- name: keycloak.slis
|
||||
rules:
|
||||
- alert: keycloak-down
|
||||
expr: up{job="keycloak"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: keycloak
|
||||
annotations:
|
||||
summary: keycloak is down
|
||||
description: 'keycloak has been unreachable for 5m (SLI: up).'
|
||||
@@ -35,6 +35,34 @@ spec:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PersistentVolume nextcloud-legacy-archive - static external NFS (bulk tier)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: nextcloud-legacy-archive
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud
|
||||
spec:
|
||||
capacity:
|
||||
storage: 270Gi
|
||||
accessModes:
|
||||
- ReadOnlyMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: ''
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
- hard
|
||||
- noatime
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
nfs:
|
||||
server: ${BULK_NFS_SERVER}
|
||||
path: /mnt/Storage/nextcloud
|
||||
readOnly: true
|
||||
claimRef:
|
||||
namespace: nextcloud
|
||||
name: nextcloud-legacy-archive
|
||||
---
|
||||
# PVC nextcloud-data (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
@@ -51,6 +79,23 @@ spec:
|
||||
requests:
|
||||
storage: 20Gi
|
||||
---
|
||||
# PVC nextcloud-legacy-archive (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: nextcloud-legacy-archive
|
||||
namespace: nextcloud
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadOnlyMany
|
||||
storageClassName: ''
|
||||
volumeName: nextcloud-legacy-archive
|
||||
resources:
|
||||
requests:
|
||||
storage: 270Gi
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -104,13 +149,13 @@ spec:
|
||||
name: nextcloud-bootstrap
|
||||
key: admin-password
|
||||
- name: NEXTCLOUD_TRUSTED_DOMAINS
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
- name: OVERWRITEHOST
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
- name: OVERWRITEPROTOCOL
|
||||
value: https
|
||||
- name: OVERWRITECLIURL
|
||||
value: https://nextcloud.home.keithjackson.dev
|
||||
value: https://nextcloud.${BASE_DOMAIN}
|
||||
- name: NEXTCLOUD_OIDC_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
@@ -119,19 +164,26 @@ spec:
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /var/www/html
|
||||
- name: legacy-archive
|
||||
mountPath: /mnt/legacy-nextcloud
|
||||
subPath: nextcloud
|
||||
readOnly: true
|
||||
lifecycle:
|
||||
postStart:
|
||||
exec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"\
|
||||
[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"\
|
||||
$ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n"
|
||||
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc]\
|
||||
\ client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
|
||||
); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: nextcloud-data
|
||||
- name: legacy-archive
|
||||
persistentVolumeClaim:
|
||||
claimName: nextcloud-legacy-archive
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
@@ -228,7 +280,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- nextcloud.home.keithjackson.dev
|
||||
- nextcloud.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
+67
-8
@@ -35,6 +35,33 @@ spec:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PersistentVolume radarr-media - static external NFS (bulk tier)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: radarr-media
|
||||
labels:
|
||||
app.kubernetes.io/name: radarr
|
||||
spec:
|
||||
capacity:
|
||||
storage: 9Ti
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: ''
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
- hard
|
||||
- noatime
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
nfs:
|
||||
server: ${BULK_NFS_SERVER}
|
||||
path: /mnt/Storage/plex
|
||||
claimRef:
|
||||
namespace: radarr
|
||||
name: radarr-media
|
||||
---
|
||||
# PVC radarr-config (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
@@ -51,6 +78,23 @@ spec:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
# PVC radarr-media (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: radarr-media
|
||||
namespace: radarr
|
||||
labels:
|
||||
app.kubernetes.io/name: radarr
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: ''
|
||||
volumeName: radarr-media
|
||||
resources:
|
||||
requests:
|
||||
storage: 9Ti
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -74,16 +118,25 @@ spec:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
spec:
|
||||
containers:
|
||||
- name: radarr
|
||||
image: lscr.io/linuxserver/radarr:5.21.1
|
||||
ports:
|
||||
- containerPort: 7878
|
||||
env:
|
||||
- name: PUID
|
||||
value: '1000'
|
||||
- name: PGID
|
||||
value: '3000'
|
||||
- name: TZ
|
||||
value: Etc/UTC
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
mountPath: /config
|
||||
- name: media
|
||||
mountPath: /media
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
@@ -91,16 +144,17 @@ spec:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:7878
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://radarr.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
@@ -148,10 +202,15 @@ spec:
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
securityContext:
|
||||
fsGroup: 3000
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
claimName: radarr-config
|
||||
- name: media
|
||||
persistentVolumeClaim:
|
||||
claimName: radarr-media
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
@@ -288,14 +347,14 @@ metadata:
|
||||
app.kubernetes.io/name: radarr
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- radarr.home.keithjackson.dev
|
||||
- radarr.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -5,6 +5,15 @@
|
||||
# credential from its own namespace (a Pod cannot read a Secret in another namespace).
|
||||
# access-key-encryption: FIXED base64 32-byte AES key Semaphore uses to encrypt stored SSH keys /
|
||||
# credentials; MUST stay stable across restarts or stored secrets become undecryptable.
|
||||
# oidc-client-secret: the Keycloak `semaphore` CONFIDENTIAL client secret, and ONLY that. It MIRRORS
|
||||
# `semaphore-client-secret` in platform/keycloak/keycloak-realm-clients.sops.yaml (the realm import
|
||||
# injects that copy into the client), same cross-namespace mirror pattern as db-password above.
|
||||
# It replaced the former `oidc-providers-json` key (2026-09-07): that key held the WHOLE
|
||||
# SEMAPHORE_OIDC_PROVIDERS blob, which buried the environment-specific realm-issuer and redirect
|
||||
# URLs inside ciphertext where Flux's ${BASE_DOMAIN} postBuild substitution could not reach them,
|
||||
# so prod ran with dev's URLs. Those URLs are PUBLIC and now live in plaintext in
|
||||
# catalog/semaphore.yaml (rendered to apps/wave-1/semaphore.yaml); the kubelet splices THIS value
|
||||
# back into the blob via $(SEMAPHORE_OIDC_CLIENT_SECRET). Only the secret half is encrypted.
|
||||
# Recover: infra/devcontainer/dev.sh run sops -d apps/semaphore-bootstrap-secret.sops.yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
@@ -15,11 +24,11 @@ metadata:
|
||||
app.kubernetes.io/name: semaphore
|
||||
type: Opaque
|
||||
stringData:
|
||||
admin-username: ENC[AES256_GCM,data:uD/vO9A9MjerFg==,iv:i/NAqVD6rWg9oWWSGBrRkxzwcPp2IW1HLIKY0pdcPPg=,tag:r27tKLAJ+ydE7Ty3ccMNUw==,type:str]
|
||||
admin-password: ENC[AES256_GCM,data:544nWY1UmBnlsdPJAQZJMXRumu4W9X7UXZvZ6vHN/ArjRgLXKTArOe5JkT/nNrNw,iv:AC1DDbZj2HfpU19EjzA7T+WpIaR4nZc3PNN61y8bAzU=,tag:r28LFi5w6ZeCCaRaX2jRdg==,type:str]
|
||||
db-password: ENC[AES256_GCM,data:ySkdvaGRCW9fMQqChs5l8dY4zCG19i4HOEinRC48mgLuh9WOrjd16T6jwFgiwB55,iv:+1gztJ22HbwgQJiolZMMWOcKIsxrBJpnruJdtxdooOU=,tag:8CfJGwyPTjDGyr0CvalXMQ==,type:str]
|
||||
access-key-encryption: ENC[AES256_GCM,data:1YAMLo1nVkpJWZc+KBZ7B4ia+Y8P+xMyOhHMcEGGJOXd3o+WC5FqirLBVh4=,iv:j28ozeaDhGWDjoaxFOhn2t9YbYE7L3ah/Mfi+i2DWwY=,tag:czG9Wxc/7Nsuh1W5iATOww==,type:str]
|
||||
oidc-providers-json: ENC[AES256_GCM,data:90yt5mIVWLVI2KrlJWpxhL/BMDpEgRhv7+BL+mBRt79diRvU7pN9vtzhZprAKzcU1sEywlT4EwFlsvJpYyo5Qbkp+ZVknbyhAlGvHhJOBHukOPWgf4LignLerO2GzhXjOOzFqWfZ4EyUxNMWTbSe/xc0zIVHDDNQiYplGVRK+E/ukiI7sD/Tn2RrbrdvpC5GyK0NXWRsXn9ffhE3fCoiRHXagj9i/q+uUrQI5EKpPobrDUAGqvKoBXTl/wJ4phnWg9IqtCTE/c3yA7fLLz4j5dCtPZNSVPT4TKDx8Mdv8CfCgziVicjG2s+yHuWpsQn8lf6nvdJmpYPvIqTtLWdxFWuD8Vfo6Qbq2mqPbKR+KmDOGUnjl4gbZdL1SkE9JiqBUobz7UXeLXwYzxMIsinOfYZLcCk7alrD5Yndj9JDxDJZQ54sMMBfVbxDl94W1VHy03RxAR46KKAbPaxOP5wH4Q4z3SDceNwmuGCDRGChRXiHhN5X2X0GqYCbc8E6GdkIMcpmD80e26ydEmAjw7F89mCF4F/yMzJNiG82qaZIn8HeqgGA8urous9r9bYJNlitYTYQ,iv:jbfVdrrZ9fGiOxhQeoTJfFg1QwiOMkoQyaadiVfxX/w=,tag:JkepcRs34EuEgNb4buTTDQ==,type:str]
|
||||
admin-username: ENC[AES256_GCM,data:ju3wVdjNzRUP/w==,iv:d/5ZxgQZ5bympu3gT2ck3DJy9iUPPBAtZYsJvD3To7I=,tag:0UtBaDVqNSK4RCJrQicchw==,type:str]
|
||||
admin-password: ENC[AES256_GCM,data:pxIwpgh3BoND2TI0mPTixOTJoictzT77dqtuRdMKgY/2WaW4wM1Ra6H6n/dxnyR2,iv:eYbFQGFYVJsnYGwn6cOI3BgAdzMcin9YggAhiuL+xqM=,tag:/sPWGv4gue0vjc2nMk7wWA==,type:str]
|
||||
db-password: ENC[AES256_GCM,data:hBFpWXdpk1R5a23Pntup03t56NBRPS+SZU8CrHVb/ypIaIpgf7kjYtVy6nhTF6u0,iv:eBzoQgH5ZhWWpfKZ6AQmwaSal5QaiuN9gfexQ7GsVFs=,tag:j2k8EwhKDlj/Ud6O39ioVQ==,type:str]
|
||||
access-key-encryption: ENC[AES256_GCM,data:SKtAV4HxJ9pNYDqUK/1uOdI1rtMji3pTfprI2rhQxFQyhc6dFlhqI+OUEcY=,iv:rKCP+FLidcOltOwmMMcO5M8tz4qiifIMI+W5soGO2M8=,tag:HaHzkX/JI+9wmcg2SD9GAQ==,type:str]
|
||||
oidc-client-secret: ENC[AES256_GCM,data:uCo7TNzbM2QKw9ie+xzyUG9MFKco1OAlwr5qPn/AGpNvo1YyXSfbZyyM0CLJ1NS821hGg5Zr/m3R/xpy84s3FQ==,iv:O8a4haskufpmorPaxOkJ1Q99Y5GTlzmDFJAsHS+irLk=,tag:VojJpA9jNMbNQmIe1Qr44w==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
@@ -29,14 +38,14 @@ sops:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuRDlqRFJ0dk14UHluSGRP
|
||||
S1pESy9PWlpMakU0aTkzNVhqUGdtbWNEdTJjCm5RNG14azJHc28rdEJ4L0RzR0xO
|
||||
MGRocGZ1V1JZTENWQkN3VGZzamp2eTgKLS0tIGt6QkNNaW5hdTBJVHY0SVo5d3JX
|
||||
elp0SWxHcjg0RWF3aW9mWTJHZ0lhcUEKc0rU5VilAVDleCwqq4SOF+Y9sniu1Pka
|
||||
w/MGDA9THSB0b+YgWRWiX/KiimFqp0ZFODOJ9cHVwFGY71+VadPwOg==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyTzF6NTBtcFVWK1Vtc0I2
|
||||
VGdySHBsRGgvc1h0VzgvQ01xYVBJV3g2Q1NRCjJyZHVYZGtuamo1MFJHWlBwWkxt
|
||||
cHA4VkY1TkRpN3M4UTdMOWZTbFVGemMKLS0tIGVNdDBrWVJveU56VjVBU2JWdHF4
|
||||
RGtaYStXYUpONTZjZCtmM0NyZ3Rka1UKIQpmQai8dz+uRAsXxyeuGaRlRwsFqIPQ
|
||||
9AfrZJcslYwlaK3ptm1oqaCdrlxcMar2nI7thSqzbLe+e1nzA2z7vQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-27T22:12:32Z"
|
||||
mac: ENC[AES256_GCM,data:nHftJTGnTr0O5h8yGXwgz3eflggXJfIXM3m8n3bzDJE/P4nvSpZzFeMb+sc2on+AL+t+SeKrIl8blzm33XsQ1lkaY85Zm+bc5cnED1g04BMDVmmK3brh/8w50Bp2wPEVzb4I01MDzhhyIbOYg7pFvZAH3/VuqvM1t6XmfF1NrOA=,iv:4KBa9PH/dfdGwLC16RenZIVB6BtSoGo6x3qt3h1R43I=,tag:KULiA1klP/RZMNigkloGDA==,type:str]
|
||||
lastmodified: "2026-09-07T07:01:50Z"
|
||||
mac: ENC[AES256_GCM,data:Xutm1rQt57J4RILElD8FkNOsQT1HK5jZlPtqqLyZ/VqOoDwo+IS1wp7hROprlUvQC0bUXy79tRRIl5Q0CCOD+fvAGIdmade+vB7qXlYN6Q1441dGhuSTPvckso6FaB2ddRjKhKGLc/Z4bJoz9VaRNNxJNpWXIrENTg+I+QhNhlo=,iv:klJOrEQ6QN5dTGkQfGnbMbgCWsgWwZPb5YXDMeWeE34=,tag:fuJr4t3Tt3dT9u3/0SFo8Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -110,7 +110,7 @@ spec:
|
||||
- name: SEMAPHORE_ADMIN_NAME
|
||||
value: Semaphore Admin
|
||||
- name: SEMAPHORE_ADMIN_EMAIL
|
||||
value: admin@home.keithjackson.dev
|
||||
value: admin@${BASE_DOMAIN}
|
||||
- name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
@@ -118,11 +118,13 @@ spec:
|
||||
key: access-key-encryption
|
||||
- name: SEMAPHORE_PORT
|
||||
value: :3000
|
||||
- name: SEMAPHORE_OIDC_PROVIDERS
|
||||
- name: SEMAPHORE_OIDC_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: semaphore-bootstrap
|
||||
key: oidc-providers-json
|
||||
key: oidc-client-secret
|
||||
- name: SEMAPHORE_OIDC_PROVIDERS
|
||||
value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/semaphore
|
||||
@@ -231,7 +233,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- semaphore.home.keithjackson.dev
|
||||
- semaphore.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
+67
-8
@@ -35,6 +35,33 @@ spec:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PersistentVolume sonarr-media - static external NFS (bulk tier)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: sonarr-media
|
||||
labels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
capacity:
|
||||
storage: 9Ti
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: ''
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
- hard
|
||||
- noatime
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
nfs:
|
||||
server: ${BULK_NFS_SERVER}
|
||||
path: /mnt/Storage/plex
|
||||
claimRef:
|
||||
namespace: sonarr
|
||||
name: sonarr-media
|
||||
---
|
||||
# PVC sonarr-config (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
@@ -51,6 +78,23 @@ spec:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
# PVC sonarr-media (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: sonarr-media
|
||||
namespace: sonarr
|
||||
labels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: ''
|
||||
volumeName: sonarr-media
|
||||
resources:
|
||||
requests:
|
||||
storage: 9Ti
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -74,16 +118,25 @@ spec:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
spec:
|
||||
containers:
|
||||
- name: sonarr
|
||||
image: lscr.io/linuxserver/sonarr:4.0.15
|
||||
ports:
|
||||
- containerPort: 8989
|
||||
env:
|
||||
- name: PUID
|
||||
value: '1000'
|
||||
- name: PGID
|
||||
value: '3000'
|
||||
- name: TZ
|
||||
value: Etc/UTC
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
mountPath: /config
|
||||
- name: media
|
||||
mountPath: /media
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
@@ -91,16 +144,17 @@ spec:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:8989
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
@@ -148,10 +202,15 @@ spec:
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
securityContext:
|
||||
fsGroup: 3000
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
claimName: sonarr-config
|
||||
- name: media
|
||||
persistentVolumeClaim:
|
||||
claimName: sonarr-media
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
@@ -288,14 +347,14 @@ metadata:
|
||||
app.kubernetes.io/name: sonarr
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- sonarr.home.keithjackson.dev
|
||||
- sonarr.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -223,7 +223,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- seerr.home.keithjackson.dev
|
||||
- seerr.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
|
||||
# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
|
||||
# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
|
||||
# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
|
||||
# same script bootstraps dev and prod with no fork (R-ENV-02).
|
||||
#
|
||||
# Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
|
||||
# =============================================================================
|
||||
@@ -33,7 +36,7 @@
|
||||
# 2. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
|
||||
# decryption works on first pass (../sops/age-key-secret.md). This script
|
||||
# only VERIFIES it exists; it never handles key material.
|
||||
# 3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
|
||||
# 3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
|
||||
# GitRepository `flux-system` (source) -> Kustomization `platform` ->
|
||||
# `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
|
||||
# (chained dependsOn) + `apps-vms`, with cluster-vars overlay
|
||||
@@ -52,7 +55,10 @@
|
||||
set -euo pipefail
|
||||
|
||||
# --- config (env-overridable) ------------------------------------------------
|
||||
FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}" # dev is the ONLY Phase-1 build target
|
||||
# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
|
||||
# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
|
||||
# every existing caller — override it, never edit it.
|
||||
FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
|
||||
FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
|
||||
SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
|
||||
# The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
|
||||
@@ -62,7 +68,7 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
|
||||
|
||||
echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
|
||||
echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
|
||||
|
||||
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
|
||||
for bin in flux kubectl; do
|
||||
@@ -134,7 +140,7 @@ if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- 3. apply the authored dev entrypoint ------------------------------------
|
||||
# --- 3. apply the authored environment entrypoint ----------------------------
|
||||
echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
|
||||
kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
|
||||
|
||||
|
||||
@@ -1,11 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
|
||||
# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
|
||||
# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
|
||||
# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
|
||||
# CONTRIBUTING.md).
|
||||
# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
|
||||
# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
|
||||
# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
|
||||
# using it now.
|
||||
#
|
||||
# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
|
||||
# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
|
||||
# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
|
||||
#
|
||||
# PROD (bare metal, current use): HOST_NAT_IP=192.168.2.132
|
||||
# The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
|
||||
# NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
|
||||
# three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
|
||||
# apply to prod, so do not "fix" prod by opening zones.
|
||||
#
|
||||
# DEV (libvirt VMs, HISTORICAL): HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
|
||||
# *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
|
||||
# bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
|
||||
# recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
|
||||
#
|
||||
# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
|
||||
# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
|
||||
# author-only work and must never be fired off casually or "to see if it works": it starts a
|
||||
# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
|
||||
# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
|
||||
# it will then reconcile. Know which HEAD you are publishing before you run it.
|
||||
set -euo pipefail
|
||||
ENGINE="${ENGINE:-podman}"
|
||||
GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}" # pinned; bump if needed
|
||||
@@ -16,11 +35,24 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
|
||||
# subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
|
||||
DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
|
||||
DATA_DIR="podman volume ${DATA_VOLUME}" # display label only (data lives in the named volume)
|
||||
# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
|
||||
# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
|
||||
# clone/redirect URLs are correct from inside the cluster.
|
||||
# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
|
||||
#
|
||||
# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
|
||||
# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
|
||||
# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
|
||||
# Pass it explicitly, e.g. for prod: HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
|
||||
#
|
||||
# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
|
||||
# VM->host:3000 by default and had to be opened once with
|
||||
# sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
|
||||
HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
|
||||
# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
|
||||
if [ -z "${HOST_NAT_IP:-}" ]; then
|
||||
echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
|
||||
echo " prod: HOST_NAT_IP=192.168.2.132 (same LAN as the nodes, no NAT, no firewall change)" >&2
|
||||
echo " There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
|
||||
exit 1
|
||||
fi
|
||||
ORG="${GITEA_ORG:-Home-Network}"
|
||||
REPO="${GITEA_REPO:-home-network-k3s-impl}"
|
||||
|
||||
@@ -75,8 +107,17 @@ cat <<EOF
|
||||
|
||||
Gitea bootstrap source ready (${DATA_DIR}).
|
||||
UI (host): ${BASE}/${ORG}/${REPO}
|
||||
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git (clusters/dev)
|
||||
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
|
||||
(patch this url into the overlay being bootstrapped:
|
||||
clusters/prod today; clusters/dev has already cut over)
|
||||
Admin creds (host, 0600): ${ADMIN_CRED_FILE}
|
||||
Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
|
||||
Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
|
||||
firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
|
||||
must allow VM->host:${HTTP_PORT} —
|
||||
ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
|
||||
REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
|
||||
at this source will reconcile THAT revision.
|
||||
TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
|
||||
any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
|
||||
procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
|
||||
EOF
|
||||
+39
-5
@@ -24,9 +24,9 @@ env:
|
||||
name: gitea-bootstrap
|
||||
key: secret-key
|
||||
- name: GITEA__server__DOMAIN # correct public clone/redirect URLs for the TLS-terminated host
|
||||
value: gitea.home.keithjackson.dev
|
||||
value: gitea.${BASE_DOMAIN}
|
||||
- name: GITEA__server__ROOT_URL
|
||||
value: https://gitea.home.keithjackson.dev/
|
||||
value: https://gitea.${BASE_DOMAIN}/
|
||||
- name: GITEA__actions__ENABLED # Phase 6 CI/CD: declared (not implicit upstream default) so the
|
||||
value: "true" # in-cluster act_runner (ns gitea-actions) registers against it
|
||||
- name: GITEA__actions__DEFAULT_ACTIONS_URL # resolve `uses: <org>/<action>@ref` from github.com (cluster
|
||||
@@ -104,7 +104,7 @@ lifecycle:
|
||||
# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
|
||||
# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
|
||||
# usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
|
||||
GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
||||
GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
|
||||
gitea_oidc_fail() {
|
||||
echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
|
||||
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
|
||||
@@ -162,7 +162,13 @@ lifecycle:
|
||||
gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
|
||||
fi
|
||||
fi
|
||||
exposure: public # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
|
||||
# A postStart hook MUST exit 0 — a non-zero exit makes the kubelet KILL the container
|
||||
# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but
|
||||
# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to
|
||||
# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
|
||||
# the moment its OIDC guard was fixed and the block actually ran for the first time.
|
||||
exit 0
|
||||
exposure: public # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
||||
storage:
|
||||
- name: data
|
||||
@@ -172,8 +178,36 @@ storage:
|
||||
# the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
|
||||
dependencies: [] # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: gitea.home.keithjackson.dev
|
||||
- host: gitea.${BASE_DOMAIN}
|
||||
port: 3000 # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
||||
# THIS IS WHAT KEEPS PROD'S GITOPS OUT OF A DEADLOCK — it is not an incidental allowance. Remove it
|
||||
# and prod cannot reconcile AT ALL, in a way that is very hard to diagnose from cold.
|
||||
#
|
||||
# Prod's Flux source is THIS SERVICE, not a public hostname:
|
||||
# clusters/prod/kustomization.yaml -> GitRepository.spec.url =
|
||||
# http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
|
||||
# chosen deliberately so prod's GitOps depends on NEITHER public DNS, Cloudflare, prod's own
|
||||
# Gateway, NOR ${BASE_DOMAIN} — a future domain change cannot sever prod from its own source. That
|
||||
# file documents the other half of this pair at length; the two must be read together.
|
||||
#
|
||||
# The only other ingress this entry renders is gitea-allow-gateway-ingress, which admits
|
||||
# `fromEntities: [ingress]` — the shared Gateway's Envoy proxy — and nothing else. Flux's
|
||||
# source-controller dials the ClusterIP DIRECTLY, so under the namespace default-deny (R-NET-01) its
|
||||
# clone was DROPPED (`wget` from source-controller timed out; srcReady=False, "unable to list
|
||||
# remote"). The deadlock that made this urgent: the HTTPRoute that would have served the new public
|
||||
# name after the 2026-09-13 apex flip lives in a DOWNSTREAM tier, which cannot apply while the
|
||||
# source is down — source down -> routes never rename -> source stays down. It was broken on
|
||||
# 2026-09-13 with a live `kubectl apply` of a hand-written gitea-allow-flux-ingress, i.e. with a
|
||||
# cluster object that existed OUTSIDE git. This declaration renders that object, so one
|
||||
# `kubectl delete` or one cluster rebuild can no longer put prod back in the deadlock.
|
||||
inClusterClients:
|
||||
# `name` is pinned to `flux` (rather than defaulting to the namespace) so the rendered policy name
|
||||
# equals the live object's, `gitea-allow-flux-ingress`, and Flux ADOPTS it in place: no
|
||||
# near-duplicate policy beside it, and no window in which prod's own source has no allow.
|
||||
- name: flux
|
||||
namespace: flux-system # matched by NAMESPACE alone — the flux controllers' pod labels belong
|
||||
# to upstream's manifests, not to this platform's contract
|
||||
port: 3000 # ONLY the Gitea HTTP port — the same single port the Gateway allow opens
|
||||
auth:
|
||||
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
|
||||
groups:
|
||||
|
||||
+22
-2
@@ -17,20 +17,40 @@ name: jackett
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/jackett:latest
|
||||
exposure: internal # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
|
||||
exposure: internal # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
|
||||
egress: vpn # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
- name: config
|
||||
size: 2Gi
|
||||
class: longhorn # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
|
||||
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
|
||||
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
|
||||
# /config), so without this the Longhorn PVC mounted at /data/config sat
|
||||
# holding nothing but lost+found while the app wrote its entire
|
||||
# configuration to the container's EPHEMERAL writable layer — destroyed on
|
||||
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
|
||||
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
|
||||
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
|
||||
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
|
||||
# The live /config contents were copied onto each PVC before this landed,
|
||||
# so API keys and credentials other apps reference were preserved.
|
||||
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: jackett.home.keithjackson.dev
|
||||
- host: jackett.${BASE_DOMAIN}
|
||||
port: 9117 # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
|
||||
auth:
|
||||
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
|
||||
groups:
|
||||
- media-users # authenticated AND in an authorized group (R-IDP-06)
|
||||
# platform-admins is a SUPERSET, not a widening. --allowed-group is repeatable and OR-ed, so
|
||||
# listing both means "media consumers OR platform operators". Without it the media UIs are
|
||||
# reachable by NOBODY the moment media-users is empty: on prod the realm's only user is the
|
||||
# operator, who is in platform-admins only, so jackett/qbittorrent/radarr/sonarr answered an
|
||||
# immediate 403 at the oauth2-proxy callback (no login prompt, because the Keycloak SSO session
|
||||
# already existed) while pihole/crafty — which already gate on platform-admins — worked.
|
||||
# This grants no real privilege: platform-admins already maps to cluster-admin via
|
||||
# k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
|
||||
- platform-admins # operators of these services must not be locked out of them
|
||||
update:
|
||||
channel: stable # R-UPD-01
|
||||
range: ">=0.22.0" # stay on the Jackett 0.22.x stable line
|
||||
|
||||
@@ -18,7 +18,7 @@ dependencies:
|
||||
port: 5432
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: keycloak.home.keithjackson.dev
|
||||
- host: keycloak.${BASE_DOMAIN}
|
||||
port: 8080 # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
|
||||
auth:
|
||||
mode: native # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.
|
||||
|
||||
+54
-6
@@ -50,13 +50,13 @@ env:
|
||||
# the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
|
||||
# though it speaks plain HTTP behind the proxy. ---
|
||||
- name: NEXTCLOUD_TRUSTED_DOMAINS
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
- name: OVERWRITEHOST
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
- name: OVERWRITEPROTOCOL
|
||||
value: https
|
||||
- name: OVERWRITECLIURL
|
||||
value: https://nextcloud.home.keithjackson.dev
|
||||
value: https://nextcloud.${BASE_DOMAIN}
|
||||
# --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
|
||||
# `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
|
||||
# (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
|
||||
@@ -88,7 +88,7 @@ lifecycle:
|
||||
# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
|
||||
# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
|
||||
OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
|
||||
NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
||||
NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
|
||||
nextcloud_oidc_fail() {
|
||||
echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
|
||||
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
|
||||
@@ -203,7 +203,17 @@ lifecycle:
|
||||
[ "$missing" = 0 ] && break
|
||||
p=$((p + 1)); sleep 2
|
||||
done
|
||||
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
|
||||
if [ "$p" -ge 15 ]; then
|
||||
nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
|
||||
fi
|
||||
# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container
|
||||
# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:
|
||||
# the final statement used to be `[ "$p" -ge 15 ] && nextcloud_oidc_fail ...`, so on the
|
||||
# SUCCESS path the test returned 1 and became the script's exit status. That was latent for
|
||||
# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard
|
||||
# exposed it immediately. Every failure path here logs loudly and writes a marker file and
|
||||
# deliberately does NOT crash the pod — the break-glass local admin must stay usable.
|
||||
exit 0
|
||||
exposure: public # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks
|
||||
storage:
|
||||
@@ -213,12 +223,50 @@ storage:
|
||||
mountPath: /var/www/html # 3-replica tier; irreplaceable, DR-critical (docs/04; R-STATE-02).
|
||||
# The whole html dir is persisted so config.php + user data survive a
|
||||
# rebuild (the default /data/<name> would strand them on ephemeral rootfs).
|
||||
# THE LEGACY NEXTCLOUD DATASET (Storage/nextcloud, 270 G) — MOUNTED READ-ONLY, DELIBERATELY, AND
|
||||
# NOT AS THIS INSTANCE'S DATA DIRECTORY. Read this before changing either decision.
|
||||
#
|
||||
# WHAT IS ACTUALLY IN THERE (measured 2026-09-07, not assumed):
|
||||
# /mnt/Storage/nextcloud/nextcloud/config 103 M
|
||||
# /mnt/Storage/nextcloud/nextcloud/config_old 3.4 G
|
||||
# /mnt/Storage/nextcloud/nextcloud/config_backup 267 G <- 99% of the dataset
|
||||
# There is NO `data/` directory. This is the OLD, non-k3s Nextcloud's configuration plus a backup
|
||||
# blob — an ARCHIVE, not a live user-data tree.
|
||||
#
|
||||
# WHY READ-ONLY. Two independent reasons, either sufficient:
|
||||
# 1. The old instance is STILL LIVE on it. The export is currently shared rw to 192.168.1.6 and
|
||||
# 192.168.1.5, and the operator has said the existing non-k3s prod must remain active. Two
|
||||
# Nextcloud instances writing one instance's config directory, each with its own database and
|
||||
# therefore its own oc_filecache, is a corruption scenario with no clean recovery.
|
||||
# 2. Nothing here needs writing. An archive is read.
|
||||
# "Do not grant more than needed" therefore lands on `ro`. If the operator's intent is in fact a
|
||||
# MIGRATION (read this data in, then decommission the old instance), that is a different and
|
||||
# larger change than a mount, and it should be a deliberate one — flip `readOnly` and update the
|
||||
# export at that point, not before. Enforced by the PV's nfs.readOnly, so it is a real `ro` mount.
|
||||
#
|
||||
# WHY NOT /var/www/html/data. That path is the LONGHORN PVC above and is this instance's own
|
||||
# state; mounting a foreign tree into it would have Nextcloud scan and index an archive it does
|
||||
# not own. A neutral path keeps the two completely separate. To expose it to users, add it as a
|
||||
# Nextcloud External Storage (Local) mount pointing at this path — an in-app, per-user decision.
|
||||
#
|
||||
# subPath skips the redundant single-child directory: the dataset root holds exactly one entry,
|
||||
# `nextcloud/`, so mounting that subtree puts config/ config_old/ config_backup/ at the top.
|
||||
#
|
||||
# UID: nothing to do. The tree is 0755 owned 1000:3000 and the mount is read-only, so www-data
|
||||
# (uid 33) can read and traverse it. (fsGroup would be a no-op on an NFS volume in any case.)
|
||||
- name: legacy-archive
|
||||
class: bulk
|
||||
export: /mnt/Storage/nextcloud # the ZFS dataset Storage/nextcloud, on ${BULK_NFS_SERVER}
|
||||
subPath: nextcloud
|
||||
mountPath: /mnt/legacy-nextcloud
|
||||
readOnly: true
|
||||
size: 270Gi # ADVISORY ONLY on a static NFS PV — the dataset's used size.
|
||||
dependencies:
|
||||
- to: nextcloud-db # explicit allow: Nextcloud -> its PostgreSQL (docs/13 graph; R-NET-02).
|
||||
port: 5432 # Renders the egress half; the DB namespace mirrors keycloak-db (no
|
||||
protocol: tcp # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
|
||||
ingressRoutes:
|
||||
- host: nextcloud.home.keithjackson.dev
|
||||
- host: nextcloud.${BASE_DOMAIN}
|
||||
port: 80 # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
|
||||
auth:
|
||||
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
|
||||
|
||||
+1
-1
@@ -14,7 +14,7 @@ storage:
|
||||
class: longhorn # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
|
||||
dependencies: [] # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: pihole.home.keithjackson.dev
|
||||
- host: pihole.${BASE_DOMAIN}
|
||||
port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09)
|
||||
auth:
|
||||
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
|
||||
|
||||
+31
-16
@@ -4,21 +4,11 @@
|
||||
# auth: native (Plex accounts, NOT Keycloak — docs/13 + docs/15 documented exception).
|
||||
# Validates against schemas/servicedefinition.schema.json.
|
||||
#
|
||||
# BULK MEDIA (spec bulk-tier requirement, docs/04) — OPEN PREREQUISITE, intentionally NOT declared
|
||||
# here yet (the requirement ID is cited in the WS-APPS report, not as a token here: this change does
|
||||
# NOT exercise the bulk tier, so it must not be selected as a "touched" requirement whose in-cluster
|
||||
# conformance test is still a Phase-1 gap).
|
||||
# The Plex library belongs on the `bulk` tier (docs/04 / docs/13 cross-cutting note: metadata DB
|
||||
# on replicated Longhorn, large library on bulk). The `bulk` StorageClass exists
|
||||
# (platform/longhorn/storageclasses.yaml) and the schema+renderer both support `class: bulk`, but
|
||||
# that class is a documented STUB: OQ-2 (external NFS/SMB backing) is UNRESOLVED and the NFS CSI
|
||||
# driver is not installed, so a `bulk` PVC will not bind and would leave the Plex pod Pending.
|
||||
# To onboard the running Plex shell now (compute + config + public exposure), the media volume is
|
||||
# omitted. Once OQ-2 is resolved and `bulk` provisions, attach the library with a one-line add:
|
||||
# storage:
|
||||
# - name: media
|
||||
# class: bulk # external mount; size omitted (sized outside the cluster)
|
||||
# then re-render. See the WS-APPS report for the full bulk-media finding.
|
||||
# BULK MEDIA — NOW ATTACHED (docs/04 bulk tier; the long-standing OQ-2 prerequisite, resolved for
|
||||
# the STATIC case). The library is 8.33 T of ZFS on the Proxmox host at /mnt/Storage/plex, and it
|
||||
# already exists: nothing is provisioned, an existing dataset is mounted. See
|
||||
# docs/BULK-NFS-STORAGE.md for the layout, the export change and the uid analysis. (The bulk-tier
|
||||
# requirement ID is deliberately still not typed here — see that document's "gate" note.)
|
||||
name: plex
|
||||
kind: container
|
||||
source:
|
||||
@@ -70,9 +60,34 @@ storage:
|
||||
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
|
||||
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
|
||||
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
|
||||
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
|
||||
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
|
||||
# downloads/ movies/ photos/ tv/).
|
||||
#
|
||||
# WHY THE ROOT AND NOT THREE subPath MOUNTS. Nothing is gained by narrowing: the mount is `ro`,
|
||||
# so no subPath could remove a privilege that matters, while three mounts would triple the NFS
|
||||
# mounts per pod and mean a manifest change every time a top-level library is added.
|
||||
#
|
||||
# WHY READ-ONLY, when the export is `rw` and the other consumers write. Plex is a SERVER: it
|
||||
# indexes and streams. Every write on that tree belongs to Sonarr and Radarr. Marking it here
|
||||
# costs nothing and removes Plex from the set of things that can damage 8.33 T. Enforced by the
|
||||
# PV's nfs.readOnly (a real `ro` mount), not only by the volumeMount flag.
|
||||
#
|
||||
# UID: no change needed. The tree is 0755 owned 1000:3000 throughout, so it is world-readable and
|
||||
# world-traversable, and PLEX_UID/PLEX_GID above already make the process uid 1000 in any case.
|
||||
# (fsGroup would have been a no-op here regardless — the in-tree NFS plugin does no ownership
|
||||
# management. That is the trap the arrs' PUID/PGID exists to avoid; Plex never hits it.)
|
||||
- name: media
|
||||
class: bulk
|
||||
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
|
||||
mountPath: /media
|
||||
readOnly: true
|
||||
size: 9Ti # ADVISORY ONLY (nothing enforces a static NFS PV's capacity): the
|
||||
# dataset's 8.33 T used, rounded up, so `kubectl get pv` reports
|
||||
# something true. The real limit is the ZFS pool.
|
||||
dependencies: [] # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: plex.home.keithjackson.dev
|
||||
- host: plex.${BASE_DOMAIN}
|
||||
port: 32400 # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
|
||||
auth:
|
||||
mode: native # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
|
||||
|
||||
+129
-16
@@ -4,19 +4,18 @@
|
||||
# internal download client | egress VPN + kill-switch | config on Longhorn | auth forward-auth
|
||||
# (bare web UI, no native OIDC).
|
||||
#
|
||||
# BULK DOWNLOADS TIER — OPEN PREREQUISITE, intentionally NOT declared here (mirrors plex + sonarr/radarr):
|
||||
# docs/13 row 13 places qBittorrent's downloads/staging volume on the `bulk` tier (large, non-3x-
|
||||
# replicated media handoff to the arr stack). The `bulk` StorageClass exists
|
||||
# (platform/longhorn/storageclasses.yaml) and both the schema and renderer support `class: bulk`, BUT
|
||||
# it is a NON-PROVISIONING STUB pending OQ-2 (its NFS/SMB backing is REPLACE-ME and the NFS CSI driver
|
||||
# is not installed) — a `bulk` PVC would never bind and would leave the pod Pending. Declaring it now
|
||||
# would also pull the bulk-tier requirement into the change's "touched" set, whose in-cluster
|
||||
# conformance test is not yet scaffolded, blocking the merge gate. So the downloads volume is DEFERRED
|
||||
# and this onboarding ships config-only. Once OQ-2 is resolved and `bulk` provisions, attach downloads
|
||||
# with a one-line add under storage:
|
||||
# - name: downloads
|
||||
# class: bulk # external mount; size omitted (sized outside the cluster)
|
||||
# then re-render. See the WS-APPS report for the full bulk-downloads finding.
|
||||
# BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record).
|
||||
# This entry used to ship config-only, with a comment promising to attach the downloads volume "once
|
||||
# OQ-2 is resolved". OQ-2 was resolved for STATIC pre-existing datasets (schema 1.5.0 / renderer
|
||||
# `class: bulk` + `export` -> a static, pre-bound NFS PV), and plex/sonarr/radarr/nextcloud all took
|
||||
# their media mounts — but qBittorrent's was never added, so the DOWNLOAD CLIENT was the one member
|
||||
# of the chain with no media volume at all. Measured on prod 2026-09-12:
|
||||
# Session\DefaultSavePath=/downloads/ Session\TempPath=/downloads/incomplete/
|
||||
# Downloads\SavePath=/downloads/ `ls /downloads` -> No such file or directory
|
||||
# i.e. the app was configured to write a path that DID NOT EXIST, in a container with no volume there.
|
||||
# Anything it did manage to create would have been invisible to sonarr/radarr, lost on every pod
|
||||
# recreation, and filling the NODE's ephemeral disk in the meantime. The `downloads` entry under
|
||||
# storage: below closes that. See docs/BULK-NFS-STORAGE.md and the bulk-tier L0 conformance test.
|
||||
#
|
||||
# VPN-class (docs/13 row 13; docs/03 "VPN-class set"): ALL of this pod's public-internet egress
|
||||
# (peer/tracker/DHT traffic) MUST leave via the node PIA tunnel, never the WAN. `egress: vpn` renders
|
||||
@@ -32,20 +31,134 @@ name: qbittorrent
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/qbittorrent:5.1.2 # pinned stable tag (R-UPD-01); update.range governs advancement
|
||||
exposure: internal # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
|
||||
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
|
||||
# LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
|
||||
# 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
|
||||
# uid=911 gid=911, and /mnt/Storage/plex/downloads is 0755 owned 1000:1000. 0755 grants write to the
|
||||
# OWNER ONLY, so uid 911 would have mounted the export cleanly and then failed every write with
|
||||
# EACCES — the "mounts fine, cannot write" failure the sonarr entry warns about at length.
|
||||
env:
|
||||
- name: PUID
|
||||
value: "1000" # owner uid of /mnt/Storage/plex/downloads (and of every dir under the export)
|
||||
- name: PGID
|
||||
value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay
|
||||
# readable to the EXISTING non-k3s consumers of the same dataset, and land
|
||||
# group-owned the same way the arrs' own writes do
|
||||
podSecurityContext:
|
||||
# For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
|
||||
# on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
|
||||
# the kubelet applies it, so /config becomes group-writable by the now-uid-1000 process without
|
||||
# depending on the image chowning it. (This image DOES chown: init-qbittorrent-config runs
|
||||
# `lsiown -R abc:abc /config`. Belt and braces — the live /config was 911:911 before this change,
|
||||
# and losing write access to it would lose the WebUI password hash and the session state.)
|
||||
fsGroup: 3000
|
||||
exposure: internal # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
|
||||
egress: vpn # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
- name: config
|
||||
size: 5Gi
|
||||
class: longhorn # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
|
||||
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
|
||||
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
|
||||
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
|
||||
# /config), so without this the Longhorn PVC mounted at /data/config sat
|
||||
# holding nothing but lost+found while the app wrote its entire
|
||||
# configuration to the container's EPHEMERAL writable layer — destroyed on
|
||||
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
|
||||
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
|
||||
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
|
||||
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
|
||||
# The live /config contents were copied onto each PVC before this landed,
|
||||
# so API keys and credentials other apps reference were preserved.
|
||||
# THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
|
||||
# (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
|
||||
# /downloads, which is precisely where this app is ALREADY configured to write.
|
||||
#
|
||||
# WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
|
||||
# * LEAST PRIVILEGE, AND IT COSTS NOTHING HERE. A download client has no business writing the
|
||||
# LIBRARY. The arrs need the root mount because THEY hardlink downloads/ -> tv/ or movies/ and
|
||||
# link() is a same-MOUNT operation (EXDEV otherwise) — their smallest common ancestor genuinely
|
||||
# IS the export root. qBittorrent performs no such cross-directory link: it writes into the
|
||||
# downloads tree and nothing else, so narrowing to that subtree removes movies/, tv/ and photos/
|
||||
# from this pod's mount namespace without removing any operation it actually performs.
|
||||
# * HARDLINKS STILL WORK, because they are not done here. The arrs still see ONE mount containing
|
||||
# both /media/downloads and /media/tv, so their link() stays intra-mount. Proven live 2026-09-12:
|
||||
# a file created from this pod under /downloads was hardlinked from the sonarr pod between
|
||||
# /media/downloads/... and /media/tv/... with st_nlink 1 -> 2 and no EXDEV.
|
||||
# * IT IS ONE MOUNT, so the L0 test `bulk-tier-distinct-from-replicated` assertion (6) — no
|
||||
# workload mounts one export twice — holds. That assertion is about TWO mounts inside ONE pod;
|
||||
# a single subPath mount is not the EXDEV trap, two are.
|
||||
# * The narrowing is a CLIENT/kubelet guarantee, stated honestly: the NFS server still exports the
|
||||
# whole dataset to this node, so the containment is the mount namespace, not the server's ACLs —
|
||||
# the same class of caveat docs/BULK-NFS-STORAGE.md §5 records for plex's readOnly.
|
||||
#
|
||||
# NOT readOnly — this is the one workload whose whole job is WRITING here.
|
||||
#
|
||||
# KNOWN, DELIBERATE SIDE EFFECT ON SHARED STATE: the image's init-qbittorrent-config runs
|
||||
# `lsiown abc:abc /downloads` (non-recursive) whenever /downloads is a mount, so starting this pod
|
||||
# chgrps /mnt/Storage/plex/downloads from 1000:1000 to 1000:3000 on the SERVER. Benign and verified:
|
||||
# the directory is mode 0755, whose group bits (r-x) are identical to its other bits, so no consumer
|
||||
# — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
|
||||
# non-cluster consumers, which should never be a surprise.
|
||||
#
|
||||
# PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path
|
||||
# Mappings". This pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
|
||||
# /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
|
||||
# ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote
|
||||
# path /downloads/, local path /media/downloads/).
|
||||
#
|
||||
# THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent
|
||||
# download client itself — which did not exist in EITHER arr until then (both
|
||||
# `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up
|
||||
# at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed
|
||||
# reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it
|
||||
# present — same torrent, same client.
|
||||
#
|
||||
# BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild
|
||||
# from git alone does NOT restore them and no gate here notices their absence. The alternative —
|
||||
# repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
|
||||
# buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
|
||||
# not chosen because it changes a running download client's save path, which the deferral above
|
||||
# already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the
|
||||
# mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator
|
||||
# decision, not a technical blocker.
|
||||
#
|
||||
# RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username /
|
||||
# WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every
|
||||
# start. The arrs' saved download-client password therefore goes stale on every pod restart. See the
|
||||
# same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally.
|
||||
- name: downloads
|
||||
class: bulk
|
||||
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export
|
||||
# plex (ro) and sonarr/radarr (rw) mount; one dataset, one set of inodes
|
||||
subPath: downloads # the handoff subtree ONLY: no movies/, no tv/, no photos/
|
||||
mountPath: /downloads # matches the app's EXISTING Session\DefaultSavePath / TempPath, so no
|
||||
# change to config-PVC app state is needed to make this take effect
|
||||
size: 9Ti # ADVISORY ONLY on a static NFS PV (the ZFS pool is the real limit): the
|
||||
# dataset's 8.4 T used rounded up, stated the same way plex/sonarr/radarr
|
||||
# state it so `kubectl get pv` reports something true
|
||||
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02).
|
||||
# STILL EMPTY WITH THE NFS MOUNT ABOVE, AND THAT IS CORRECT, NOT AN
|
||||
# OVERSIGHT. An NFS PersistentVolume is mounted by the NODE KERNEL in the
|
||||
# HOST netns, not by this pod: there is no pod-side socket to 2049, so no
|
||||
# CiliumNetworkPolicy can permit or deny it and none is needed. Verified by
|
||||
# construction — sonarr and radarr have mounted this same export read-write
|
||||
# for weeks and neither has any CNP naming the NFS server or port 2049
|
||||
# (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
|
||||
# egress gateway off this traffic (its selector is a podSelector), and
|
||||
# 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
|
||||
# platform/vpn-egress/egress-steering.yaml, "BULK NFS".
|
||||
ingressRoutes:
|
||||
- host: qbittorrent.home.keithjackson.dev
|
||||
- host: qbittorrent.${BASE_DOMAIN}
|
||||
port: 8080 # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
|
||||
auth:
|
||||
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
|
||||
groups:
|
||||
- media-users # authenticated AND in an authorized group (R-IDP-06)
|
||||
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
|
||||
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
|
||||
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
|
||||
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
|
||||
- platform-admins # operators of these services must not be locked out of them
|
||||
update:
|
||||
channel: stable # R-UPD-01
|
||||
range: ">=5.0.0 <6.0.0" # stay on the qBittorrent 5.x stable line
|
||||
|
||||
+84
-7
@@ -9,18 +9,90 @@ name: radarr
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/radarr:5.21.1 # pinned stable v5 tag (R-UPD-01); update.range governs advancement
|
||||
exposure: internal # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
|
||||
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
|
||||
# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
|
||||
# which cannot write a 1000:3000-owned 0755 tree.
|
||||
env:
|
||||
- name: PUID
|
||||
value: "1000" # owner uid of every directory under /mnt/Storage/plex
|
||||
- name: PGID
|
||||
value: "3000" # dominant group there; keeps files this writes readable to the
|
||||
# EXISTING non-k3s consumers of the same export, which stay live
|
||||
- name: TZ
|
||||
value: Etc/UTC # same safe default plex uses; the arrs schedule by local time
|
||||
podSecurityContext:
|
||||
# For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
|
||||
# does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
|
||||
# what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
|
||||
# /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
|
||||
fsGroup: 3000
|
||||
exposure: internal # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
|
||||
egress: vpn # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
|
||||
# kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
- name: config
|
||||
size: 5Gi
|
||||
class: longhorn # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
|
||||
# NOTE (bulk-media PREREQUISITE): docs/13 row 3 places the media library / downloads volume on the
|
||||
# `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
|
||||
# but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
|
||||
# volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
|
||||
# therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Radarr deployable today.
|
||||
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
|
||||
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
|
||||
# /config), so without this the Longhorn PVC mounted at /data/config sat
|
||||
# holding nothing but lost+found while the app wrote its entire
|
||||
# configuration to the container's EPHEMERAL writable layer — destroyed on
|
||||
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
|
||||
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
|
||||
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
|
||||
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
|
||||
# The live /config contents were copied onto each PVC before this landed,
|
||||
# so API keys and credentials other apps reference were preserved.
|
||||
# THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
|
||||
# /media/movies (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
|
||||
#
|
||||
# *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Radarr imports by HARDLINKING or
|
||||
# atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
|
||||
# and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
|
||||
# /mnt/Storage/plex/movies/ the library
|
||||
# /mnt/Storage/plex/downloads/complete/radarr/ where finished releases land
|
||||
# — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
|
||||
# them as two subPath volumes (subPath: movies at /media/movies, subPath: downloads elsewhere) creates
|
||||
# TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Radarr then
|
||||
# silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
|
||||
# seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
|
||||
# is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
|
||||
# requirement, not a shortcut. (The cost is stated plainly: Radarr can also write /media/tv,
|
||||
# which is sonarr's. Narrowing that would cost hardlinks, which is a worse trade.)
|
||||
#
|
||||
# *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
|
||||
# renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
|
||||
# config volume entirely; the pod would come up with its app database shadowed by an 8 T media
|
||||
# tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
|
||||
# the same export, so when qbittorrent is eventually given this mount too, the arrs and the
|
||||
# download client will already agree on absolute paths — which is the other precondition for
|
||||
# hardlinked imports (otherwise every import needs a Remote Path Mapping).
|
||||
#
|
||||
# SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
|
||||
# is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
|
||||
# with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
|
||||
# Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
|
||||
# documents having already been bitten by. Fixing it means starting from an empty config volume, so
|
||||
# it belongs in its own change with the operator present.
|
||||
#
|
||||
# UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
|
||||
# + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
|
||||
# the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
|
||||
# write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
|
||||
# every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
|
||||
#
|
||||
# AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
|
||||
# ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
|
||||
# lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
|
||||
# that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
|
||||
# the image re-chowning /config for itself.
|
||||
- name: media
|
||||
class: bulk
|
||||
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
|
||||
mountPath: /media # => /media/movies (Root Folder) and /media/downloads (import source)
|
||||
size: 9Ti # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
|
||||
# rounded up, so `kubectl get pv` reports something true.
|
||||
dependencies:
|
||||
- to: qbittorrent # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
|
||||
port: 8080
|
||||
@@ -29,12 +101,17 @@ dependencies:
|
||||
port: 9117
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: radarr.home.keithjackson.dev
|
||||
- host: radarr.${BASE_DOMAIN}
|
||||
port: 7878 # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
||||
auth:
|
||||
mode: forward-auth # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
|
||||
groups:
|
||||
- media-users # authenticated AND in the media group (R-IDP-06)
|
||||
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
|
||||
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
|
||||
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
|
||||
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
|
||||
- platform-admins # operators of these services must not be locked out of them
|
||||
update:
|
||||
channel: stable # R-UPD-01
|
||||
range: ">=5.0.0 <6.0.0" # stay on the Radarr v5 stable line
|
||||
|
||||
+383
-10
@@ -4,7 +4,8 @@
|
||||
Consumes ONE ServiceDefinition (schemas/servicedefinition.schema.json, catalog/*.yaml)
|
||||
and emits the concrete manifest set the renderer contract specifies
|
||||
(schemas/renderer-contract.md): Namespace + default-deny baseline, Deployment/Service or
|
||||
KubeVirt VirtualMachine + CDI DataVolume, PVCs, per-dependency CiliumNetworkPolicy allows,
|
||||
KubeVirt VirtualMachine + CDI DataVolume, PVCs, per-dependency and per-in-cluster-client
|
||||
CiliumNetworkPolicy allows,
|
||||
egress-class steering label, HTTPRoute(s), ServiceMonitor/alerts, Flux image-automation,
|
||||
and the auth-mode / backup wiring hints.
|
||||
|
||||
@@ -44,7 +45,39 @@ except ImportError: # pragma: no cover - explicit, actionable failure
|
||||
|
||||
# --- Label / naming conventions (match platform/kyverno/policies + platform/*/) --------
|
||||
LBL = "homelab.keithjackson.dev"
|
||||
ZONE = "home.keithjackson.dev"
|
||||
# The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
|
||||
# Every rendered manifest lands under a Flux path whose Kustomization declares
|
||||
# postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
|
||||
# reconcile time: dev renders home.keithjackson.dev, prod renders prod.home.keithjackson.dev.
|
||||
# It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
|
||||
# the escape contract below reserves `$${...}` for expansions meant for a container's shell.
|
||||
# Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
|
||||
# and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
|
||||
ZONE = "${BASE_DOMAIN}"
|
||||
# The bulk (external NFS) tier's SERVER, emitted as a Flux post-build TOKEN for exactly the same
|
||||
# reason as ZONE: dev and prod reach the same Proxmox ZFS host by DIFFERENT addresses (dev's nodes
|
||||
# are on the host's own LAN and dial it directly; prod's nodes are on the far side of pfSense and
|
||||
# must be given whichever address the still-pending routing decision produces). Hardcoding either
|
||||
# one authors a manifest that is silently wrong in the other environment. Declared in
|
||||
# clusters/base/cluster-vars.yaml and patched by BOTH overlays; written BARE (it is a real knob,
|
||||
# not a shell expansion), so StrictPostBuildSubstitutions fails the tier if an overlay forgets it.
|
||||
BULK_NFS_SERVER = "${BULK_NFS_SERVER}"
|
||||
# Mount options for every bulk NFS volume. These are a PLATFORM decision, not a per-service one,
|
||||
# so they live here rather than on the ServiceDefinition surface:
|
||||
# nfsvers=4.1 NFSv4 carries mount+lock IN the protocol, so it needs ONE port (tcp/2049) and no
|
||||
# rpcbind/111, mountd or statd. That is what makes the prod path a single pfSense
|
||||
# rule instead of four, and it is why v4 is chosen over v3. The server advertises
|
||||
# +3 +4 +4.1 +4.2; 4.1 is the conservative floor every node kernel already speaks.
|
||||
# hard A `soft` mount turns a server hiccup into a short read / EIO handed to the
|
||||
# application, which for a media library and an archive is silent corruption.
|
||||
# `hard` blocks instead. The cost is explicit: an NFS outage HANGS I/O rather than
|
||||
# failing it, so an outage shows up as stuck pods, not as damaged files. Chosen
|
||||
# deliberately -- hung is recoverable, truncated is not.
|
||||
# noatime The datasets are relatime, so every read would otherwise generate a metadata
|
||||
# WRITE back to the pool. Plex walking an 8 T library must not do that.
|
||||
# timeo=600/retrans=2 the TCP defaults, stated explicitly so a future reader knows the retry
|
||||
# budget was considered rather than inherited.
|
||||
BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"]
|
||||
GATEWAY_NAME = "shared"
|
||||
GATEWAY_NS = "gateway"
|
||||
KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery label
|
||||
@@ -84,6 +117,7 @@ def apply_defaults(sd: dict) -> dict:
|
||||
s.setdefault("storage", [])
|
||||
s.setdefault("dependencies", [])
|
||||
s.setdefault("ingressRoutes", [])
|
||||
s.setdefault("inClusterClients", [])
|
||||
|
||||
# auth.mode default posture: forward-auth for any exposed UI, else none (internal, no
|
||||
# UI). An explicit auth block always wins (R-IDP-02).
|
||||
@@ -103,10 +137,19 @@ def apply_defaults(sd: dict) -> dict:
|
||||
obs.setdefault("slis", [])
|
||||
s["observability"] = obs
|
||||
|
||||
# storage entry class default is longhorn (replicated tier).
|
||||
s["storage"] = [{**{"class": "longhorn"}, **v} for v in s["storage"]]
|
||||
# storage entry class default is longhorn (replicated tier); readOnly default is False
|
||||
# (grant-nothing-extra is expressed by SETTING it, so the default stays back-compatible and
|
||||
# every pre-existing golden renders byte-identical).
|
||||
s["storage"] = [{**{"class": "longhorn", "readOnly": False}, **v} for v in s["storage"]]
|
||||
# dependency protocol default is tcp.
|
||||
s["dependencies"] = [{**{"protocol": "tcp"}, **d} for d in s["dependencies"]]
|
||||
# in-cluster client protocol default is tcp (as for a dependency); its `name` — which only
|
||||
# ever appears in the rendered POLICY name — defaults to the namespace it admits, so the
|
||||
# common case declares just `namespace` + `port` and reads like a dependency edge.
|
||||
s["inClusterClients"] = [
|
||||
{**{"protocol": "tcp", "name": c.get("namespace", "")}, **c}
|
||||
for c in s["inClusterClients"]
|
||||
]
|
||||
return s
|
||||
|
||||
|
||||
@@ -192,18 +235,139 @@ def _service_ports(s: dict) -> list:
|
||||
return [{"name": f"port-{p}", "port": p, "targetPort": p} for p in ports]
|
||||
|
||||
|
||||
def _bulk_access_mode(vol: dict) -> str:
|
||||
"""A bulk volume's access mode follows its read/write posture.
|
||||
|
||||
Every bulk volume is `...Many`: the SAME export is consumed by several namespaces at once
|
||||
(plex, sonarr and radarr all sit on /mnt/Storage/plex) and by several nodes at once, which is
|
||||
precisely what ReadWriteOnce cannot express. WHICH `Many` it is says whether this consumer
|
||||
writes -- so `kubectl get pv` reports the intent, not merely the topology.
|
||||
"""
|
||||
return "ReadOnlyMany" if vol.get("readOnly") else "ReadWriteMany"
|
||||
|
||||
|
||||
def r_bulk_pvs(s: dict) -> list:
|
||||
"""One STATIC PersistentVolume per `class: bulk` storage entry (docs/04 bulk tier).
|
||||
|
||||
WHY A PV AT ALL, RATHER THAN AN INLINE `nfs:` POD VOLUME
|
||||
-------------------------------------------------------
|
||||
An inline volume is fewer objects and it would work. It is rejected for four reasons, in
|
||||
descending order of what they would cost to discover later:
|
||||
|
||||
1. RECLAIM POLICY. `persistentVolumeReclaimPolicy: Retain` is a declared, reviewable
|
||||
property of a PV. There is no equivalent for an inline volume -- and the data behind
|
||||
these mounts is 8.33 TiB of media and a 267 GiB archive that ALSO has live non-k3s
|
||||
consumers. A statement in git that nothing in this cluster may reclaim it is worth an
|
||||
object.
|
||||
2. READ-ONLY LIVES IN ONE PLACE. `spec.nfs.readOnly: true` becomes a real `ro` mount option
|
||||
for every pod that ever binds this volume. The same intent expressed on an inline volume
|
||||
is one Deployment edit away from being dropped silently, with no diff anywhere near the
|
||||
storage. (It is still a CLIENT-side guarantee -- the export cannot distinguish plex from
|
||||
sonarr, since both may run on any node -- but it is one that lives in one reviewable
|
||||
place.)
|
||||
3. VISIBILITY. `kubectl get pv,pvc` is where an operator looks for "what external storage is
|
||||
attached, and who holds it". An inline volume is invisible there, and invisible to
|
||||
anything that reasons over PVCs.
|
||||
4. IT IS THE CONTRACT THE REPO ALREADY HAS. Every other storage entry is a PVC; the
|
||||
renderer, the Kyverno ServiceDefinition policy and the backup wiring all key on that. An
|
||||
inline volume would be a second, parallel storage model inside the same renderer.
|
||||
|
||||
CROSS-NAMESPACE. A PV binds 1:1 to a PVC, so "plex, sonarr and radarr share one export" is NOT
|
||||
one shared PV -- it is three PVs, one per namespace, each naming the same server+path. That is
|
||||
the correct model rather than a workaround: the sharing happens on the NFS server, which is
|
||||
built for concurrent clients, and each namespace keeps its own independently reviewable access
|
||||
mode (plex ReadOnlyMany, the arrs ReadWriteMany). One shared PV would have forced a single
|
||||
posture on all three.
|
||||
|
||||
BINDING IS PINNED FROM BOTH ENDS -- `claimRef` here and `volumeName` on the PVC. Either alone
|
||||
leaves a race in which some other PVC of matching size and mode grabs the volume first;
|
||||
together the binding is deterministic. `storageClassName: ""` (explicitly empty, NOT the `bulk`
|
||||
StorageClass) is what tells the binder this volume is statically provisioned: it suppresses the
|
||||
default-StorageClass annotation and guarantees no provisioner is ever asked to create anything.
|
||||
It also sidesteps a trap -- a StorageClass's `provisioner` is IMMUTABLE, so re-pointing the
|
||||
existing `bulk` class (whose stub names an NFS CSI driver that is not installed) at this data
|
||||
would make Flux fail forever on that one object.
|
||||
"""
|
||||
out = []
|
||||
for vol in s["storage"]:
|
||||
if vol.get("class") != "bulk":
|
||||
continue
|
||||
name = f"{s['name']}-{vol['name']}"
|
||||
# `export` and `size` are both structurally required of a bulk entry, and the JSON
|
||||
# Schema only enforces the first. A PersistentVolume with no `capacity` is rejected by
|
||||
# the API server, so a missing size must fail HERE with a sentence a human can act on
|
||||
# rather than as a bare KeyError three frames down.
|
||||
for field, why in (
|
||||
("export", "the absolute NFS export path this volume attaches"),
|
||||
("size", "the PV capacity (advisory, but the API requires the field)"),
|
||||
):
|
||||
if not vol.get(field):
|
||||
raise ValueError(
|
||||
f"{s['name']}: storage entry {vol['name']!r} has class 'bulk' but no "
|
||||
f"'{field}' — {why}. A bulk volume attaches storage that already exists "
|
||||
f"outside the cluster, so it cannot be rendered without it."
|
||||
)
|
||||
nfs = {"server": BULK_NFS_SERVER, "path": vol["export"]}
|
||||
if vol.get("readOnly"):
|
||||
nfs["readOnly"] = True
|
||||
spec = {
|
||||
"capacity": {"storage": vol["size"]},
|
||||
"accessModes": [_bulk_access_mode(vol)],
|
||||
# Retain: deleting the PVC must NEVER be able to reach the data. There is nothing to
|
||||
# "reclaim" here in any case -- the volume was not provisioned by the cluster.
|
||||
"persistentVolumeReclaimPolicy": "Retain",
|
||||
"storageClassName": "",
|
||||
"mountOptions": list(BULK_NFS_MOUNT_OPTIONS),
|
||||
"nfs": nfs,
|
||||
"claimRef": {"namespace": s["name"], "name": name},
|
||||
}
|
||||
out.append(
|
||||
{
|
||||
"apiVersion": "v1",
|
||||
"kind": "PersistentVolume",
|
||||
"metadata": {
|
||||
"name": name,
|
||||
"labels": {"app.kubernetes.io/name": s["name"]},
|
||||
},
|
||||
"spec": spec,
|
||||
}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def r_pvcs(s: dict) -> list:
|
||||
"""One PVC per storage entry (R-STATE-01/02/04).
|
||||
|
||||
longhorn -> 3-replica replicated tier; bulk -> external media tier (not 3x replicated).
|
||||
longhorn -> 3-replica replicated tier, dynamically provisioned, ReadWriteOnce.
|
||||
bulk -> the STATIC external NFS PV rendered by r_bulk_pvs, pre-bound by volumeName.
|
||||
"""
|
||||
out = []
|
||||
for vol in s["storage"]:
|
||||
cls = vol.get("class", "longhorn")
|
||||
storage_class = "longhorn" if cls == "longhorn" else "bulk"
|
||||
if cls == "bulk":
|
||||
# Mirror image of the PV: the same explicitly-empty class (so nothing dynamically
|
||||
# provisions), the same access mode, and volumeName completing the two-sided binding.
|
||||
out.append(
|
||||
{
|
||||
"apiVersion": "v1",
|
||||
"kind": "PersistentVolumeClaim",
|
||||
"metadata": {
|
||||
"name": f"{s['name']}-{vol['name']}",
|
||||
"namespace": s["name"],
|
||||
"labels": {"app.kubernetes.io/name": s["name"]},
|
||||
},
|
||||
"spec": {
|
||||
"accessModes": [_bulk_access_mode(vol)],
|
||||
"storageClassName": "",
|
||||
"volumeName": f"{s['name']}-{vol['name']}",
|
||||
"resources": {"requests": {"storage": vol["size"]}},
|
||||
},
|
||||
}
|
||||
)
|
||||
continue
|
||||
spec = {
|
||||
"accessModes": ["ReadWriteOnce"],
|
||||
"storageClassName": storage_class,
|
||||
"storageClassName": "longhorn",
|
||||
}
|
||||
if vol.get("size"):
|
||||
spec["resources"] = {"requests": {"storage": vol["size"]}}
|
||||
@@ -249,7 +413,17 @@ def r_deployment(s: dict) -> dict:
|
||||
# it verbatim for apps that need a fixed on-disk layout (e.g. Gitea keeps all state
|
||||
# directly under /data). Omitting it preserves the /data/<name> default (back-compat).
|
||||
mount_path = vol.get("mountPath", f"/data/{vol['name']}")
|
||||
volume_mounts.append({"name": vol["name"], "mountPath": mount_path})
|
||||
mount = {"name": vol["name"], "mountPath": mount_path}
|
||||
# subPath mounts a subtree of the volume instead of its root. Emitted only when declared,
|
||||
# so every pre-existing golden stays byte-identical.
|
||||
if vol.get("subPath"):
|
||||
mount["subPath"] = vol["subPath"]
|
||||
# readOnly on the mount as well as on the PV's nfs source. Belt and braces on purpose: the
|
||||
# PV field is what makes the kernel mount `ro`; this one is what a reviewer reading the
|
||||
# Deployment sees. An edit that drops one still leaves the other.
|
||||
if vol.get("readOnly"):
|
||||
mount["readOnly"] = True
|
||||
volume_mounts.append(mount)
|
||||
volumes.append(
|
||||
{
|
||||
"name": vol["name"],
|
||||
@@ -535,6 +709,82 @@ def r_gateway_ingress_policy(s: dict):
|
||||
}
|
||||
|
||||
|
||||
def r_in_cluster_client_policies(s: dict) -> list:
|
||||
"""East-west allow: let a declared IN-CLUSTER client reach this service DIRECTLY (R-NET-02).
|
||||
|
||||
`ingressRoutes` opens the NORTH-SOUTH path only — r_gateway_ingress_policy admits the
|
||||
reserved `ingress` identity (the Gateway proxy) and nothing else — and `dependencies`
|
||||
compiles edges only BETWEEN CATALOG SERVICES (both halves are rendered from a catalog
|
||||
entry's own declaration). A PLATFORM CONTROLLER that dials this service's ClusterIP
|
||||
directly therefore had no authorable form at all, and under default-deny (R-NET-01) it is
|
||||
DROPPED.
|
||||
|
||||
That gap is not theoretical: prod's `flux-system` source-controller clones the platform's
|
||||
own repo from the in-cluster Gitea Service (clusters/prod/kustomization.yaml pins
|
||||
`url: http://gitea.gitea.svc.cluster.local:3000/...` so prod's GitOps depends on neither
|
||||
public DNS, Cloudflare, its own Gateway nor ${BASE_DOMAIN}). gitea-allow-gateway-ingress
|
||||
admits `fromEntities: [ingress]` only, so the clone was denied, the source could not
|
||||
reconcile — and the HTTPRoute that would have served a public name instead lives in a
|
||||
DOWNSTREAM tier that cannot apply while the source is down. Source down -> routes never
|
||||
rename -> source stays down. It had to be broken with a live `kubectl apply`, i.e. with a
|
||||
cluster object outside git, which one `kubectl delete` or one rebuild would take away.
|
||||
|
||||
Each entry renders ONE least-privilege ingress CiliumNetworkPolicy, shaped exactly like the
|
||||
reciprocal-dependency half (r_reverse_dependency_policies) but selecting the client by
|
||||
NAMESPACE alone, because a platform controller's pod labels are the upstream chart's to
|
||||
change and are not part of this platform's contract: `fromEndpoints` carrying ONLY the
|
||||
reserved `k8s:io.kubernetes.pod.namespace` label, `toPorts` carrying ONLY the declared
|
||||
port/protocol. Never `fromEntities: all`, and never the whole service. Additive: the
|
||||
namespace default-deny baseline keeps denying every undeclared source and port.
|
||||
|
||||
The policy NAME is `<service>-allow-<name>-ingress`, `name` defaulting to the namespace —
|
||||
it is operator-facing (`kubectl get cnp`, a Hubble drop), and letting an author pin it is
|
||||
what allows an allowance that already had to be applied live to be authored under its
|
||||
EXISTING name, so Flux adopts that object in place rather than creating a near-duplicate
|
||||
beside it and leaving the live one unmanaged.
|
||||
"""
|
||||
out = []
|
||||
for client in s["inClusterClients"]:
|
||||
proto = client.get("protocol", "tcp").upper()
|
||||
out.append(
|
||||
{
|
||||
"apiVersion": "cilium.io/v2",
|
||||
"kind": "CiliumNetworkPolicy",
|
||||
"metadata": {
|
||||
"name": f"{s['name']}-allow-{client['name']}-ingress",
|
||||
"namespace": s["name"],
|
||||
"labels": {"app.kubernetes.io/name": s["name"]},
|
||||
},
|
||||
"spec": {
|
||||
"endpointSelector": {
|
||||
"matchLabels": {"app.kubernetes.io/name": s["name"]}
|
||||
},
|
||||
"ingress": [
|
||||
{
|
||||
"fromEndpoints": [
|
||||
{
|
||||
"matchLabels": {
|
||||
"k8s:io.kubernetes.pod.namespace": client[
|
||||
"namespace"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"toPorts": [
|
||||
{
|
||||
"ports": [
|
||||
{"port": str(client["port"]), "protocol": proto}
|
||||
]
|
||||
}
|
||||
],
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def _route_meta(s: dict, route: dict) -> dict:
|
||||
meta = {
|
||||
"name": f"{s['name']}-{route['port']}",
|
||||
@@ -1087,6 +1337,14 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
|
||||
if s["kind"] == "container":
|
||||
# Container PVCs. (For a vm, the CDI DataVolume owns/creates the disk PVC, so we
|
||||
# never render a plain PVC for the VM root disk — it would collide with CDI.)
|
||||
# The static bulk PV is rendered BEFORE the PVC that names it. Document order inside one
|
||||
# kustomize build is not an apply-ordering guarantee, but a PVC whose PV does not exist
|
||||
# yet merely stays Pending and binds on the next reconcile, so this is a readability
|
||||
# convention rather than a correctness requirement.
|
||||
for pv in r_bulk_pvs(s):
|
||||
docs.append(
|
||||
(f"PersistentVolume {pv['metadata']['name']} - static external NFS (bulk tier)", pv)
|
||||
)
|
||||
for pvc in r_pvcs(s):
|
||||
docs.append((f"PVC {pvc['metadata']['name']} (R-STATE)", pvc))
|
||||
docs.append(("Deployment (R-PLAT-02)", r_deployment(s)))
|
||||
@@ -1119,6 +1377,15 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
|
||||
)
|
||||
)
|
||||
|
||||
for cnp in r_in_cluster_client_policies(s):
|
||||
docs.append(
|
||||
(
|
||||
"CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): "
|
||||
f"{cnp['metadata']['name']}",
|
||||
cnp,
|
||||
)
|
||||
)
|
||||
|
||||
ie = r_internet_egress(s)
|
||||
docs.append(
|
||||
(
|
||||
@@ -1313,6 +1580,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
||||
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
||||
- `<svc>-db-backup-credentials.sops.yaml` -> wave(svc-db) (CNPG Barman S3 backup secret)
|
||||
- `<svc>-db-backup-credentials-prod.sops.yaml` -> wave(svc-db) (the PROD half of that
|
||||
secret: the two environments carry DIFFERENT object-store credentials, selected in
|
||||
the Cluster CR by `${BACKUP_CREDENTIALS_SUFFIX}`. Both objects ship to both clusters,
|
||||
so both must be placed in the DB wave — the Secret has to exist before the Cluster
|
||||
that names it.)
|
||||
Non-catalog units (EXTRA_UNITS, e.g. gitea-actions) place by their declared anchor+offset.
|
||||
Only files that actually exist in `present_files` are placed, so the plan is robust to a
|
||||
service having no bootstrap secret or no DB. Deterministic (sorted).
|
||||
@@ -1340,6 +1612,9 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
add(wdb, f"{db}.yaml")
|
||||
add(wdb, f"{db}-app.sops.yaml")
|
||||
add(wdb, f"{db}-backup-credentials.sops.yaml") # CNPG Barman S3 creds (R-STATE-03)
|
||||
# PROD half of the same credential (2026-09-07 per-env split). Placed only when
|
||||
# the file exists, so a DB that has not been split is unaffected.
|
||||
add(wdb, f"{db}-backup-credentials-prod.sops.yaml")
|
||||
|
||||
for unit, cfg in EXTRA_UNITS.items():
|
||||
anchor = cfg["anchor"]
|
||||
@@ -1627,6 +1902,101 @@ def _selftest() -> int:
|
||||
"a non-exposed service must not render a gateway-ingress allow (least privilege)",
|
||||
)
|
||||
|
||||
# 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a
|
||||
# PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE
|
||||
# declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service
|
||||
# reconcilable: fromEntities[ingress] does not cover a controller dialling the ClusterIP.
|
||||
icc_docs = render(
|
||||
{
|
||||
"name": "g",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "g.home.keithjackson.dev", "port": 3000}],
|
||||
"auth": {"mode": "none"},
|
||||
"inClusterClients": [
|
||||
{"name": "flux", "namespace": "flux-system", "port": 3000}
|
||||
],
|
||||
}
|
||||
)
|
||||
icc = [
|
||||
d[1]
|
||||
for d in icc_docs
|
||||
if d[1]["kind"] == "CiliumNetworkPolicy"
|
||||
and d[1]["metadata"]["name"] == "g-allow-flux-ingress"
|
||||
]
|
||||
check(
|
||||
len(icc) == 1,
|
||||
"an inClusterClients entry must render one ingress CNP named "
|
||||
"<service>-allow-<name>-ingress",
|
||||
)
|
||||
if icc:
|
||||
p = icc[0]
|
||||
check(
|
||||
p["metadata"]["namespace"] == "g"
|
||||
and p["spec"]["endpointSelector"]["matchLabels"]
|
||||
== {"app.kubernetes.io/name": "g"},
|
||||
"in-cluster client allow must live in the service's namespace and select the service",
|
||||
)
|
||||
ing = p["spec"]["ingress"]
|
||||
check(
|
||||
ing[0]["fromEndpoints"]
|
||||
== [{"matchLabels": {"k8s:io.kubernetes.pod.namespace": "flux-system"}}],
|
||||
"in-cluster client allow must match the client NAMESPACE only (reserved k8s: label)",
|
||||
)
|
||||
check(
|
||||
"fromEntities" not in ing[0],
|
||||
"in-cluster client allow must not widen to an entity (never fromEntities: all)",
|
||||
)
|
||||
icc_ports = {
|
||||
(pt["port"], pt["protocol"])
|
||||
for r in ing
|
||||
for tp in r["toPorts"]
|
||||
for pt in tp["ports"]
|
||||
}
|
||||
check(
|
||||
icc_ports == {("3000", "TCP")},
|
||||
"in-cluster client allow must be least-privilege (only the declared port/protocol)",
|
||||
)
|
||||
# ADDITIVE: default-deny ingress stays [] and the gateway allow is untouched.
|
||||
icc_dd = next(d[1] for d in icc_docs if d[1]["kind"] == "NetworkPolicy")
|
||||
check(
|
||||
icc_dd["spec"]["ingress"] == [],
|
||||
"in-cluster client allow must be ADDITIVE — default-deny ingress stays [] (R-NET-01)",
|
||||
)
|
||||
check(
|
||||
any(
|
||||
d[1]["metadata"].get("name") == "g-allow-gateway-ingress"
|
||||
for d in icc_docs
|
||||
if d[1]["kind"] == "CiliumNetworkPolicy"
|
||||
),
|
||||
"in-cluster client allow must not displace the north-south gateway allow",
|
||||
)
|
||||
# `name` is OPTIONAL and defaults to the namespace, so the common case is namespace + port.
|
||||
icc_default = render(
|
||||
{
|
||||
"name": "g",
|
||||
"source": {"image": "a:1"},
|
||||
"inClusterClients": [{"namespace": "monitoring", "port": 9090}],
|
||||
}
|
||||
)
|
||||
check(
|
||||
any(
|
||||
d[1]["metadata"].get("name") == "g-allow-monitoring-ingress"
|
||||
for d in icc_default
|
||||
if d[1]["kind"] == "CiliumNetworkPolicy"
|
||||
),
|
||||
"inClusterClients.name must default to the admitted namespace",
|
||||
)
|
||||
# A service that declares none must render NO such policy (grant-nothing-extra).
|
||||
check(
|
||||
not any(
|
||||
d[1]["kind"] == "CiliumNetworkPolicy"
|
||||
and d[1]["metadata"]["name"].endswith("-ingress")
|
||||
and "fromEndpoints" in d[1]["spec"].get("ingress", [{}])[0]
|
||||
for d in render({"name": "q", "source": {"image": "a:1"}})
|
||||
),
|
||||
"a service declaring no inClusterClients must render no in-cluster client allow",
|
||||
)
|
||||
|
||||
# 5) vm kind -> VirtualMachine + standalone DataVolume.
|
||||
vm_docs = render(
|
||||
{
|
||||
@@ -1822,6 +2192,7 @@ def _selftest() -> int:
|
||||
present = {
|
||||
"a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
|
||||
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
|
||||
"svc-db-backup-credentials-prod.sops.yaml",
|
||||
"gitea.yaml",
|
||||
"gitea-actions.yaml", "gitea-actions-runner-secret.sops.yaml",
|
||||
}
|
||||
@@ -1831,8 +2202,10 @@ def _selftest() -> int:
|
||||
flat = {f: wv for wv, fs in plan.items() for f in fs}
|
||||
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
|
||||
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
|
||||
and flat.get("svc-db-backup-credentials.sops.yaml") == 0,
|
||||
"CNPG DB manifest + its owner secret + its Barman backup secret must be in the DB's (lower) wave")
|
||||
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
|
||||
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
|
||||
"CNPG DB manifest + its owner secret + BOTH halves of its per-env Barman backup secret "
|
||||
"must be in the DB's (lower) wave")
|
||||
check(flat.get("svc.yaml") == 1 and flat.get("svc-bootstrap-secret.sops.yaml") == 1,
|
||||
"a DB-backed app + its bootstrap secret must be in the consumer wave, above the DB")
|
||||
check(flat.get("gitea.yaml") == 0, "gitea (leaf) must be wave 0")
|
||||
|
||||
+1
-1
@@ -57,7 +57,7 @@ dependencies:
|
||||
port: 32400
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: seerr.home.keithjackson.dev
|
||||
- host: seerr.${BASE_DOMAIN}
|
||||
port: 5055 # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
|
||||
auth:
|
||||
mode: native # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
|
||||
|
||||
+33
-13
@@ -58,7 +58,7 @@ env:
|
||||
- name: SEMAPHORE_ADMIN_NAME
|
||||
value: Semaphore Admin
|
||||
- name: SEMAPHORE_ADMIN_EMAIL
|
||||
value: admin@home.keithjackson.dev
|
||||
value: admin@${BASE_DOMAIN}
|
||||
# --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
|
||||
# SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
|
||||
- name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
|
||||
@@ -78,16 +78,34 @@ env:
|
||||
# (util ConfigType `oidc_providers`, env-tagged SEMAPHORE_OIDC_PROVIDERS). The value is a single
|
||||
# JSON blob for the `keycloak` provider — provider_url is the homelab realm issuer (auto-discovery),
|
||||
# client_id=semaphore, redirect_url=/api/auth/oidc/keycloak/redirect, claims map to the realm's
|
||||
# token. It CONTAINS the confidential client secret, so the whole blob is held SOPS-encrypted in the
|
||||
# semaphore-bootstrap Secret (key oidc-providers-json; same secret value SOPS-held in
|
||||
# keycloak-realm-clients) and injected here by reference — never in cleartext. The name is NOT a
|
||||
# kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / SEMAPHORE_PORT_*), so no collision.
|
||||
# The LOCAL SEMAPHORE_ADMIN above stays as the break-glass path (docs/15). ---
|
||||
- name: SEMAPHORE_OIDC_PROVIDERS
|
||||
# token. The name is NOT a kubelet service-link var (those are SEMAPHORE_SERVICE_HOST /
|
||||
# SEMAPHORE_PORT_*), so no collision. The LOCAL SEMAPHORE_ADMIN above stays break-glass (docs/15).
|
||||
#
|
||||
# PER-ENVIRONMENT URLS (2026-09-07 fix). The blob USED to live whole inside the SOPS-encrypted
|
||||
# semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
|
||||
# client secret. That hid two environment-specific, NON-SECRET URLs — the realm issuer and the
|
||||
# redirect URI — inside ciphertext, where Flux's ${BASE_DOMAIN} postBuild substitution CANNOT
|
||||
# reach them. Both were frozen at the apex `home.keithjackson.dev`, so PROD (BASE_DOMAIN =
|
||||
# prod.home.keithjackson.dev) pointed its OIDC at DEV's Keycloak — unroutable from prod's
|
||||
# 192.168.2.0/24 (the identical root cause as the Headlamp login timeout; see
|
||||
# platform/headlamp/README.md). Dev was only ever "right" because dev IS the apex.
|
||||
#
|
||||
# THE SPLIT: the URLs are public, so the blob is now a PLAINTEXT literal here and gets
|
||||
# ${BASE_DOMAIN}-substituted per environment like every other hostname. Only the confidential
|
||||
# client secret stays SOPS-encrypted, in its own key (`oidc-client-secret`), and is spliced back
|
||||
# in by the KUBELET's dependent-env-var expansion: an env `value` may reference an EARLIER env
|
||||
# var of the same container as $(NAME), and secretKeyRef-sourced vars are eligible. So
|
||||
# SEMAPHORE_OIDC_CLIENT_SECRET MUST STAY IMMEDIATELY ABOVE — the kubelet resolves env in list
|
||||
# order, and a reference to a not-yet-defined name is left as the LITERAL text $(...). It is
|
||||
# fail-loud in the way that matters: if the Secret key is missing the Pod never starts
|
||||
# (CreateContainerConfigError), rather than silently authenticating against nothing. ---
|
||||
- name: SEMAPHORE_OIDC_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: semaphore-bootstrap
|
||||
key: oidc-providers-json
|
||||
key: oidc-client-secret
|
||||
- name: SEMAPHORE_OIDC_PROVIDERS
|
||||
value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
|
||||
# The semaphoreui/semaphore image runs as the non-root user UID 1001 / GID 0 (tini -> server-wrapper),
|
||||
# and on boot the wrapper writes its generated config.json into the config dir. A mounted Longhorn PVC
|
||||
# is root:root, so UID 1001 cannot write it -> the pod panics ("permission denied"). fsGroup=1001 makes
|
||||
@@ -110,17 +128,19 @@ dependencies:
|
||||
port: 5432 # Renders the egress half; the DB namespace mirrors keycloak-db (no
|
||||
protocol: tcp # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
|
||||
ingressRoutes:
|
||||
- host: semaphore.home.keithjackson.dev
|
||||
- host: semaphore.${BASE_DOMAIN}
|
||||
port: 3000 # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
|
||||
auth:
|
||||
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
|
||||
groups:
|
||||
- platform-admins # access scoped to an authorized group (R-IDP-06)
|
||||
# WIRED (native OIDC): the Keycloak `semaphore` confidential client (platform/keycloak/keycloak-realm-
|
||||
# import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env (JSON blob held encrypted
|
||||
# in the semaphore-bootstrap Secret, oidc-providers-json). The `groups` claim is exposed for
|
||||
# authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the break-glass path so an IdP outage
|
||||
# never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here.
|
||||
# import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env — a PLAINTEXT JSON blob
|
||||
# (see `env` above) whose per-environment URLs are ${BASE_DOMAIN}-substituted, with the confidential
|
||||
# client secret spliced in at runtime from the semaphore-bootstrap Secret key `oidc-client-secret`.
|
||||
# The `groups` claim is exposed for authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the
|
||||
# break-glass path so an IdP outage never locks the operator out (docs/15 "Availability caveat").
|
||||
# No secret is embedded here.
|
||||
update:
|
||||
channel: stable # R-UPD-01
|
||||
range: ">=2.19.0 <3.0.0" # stay on the 2.x stable line
|
||||
|
||||
+84
-7
@@ -9,18 +9,90 @@ name: sonarr
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/sonarr:4.0.15 # pinned stable v4 tag (R-UPD-01); update.range governs advancement
|
||||
exposure: internal # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
|
||||
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
|
||||
# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
|
||||
# which cannot write a 1000:3000-owned 0755 tree.
|
||||
env:
|
||||
- name: PUID
|
||||
value: "1000" # owner uid of every directory under /mnt/Storage/plex
|
||||
- name: PGID
|
||||
value: "3000" # dominant group there; keeps files this writes readable to the
|
||||
# EXISTING non-k3s consumers of the same export, which stay live
|
||||
- name: TZ
|
||||
value: Etc/UTC # same safe default plex uses; the arrs schedule by local time
|
||||
podSecurityContext:
|
||||
# For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
|
||||
# does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
|
||||
# what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
|
||||
# /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
|
||||
fsGroup: 3000
|
||||
exposure: internal # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
|
||||
egress: vpn # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
|
||||
# kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
- name: config
|
||||
size: 5Gi
|
||||
class: longhorn # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
|
||||
# NOTE (bulk-media PREREQUISITE): docs/13 row 2 places the media library / downloads volume on the
|
||||
# `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
|
||||
# but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
|
||||
# volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
|
||||
# therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Sonarr deployable today.
|
||||
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
|
||||
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
|
||||
# /config), so without this the Longhorn PVC mounted at /data/config sat
|
||||
# holding nothing but lost+found while the app wrote its entire
|
||||
# configuration to the container's EPHEMERAL writable layer — destroyed on
|
||||
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
|
||||
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
|
||||
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
|
||||
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
|
||||
# The live /config contents were copied onto each PVC before this landed,
|
||||
# so API keys and credentials other apps reference were preserved.
|
||||
# THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
|
||||
# /media/tv (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
|
||||
#
|
||||
# *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Sonarr imports by HARDLINKING or
|
||||
# atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
|
||||
# and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
|
||||
# /mnt/Storage/plex/tv/ the library
|
||||
# /mnt/Storage/plex/downloads/complete/tv-sonarr/ where finished releases land
|
||||
# — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
|
||||
# them as two subPath volumes (subPath: tv at /media/tv, subPath: downloads elsewhere) creates
|
||||
# TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Sonarr then
|
||||
# silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
|
||||
# seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
|
||||
# is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
|
||||
# requirement, not a shortcut. (The cost is stated plainly: Sonarr can also write /media/movies,
|
||||
# which is radarr's. Narrowing that would cost hardlinks, which is a worse trade.)
|
||||
#
|
||||
# *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
|
||||
# renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
|
||||
# config volume entirely; the pod would come up with its app database shadowed by an 8 T media
|
||||
# tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
|
||||
# the same export, so when qbittorrent is eventually given this mount too, the arrs and the
|
||||
# download client will already agree on absolute paths — which is the other precondition for
|
||||
# hardlinked imports (otherwise every import needs a Remote Path Mapping).
|
||||
#
|
||||
# SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
|
||||
# is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
|
||||
# with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
|
||||
# Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
|
||||
# documents having already been bitten by. Fixing it means starting from an empty config volume, so
|
||||
# it belongs in its own change with the operator present.
|
||||
#
|
||||
# UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
|
||||
# + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
|
||||
# the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
|
||||
# write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
|
||||
# every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
|
||||
#
|
||||
# AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
|
||||
# ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
|
||||
# lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
|
||||
# that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
|
||||
# the image re-chowning /config for itself.
|
||||
- name: media
|
||||
class: bulk
|
||||
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
|
||||
mountPath: /media # => /media/tv (Root Folder) and /media/downloads (import source)
|
||||
size: 9Ti # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
|
||||
# rounded up, so `kubectl get pv` reports something true.
|
||||
dependencies:
|
||||
- to: qbittorrent # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
|
||||
port: 8080
|
||||
@@ -29,12 +101,17 @@ dependencies:
|
||||
port: 9117
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: sonarr.home.keithjackson.dev
|
||||
- host: sonarr.${BASE_DOMAIN}
|
||||
port: 8989 # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
||||
auth:
|
||||
mode: forward-auth # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
|
||||
groups:
|
||||
- media-users # authenticated AND in the media group (R-IDP-06)
|
||||
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
|
||||
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
|
||||
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
|
||||
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
|
||||
- platform-admins # operators of these services must not be locked out of them
|
||||
update:
|
||||
channel: stable # R-UPD-01
|
||||
range: ">=4.0.0 <5.0.0" # stay on the Sonarr v4 stable line
|
||||
|
||||
+80
-3
@@ -27,9 +27,31 @@ make -C ci test # unit tests for these scripts themselves (ci/tests/*.tes
|
||||
```
|
||||
|
||||
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
|
||||
(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
|
||||
per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
|
||||
spec-immutability enforcement). See `conformance/tests/README.md`.
|
||||
(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
|
||||
L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
|
||||
least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
|
||||
|
||||
### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
|
||||
|
||||
R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
|
||||
worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
|
||||
coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
|
||||
selected *by* the requirements a change touches — because **the change most likely to break it
|
||||
attributes no requirement at all**. Requirement-scoped selection then filters the check out of
|
||||
exactly that change.
|
||||
|
||||
This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
|
||||
3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
|
||||
`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
|
||||
failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
|
||||
a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
|
||||
and the runner unions every declaring test into the run. Adding the annotation is the only step.
|
||||
|
||||
`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
|
||||
tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
|
||||
gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
|
||||
by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
|
||||
full rationale and the incidents behind it.
|
||||
|
||||
## Exit-code contract
|
||||
|
||||
@@ -37,6 +59,47 @@ Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIP
|
||||
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
||||
the gate stays honest in Phase 0 (no fake green).
|
||||
|
||||
### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
|
||||
|
||||
**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
|
||||
used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
|
||||
warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
|
||||
`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
|
||||
unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
|
||||
merge-approved. Fail-open in the one component whose whole job is to fail closed.
|
||||
|
||||
| verdict | exit | meaning |
|
||||
|---|---|---|
|
||||
| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
|
||||
| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
|
||||
| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
|
||||
| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
|
||||
|
||||
Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
|
||||
|
||||
**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
|
||||
`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
|
||||
caller that must distinguish them invokes `./ci/gate.sh` directly.
|
||||
|
||||
### Which skips block, and why — a skip is not one thing
|
||||
|
||||
| stage | a skip there means | verdict |
|
||||
|---|---|---|
|
||||
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
|
||||
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
|
||||
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
|
||||
|
||||
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
|
||||
|
||||
`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
|
||||
sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
|
||||
back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
|
||||
cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
|
||||
strictness.
|
||||
|
||||
Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
|
||||
stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
|
||||
|
||||
### The touched-requirement subset can also be *undeterminable* (exit 3)
|
||||
|
||||
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
|
||||
@@ -87,6 +150,8 @@ ci/
|
||||
lint.sh Stage: yaml/lint
|
||||
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
|
||||
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
||||
(harness: platform/monitoring/tests/). Catches the two alerting bugs no
|
||||
other stage can see — permanently-firing and structurally INERT rules.
|
||||
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
|
||||
exits 3 LOUDLY rather than guessing when the diff base is unreliable
|
||||
@@ -122,3 +187,15 @@ its owner delivers a runnable entrypoint.
|
||||
`CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
|
||||
be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
|
||||
turns any unverified cluster-needing touched requirement into a hard block.
|
||||
|
||||
All four switches are read through one `truthy()` in `lib.sh`, so they spell "on" identically
|
||||
(`true|1|yes|y|on`, any case) and — the direction that matters — **anything else is OFF**: a
|
||||
misspelled `CLUSTER_AVAILABLE=ture` leaves the capability undeclared instead of silently claiming
|
||||
it, which can only make the gate stricter about what it says it verified.
|
||||
|
||||
| switch | default | effect |
|
||||
|---|---|---|
|
||||
| `CLUSTER_AVAILABLE` | `false` | declares a cluster; L2/L3 then run and are required |
|
||||
| `REQUIRE_CLUSTER_TESTS` | `false` | Phase 1+: an unverified cluster-needing touched requirement BLOCKS |
|
||||
| `REQUIRE_SPEC_VERIFICATION` | `false` | an unverifiable spec repo BLOCKS instead of UNDETERMINED |
|
||||
| `GATE_STRICT_SKIPS` | `auto` | `true` = a missing tool / unwired seam BLOCKS (CI); `auto` follows `GITHUB_ACTIONS`/`CI` |
|
||||
+5
-1
@@ -47,7 +47,11 @@ fi
|
||||
# (conformance/lib/common.sh) now unions a comma/space list of layers, so this is a single
|
||||
# invocation with one aggregated report (no per-layer looping needed). Artifacts are then
|
||||
# COLLECTED from the fixed location below.
|
||||
"$CONFORMANCE_CMD" --layer l2,l3 "${req_args[@]}"
|
||||
# --include-scope: union the touched subset with every L2/L3 test DECLARING
|
||||
# `conformance.home.k3s/scope: whole-repo` (none today — wired once, here, so a future
|
||||
# whole-repo behavioural invariant needs only the annotation; see conformance/lib/common.sh).
|
||||
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
||||
"$CONFORMANCE_CMD" --layer l2,l3 --include-scope "$WHOLE_REPO_SCOPE" "${req_args[@]}"
|
||||
|
||||
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
||||
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
||||
|
||||
+163
-10
@@ -7,6 +7,75 @@
|
||||
# skip (3) not run (cluster-gated stage w/ no cluster, or unwired seam) —
|
||||
# NOT counted as green (HARD RULE 4: don't fake green)
|
||||
#
|
||||
# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that
|
||||
# SKIPPED only printed a warning ("— not green") and the gate then printed
|
||||
# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose
|
||||
# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently
|
||||
# becoming unwired, produced exit 0 and read as merge-approved. Observed live:
|
||||
# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and
|
||||
# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already
|
||||
# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly
|
||||
# this shape of non-answer — the offline stages just did not use it.
|
||||
#
|
||||
# MERGE GATE: PASS exit 0 every stage that was SUPPOSED TO RUN ran and was
|
||||
# green, and the R-TEST-03 subset was checked.
|
||||
# MERGE GATE: BLOCKED exit 1 a stage FAILED, or a stage did not run WHERE IT
|
||||
# MUST (see the per-stage rules below), or the
|
||||
# decision input is unusable.
|
||||
# MERGE GATE: UNDETERMINED exit 3 nothing failed, but a stage did not run, so
|
||||
# greenness was NOT ESTABLISHED. Never a pass;
|
||||
# non-zero so no automation can read it as approval.
|
||||
# `3` is this repo's existing "not run / not
|
||||
# established" code (ci/lib.sh), now spoken by the
|
||||
# gate as a whole and not only by its stages.
|
||||
# MERGE GATE: NOT APPLICABLE exit 0 there is no branch under gate at all (HEAD is on
|
||||
# base) — a push-to-main CI run. Stages were green;
|
||||
# no subset was checked, and the headline says so.
|
||||
#
|
||||
# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so
|
||||
# `make -C ci gate` collapses 1 and 3 into 2 — a human sees the headline and that is enough.
|
||||
# A caller that must tell BLOCKED from UNDETERMINED invokes ./ci/gate.sh directly; to tolerate
|
||||
# an UNDETERMINED run it writes `./ci/gate.sh || [ $? -eq 3 ]`, the same explicit idiom
|
||||
# .gitea/workflows/gate.yml already used for its spec-immutability pre-step and now uses for
|
||||
# the gate itself (with GATE_STRICT_SKIPS=true, so that tolerance can ONLY ever cover the
|
||||
# declared-environmental skip below — a missing tool on the runner is exit 1, untolerated).
|
||||
# TOLERATING A NON-ANSWER IS A VISIBLE, REVIEWED CHOICE IN THE CALLER. There is deliberately no
|
||||
# knob inside this script that turns UNDETERMINED into exit 0: that would be the fail-open this
|
||||
# change exists to remove, just relocated.
|
||||
#
|
||||
# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
|
||||
#
|
||||
# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired
|
||||
# seam went unwired. On a CI runner the pinned toolchain is installed for us by
|
||||
# .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
|
||||
# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
|
||||
# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
|
||||
# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
|
||||
#
|
||||
# spec-immutability (R-TEST-06) A skip means the sibling spec repo is not checked out, so
|
||||
# the spec COULD NOT BE VERIFIED. Never a pass — unverified is unverified — so it denies a
|
||||
# PASS and yields UNDETERMINED. It is deliberately NOT upgraded to BLOCKED by CI strictness,
|
||||
# because on the live runner it is not a defect anyone can fix from inside this repo: the
|
||||
# workflow does not check the spec repo out at all (SPEC_REPO_SLUG is unwired — the blueprint
|
||||
# lives in merge-gate.yaml), and blocking would leave the only enforced status check
|
||||
# permanently red for a reason no PR could clear. So THIS is the one skip whose non-answer the
|
||||
# CALLER may knowingly tolerate, and gate.yml does so in one visible line. Locally the
|
||||
# remedy exists and the message names it: check the spec repo out beside this one, or point
|
||||
# SPEC_REPO at it (infra/devcontainer/dev.sh mounts it automatically when it is a sibling).
|
||||
# Set REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a
|
||||
# hard block once that checkout is wired.
|
||||
#
|
||||
# conformance (L2/L3) A skip when CLUSTER_AVAILABLE!=true is NOT a defect and
|
||||
# NOT an accident: it is the environment DECLARING that the capability is absent, which
|
||||
# is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed
|
||||
# to run", so it does not deny a PASS — the touched requirements it would have covered
|
||||
# are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that
|
||||
# into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as
|
||||
# it already did. That asymmetry is the whole distinction this file now draws:
|
||||
# DECLARED-not-applicable may pass; COULD-NOT-RUN may not. A declared gap is still NAMED in
|
||||
# the PASS headline (see DECLARED_GAPS below), so a PASS never reads as "everything was
|
||||
# verified" when a layer was declared out of scope.
|
||||
#
|
||||
# Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
|
||||
# MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
|
||||
# available, the gate reports those requirements UNVERIFIED and — unless
|
||||
@@ -20,6 +89,7 @@ source "${CI_DIR}/lib.sh"
|
||||
set +e # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
|
||||
|
||||
: "${REQUIRE_CLUSTER_TESTS:=false}"
|
||||
: "${REQUIRE_SPEC_VERIFICATION:=false}"
|
||||
|
||||
declare -A OUTCOME
|
||||
run_stage() {
|
||||
@@ -47,6 +117,13 @@ run_stage conformance "${CI_DIR}/conformance.sh"
|
||||
echo >&2
|
||||
log "==== gate decision ===="
|
||||
BLOCK=0
|
||||
# Stages that did NOT run and are tolerated here — but never as green. A non-empty list
|
||||
# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3.
|
||||
NOT_ESTABLISHED=()
|
||||
# Stages the environment DECLARED inapplicable (today: L2/L3 with no cluster declared). These
|
||||
# do not deny a PASS — nothing was supposed to run — but they are named IN the PASS headline, so
|
||||
# "PASS" is never mistaken for "everything was verified".
|
||||
DECLARED_GAPS=()
|
||||
|
||||
# --- R-TEST-03 input: the touched-requirement subset --------------------------
|
||||
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
|
||||
@@ -64,37 +141,80 @@ rm -f "$_reason_file"
|
||||
TOUCHED=()
|
||||
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
|
||||
UNDECIDED=0
|
||||
SUBSET_BROKEN=0
|
||||
if [ "$TOUCHED_RC" -eq 3 ]; then
|
||||
UNDECIDED=1
|
||||
echo >&2
|
||||
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
|
||||
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
|
||||
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
|
||||
elif [ "$TOUCHED_RC" -ne 0 ]; then
|
||||
# Any OTHER non-zero exit is a hard defect in the subset computation itself — an
|
||||
# incomplete path->requirement map (a changed path matches no row) or an invalid row
|
||||
# would report exactly this. The reasoning is taken from the unlanded
|
||||
# fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so
|
||||
# any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below
|
||||
# printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on
|
||||
# `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is
|
||||
# already closed the moment a fail-closed path map makes exit 1 reachable. A subset that
|
||||
# cannot be computed is NOT an empty subset.
|
||||
SUBSET_BROKEN=1
|
||||
echo >&2
|
||||
err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}"
|
||||
err " (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix."
|
||||
summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied."
|
||||
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED[*]}"
|
||||
fi
|
||||
|
||||
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
|
||||
# A skip means the spec repo wasn't available to verify — surfaced, not green.
|
||||
# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it
|
||||
# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the
|
||||
# file header for why this one is not upgraded to a block by CI detection alone.
|
||||
case "${OUTCOME[spec-immutability]}" in
|
||||
fail) err "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
|
||||
skip) warn "spec-immutability: skipped (spec repo unavailable) — not verified" ;;
|
||||
skip)
|
||||
if truthy "$REQUIRE_SPEC_VERIFICATION"; then
|
||||
err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge"
|
||||
summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)"
|
||||
BLOCK=1
|
||||
else
|
||||
warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so"
|
||||
warn " this run cannot be a PASS. Check the spec repo out beside this one (or set"
|
||||
warn " SPEC_REPO) to verify it."
|
||||
summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)"
|
||||
NOT_ESTABLISHED+=("spec-immutability")
|
||||
fi ;;
|
||||
pass) ok "spec-immutability: green" ;;
|
||||
esac
|
||||
|
||||
# L0/L1 stages: any failure blocks. A SKIP here means a seam is unwired — in
|
||||
# Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
|
||||
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
|
||||
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
|
||||
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
|
||||
# no-cluster reproducibility gate (canonical render == committed apps/ goldens):
|
||||
# a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
|
||||
for s in lint l0 render-golden l1; do
|
||||
case "${OUTCOME[$s]}" in
|
||||
fail) err "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
|
||||
skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;
|
||||
skip)
|
||||
if strict_skips; then
|
||||
err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
|
||||
err " toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
|
||||
err " seam is on main — so this is a DEFECT, not an environment -> blocks merge"
|
||||
summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
|
||||
BLOCK=1
|
||||
else
|
||||
warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS"
|
||||
summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green"
|
||||
NOT_ESTABLISHED+=("$s")
|
||||
fi ;;
|
||||
pass) ok "${s}: green" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# L2/L3 conformance: cluster-gated.
|
||||
# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS,
|
||||
# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather
|
||||
# than the stage failing to run where it could have — see the file header.
|
||||
case "${OUTCOME[conformance]}" in
|
||||
fail)
|
||||
err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
|
||||
@@ -105,18 +225,22 @@ case "${OUTCOME[conformance]}" in
|
||||
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
|
||||
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
|
||||
else
|
||||
if [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
reqs=("${TOUCHED[@]}")
|
||||
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
|
||||
warn " (no cluster — Phase 0). These are NOT certified green."
|
||||
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
|
||||
if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then
|
||||
if truthy "$REQUIRE_CLUSTER_TESTS"; then
|
||||
err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
|
||||
BLOCK=1
|
||||
else
|
||||
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared; ${#TOUCHED[@]} touched reqs unverified)")
|
||||
fi
|
||||
else
|
||||
skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
|
||||
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
|
||||
fi
|
||||
fi ;;
|
||||
esac
|
||||
@@ -132,13 +256,34 @@ if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
|
||||
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
||||
BLOCK=1
|
||||
fi
|
||||
# An incomplete/invalid path map is not an environment problem to skip past: the change under
|
||||
# gate contains paths whose requirements are UNKNOWN, so no subset can be certified.
|
||||
if [ "$SUBSET_BROKEN" -eq 1 ]; then
|
||||
err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
||||
BLOCK=1
|
||||
fi
|
||||
|
||||
echo >&2
|
||||
# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so
|
||||
# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED
|
||||
# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two
|
||||
# reasons this is not a green run, and the stronger, non-zero one must be the headline.
|
||||
if [ "$BLOCK" -ne 0 ]; then
|
||||
err "MERGE GATE: BLOCKED"
|
||||
summary ""
|
||||
summary "**Result: BLOCKED** — see failures above."
|
||||
exit 1
|
||||
elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then
|
||||
err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED."
|
||||
err " Stages that did not run: ${NOT_ESTABLISHED[*]}"
|
||||
err " Nothing FAILED, and nothing is certified either: a stage that did not run is not"
|
||||
err " green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as"
|
||||
err " merge-approved. Fix the environment (install the pinned tool / wire the seam /"
|
||||
err " check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:"
|
||||
err " ./ci/gate.sh || [ \$? -eq 3 ]"
|
||||
summary ""
|
||||
summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)."
|
||||
exit 3
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
|
||||
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
|
||||
@@ -148,8 +293,16 @@ elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
|
||||
exit 0
|
||||
else
|
||||
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
|
||||
summary ""
|
||||
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
|
||||
ok "MERGE GATE: PASS — every stage that was supposed to run RAN and was GREEN, and the"
|
||||
ok " R-TEST-03 touched-requirement subset was checked."
|
||||
if [ "${#DECLARED_GAPS[@]}" -gt 0 ]; then
|
||||
ok " NOT verified here, because the environment DECLARED it out of scope (never a silent"
|
||||
ok " gap — Phase 0 by policy, see the header): ${DECLARED_GAPS[*]}"
|
||||
summary ""
|
||||
summary "**Result: PASS** — every stage that was supposed to run ran green. Declared out of scope (NOT verified): \`${DECLARED_GAPS[*]}\`."
|
||||
else
|
||||
summary ""
|
||||
summary "**Result: PASS** — every stage that was supposed to run ran green; nothing was declared out of scope."
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
@@ -10,6 +10,18 @@
|
||||
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
|
||||
# a per-resource policy cannot express (secret hygiene across every
|
||||
# tracked file, least-privilege RBAC, spec-immutability enforcement).
|
||||
# Selection = the TOUCHED requirement subset (R-TEST-03) UNION every L0
|
||||
# test DECLARING `conformance.home.k3s/scope: whole-repo`. That union
|
||||
# replaces what used to be a hand-written "UNCONDITIONAL" stage per
|
||||
# whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e
|
||||
# bulk-tier): three near-identical 12-line rationales, three hardcoded
|
||||
# check.sh paths and three copies of the same rc 0/3/* case block, which
|
||||
# did not scale and — worse — failed SILENTLY, because whoever added the
|
||||
# next whole-repo invariant had to REMEMBER to hand-write a stage. Now the
|
||||
# test declares its own scope and the runner enforces it. The reasoning and
|
||||
# the incidents those stages recorded were moved INTO the tests they
|
||||
# describe (see each chainsaw-test.yaml's `scope` annotation) and the
|
||||
# general mechanism is documented in conformance/lib/common.sh.
|
||||
#
|
||||
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
|
||||
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
|
||||
@@ -83,58 +95,45 @@ fi
|
||||
# binding, the spec-immutability gate's own behaviour) live in the conformance
|
||||
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
|
||||
# through the same single-command runner CI and a laptop both use (R-TEST-02).
|
||||
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
|
||||
#
|
||||
# THE UNION, AND WHY IT IS NOT OPTIONAL (R-TEST-03's blind spot)
|
||||
# --------------------------------------------------------------
|
||||
# 3b scopes the conformance run to the TOUCHED requirements. That is right for an invariant
|
||||
# whose inputs live under the subtree its requirements map to. It is WRONG for one that
|
||||
# quantifies over the WHOLE tracked tree, or over subtrees (apps/, infra/, most of platform/)
|
||||
# that ci/touched-requirements.sh's coarse path map keys to NOTHING: for those, the change
|
||||
# most likely to break the invariant attributes no requirement at all, so requirement-scoped
|
||||
# selection filters the check out of EXACTLY that change. `--include-scope whole-repo` unions
|
||||
# in every L0 test that DECLARES that scope, so those run on every gate invocation.
|
||||
#
|
||||
# rc 3 IS NOT A PASS. The runner returns 3 when a scope-declared check SKIPPED (its subject
|
||||
# subtree is absent, or it has no check script): UNWIRED, not green, exactly as the deleted
|
||||
# per-check stages had it ("check SKIPPED (clusters/ absent) — not a pass"). It is mapped to
|
||||
# the gate's own rc 3 below, never to rc 0.
|
||||
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
||||
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0 --include-scope ${WHOLE_REPO_SCOPE})"
|
||||
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
|
||||
summary "- l0-static: **SKIPPED** (conformance runner missing)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
else
|
||||
load_touched_requirements
|
||||
if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
|
||||
ok "L0 static conformance checks passed"
|
||||
summary "- l0-static: **passed**"
|
||||
else
|
||||
err "L0 static conformance checks failed"
|
||||
summary "- l0-static: **FAILED**"
|
||||
rc=1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
|
||||
# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
|
||||
# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
|
||||
# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
|
||||
# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
|
||||
# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
|
||||
# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
|
||||
# a legitimate manifest, and that no expansion silently empties today.
|
||||
#
|
||||
# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
|
||||
# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
|
||||
# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
|
||||
# attributes no requirement, so the requirement-scoped selection would filter this check
|
||||
# out of exactly the change that introduces the bug. The invariant is whole-repo (same
|
||||
# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
|
||||
# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
|
||||
# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
|
||||
SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
|
||||
log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
|
||||
if [ ! -f "$SUBST_CHECK" ]; then
|
||||
skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
|
||||
summary "- subst-safety: **SKIPPED** (check missing)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
else
|
||||
subst_rc=0
|
||||
bash "$SUBST_CHECK" || subst_rc=$?
|
||||
case "$subst_rc" in
|
||||
0) ok "Flux postBuild substitution safety passed"
|
||||
summary "- subst-safety: **passed**" ;;
|
||||
3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
|
||||
summary "- subst-safety: **SKIPPED** (nothing to audit)"
|
||||
l0_static_rc=0
|
||||
"$CONFORMANCE_CMD" --layer l0 --include-scope "$WHOLE_REPO_SCOPE" \
|
||||
${REQ_ARGS[@]+"${REQ_ARGS[@]}"} || l0_static_rc=$?
|
||||
case "$l0_static_rc" in
|
||||
0) ok "L0 static conformance checks passed"
|
||||
summary "- l0-static: **passed**" ;;
|
||||
3) skip "an L0 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
|
||||
summary "- l0-static: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
|
||||
*) err "Flux postBuild substitution safety FAILED"
|
||||
summary "- subst-safety: **FAILED**"; rc=1 ;;
|
||||
*) err "L0 static conformance checks failed"
|
||||
summary "- l0-static: **FAILED**"
|
||||
rc=1 ;;
|
||||
esac
|
||||
# Per-check step-summary lines, so the gate output still NAMES what ran — the generated
|
||||
# replacement for the one hardcoded `summary` line each deleted stage carried.
|
||||
emit_offline_summary
|
||||
fi
|
||||
|
||||
case "$rc" in
|
||||
|
||||
@@ -29,7 +29,16 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
||||
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
|
||||
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
||||
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
||||
"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
|
||||
# --include-scope: union the touched subset with every L1 test DECLARING
|
||||
# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
|
||||
# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
|
||||
# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
|
||||
# "someone must remember" is the failure mode this whole change removes; see
|
||||
# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
|
||||
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
||||
l1_rc=0
|
||||
"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
|
||||
${req_args[@]+"${req_args[@]}"} || l1_rc=$?
|
||||
|
||||
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
||||
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
||||
@@ -39,5 +48,16 @@ if [ -d "${CONF_ART}/coverage" ]; then
|
||||
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
|
||||
fi
|
||||
|
||||
ok "L1 rendering checks passed"
|
||||
summary "- l1-render: **passed**"
|
||||
case "$l1_rc" in
|
||||
0) ok "L1 rendering checks passed"
|
||||
summary "- l1-render: **passed**"
|
||||
emit_offline_summary ;;
|
||||
3) skip "an L1 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
|
||||
summary "- l1-render: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
|
||||
emit_offline_summary
|
||||
exit 3 ;;
|
||||
*) err "L1 rendering checks FAILED (rc=${l1_rc})"
|
||||
summary "- l1-render: **FAILED**"
|
||||
emit_offline_summary
|
||||
exit "$l1_rc" ;;
|
||||
esac
|
||||
@@ -72,20 +72,83 @@ entrypoint_ready() {
|
||||
return 1
|
||||
}
|
||||
|
||||
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
|
||||
# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a
|
||||
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
|
||||
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
|
||||
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
|
||||
cluster_available() {
|
||||
case "${CLUSTER_AVAILABLE:-false}" in
|
||||
true|1|yes) return 0 ;;
|
||||
# truthy <value> — the ONE place the repo's "is this switch on" spelling lives, so
|
||||
# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS
|
||||
# all agree. Anything not listed is OFF, and that direction matters: a misspelled value
|
||||
# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than
|
||||
# silently declaring a capability "available", so a typo can only make the gate stricter about
|
||||
# what it claims to have verified, never looser.
|
||||
truthy() {
|
||||
case "${1:-}" in
|
||||
true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
|
||||
# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a
|
||||
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
|
||||
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
|
||||
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
|
||||
# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip —
|
||||
# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh.
|
||||
cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
|
||||
|
||||
# --- Skip strictness: is this a CI runner, or a workstation? -------------------
|
||||
# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
|
||||
#
|
||||
# CI the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
|
||||
# before the gate runs, and every seam is on `main`. A stage that skips here is a
|
||||
# DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
|
||||
# local a missing pinned tool is a real, ordinary local condition -> the gate reports
|
||||
# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
|
||||
#
|
||||
# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback:
|
||||
#
|
||||
# 1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on
|
||||
# the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE
|
||||
# WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no
|
||||
# env-var sniffing can get it wrong.
|
||||
# 2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS /
|
||||
# CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do
|
||||
# NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks
|
||||
# like CI" is exactly the unreliable inference this gate must not contain.
|
||||
#
|
||||
# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI
|
||||
# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still
|
||||
# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly
|
||||
# and only for the one declared-environmental skip (see its comment). A workstation misread as
|
||||
# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true
|
||||
# locally to rehearse CI strictness.
|
||||
: "${GATE_STRICT_SKIPS:=auto}"
|
||||
strict_skips() {
|
||||
case "$GATE_STRICT_SKIPS" in
|
||||
auto) truthy "${GITHUB_ACTIONS:-}" || truthy "${CI:-}" ;;
|
||||
*) truthy "$GATE_STRICT_SKIPS" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
|
||||
|
||||
# --- Offline (L0/L1) per-check step summary -----------------------------------
|
||||
# conformance/lib/run_l1_offline.py writes one Markdown line per check it executed to
|
||||
# conformance/_artifacts/offline-summary.md. Splice those into $GITHUB_STEP_SUMMARY so the
|
||||
# gate's output still NAMES every check that ran and its verdict.
|
||||
#
|
||||
# WHY THIS EXISTS: ci/l0.sh used to carry a hand-written stage per whole-repo check, each with
|
||||
# its own `summary "- subst-safety: **passed**"` line. Those stages are gone (the tests declare
|
||||
# `conformance.home.k3s/scope: whole-repo` instead and the runner unions them in), so the
|
||||
# naming is generated for EVERY offline check rather than for the three that had a stage.
|
||||
: "${OFFLINE_SUMMARY_FILE:=${CONFORMANCE_DIR}/_artifacts/offline-summary.md}"
|
||||
emit_offline_summary() {
|
||||
local f="${1:-$OFFLINE_SUMMARY_FILE}" line
|
||||
[ -f "$f" ] || return 0
|
||||
while IFS= read -r line; do
|
||||
[ -n "$line" ] && summary " $line"
|
||||
done < "$f"
|
||||
return 0
|
||||
}
|
||||
|
||||
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
|
||||
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
|
||||
# (ready to splice into a conformance/run.sh invocation).
|
||||
@@ -111,7 +174,16 @@ load_touched_requirements() {
|
||||
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
|
||||
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
|
||||
# Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a
|
||||
# path->requirement map that cannot classify a changed path, once that map fails closed).
|
||||
# The FULL layer still runs here as a superset — nothing goes untested — but this stage must
|
||||
# not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED
|
||||
# merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed
|
||||
# (47068ee), whose gate-side half this branch lands.
|
||||
warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)."
|
||||
warn " Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen."
|
||||
warn " ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in"
|
||||
warn " ci/touched-requirements.sh so every changed path is classified."
|
||||
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED_REQS[*]}"
|
||||
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
|
||||
|
||||
Executable
+214
@@ -0,0 +1,214 @@
|
||||
#!/usr/bin/env python3
|
||||
# =============================================================================
|
||||
# ci/render-keycloak-client-urls.py
|
||||
#
|
||||
# Render platform/keycloak/client-urls.yaml FROM platform/keycloak/keycloak-realm-import.yaml.
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
# ---------------
|
||||
# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
|
||||
# Done=True / Started=False it never re-runs, so a redirect URI changed in git is
|
||||
# never applied to the realm. The client-secret half of that failure class is
|
||||
# already healed by a level-triggered CronJob (platform/keycloak/client-secret-
|
||||
# reconciler.yaml). This renderer supplies the DESIRED STATE that the same CronJob
|
||||
# needs to heal the URL half.
|
||||
#
|
||||
# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
|
||||
# automountServiceAccountToken: false. It has NO Kubernetes API access at all, by
|
||||
# design (R-SEC-02) — so it cannot read the KeycloakRealmImport CR to learn what
|
||||
# the realm is supposed to look like. Its inputs must be MOUNTED. Hence a
|
||||
# ConfigMap.
|
||||
#
|
||||
# A hand-authored ConfigMap would be a SECOND SOURCE OF TRUTH for every public
|
||||
# hostname in the realm, free to disagree with the realm import — and "two
|
||||
# declarations that can disagree" is the same shape as the bug this whole effort
|
||||
# is closing. So the ConfigMap is GENERATED from the realm import and the
|
||||
# generation is GATED: conformance/tests/l0-static/keycloak-client-secret-
|
||||
# reconciler/check.sh runs this script with --check on every merge, and the merge
|
||||
# fails if the committed file is not what this renderer produces. That is the same
|
||||
# render==golden idiom catalog/render.py + ci/render-golden.sh already use for
|
||||
# apps/.
|
||||
#
|
||||
# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. If a URL in the realm import is
|
||||
# written as a Flux substitution token (https://grafana.${BASE_DOMAIN}/...), the
|
||||
# token is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files
|
||||
# are rendered by the SAME Flux Kustomization (./platform-config, whose
|
||||
# postBuild.substituteFrom reads cluster-vars), so both are substituted in the
|
||||
# same pass with the same values and cannot disagree about what BASE_DOMAIN is.
|
||||
# This renderer never parses, splits, or reconstructs a hostname, and never reads
|
||||
# cluster-vars — so there is no second place for the domain to be decided.
|
||||
#
|
||||
# USAGE
|
||||
# python3 ci/render-keycloak-client-urls.py # print the rendered file
|
||||
# python3 ci/render-keycloak-client-urls.py --write # write it (regenerate)
|
||||
# python3 ci/render-keycloak-client-urls.py --check # exit 1 if it has drifted
|
||||
# =============================================================================
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import difflib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
|
||||
OUT_FILE = "platform/keycloak/client-urls.yaml"
|
||||
DATA_KEY = "client-urls.json"
|
||||
CONFIGMAP_NAME = "keycloak-client-urls"
|
||||
NAMESPACE = "keycloak"
|
||||
|
||||
# The client fields this renderer exports, and therefore the ONLY fields the
|
||||
# reconciler may write. Every one of them is an environment-derived URL: it changes
|
||||
# when BASE_DOMAIN changes, and a stale value breaks OIDC login (a wrong
|
||||
# redirectUri) or silently widens trust (a stale wildcard left behind).
|
||||
#
|
||||
# DELIBERATELY ABSENT: `secret` (healed by the secret pass from the SOPS Secret,
|
||||
# and it must never appear in a ConfigMap), and every non-URL field —
|
||||
# protocolMappers, defaultClientScopes, attributes, flags. Those are not
|
||||
# environment-derived, so nothing re-renders them on a domain flip; keeping them
|
||||
# out keeps the reconciler's write surface as narrow as the problem.
|
||||
LIST_FIELDS = ("redirectUris", "webOrigins")
|
||||
STR_FIELDS = ("rootUrl", "baseUrl", "adminUrl")
|
||||
|
||||
|
||||
def render(root: Path) -> str:
|
||||
"""Return the full text of client-urls.yaml as it should be committed."""
|
||||
docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
|
||||
imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
|
||||
if imp is None:
|
||||
raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
|
||||
clients = (imp.get("spec") or {}).get("realm", {}).get("clients") or []
|
||||
if not clients:
|
||||
# Fail closed. Emitting an empty desired state would make the reconciler
|
||||
# believe every client is unmanaged and silently stop reconciling URLs.
|
||||
raise SystemExit("FATAL: %s declares zero clients — refusing to render an "
|
||||
"empty desired state" % IMPORT_FILE)
|
||||
|
||||
want: dict[str, dict] = {}
|
||||
for c in clients:
|
||||
cid = c.get("clientId")
|
||||
if not isinstance(cid, str) or not cid:
|
||||
raise SystemExit("FATAL: %s has a client with no clientId" % IMPORT_FILE)
|
||||
entry: dict[str, object] = {}
|
||||
for f in LIST_FIELDS:
|
||||
if f not in c:
|
||||
continue # not declared => NOT MANAGED (never forced empty)
|
||||
v = c[f]
|
||||
if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
|
||||
raise SystemExit("FATAL: %s client %r field %s is not a list of strings"
|
||||
% (IMPORT_FILE, cid, f))
|
||||
entry[f] = list(v) # verbatim, order as authored
|
||||
for f in STR_FIELDS:
|
||||
if f not in c:
|
||||
continue
|
||||
v = c[f]
|
||||
if not isinstance(v, str):
|
||||
raise SystemExit("FATAL: %s client %r field %s is not a string"
|
||||
% (IMPORT_FILE, cid, f))
|
||||
entry[f] = v
|
||||
if entry:
|
||||
want[cid] = entry
|
||||
if not want:
|
||||
raise SystemExit("FATAL: no client in %s declares any of %s — refusing to render "
|
||||
"an empty desired state"
|
||||
% (IMPORT_FILE, ", ".join(LIST_FIELDS + STR_FIELDS)))
|
||||
|
||||
body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
|
||||
indented = "\n".join((" " + ln) if ln else "" for ln in body.splitlines())
|
||||
return HEADER + indented + "\n"
|
||||
|
||||
|
||||
HEADER = """---
|
||||
# =============================================================================
|
||||
# GENERATED FILE — DO NOT EDIT BY HAND.
|
||||
#
|
||||
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm.clients)
|
||||
# regenerate: python3 ci/render-keycloak-client-urls.py --write
|
||||
# gated by: conformance/tests/l0-static/keycloak-client-secret-reconciler
|
||||
# (runs this renderer with --check on every merge)
|
||||
#
|
||||
# WHAT THIS IS: the DESIRED STATE for every environment-derived client URL in the
|
||||
# `homelab` realm — redirectUris, webOrigins, rootUrl, baseUrl, adminUrl. The
|
||||
# client-secret reconciler CronJob mounts it and makes the live realm match it.
|
||||
#
|
||||
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
|
||||
# once Done=True, so a redirect URI edited in git never reaches the realm. Prod's
|
||||
# stored redirect URIs were measured still pointing at the pre-migration hostname
|
||||
# while git declared the new one, and every OIDC login would have failed with a
|
||||
# redirect_uri mismatch. The reconciler is the level-triggered loop that closes
|
||||
# that; this file is what it reconciles TOWARDS.
|
||||
#
|
||||
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of every public
|
||||
# hostname would be a second source of truth, free to disagree with the realm
|
||||
# import. It is rendered from the import and the render is gated, so the two
|
||||
# cannot drift. URL strings are copied VERBATIM — if the source uses a
|
||||
# ${BASE_DOMAIN} token it appears here unchanged, and both files are substituted
|
||||
# by the SAME Flux Kustomization (./platform-config), so the domain is decided in
|
||||
# exactly one place.
|
||||
#
|
||||
# A field that is NOT declared on a client in the realm import is simply ABSENT
|
||||
# here, and the reconciler leaves it alone. Absence means UNMANAGED, never
|
||||
# "force to empty".
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: %s
|
||||
namespace: %s
|
||||
labels:
|
||||
app.kubernetes.io/name: %s
|
||||
app.kubernetes.io/part-of: keycloak
|
||||
data:
|
||||
%s: |
|
||||
""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME, DATA_KEY)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
g = ap.add_mutually_exclusive_group()
|
||||
g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
|
||||
g.add_argument("--check", action="store_true",
|
||||
help="exit 1 if the committed file has drifted from the source")
|
||||
args = ap.parse_args()
|
||||
|
||||
want = render(ROOT)
|
||||
target = ROOT / OUT_FILE
|
||||
|
||||
if args.write:
|
||||
target.write_text(want, encoding="utf-8")
|
||||
print("wrote %s" % OUT_FILE)
|
||||
return 0
|
||||
|
||||
if args.check:
|
||||
if not target.is_file():
|
||||
print("CHECK FAIL: %s does not exist. Run: python3 "
|
||||
"ci/render-keycloak-client-urls.py --write" % OUT_FILE, file=sys.stderr)
|
||||
return 1
|
||||
have = target.read_text(encoding="utf-8")
|
||||
if have == want:
|
||||
print("client-urls.yaml is in sync with %s" % IMPORT_FILE)
|
||||
return 0
|
||||
print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
|
||||
"realm on a desired state the realm import does not declare.\n"
|
||||
"Regenerate with: python3 ci/render-keycloak-client-urls.py --write\n"
|
||||
% (OUT_FILE, IMPORT_FILE), file=sys.stderr)
|
||||
for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
|
||||
fromfile="committed " + OUT_FILE,
|
||||
tofile="rendered from " + IMPORT_FILE):
|
||||
sys.stderr.write(line)
|
||||
return 1
|
||||
|
||||
sys.stdout.write(want)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Executable
+189
@@ -0,0 +1,189 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/gate-decision.test.sh — the merge gate's OWN decision table.
|
||||
#
|
||||
# WHY THIS SUITE EXISTS (2026-09-13). ci/gate.sh used to set BLOCK=1 for a FAILED stage but
|
||||
# only `warn` for a SKIPPED one, so it printed the words "— not green" and then concluded
|
||||
# `MERGE GATE: PASS` and exited 0. Observed live twice: `prom-rules` skipped for a missing
|
||||
# promtool, and `l0` skipped for a missing pinned kyverno CLI — both read by automation as
|
||||
# merge-approved. That is fail-open in the one component whose entire job is to fail closed,
|
||||
# and nothing tested it: every other ci/tests suite covers the INPUT (touched-requirements),
|
||||
# none covered the DECISION.
|
||||
#
|
||||
# WHAT IS UNDER TEST: the real ci/gate.sh and ci/lib.sh, copied into a throwaway CI_DIR beside
|
||||
# STUB stage scripts whose exit codes this suite dictates (STAGE_RC_<stage>) and a stub
|
||||
# touched-requirements.sh whose rc/reason it dictates (TR_RC / TR_REASON / TR_OUT). Stubbing the
|
||||
# stages is the point: this asserts how the gate REASONS about pass/fail/skip, independently of
|
||||
# what any stage happens to do on this machine — so it runs anywhere, with no tools, no cluster,
|
||||
# no network, in milliseconds.
|
||||
#
|
||||
# THE INVARIANT: `MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and
|
||||
# was GREEN. A stage that could not run is never a pass, and the exit code says so too.
|
||||
#
|
||||
# Run: ci/tests/gate-decision.test.sh (or `make -C ci test`)
|
||||
set -uo pipefail
|
||||
|
||||
REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
CI="${TMP}/ci"; mkdir -p "$CI"
|
||||
REPO="${TMP}/repo"; mkdir -p "$REPO"
|
||||
|
||||
cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
|
||||
|
||||
# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
|
||||
for stage in spec-immutability lint l0 render-golden l1 conformance; do
|
||||
cat > "${CI}/${stage}.sh" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
|
||||
exit "${!v:-0}"
|
||||
STUB
|
||||
chmod +x "${CI}/${stage}.sh"
|
||||
done
|
||||
|
||||
# Stub decision input. TR_RC (default 0), TR_REASON (written to $TOUCHED_REASON_FILE like the
|
||||
# real script does), TR_OUT (the touched requirement list on stdout).
|
||||
cat > "${CI}/touched-requirements.sh" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "${TR_REASON:-}" > "$TOUCHED_REASON_FILE"; fi
|
||||
[ -n "${TR_OUT:-}" ] && printf '%s\n' "$TR_OUT"
|
||||
exit "${TR_RC:-0}"
|
||||
STUB
|
||||
chmod +x "${CI}/touched-requirements.sh"
|
||||
|
||||
# run_gate <VAR=VAL>... — run the gate with a clean, explicit environment. Sets OUT and RC.
|
||||
# GATE_STRICT_SKIPS is pinned false unless a case says otherwise, so the suite never depends on
|
||||
# whether the machine running it happens to look like CI.
|
||||
OUT="" RC=0
|
||||
run_gate() {
|
||||
local summary_file="${TMP}/summary.md"; : > "$summary_file"
|
||||
OUT="$(env -i PATH="$PATH" HOME="$HOME" \
|
||||
REPO_ROOT="$REPO" GITHUB_STEP_SUMMARY="$summary_file" \
|
||||
GATE_STRICT_SKIPS=false TR_OUT="R-TEST-03" \
|
||||
"$@" bash "${CI}/gate.sh" 2>&1)"
|
||||
RC=$?
|
||||
SUMMARY="$(cat "$summary_file")"
|
||||
}
|
||||
|
||||
# assert_verdict <headline> <expected-rc> <label>
|
||||
assert_verdict() {
|
||||
local want="$1" want_rc="$2" label="$3"
|
||||
if ! grep -qF "MERGE GATE: ${want}" <<<"$OUT"; then
|
||||
bad "${label}: headline should be '${want}' — got: $(grep -oE 'MERGE GATE: [A-Z ]+' <<<"$OUT" | tail -1)"
|
||||
elif [ "$RC" -ne "$want_rc" ]; then
|
||||
bad "${label}: headline '${want}' correct but exit code ${RC} != ${want_rc}"
|
||||
else
|
||||
ok "${label} -> ${want}, exit ${want_rc}"
|
||||
fi
|
||||
}
|
||||
# The load-bearing negative: nothing may say PASS, and nothing may exit 0.
|
||||
assert_not_pass() {
|
||||
local label="$1"
|
||||
if grep -qF "MERGE GATE: PASS" <<<"$OUT"; then bad "${label}: printed 'MERGE GATE: PASS'"
|
||||
elif [ "$RC" -eq 0 ]; then bad "${label}: exited 0"
|
||||
else ok "${label}: not a PASS, non-zero exit (${RC})"; fi
|
||||
}
|
||||
|
||||
echo "== 1. the happy path: every stage green"
|
||||
run_gate
|
||||
assert_verdict PASS 0 "all stages green + a touched subset"
|
||||
grep -qF "nothing was declared out of scope" <<<"$SUMMARY" \
|
||||
&& ok "PASS summary states nothing was declared out of scope" \
|
||||
|| bad "PASS summary does not state the absence of declared gaps"
|
||||
|
||||
echo
|
||||
echo "== 2. a FAILED stage blocks (unchanged behaviour, locked down)"
|
||||
for s in lint l0 render-golden l1 conformance spec-immutability; do
|
||||
run_gate "STAGE_RC_${s//-/_}=1"
|
||||
assert_verdict BLOCKED 1 "${s} FAILED"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "== 3. THE REGRESSION: a SKIPPED stage is not a PASS and not exit 0"
|
||||
for s in lint l0 render-golden l1; do
|
||||
run_gate "STAGE_RC_${s//-/_}=3"
|
||||
assert_not_pass "${s} SKIPPED (local)"
|
||||
assert_verdict UNDETERMINED 3 "${s} SKIPPED (local)"
|
||||
grep -qF "$s" <<<"$SUMMARY" && ok "${s}: the step summary names the stage that did not run" \
|
||||
|| bad "${s}: step summary does not name the skipped stage"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "== 4. under STRICT skips (CI declares it) a could-not-run skip BLOCKS, never merely 3"
|
||||
for s in lint l0 render-golden l1; do
|
||||
run_gate GATE_STRICT_SKIPS=true "STAGE_RC_${s//-/_}=3"
|
||||
assert_verdict BLOCKED 1 "${s} SKIPPED with GATE_STRICT_SKIPS=true"
|
||||
done
|
||||
# ...which is what makes gate.yml's `|| [ $? -eq 3 ]` safe: in CI, exit 3 cannot be a missing
|
||||
# tool or an unwired seam. Only the declared-environmental skip below can produce it.
|
||||
run_gate GATE_STRICT_SKIPS=true STAGE_RC_l0=3 STAGE_RC_lint=3
|
||||
assert_verdict BLOCKED 1 "several stages SKIPPED under strict skips"
|
||||
|
||||
echo
|
||||
echo "== 5. spec-immutability: unverified is never green, but is not upgraded to a block by CI"
|
||||
run_gate STAGE_RC_spec_immutability=3
|
||||
assert_not_pass "spec-immutability SKIPPED"
|
||||
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED (local)"
|
||||
run_gate GATE_STRICT_SKIPS=true STAGE_RC_spec_immutability=3
|
||||
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED under strict skips (deliberate exception)"
|
||||
run_gate REQUIRE_SPEC_VERIFICATION=true STAGE_RC_spec_immutability=3
|
||||
assert_verdict BLOCKED 1 "spec-immutability SKIPPED + REQUIRE_SPEC_VERIFICATION=true"
|
||||
|
||||
echo
|
||||
echo "== 6. conformance L2/L3: a DECLARED-absent capability may still pass — and is NAMED"
|
||||
run_gate STAGE_RC_conformance=3
|
||||
assert_verdict PASS 0 "conformance SKIPPED, no cluster declared"
|
||||
grep -qF "conformance-l2-l3" <<<"$OUT" \
|
||||
&& ok "the PASS headline names the declared L2/L3 gap" \
|
||||
|| bad "the PASS headline hides the declared L2/L3 gap"
|
||||
grep -qF "conformance-l2-l3" <<<"$SUMMARY" \
|
||||
&& ok "the step summary names the declared L2/L3 gap" \
|
||||
|| bad "the step summary hides the declared L2/L3 gap"
|
||||
run_gate CLUSTER_AVAILABLE=true STAGE_RC_conformance=3
|
||||
assert_verdict BLOCKED 1 "conformance SKIPPED despite CLUSTER_AVAILABLE=true"
|
||||
run_gate REQUIRE_CLUSTER_TESTS=true STAGE_RC_conformance=3
|
||||
assert_verdict BLOCKED 1 "conformance SKIPPED + REQUIRE_CLUSTER_TESTS=true with a touched subset"
|
||||
# A misspelled switch must leave the guard OFF (truthy()), never half-on.
|
||||
run_gate CLUSTER_AVAILABLE=ture STAGE_RC_conformance=3
|
||||
assert_verdict PASS 0 "CLUSTER_AVAILABLE=ture is NOT 'available' (no cluster was declared)"
|
||||
run_gate CLUSTER_AVAILABLE=TRUE STAGE_RC_conformance=3
|
||||
assert_verdict BLOCKED 1 "CLUSTER_AVAILABLE=TRUE IS a declaration (skip then blocks)"
|
||||
|
||||
echo
|
||||
echo "== 7. the undetermined touched subset: unchanged (2026-08-31 false-green paths)"
|
||||
run_gate TR_RC=3 TR_REASON=on-base TR_OUT=
|
||||
assert_verdict "NOT APPLICABLE" 0 "subset undeterminable, reason on-base, all stages green"
|
||||
run_gate TR_RC=3 TR_REASON=no-base TR_OUT=
|
||||
assert_verdict BLOCKED 1 "subset undeterminable, reason no-base"
|
||||
run_gate TR_RC=3 TR_REASON=diff-failed TR_OUT=
|
||||
assert_verdict BLOCKED 1 "subset undeterminable, reason diff-failed"
|
||||
# Precedence: a stage that did not run outranks "nothing to gate" — the stronger, non-zero
|
||||
# headline must win, or a push-to-main run could hide a missing tool behind NOT APPLICABLE.
|
||||
run_gate TR_RC=3 TR_REASON=on-base TR_OUT= STAGE_RC_l1=3
|
||||
assert_not_pass "on-base AND l1 SKIPPED"
|
||||
assert_verdict UNDETERMINED 3 "on-base AND l1 SKIPPED (UNDETERMINED outranks NOT APPLICABLE)"
|
||||
|
||||
echo
|
||||
echo "== 8. an UNUSABLE subset blocks (carried over from 47068ee): any other non-zero exit"
|
||||
for rc in 1 2 4; do
|
||||
run_gate "TR_RC=${rc}" TR_REASON=unclassified-path TR_OUT=
|
||||
assert_verdict BLOCKED 1 "touched-requirements.sh exited ${rc} (subset unusable, not empty)"
|
||||
done
|
||||
run_gate TR_RC=1 TR_REASON=unclassified-path TR_OUT=
|
||||
grep -qF "UNUSABLE" <<<"$SUMMARY" && ok "the step summary calls the subset UNUSABLE" \
|
||||
|| bad "the step summary does not report an unusable subset"
|
||||
|
||||
echo
|
||||
echo "== 9. combinations: a FAILURE outranks a non-run, and the headline is the worst news"
|
||||
run_gate STAGE_RC_l0=1 STAGE_RC_l1=3
|
||||
assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
|
||||
run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
|
||||
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
|
||||
else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
+164
@@ -0,0 +1,164 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/touched-requirements-test-attribution.test.sh — regression tests for DETECTION
|
||||
# RULE 3 in ci/touched-requirements.sh: a change anywhere under `conformance/tests/<test>/`
|
||||
# attributes THAT TEST'S OWN `conformance.home.k3s/R-*` labels.
|
||||
#
|
||||
# THE GAP THESE LOCK DOWN (2026-09-12). The coarse path map keys `conformance/*` to
|
||||
# R-TEST-01..04 — the requirements about the HARNESS — and says nothing about what a given
|
||||
# test PROVES. So weakening `conformance/tests/.../check.sh` attributed R-TEST-01..04 and
|
||||
# nothing else: the merge gate ran the harness's self-tests and NOT the weakened test's own
|
||||
# requirement subset. Weakening (or deleting) a test is the one change that most needs the
|
||||
# requirement it certifies to be re-verified, and it was the change least likely to select it.
|
||||
# The content scan cannot close this: it reads only ADDED/REMOVED lines, so it sees a change
|
||||
# to the LABEL BLOCK but not one to the test BODY — exactly the case that matters.
|
||||
#
|
||||
# Also covers the two path-map entries added in the same change (infra/* -> R-ENV-01, and
|
||||
# R-TEST-06 on .gitea/*|ci/*), which is how the two L0 checks that are DELIBERATELY not
|
||||
# whole-repo-scoped get selected by the paths that can break them.
|
||||
#
|
||||
# Same harness as the sibling suites: the real script, pointed at a THROWAWAY git repo via
|
||||
# the REPO_ROOT override. No network, no cluster, the working repo is never touched.
|
||||
#
|
||||
# Run: ci/tests/touched-requirements-test-attribution.test.sh (or `make -C ci test`)
|
||||
set -euo pipefail
|
||||
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
|
||||
|
||||
assert_present() {
|
||||
if grep -qxF "$1" <<<"$2"; then ok "$3"
|
||||
else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
|
||||
}
|
||||
assert_absent() {
|
||||
if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))"
|
||||
else ok "$3"; fi
|
||||
}
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
git -C "$TMP" init -q -b main
|
||||
git -C "$TMP" config user.email [email protected]
|
||||
git -C "$TMP" config user.name t
|
||||
git -C "$TMP" config commit.gpgsign false
|
||||
|
||||
# --- Baseline: two conformance tests, each declaring its own requirement labels ----------
|
||||
# TEST A: a flat l0-static test. TEST B: a test with a NESTED fixture dir, to prove the
|
||||
# walk-up from an arbitrary changed file to the enclosing test dir.
|
||||
mkdir -p "$TMP/conformance/tests/l0-static/alpha" \
|
||||
"$TMP/conformance/tests/l2-behavioral/beta/fixtures" \
|
||||
"$TMP/conformance/lib" "$TMP/infra" "$TMP/ci"
|
||||
|
||||
cat > "$TMP/conformance/tests/l0-static/alpha/chainsaw-test.yaml" <<'YAML'
|
||||
# HEADER PROSE. This test used to also cover conformance.home.k3s/R-STALE-09: "true",
|
||||
# which is a REFERENCE in a comment, not a declaration, and must NOT be attributed.
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: alpha
|
||||
labels:
|
||||
conformance.home.k3s/R-ALPHA-01: "true"
|
||||
conformance.home.k3s/R-ALPHA-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/layer: "L0"
|
||||
spec: {}
|
||||
YAML
|
||||
# The check BODY deliberately contains NO R-ID text, so anything the scan finds for a
|
||||
# body-only edit came from rule 3 and not from the content scan.
|
||||
cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
echo "assert something"
|
||||
SH
|
||||
|
||||
cat > "$TMP/conformance/tests/l2-behavioral/beta/chainsaw-test.yaml" <<'YAML'
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: beta
|
||||
labels:
|
||||
conformance.home.k3s/R-BETA-07: "true"
|
||||
spec: {}
|
||||
YAML
|
||||
echo "seed: 1" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
|
||||
|
||||
echo "harness helper" > "$TMP/conformance/lib/helper.sh"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
|
||||
|
||||
# ========================================================================================
|
||||
echo "== case 1: a BODY-ONLY edit to a test attributes that test's own labels"
|
||||
git -C "$TMP" checkout -q -b body-edit
|
||||
cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
echo "assert something weaker"
|
||||
SH
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm weaken
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-ALPHA-01" "$OUT" "body-only test edit attributes the test's own R-ALPHA-01"
|
||||
assert_present "R-ALPHA-02" "$OUT" "body-only test edit attributes the test's own R-ALPHA-02"
|
||||
assert_present "R-TEST-03" "$OUT" "the conformance/* path map still applies (R-TEST-03)"
|
||||
assert_absent "R-STALE-09" "$OUT" "an R-ID cited only in the test's HEADER COMMENT is NOT attributed"
|
||||
assert_absent "R-BETA-07" "$OUT" "an UNRELATED test's labels are NOT attributed"
|
||||
|
||||
# ========================================================================================
|
||||
echo "== case 2: an edit to a NESTED file walks up to the enclosing test"
|
||||
git -C "$TMP" checkout -q -b nested-edit main
|
||||
echo "seed: 2" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm fixture
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-BETA-07" "$OUT" "nested fixture edit attributes the enclosing test's R-BETA-07"
|
||||
assert_absent "R-ALPHA-01" "$OUT" "nested fixture edit does NOT attribute a sibling test's labels"
|
||||
|
||||
# ========================================================================================
|
||||
echo "== case 3: DELETING a test still attributes what it used to certify"
|
||||
# Removing the only verifier of a requirement must not be the cheapest way to make the gate
|
||||
# stop checking it — so the labels are read from the BASE when HEAD no longer has them.
|
||||
git -C "$TMP" checkout -q -b delete-test main
|
||||
git -C "$TMP" rm -rq "$TMP/conformance/tests/l0-static/alpha"
|
||||
git -C "$TMP" commit -qm "delete the alpha test"
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-ALPHA-01" "$OUT" "deleting a test STILL attributes its R-ALPHA-01 (read from base)"
|
||||
assert_present "R-ALPHA-02" "$OUT" "deleting a test STILL attributes its R-ALPHA-02 (read from base)"
|
||||
|
||||
# ========================================================================================
|
||||
echo "== case 4: a change OUTSIDE conformance/tests/ gains no test labels"
|
||||
git -C "$TMP" checkout -q -b lib-edit main
|
||||
echo "harness helper v2" > "$TMP/conformance/lib/helper.sh"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm lib
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-TEST-02" "$OUT" "a conformance/lib edit still attributes the harness requirements"
|
||||
assert_absent "R-ALPHA-01" "$OUT" "a conformance/lib edit attributes NO individual test's labels"
|
||||
assert_absent "R-BETA-07" "$OUT" "a conformance/lib edit attributes NO individual test's labels (B)"
|
||||
|
||||
# ========================================================================================
|
||||
echo "== case 5: path-map entries for the two DELIBERATELY path-scoped L0 checks"
|
||||
# infra/* -> R-ENV-01 (dev-nodes-provisioned-as-code reads nothing but infra/).
|
||||
git -C "$TMP" checkout -q -b infra-edit main
|
||||
printf 'resource "x" "y" {}\n' > "$TMP/infra/main.tf"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm infra
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-ENV-01" "$OUT" "an infra/ edit attributes R-ENV-01 (was UNMAPPED: attributed nothing)"
|
||||
|
||||
# ci/* -> ... R-TEST-06 (spec-immutability-enforced reads nothing but ci/ + the spec repo).
|
||||
git -C "$TMP" checkout -q -b ci-edit main
|
||||
printf 'echo hello\n' > "$TMP/ci/thing.sh"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm ci
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-TEST-06" "$OUT" "a ci/ edit attributes R-TEST-06 (the spec-immutability test's own ID)"
|
||||
assert_present "R-GITOPS-01" "$OUT" "a ci/ edit still attributes R-GITOPS-01"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then
|
||||
echo "touched-requirements test-attribution tests: PASS (${PASS} assertions)"; exit 0
|
||||
fi
|
||||
echo "touched-requirements test-attribution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
|
||||
exit 1
|
||||
@@ -93,6 +93,49 @@ printf 'touched set (removal):\n%s\n---\n' "$OUT2"
|
||||
assert_absent "R-DELDOC-01" "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
|
||||
assert_present "R-DELREAL-01" "$OUT2" "REMOVED real-line R-ID IS attributed"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# PATH MAP — bootstrap/* (added 2026-09-13)
|
||||
#
|
||||
# WHY THIS IS LOCKED DOWN. path_requirements() keys `bootstrap/*` to R-STATE-03 +
|
||||
# R-GITOPS-01, and until 2026-09-13 the comment justifying it described
|
||||
# `bootstrap/standup.sh` as a file that carries the placeholder guards. That file has never
|
||||
# been on `main` (it landed only on unmerged prod-standup branches), so the comment was
|
||||
# stale and the entry looked like dead weight someone would tidy away. It is not dead: it
|
||||
# is what SELECTS conformance/tests/l0-static/placeholder-guard-coupling for the change
|
||||
# that eventually lands a placeholder guard under bootstrap/ — and that test discovers its
|
||||
# guard by content rather than by path, so it activates on exactly that change. Removing
|
||||
# the entry would let the change introducing a guard be gated by nothing, which is the
|
||||
# 2026-09-05 class. These assertions make the entry's deletion a test failure rather than a
|
||||
# silent loss of selection.
|
||||
#
|
||||
# NOTE the R-ID content scan must not be what produces these: `notes/` is outside the path
|
||||
# map precisely so the other cases isolate the scan, and here the file content cites no
|
||||
# R-ID at all, so any R-ID in the output came from the PATH.
|
||||
git -C "$TMP" checkout -q -b feature3 main
|
||||
mkdir -p "$TMP/bootstrap/flux"
|
||||
cat > "$TMP/bootstrap/flux/bootstrap.sh" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
# a bootstrap-path change that cites no requirement id of its own
|
||||
echo "flux bootstrap"
|
||||
SH
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch bootstrap/"
|
||||
OUT3="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (bootstrap/ path map):\n%s\n---\n' "$OUT3"
|
||||
assert_present "R-STATE-03" "$OUT3" "bootstrap/* attributes R-STATE-03 (selects placeholder-guard-coupling)"
|
||||
assert_present "R-GITOPS-01" "$OUT3" "bootstrap/* attributes R-GITOPS-01 (it IS the GitOps bootstrap path)"
|
||||
|
||||
# And the neighbouring entry the same test leans on for its other half: clusters/* must
|
||||
# attribute R-ENV-02, which is what selects placeholder-guard-coupling's HALF A (the
|
||||
# authored-values sweep over clusters/<env> effective cluster-vars).
|
||||
git -C "$TMP" checkout -q -b feature4 main
|
||||
mkdir -p "$TMP/clusters/base"
|
||||
printf 'kind: ConfigMap\nmetadata:\n name: cluster-vars\n' > "$TMP/clusters/base/cluster-vars.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch clusters/"
|
||||
OUT4="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
|
||||
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
|
||||
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
|
||||
else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
+123
-2
@@ -21,8 +21,12 @@
|
||||
# the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011).
|
||||
# Untouched lines and prose are NOT scanned; see changed_diff() for why.
|
||||
# 2. A coarse path->requirement fallback for changes that don't self-declare IDs.
|
||||
# 3. For a change anywhere under `conformance/tests/<test>/`: THAT TEST'S OWN requirement
|
||||
# labels, read from its chainsaw-test.yaml. See test_requirements_for_path().
|
||||
# The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for
|
||||
# untagged changes so nothing silently escapes the gate.
|
||||
# untagged changes so nothing silently escapes the gate. (3) is the same principle applied
|
||||
# to the tests themselves: editing or weakening a test must be gated by the requirements
|
||||
# that test CERTIFIES, not merely by R-TEST-0x.
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
@@ -180,14 +184,128 @@ path_requirements() {
|
||||
platform/externaldns/*) echo "R-DNS-02" ;;
|
||||
platform/velero/*) echo "R-STATE-03" ;;
|
||||
platform/monitoring/*) echo "R-OBS-01 R-OBS-02" ;;
|
||||
# platform/cnpg was UNMAPPED until 2026-09-06. It is the PostgreSQL DR path for keycloak-db /
|
||||
# nextcloud-db / semaphore-db (barmanObjectStore WAL archiving + ScheduledBackup), so a change
|
||||
# there is an R-IDP-05 / R-STATE-03 change; it now also carries the platform-health alert rules
|
||||
# for that path (R-OBS-04). Without this line an edit under platform/cnpg/ contributed ZERO
|
||||
# requirements unless it happened to retype an R-ID on a non-comment line, and escaped the gate.
|
||||
platform/cnpg/*) echo "R-IDP-05 R-STATE-03 R-OBS-04" ;;
|
||||
# vpn-egress was UNMAPPED until 2026-09-06. platform/vpn-egress/egress-steering.yaml IS the
|
||||
# R-NET-04 datapath (the rule that selects between the two egress paths) and the R-NET-05/06/07
|
||||
# steering policy — it even self-declares those IDs in its `homelab.keithjackson.dev/requirements`
|
||||
# annotation. But the content scan only reads ADDED/REMOVED lines, so any edit that does not
|
||||
# happen to retype an R-ID (a selector change, an excludedCIDRs change, a comment) contributed
|
||||
# ZERO requirements and escaped the gate entirely. Mapped explicitly; the annotation still wins
|
||||
# when a diff does carry the tags.
|
||||
platform/vpn-egress/*) echo "R-NET-04 R-NET-05 R-NET-06 R-NET-07" ;;
|
||||
# Keycloak is the whole SSO surface: the realm import (declarative identity config),
|
||||
# the SOPS client secrets, and the reconciler that heals them. Added 2026-09-06 after
|
||||
# the client-secret incident, in which editing platform/keycloak/ attributed NO
|
||||
# requirement, so the L2 keycloak tests were filtered out of the merge gate's touched
|
||||
# subset — the change that could break every OIDC login was gated by nothing.
|
||||
# R-IDP-05 is in scope because platform/keycloak/ also carries the CNPG DB Cluster and
|
||||
# its off-cluster backup credential, not only the realm.
|
||||
platform/keycloak/*) echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
|
||||
platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;;
|
||||
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
|
||||
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
|
||||
# infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
|
||||
# nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
|
||||
# infra/, yet an infra/ edit attributed ZERO requirements, so the one check that asserts
|
||||
# the nodes are provisioned as code was filtered out of every change that could break it.
|
||||
# That test's invariant IS path-scoped (it is deliberately NOT whole-repo-scoped; see its
|
||||
# chainsaw-test.yaml), and the correct fix for a path-scoped invariant is a path-map entry.
|
||||
infra/*) echo "R-ENV-01" ;;
|
||||
clusters/*) echo "R-GITOPS-01 R-ENV-02" ;;
|
||||
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
|
||||
# bootstrap/ was UNMAPPED until 2026-09-06. The entry STAYS, but its original rationale
|
||||
# was written against a file that is not here, and that is corrected on 2026-09-13:
|
||||
# it claimed "bootstrap/standup.sh carries the placeholder guards that decide whether
|
||||
# `velero` may be enabled against a real backup endpoint". bootstrap/standup.sh HAS NEVER
|
||||
# BEEN ON `main` — it was not deleted (no deletion commit exists anywhere), it landed only
|
||||
# on unmerged prod-standup branches. `git ls-tree main bootstrap/` lists README.md, flux/,
|
||||
# gitea/, sops/. So R-STATE-03 here is ANTICIPATORY, not descriptive.
|
||||
#
|
||||
# WHY IT IS STILL RIGHT. conformance/tests/l0-static/placeholder-guard-coupling DISCOVERS
|
||||
# its guard (any tracked *.sh defining `<name>_is_placeholder()`) instead of hardcoding a
|
||||
# path, so the moment a standup guard lands under bootstrap/ that test's coupling half
|
||||
# activates — and this entry is what selects the test for the change that lands it.
|
||||
# Deleting the entry would mean the change introducing a placeholder guard is gated by
|
||||
# nothing, which is the 2026-09-05 class (a prefix glob that stopped matching when the
|
||||
# value it guards changed, in a different file) left to reintroduce itself. R-GITOPS-01 is
|
||||
# descriptive today: bootstrap/flux/, gitea/ and sops/ ARE the GitOps bootstrap path.
|
||||
# Locked down by ci/tests/touched-requirements.test.sh's path-map section.
|
||||
bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;;
|
||||
# R-TEST-06 added 2026-09-12. The comment in changed_diff()'s rule 3 above records why it
|
||||
# was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE and carried by no
|
||||
# Chainsaw test, so attributing it made the coverage `--require` check false-block. It is now
|
||||
# carried by conformance/tests/l0-static/spec-immutability-enforced — whose entire input
|
||||
# surface is ci/ — so a ci/ change must attribute it, or the test that proves the
|
||||
# spec-immutability mechanism is still load-bearing runs on nothing that can break it.
|
||||
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06" ;;
|
||||
*) : ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------------
|
||||
# A CHANGED TEST ATTRIBUTES THE REQUIREMENTS IT CERTIFIES (detection rule 3).
|
||||
#
|
||||
# THE GAP THIS CLOSES (2026-09-12). The path map keys `conformance/*` to
|
||||
# R-TEST-01..04 — the requirements about the HARNESS. It says nothing about what a given
|
||||
# test PROVES. So editing `conformance/tests/l2-behavioral/tenant-isolation/check.sh` to
|
||||
# weaken or delete an assertion attributed R-TEST-01..04 and nothing else: the merge gate
|
||||
# then ran the harness's self-tests and NOT the weakened test's own requirement subset.
|
||||
# Weakening a test is the one change that most needs the requirement it certifies to be
|
||||
# re-verified, and it was the change least likely to select it.
|
||||
#
|
||||
# The `conformance.home.k3s/R-*: "true"` labels on a test ARE the authoritative declaration of
|
||||
# what it verifies (lib/coverage.py reads the same labels); this reads them directly. Note the
|
||||
# content scan in main() only sees ADDED/REMOVED lines, so it catches a change to the LABEL
|
||||
# BLOCK but not a change to the test BODY — which is exactly the case that matters here.
|
||||
#
|
||||
# WHY IT READS HEAD *OR* THE BASE: a change that DELETES a test, or deletes its
|
||||
# chainsaw-test.yaml, must still attribute what that test used to certify — otherwise removing
|
||||
# the only verifier of a requirement is the cheapest way to make the gate stop checking it.
|
||||
# HEAD first (the post-change truth), base as the fallback (the deleted case). Fail-closed:
|
||||
# this function can only ADD requirements, never remove one.
|
||||
: "${CONF_TEST_LABEL_PREFIX:=conformance.home.k3s}" # == CONF_LABEL_PREFIX, conformance/lib/common.sh
|
||||
|
||||
file_text_head_or_base() {
|
||||
local path="$1"
|
||||
git -C "$REPO_ROOT" show "HEAD:${path}" 2>/dev/null && return 0
|
||||
git -C "$REPO_ROOT" show "${BASE}:${path}" 2>/dev/null && return 0
|
||||
return 0
|
||||
}
|
||||
|
||||
# test_requirements_for_path <changed-path> -> space-separated R-IDs (possibly empty).
|
||||
# Walks UP from the changed file to the nearest ancestor directory carrying a
|
||||
# chainsaw-test.yaml, so it works for any file in a test dir (check.sh, mutations.py,
|
||||
# fixtures, the chainsaw-test.yaml itself) and for nested test layouts.
|
||||
test_requirements_for_path() {
|
||||
local f="$1" dir yaml text
|
||||
case "$f" in conformance/tests/*/*) ;; *) return 0 ;; esac
|
||||
dir="${f%/*}"
|
||||
while :; do
|
||||
case "$dir" in
|
||||
conformance/tests|conformance|.|"") return 0 ;;
|
||||
*/*) ;;
|
||||
*) return 0 ;;
|
||||
esac
|
||||
yaml="${dir}/chainsaw-test.yaml"
|
||||
text="$(file_text_head_or_base "$yaml")"
|
||||
if [ -n "$text" ]; then
|
||||
# Whole-line `#` comments are dropped first — same "reference, not declaration" rule the
|
||||
# content scan applies (changed_diff() rule 3): a label shown inside a header example is
|
||||
# documentation. Only a real YAML mapping line counts.
|
||||
printf '%s\n' "$text" \
|
||||
| grep -vE '^[[:space:]]*#' \
|
||||
| sed -nE "s#^[[:space:]]*${CONF_TEST_LABEL_PREFIX}/(R-[A-Z0-9]+-[0-9]+):[[:space:]]*[\"']?true.*#\1#p" \
|
||||
| tr '\n' ' '
|
||||
return 0
|
||||
fi
|
||||
dir="${dir%/*}"
|
||||
done
|
||||
}
|
||||
|
||||
main() {
|
||||
local files reqs="" rc=0 base_sha head_sha local_main merge_base
|
||||
|
||||
@@ -221,6 +339,9 @@ main() {
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
reqs+=" $(path_requirements "$f")"
|
||||
# Detection rule 3: a change under conformance/tests/<test>/ attributes THAT TEST'S own
|
||||
# requirement labels, so weakening a test is gated by what the test certifies.
|
||||
reqs+=" $(test_requirements_for_path "$f")"
|
||||
done <<< "$files"
|
||||
|
||||
# Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
|
||||
|
||||
@@ -21,6 +21,15 @@ YAMLLINT_VERSION="1.35.1"
|
||||
KUBECTL_VERSION="1.36.4"
|
||||
YQ_VERSION="4.44.6"
|
||||
|
||||
# Alert-rule unit tests (promtool test rules) — the L0 check
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested runs every
|
||||
# platform/**/*.promtool-test.yaml through it. MUST stay in lockstep with
|
||||
# infra/devcontainer/versions.env (which bakes it into the image) AND with the Prometheus
|
||||
# version running on prod, because promtool IS the rule engine: verified 2026-09-12 as
|
||||
# v3.14.0-distroless via `kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`.
|
||||
# Shipped inside the prometheus release tarball, not as a separate release.
|
||||
PROMTOOL_VERSION="3.14.0"
|
||||
|
||||
# ServiceDefinition JSON-Schema validation (schemas/validate.sh). Pip-installed;
|
||||
# pinned here so CI and a laptop validate against the exact same validator.
|
||||
CHECK_JSONSCHEMA_VERSION="0.29.4"
|
||||
@@ -97,6 +97,27 @@ Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0)
|
||||
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
|
||||
```
|
||||
|
||||
### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
|
||||
|
||||
The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
|
||||
controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
|
||||
and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
|
||||
reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
|
||||
before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
|
||||
**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
|
||||
`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
|
||||
have existed for months); it cost 12h on prod's cold start.
|
||||
|
||||
So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
|
||||
platform-config`), and a rule:
|
||||
|
||||
> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
|
||||
> **controller at runtime** rather than by a manifest in git.
|
||||
|
||||
`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
|
||||
`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
|
||||
profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
|
||||
|
||||
Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
|
||||
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
|
||||
(`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
|
||||
|
||||
@@ -4,8 +4,11 @@
|
||||
# NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
|
||||
# import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
|
||||
# health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
|
||||
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
|
||||
# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
|
||||
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
|
||||
# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
|
||||
# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
|
||||
# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
|
||||
# adoption concern).
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
@@ -19,6 +22,28 @@ spec:
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config
|
||||
# LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
|
||||
# storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
|
||||
# decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
|
||||
# (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
|
||||
# live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
|
||||
# with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
|
||||
# to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
|
||||
# conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
|
||||
#
|
||||
# HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
|
||||
# derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
|
||||
# status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
|
||||
# it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
|
||||
#
|
||||
# This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
|
||||
# deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
|
||||
# dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
|
||||
#
|
||||
# The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
|
||||
# edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
|
||||
# cannot silently drop this tier's KubeVirt/StorageClass gate.
|
||||
- name: platform-config-cdi
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
|
||||
@@ -34,6 +34,9 @@ spec:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# SOPS decryption for velero-cloud.sops.yaml (the scoped MinIO `cloud` credential) is layered per
|
||||
# environment by the overlay (clusters/dev patches THIS Kustomization by name), identical to how
|
||||
# headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. See clusters/dev/kustomization.yaml.
|
||||
# SOPS decryption for the `cloud` INI credentials — velero-cloud.sops.yaml (dev's scoped MinIO key)
|
||||
# and velero-cloud-prod.sops.yaml (prod's Cloudflare R2 token) — is layered per environment by the
|
||||
# overlay (clusters/dev and clusters/prod patch THIS Kustomization by name), identical to how
|
||||
# headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. Both Secrets are applied on both
|
||||
# clusters; ${BACKUP_CREDENTIALS_SUFFIX} (substituted from cluster-vars above) decides which one the
|
||||
# velero Deployment actually mounts. See clusters/dev/kustomization.yaml.
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
|
||||
# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
|
||||
#
|
||||
# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
|
||||
#
|
||||
# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
|
||||
# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
|
||||
# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
|
||||
# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
|
||||
#
|
||||
# * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
|
||||
# * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
|
||||
# tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
|
||||
# `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
|
||||
# the valve the 2026-09-01 substitution incident wanted and did not have.
|
||||
# * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
|
||||
# `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
|
||||
#
|
||||
# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
|
||||
# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
|
||||
# expressed as an explicit edge instead of an accident of co-tenancy.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: cilium
|
||||
namespace: flux-system
|
||||
spec:
|
||||
# A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
|
||||
# kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
|
||||
interval: 1h
|
||||
retryInterval: 2m
|
||||
# 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
|
||||
# rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
|
||||
timeout: 10m
|
||||
prune: true
|
||||
# TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
|
||||
wait: true
|
||||
# Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
|
||||
# by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
|
||||
# is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
|
||||
# revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
|
||||
# `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
|
||||
dependsOn:
|
||||
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/cilium
|
||||
# MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
|
||||
# kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
|
||||
# this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
|
||||
# renders empty). That is the intended fail-closed behaviour — but it means the stanza is
|
||||
# load-bearing for the CNI coming up at all. Do not remove it.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
|
||||
# oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).
|
||||
+243
-11
@@ -17,9 +17,63 @@ data:
|
||||
CLUSTER_ENV: "base" # dev | prod (overlays set this)
|
||||
|
||||
# --- DNS / TLS ---
|
||||
BASE_DOMAIN: "home.keithjackson.dev" # platform hostname suffix (same both envs; split-horizon)
|
||||
DNS_ZONE: "keithjackson.dev" # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
|
||||
# ExternalDNS domain-filter must be the zone, not the subdomain
|
||||
# THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
|
||||
# the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
|
||||
# ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
|
||||
#
|
||||
# THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
|
||||
# hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
|
||||
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
|
||||
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
|
||||
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
|
||||
# subdomain, so the two clusters can no longer describe the same name.
|
||||
#
|
||||
# dev -> home.keithjackson.dev (UNCHANGED — see below)
|
||||
# prod -> prod.home.keithjackson.dev
|
||||
#
|
||||
# DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
|
||||
# `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
|
||||
# relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
|
||||
# the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
|
||||
# (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
|
||||
# GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
|
||||
# it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
|
||||
# tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
|
||||
BASE_DOMAIN: "home.keithjackson.dev"
|
||||
|
||||
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
|
||||
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
|
||||
# so it cannot be created, updated or deleted regardless of TXT ownership.
|
||||
#
|
||||
# DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
|
||||
# apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
|
||||
# that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
|
||||
# record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
|
||||
# from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
|
||||
#
|
||||
# PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
|
||||
# (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
|
||||
# chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
|
||||
# — it can never delete another environment's records — and the fallback is to set prod's value
|
||||
# back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
|
||||
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
|
||||
|
||||
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
|
||||
# neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
|
||||
# record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
|
||||
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
|
||||
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
|
||||
#
|
||||
# DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
|
||||
# under owner `default`; changing dev's value would make dev's own external-dns stop recognising
|
||||
# them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
|
||||
#
|
||||
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
|
||||
# forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
|
||||
# this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
|
||||
# visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
|
||||
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
|
||||
|
||||
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
||||
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
||||
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
|
||||
@@ -42,15 +96,55 @@ data:
|
||||
#
|
||||
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
|
||||
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
|
||||
# (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
|
||||
# velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
|
||||
# ship the literal string into the cluster.
|
||||
# (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
|
||||
# platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
|
||||
# tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
|
||||
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH), velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
|
||||
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
|
||||
# UNSUBSTITUTED path would ship the literal string into the cluster.
|
||||
#
|
||||
# ADDING A KNOB IS SAFE UNDER StrictPostBuildSubstitutions ONLY BECAUSE OF THE ROOT EDGE. An
|
||||
# UNDEFINED variable is a HARD Kustomization failure, so a manifest that references a knob the
|
||||
# live ConfigMap does not yet carry breaks its whole tier. `clusters/base/platform.yaml` gained
|
||||
# `dependsOn: [flux-system]` for exactly this: dependsOn is REVISION-gated, so no substituting
|
||||
# tier can reconcile revision N until the ROOT Kustomization has APPLIED revision N — i.e. until
|
||||
# the new cluster-vars key EXISTS. Every substituting consumer chains transitively through
|
||||
# `platform`, so one edge covers the graph, and a new knob plus its first reference may ship in
|
||||
# ONE commit. (A hand-run `kubectl apply -k clusters/<env>` bypasses that ordering entirely: apply
|
||||
# the overlay BEFORE the tier that references the knob, or the tier fails until it retries.)
|
||||
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
|
||||
# [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
|
||||
# main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
|
||||
# reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
|
||||
K3S_API_HOST: "192.168.1.50"
|
||||
#
|
||||
# *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
|
||||
# 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
|
||||
# control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
|
||||
# live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
|
||||
# own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
|
||||
# Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
|
||||
# outage you would notice. A virtual address that follows the control plane is the fix, and it is
|
||||
# what kube-vip was stood up to provide.
|
||||
#
|
||||
# dev -> 192.168.1.49 (= kube_vip_address, group_vars/all.yml; base default, below)
|
||||
# prod -> 192.168.2.9 (= kube_vip_address, group_vars/k3s_prod/vars.yml)
|
||||
#
|
||||
# THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
|
||||
# unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
|
||||
# move. Read it as "the API host Cilium dials", which is what it always meant.
|
||||
#
|
||||
# [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
|
||||
# (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
|
||||
# already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
|
||||
# the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
|
||||
# already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
|
||||
# block at the top of group_vars/all.yml.
|
||||
#
|
||||
# Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
|
||||
# in both envs, so it is NOT a knob.
|
||||
#
|
||||
# TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
|
||||
# both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
|
||||
# https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
|
||||
K3S_API_HOST: "192.168.1.49"
|
||||
|
||||
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
|
||||
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
|
||||
@@ -66,14 +160,59 @@ data:
|
||||
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
||||
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
||||
|
||||
# --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
|
||||
# THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
|
||||
# library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
|
||||
# (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
|
||||
# `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
|
||||
# declared in the ServiceDefinition, because the path is identical in every environment and only
|
||||
# the ADDRESS is not.
|
||||
#
|
||||
# WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
|
||||
# The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
|
||||
# dev nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
|
||||
# NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
|
||||
# /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
|
||||
# per the convention of the LAN block above.
|
||||
# prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
|
||||
# 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
|
||||
# the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
|
||||
# DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
|
||||
# So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
|
||||
# blast radius of shipping it unset, which is NOT the usual "fails to apply".
|
||||
BULK_NFS_SERVER: "192.168.1.9"
|
||||
|
||||
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
|
||||
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
|
||||
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
|
||||
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR.
|
||||
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR —
|
||||
# and, measured 2026-09-05 from a pod INSIDE prod, it CANNOT reach it anyway (prod is on the SOHO
|
||||
# network 192.168.2.0/24, which is pfSense's WAN, while this MinIO sits inside pfSense's LAN:
|
||||
# `curl http://192.168.1.9:9000/minio/health/live` timed out after 15s). Prod uses Cloudflare R2.
|
||||
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
|
||||
|
||||
# S3 region. Every S3 client REQUIRES one; MinIO ignores whatever it is; Cloudflare R2 demands
|
||||
# exactly `auto` (its `head-bucket` reports the physical hint `WNAM`, which is NOT a usable region
|
||||
# string). This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-
|
||||
# accident for MinIO and wrong for anything else, which is why it is a knob now. The base default
|
||||
# is dev's LIVE value, deliberately: dev's Velero has been signing requests with `us-east-1`
|
||||
# against that MinIO since bootstrap, and this repo's rule is that base == the known-good dev
|
||||
# value. Prod overrides to `auto`.
|
||||
S3_BACKUP_REGION: "us-east-1"
|
||||
|
||||
# --- storage (ADR-0003) ---
|
||||
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
||||
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
|
||||
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
|
||||
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
|
||||
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
|
||||
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
|
||||
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
|
||||
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
|
||||
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
|
||||
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
|
||||
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
|
||||
LONGHORN_DATA_PATH: "/var/lib/longhorn"
|
||||
|
||||
# --- observability sizing (docs/10) ---
|
||||
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
|
||||
@@ -113,6 +252,99 @@ data:
|
||||
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
|
||||
VELERO_RETENTION: "168h" # dev short | prod long
|
||||
|
||||
# ==========================================================================================
|
||||
# OBJECT-STORE TENANCY — the three knobs that keep two clusters and two backup tools from
|
||||
# writing into each other's data. THESE ARE SAFETY KNOBS, not preferences.
|
||||
# ==========================================================================================
|
||||
# Nothing about the object layout was per-environment before this: Velero's bucket was the
|
||||
# hardcoded literal `velero` with no prefix, and every CNPG Cluster used the hardcoded
|
||||
# `s3://cnpg` with barman's `serverName` defaulting to the CLUSTER NAME — and BOTH environments
|
||||
# have a keycloak-db, a nextcloud-db and a semaphore-db. Point two clusters at one bucket under
|
||||
# that layout and:
|
||||
# (a) each cluster's Velero ADOPTS the other's backups and TTL-DELETES them on retention; and
|
||||
# (b) two different Postgres servers stream WAL into ONE Barman prefix, which does not merely
|
||||
# mix files — it CORRUPTS THE CHAIN and destroys PITR for both. That is the worse half:
|
||||
# dev's DR restore is PROVEN (R-STATE-03) and this would silently un-prove it.
|
||||
#
|
||||
# WHY THERE IS NO SINGLE `S3_BACKUP_BUCKET`. The two consumers take differently-SHAPED values:
|
||||
# the Velero chart wants `bucket` and `prefix` as separate fields, while CNPG's
|
||||
# barmanObjectStore.destinationPath is ONE full s3:// URL whose first path segment IS the bucket.
|
||||
# CNPG's bucket therefore already lives inside CNPG_BACKUP_PATH; adding a second knob that had to
|
||||
# hold the same bucket name a second time would be a rotation waiting to drift — the exact class
|
||||
# of duplication this repo has been bitten by. Three knobs, each shaped for its own consumer, and
|
||||
# no two of them can contradict each other.
|
||||
#
|
||||
# BASE = DEV'S LIVE LAYOUT, BYTE-FOR-BYTE, AND MUST STAY THAT WAY. Dev's existing backups are at
|
||||
# `s3://velero/` (bucket root, no prefix) and `s3://cnpg/<clusterName>/` on that MinIO. Giving dev
|
||||
# a prefix, or renaming its buckets, would ORPHAN the very backups its proven restore depends on —
|
||||
# the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default` and its LONGHORN_DATA_PATH stays
|
||||
# /var/lib/longhorn. Verified: with an EMPTY prefix the Velero chart omits `objectStorage.prefix`
|
||||
# from the rendered BackupStorageLocation entirely (helm template, chart 12.1.0), so dev's live
|
||||
# BSL is unchanged by this parameterization rather than merely equivalent to it.
|
||||
#
|
||||
# PROD, which has no history to preserve, gets ONE bucket with TWO disjoint top-level prefixes:
|
||||
#
|
||||
# s3://k3s-prod-backup/velero/… Velero (kopia repo + backup metadata)
|
||||
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG WAL + base backups
|
||||
#
|
||||
# The operator created exactly one R2 bucket, so velero-vs-barman separation has to come from the
|
||||
# prefix rather than the bucket. The BUCKET ROOT IS DELIBERATELY LEFT EMPTY: if a prefix ever
|
||||
# renders wrong, its objects land somewhere obviously bogus instead of silently colliding with a
|
||||
# real tenant's data. Dev↔prod isolation is doubly independent — different endpoint AND different
|
||||
# bucket — so neither cluster can reach the other's objects even with a mis-substituted prefix.
|
||||
VELERO_BACKUP_BUCKET: "velero" # dev: the `velero` MinIO bucket | prod: k3s-prod-backup
|
||||
VELERO_BACKUP_PREFIX: "" # dev: bucket root (live layout) | prod: "velero"
|
||||
CNPG_BACKUP_PATH: "s3://cnpg" # dev: the `cnpg` MinIO bucket root | prod: s3://k3s-prod-backup/barman
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# WHICH CREDENTIAL SECRET THE BACKUP CONSUMERS READ (added 2026-09-07 — the per-env credential split)
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# The three knobs above say WHERE the backups go. This one says WHICH KEY signs the requests, and
|
||||
# it exists because the endpoint moved per-environment while the credential did not.
|
||||
#
|
||||
# THE BUG IT FIXES. The four backup-credential Secrets — velero-cloud.sops.yaml and the three
|
||||
# <db>-backup-credentials.sops.yaml — are SHARED BASE manifests: platform/ and apps/ are rendered
|
||||
# by the same Flux tiers on both clusters, and cluster-vars substitution is the ONLY thing that
|
||||
# differs (R-ENV-02). So writing prod's R2 token into those four files did not GIVE prod a
|
||||
# credential, it REPLACED DEV'S: dev's Velero and all three dev CNPG Clusters would have started
|
||||
# signing MinIO requests with an R2 key id. For Velero that is a broken BSL; for CNPG it is worse,
|
||||
# because `archive_mode=on` means Postgres REFUSES TO RECYCLE a WAL segment it could not archive,
|
||||
# so a 403 loop is a monotonically filling pgdata PVC ending in a PANIC — the same failure prod
|
||||
# spent its whole bring-up in with a placeholder endpoint. One shared Secret also means rotating
|
||||
# either environment's key silently breaks the other; that is the structural reason for the split,
|
||||
# independent of dev's eventual decommission.
|
||||
#
|
||||
# THE MECHANISM. Each credential is now TWO Secret objects, both applied to BOTH clusters (they are
|
||||
# base resources; nothing here forks a manifest per environment). The consumer names one:
|
||||
#
|
||||
# velero: credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
# CNPG: s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
#
|
||||
# ONE knob, not four, ON PURPOSE. Four independent secret-name knobs could be updated three-
|
||||
# quarters of the way, leaving one consumer pointed at the OTHER environment's key with nothing
|
||||
# rendering wrong — and StrictPostBuildSubstitutions cannot catch that, because it only fails on an
|
||||
# UNDEFINED variable, never on a defined-but-stale one. With a single suffix the four move together
|
||||
# or not at all. The cost is that the rendered Secret name is not greppable from the consumer; the
|
||||
# names are spelled out in full in each kustomization.yaml beside the resources.
|
||||
#
|
||||
# THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names,
|
||||
# which are the objects that already exist and that the live dev cluster is running. A default
|
||||
# naming a Secret that does not exist would be worse than the shared-Secret state it replaces:
|
||||
# Velero's pod would sit in ContainerCreating forever on a missing `existingSecret` volume, and a
|
||||
# CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — i.e. the disk-full clock
|
||||
# again — so an unresolvable default breaks the referencing cluster HARDER than a wrong key does.
|
||||
# Empty is also the value dev must carry, so base and dev agree, exactly like VELERO_BACKUP_PREFIX.
|
||||
#
|
||||
# An EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions: strict mode fails on a variable that is
|
||||
# not DEFINED, not on one defined as "". VELERO_BACKUP_PREFIX above has shipped empty since the
|
||||
# gate was enabled and proves it. Both overlays state this knob explicitly — dev "" and prod
|
||||
# "-prod" — so neither inherits it silently.
|
||||
#
|
||||
# ADDING A NEW BACKUP CREDENTIAL: create BOTH halves (`x-credentials.sops.yaml` and
|
||||
# `x-credentials-prod.sops.yaml`), list both in the kustomization, and reference
|
||||
# `x-credentials${BACKUP_CREDENTIALS_SUFFIX}`. Never a single shared Secret again.
|
||||
BACKUP_CREDENTIALS_SUFFIX: "" # dev: "" (MinIO creds) | prod: "-prod" (R2 creds)
|
||||
|
||||
# --- placement ---
|
||||
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
|
||||
PLATFORM_NODE_SELECTOR_VALUE: "linux"
|
||||
@@ -5,8 +5,24 @@ apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- flux-system.yaml # GitRepository (reconciliation source)
|
||||
# Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
|
||||
# of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
|
||||
# Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
|
||||
# `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
|
||||
# interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
|
||||
# freeze the other eleven platform components. See clusters/base/cilium.yaml.
|
||||
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
||||
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
||||
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
||||
# SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
|
||||
# manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
|
||||
# datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
|
||||
# platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
|
||||
# keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
|
||||
# CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
|
||||
# cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
|
||||
# as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
|
||||
- platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
|
||||
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
|
||||
# Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
|
||||
# (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
|
||||
@@ -34,6 +50,13 @@ resources:
|
||||
# reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
|
||||
# non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
|
||||
- crafty.yaml # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
|
||||
# Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34
|
||||
# runbooks the alert rules' `runbook_url` annotations point at, at <apex>/runbooks/<name>; before
|
||||
# this tier existed that path was a 404 and every alert handed the operator a dead link. Same
|
||||
# non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on
|
||||
# purpose — it renders documentation containing `${...}` shell examples that envsubst would empty;
|
||||
# see clusters/base/runbooks.yaml.
|
||||
- runbooks.yaml # Flux Kustomization -> ./runbooks (dependsOn platform-config)
|
||||
# Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
|
||||
# controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
|
||||
# Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
|
||||
# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
|
||||
#
|
||||
# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
|
||||
#
|
||||
# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
|
||||
# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
|
||||
# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
|
||||
# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
|
||||
# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
|
||||
# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
|
||||
# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
|
||||
#
|
||||
# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
|
||||
# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
|
||||
# graph order itself, so a cold cluster needs no human in the loop.
|
||||
#
|
||||
# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
|
||||
# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
|
||||
# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
|
||||
# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
|
||||
# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
|
||||
# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
|
||||
# kind. One edge, no pre-apply.
|
||||
#
|
||||
# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
|
||||
# and everything else waits on: putting CDI's health on that critical path is the mistake the
|
||||
# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
|
||||
# StorageProfiles that were in the wrong pass.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: platform-config-cdi
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
# 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
|
||||
# very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
|
||||
# now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
|
||||
# about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
|
||||
# cache is one retryInterval. Keep it short.
|
||||
retryInterval: 2m
|
||||
# 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
|
||||
# image to pull and no controller to become Available, so a slow apply here means a real error, not
|
||||
# a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
|
||||
timeout: 2m
|
||||
# TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
|
||||
# component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
|
||||
# It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
|
||||
# second path also declares. See the transition note at the bottom of this file.
|
||||
prune: true
|
||||
# TRUE, but understand exactly what it buys and what it does NOT.
|
||||
#
|
||||
# CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
|
||||
# provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
|
||||
# `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
|
||||
# it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
|
||||
# is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
|
||||
# If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
|
||||
# `apps-vms` waits on it.
|
||||
#
|
||||
# WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
|
||||
# provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
|
||||
# storageprofile-controller computes that status from our spec override plus inference. So Ready
|
||||
# here means "the spec override is accepted by the API server", not "CDI has folded it into
|
||||
# status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
|
||||
# below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
|
||||
# a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
|
||||
wait: true
|
||||
dependsOn:
|
||||
# platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
|
||||
# causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
|
||||
# ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
|
||||
# flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
|
||||
- name: platform-config
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform-config-cdi
|
||||
# NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
|
||||
# no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
|
||||
# oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
|
||||
# clusters/dev or clusters/prod for this tier.
|
||||
#
|
||||
# substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
|
||||
# substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
|
||||
# 1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
|
||||
# whose Kustomization declares substituteFrom. Omitting it would move
|
||||
# cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
|
||||
# could ship as a literal string, precisely the class of bug that check exists to stop.
|
||||
# 2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
|
||||
# with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
|
||||
# empty. Without the stanza there is no substitution at all and the literal ships silently.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
|
||||
# Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
|
||||
# Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
|
||||
# StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
|
||||
# the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
|
||||
# dual-ownership fight — the objects are pruned before they are re-applied under this tier's
|
||||
# `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
|
||||
#
|
||||
# Why that is acceptable here — and it is NOT the platform/cilium case:
|
||||
# * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
|
||||
# nothing else in the repo declares a StorageProfile. The stale set is exactly two
|
||||
# cluster-scoped CDI CRs.
|
||||
# * The objects do not actually go missing: CDI's storageprofile-controller creates a
|
||||
# StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
|
||||
# inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
|
||||
# absent".
|
||||
# * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
|
||||
# existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
|
||||
# is disturbed and no data path is touched.
|
||||
# * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
|
||||
# instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
|
||||
# permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
|
||||
# dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
|
||||
# created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
|
||||
# could.
|
||||
#
|
||||
# PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
|
||||
# happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
|
||||
# config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
|
||||
# healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
|
||||
# --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
|
||||
# StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
|
||||
# `flux get kustomization platform-config` must be Ready BEFORE merging.
|
||||
# If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
|
||||
# temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
|
||||
# this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
|
||||
# `prune: true` in a second commit.
|
||||
@@ -11,7 +11,18 @@ spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true # drift correction: delete resources removed from git (R-GITOPS-03)
|
||||
# Cilium was moved OUT of this tier into its own Kustomization (clusters/base/cilium.yaml)
|
||||
# on 2026-09-01. `prune: true` is restored here now that the ownership transfer is VERIFIED
|
||||
# live: Namespace/kube-system, the Cilium HelmRelease and the HelmRepository all carry
|
||||
# `kustomize.toolkit.fluxcd.io/name=cilium`, this Kustomization's inventory no longer lists
|
||||
# any of the three, and the `cilium` Kustomization's inventory holds exactly those three.
|
||||
#
|
||||
# LESSON WORTH KEEPING: `platform/cilium/namespace.yaml` was the ONLY declaration of
|
||||
# Namespace/kube-system under ./platform. Moving that path out of a prune:true Kustomization
|
||||
# made the stale set `delete namespace kube-system` — coredns, metrics-server, local-path AND
|
||||
# Cilium. Before moving ANY platform/* component in future, check whether it is the sole
|
||||
# declarer of a shared Namespace; that turns a component-scoped prune into a cluster-scoped one.
|
||||
prune: true
|
||||
wait: true # block `apps` until platform is healthy
|
||||
# ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
|
||||
#
|
||||
@@ -40,6 +51,13 @@ spec:
|
||||
# health-gates its own children (see clusters/dev/sync.yaml).
|
||||
dependsOn:
|
||||
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
|
||||
# THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
|
||||
# (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
|
||||
# suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
|
||||
# nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
|
||||
# is healthy. It is also revision-gated, which is what makes the prune-safety argument above
|
||||
# deterministic rather than a race.
|
||||
- name: cilium
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
# Flux Kustomization: alert runbooks — reconciles ./runbooks.
|
||||
#
|
||||
# WHAT THIS SERVES. Every alert rule in this repo carries
|
||||
# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names.
|
||||
# Until this tier existed that path returned 404 on the live cluster, so every alert that reached a
|
||||
# human handed them a dead link at the moment they needed help. This Kustomization renders the
|
||||
# markdown under ./runbooks into a ConfigMap and a two-replica nginx that serves it at that exact
|
||||
# prefix. See runbooks/README.md and runbooks/serving/httproute.yaml.
|
||||
#
|
||||
# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false) — the same placement and
|
||||
# the same reasoning as homepage/headlamp/oauth2-proxy. It needs the wildcard TLS cert, the shared
|
||||
# Gateway (including the `https-apex` listener) and split-horizon DNS, all of which arrive with
|
||||
# `platform` + `platform-config`; and a leaf web server must never be able to stall the platform
|
||||
# chain, which is the whole lesson of DECISIONS-LOG's "Headlamp gating stall". wait:false guarantees
|
||||
# its health can never gate anything even if something is later mistakenly pointed dependsOn it.
|
||||
#
|
||||
# NO postBuild.substituteFrom — THE ONE DELIBERATE DIVERGENCE FROM EVERY OTHER TIER, AND THE MOST
|
||||
# IMPORTANT LINE IN THIS FILE.
|
||||
# This tier renders the runbook PROSE into a ConfigMap, and runbooks are full of shell: `$POD`,
|
||||
# `$(kubectl ...)`, and — unavoidably, since they document these very annotations — the literal text
|
||||
# `${BASE_DOMAIN}`. Flux's postBuild substitution runs envsubst over the ENTIRE built stream, and it
|
||||
# FAILS OPEN: an undefined `${X}` becomes the empty string while the Kustomization still reports
|
||||
# Ready (DECISIONS-LOG 2026-09-01 — that is how `k8sServiceHost` was emptied and Cilium broke on 2
|
||||
# of 3 dev nodes). Enabled here it would silently mutilate the documentation: every braced expansion
|
||||
# in every command example blanked, in a file nobody re-reads after writing. Worse, the L0 check
|
||||
# `conformance/tests/l0-static/flux-postbuild-substitution` scans only files reachable through
|
||||
# kustomization `resources:`, not `configMapGenerator` inputs, so the damage would be invisible to
|
||||
# the gate as well.
|
||||
# The fix is to have nothing to substitute. The only knob this tier would have wanted is the apex
|
||||
# hostname, and runbooks/serving/httproute.yaml avoids needing it by declaring NO `hostnames:` and
|
||||
# inheriting the `https-apex` listener's own hostname instead. So this tier is
|
||||
# environment-independent by construction and R-ENV-02 ("authored once, promoted dev -> prod") holds
|
||||
# without substitution rather than because of it.
|
||||
# IF A FUTURE KNOB EVER GENUINELY NEEDS SUBSTITUTION HERE: do not add substituteFrom to this
|
||||
# Kustomization. Split the serving manifests into their own substituted tier and leave the content
|
||||
# ConfigMap in an unsubstituted one, or the runbooks lose their shell examples.
|
||||
#
|
||||
# NO SOPS decryption stanza: this tier carries no encrypted Secret (markdown plus five plain
|
||||
# manifests), so — like homepage — neither clusters/dev nor clusters/prod patches it.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: runbooks
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config # wildcard TLS + Gateway (incl. the https-apex listener) first
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./runbooks
|
||||
@@ -106,8 +106,11 @@ patches:
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# velero (backup controller) has its own post-platform Flux Kustomization (clusters/base/backup.yaml).
|
||||
# It carries velero-cloud.sops.yaml (the scoped MinIO `cloud` credential), so it needs its own SOPS
|
||||
# decryption stanza (same as headlamp/oauth2-proxy/crafty). The `sops-age` Secret is out-of-band (name only).
|
||||
# It carries BOTH halves of the per-env `cloud` INI credential — velero-cloud.sops.yaml (dev's scoped
|
||||
# MinIO key) and velero-cloud-prod.sops.yaml (prod's R2 token) — so it needs its own SOPS decryption
|
||||
# stanza (same as headlamp/oauth2-proxy/crafty). Which one the Deployment mounts is chosen by
|
||||
# BACKUP_CREDENTIALS_SUFFIX in the cluster-vars patch below, not here. The `sops-age` Secret is
|
||||
# out-of-band (name only).
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
@@ -148,6 +151,33 @@ patches:
|
||||
- op: replace
|
||||
path: /data/ACME_CLUSTER_ISSUER
|
||||
value: "letsencrypt-prod"
|
||||
# --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
|
||||
# DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
|
||||
# like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
|
||||
# Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
|
||||
# Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
|
||||
# (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
|
||||
# prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
|
||||
# not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
|
||||
# not a variable change: infra/PROD-PROVISIONING.md §10.
|
||||
- op: replace
|
||||
path: /data/BASE_DOMAIN
|
||||
value: "home.keithjackson.dev"
|
||||
# ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
|
||||
# and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
|
||||
# the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_DOMAIN_FILTER
|
||||
value: "keithjackson.dev"
|
||||
# DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
|
||||
# registry under owner `default` (the external-dns chart default, which is what both clusters
|
||||
# silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
|
||||
# BY this string: any other value and it stops recognising every record it manages, then
|
||||
# re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
|
||||
# whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_TXT_OWNER_ID
|
||||
value: "default"
|
||||
# --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
|
||||
# EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
|
||||
# equals the base default. These values are what the LIVE dev cluster is running right now, so
|
||||
@@ -160,11 +190,25 @@ patches:
|
||||
- op: replace
|
||||
path: /data/GATEWAY_VIP
|
||||
value: "192.168.1.53"
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials — the primary control-plane node
|
||||
# (k3s-1). [parity] equals the Ansible inventory's k3s_api_host for dev.
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials — DEV'S CONTROL-PLANE VIP,
|
||||
# the address kube-vip floats, NOT the primary node .50.
|
||||
# [parity] equals the Ansible inventory's k3s_apiserver_endpoint for dev
|
||||
# (= kube_vip_address, group_vars/all.yml). See clusters/base/cluster-vars.yaml for why.
|
||||
#
|
||||
# *** THE ONE EXCEPTION TO "NEVER LET THIS BLOCK DRIFT FROM THE LIVE CLUSTER" ***
|
||||
# Every other value in this block is what dev is running right now. This one is not:
|
||||
# the live dev cilium-agents dial 192.168.1.50 (k3s-1). Changing it here is a
|
||||
# DELIBERATE, OPERATOR-APPROVED move (2026-09-06) and reconciling it ROLLS THE CILIUM
|
||||
# AGENT DAEMONSET on dev — one node at a time (updateStrategy.maxUnavailable: 1, kept
|
||||
# in parity for exactly this reason). Do not push this to dev-live casually, and not
|
||||
# while dev's DNS migration is mid-flight; the L2 datapath survives an agent restart
|
||||
# but a wrong value here is the 2026-09-01 incident. .49 is in dev's serving-cert SANs
|
||||
# already (k3s_tls_sans) and dev's own kubeconfig validates against it via
|
||||
# tls-server-name, so TLS is not the risk — the risk is reachability, and .49 is on the
|
||||
# same LAN as the nodes and ARP-announced by kube-vip on every server.
|
||||
- op: replace
|
||||
path: /data/K3S_API_HOST
|
||||
value: "192.168.1.50"
|
||||
value: "192.168.1.49"
|
||||
# Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
|
||||
# (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
|
||||
- op: replace
|
||||
@@ -181,15 +225,81 @@ patches:
|
||||
- op: replace
|
||||
path: /data/CRAFTY_L4_VIP
|
||||
value: "192.168.1.55"
|
||||
# Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
|
||||
# (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
|
||||
# Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
|
||||
# Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
|
||||
# end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
|
||||
# change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
|
||||
- op: replace
|
||||
path: /data/BULK_NFS_SERVER
|
||||
value: "192.168.1.9"
|
||||
# Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
|
||||
# barmanObjectStore and Velero's backupStorageLocation write to.
|
||||
#
|
||||
# DEV IS DELIBERATELY STAYING ON THIS MinIO while prod goes to Cloudflare R2 (2026-09-07).
|
||||
# Prod could not use it at all — it cannot even route to it — but dev's move is a separate,
|
||||
# operator-gated decision, because R2 would start dev on an EMPTY prefix: Velero would write
|
||||
# a fresh kopia repository, CNPG would take new base backups, and the 30-day Barman window
|
||||
# would rebuild from zero. Dev's DR restore is the one PROVEN recovery path in this repo
|
||||
# (R-STATE-03); resetting its PITR window is not a side effect to take while fixing prod.
|
||||
- op: replace
|
||||
path: /data/S3_BACKUP_ENDPOINT
|
||||
value: "http://192.168.1.9:9000"
|
||||
# DO NOT CHANGE THESE FOUR. They are dev's LIVE object layout, restated verbatim for the same
|
||||
# reason as LONGHORN_DATA_PATH and EXTERNALDNS_TXT_OWNER_ID below/above: they equal the base
|
||||
# defaults today, and a silent drift in base would ORPHAN dev's existing backups — the ones
|
||||
# its proven restore depends on — without any manifest looking wrong.
|
||||
#
|
||||
# `us-east-1` is what dev's Velero has signed every request with since bootstrap; MinIO
|
||||
# ignores the value, but changing it is a needless risk against a live, working backup path.
|
||||
# An EMPTY VELERO_BACKUP_PREFIX means the bucket root, which is where dev's backups already
|
||||
# are — and the Velero chart omits `objectStorage.prefix` entirely when it is empty (verified
|
||||
# against chart 12.1.0), so dev's live BackupStorageLocation is byte-unchanged by this.
|
||||
- op: replace
|
||||
path: /data/S3_BACKUP_REGION
|
||||
value: "us-east-1"
|
||||
- op: replace
|
||||
path: /data/VELERO_BACKUP_BUCKET
|
||||
value: "velero"
|
||||
- op: replace
|
||||
path: /data/VELERO_BACKUP_PREFIX
|
||||
value: ""
|
||||
- op: replace
|
||||
path: /data/CNPG_BACKUP_PATH
|
||||
value: "s3://cnpg"
|
||||
# WHICH CREDENTIAL SIGNS THOSE REQUESTS. Empty suffix = the UNSUFFIXED Secrets
|
||||
# (velero-cloud-credentials, <db>-backup-credentials) — the scoped MinIO keys dev's Velero and
|
||||
# its three CNPG Clusters are running RIGHT NOW, byte-identical to what is on the cluster. Prod
|
||||
# renders "-prod" and reads the Cloudflare R2 token from the parallel `-prod` Secrets instead.
|
||||
#
|
||||
# RESTATED EXPLICITLY even though it equals the base default, for the same reason as
|
||||
# EXTERNALDNS_TXT_OWNER_ID and the four backup-layout knobs above: the four Secrets are SHARED
|
||||
# base manifests, so before this split prod's R2 token landed in the objects DEV reads. Dev
|
||||
# signing MinIO with an R2 key id is 403 on every barman-cloud-wal-archive, and archive_mode=on
|
||||
# turns that into a filling pgdata PVC — the disk-full clock, on the one cluster whose DR
|
||||
# restore is PROVEN (R-STATE-03). This line is what keeps dev on its own credential.
|
||||
#
|
||||
# DEV IS NOT BEING BACKED UP GOING FORWARD and is slated for decommission (operator,
|
||||
# 2026-09-07), so this is not protecting a future backup — it is protecting the EXISTING
|
||||
# backups and the WAL chain until dev is retired. Do not "simplify" it away while dev runs.
|
||||
- op: replace
|
||||
path: /data/BACKUP_CREDENTIALS_SUFFIX
|
||||
value: ""
|
||||
# dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
||||
value: "3"
|
||||
# DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
|
||||
# on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
|
||||
# mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
|
||||
# else would orphan every one of them (they are not recoverable from the old path by
|
||||
# rescheduling). Restated explicitly even though it equals the base default, exactly like the
|
||||
# LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
|
||||
# its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DATA_PATH
|
||||
value: "/var/lib/longhorn"
|
||||
# smaller retention / volumes to keep the dev workstation footprint light.
|
||||
- op: replace
|
||||
path: /data/MONITORING_PROMETHEUS_RETENTION
|
||||
|
||||
@@ -115,8 +115,20 @@ patches:
|
||||
name: sops-age
|
||||
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
|
||||
#
|
||||
# *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
|
||||
# ############################################################################################
|
||||
# ### ###
|
||||
# ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ###
|
||||
# ### ###
|
||||
# ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ###
|
||||
# ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ###
|
||||
# ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ###
|
||||
# ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ###
|
||||
# ### a live single point of failure, not a settled design. See "CUTOVER" below. ###
|
||||
# ### ###
|
||||
# ############################################################################################
|
||||
#
|
||||
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
|
||||
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
|
||||
# Two independent reasons, both consequences of the split-horizon design:
|
||||
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
|
||||
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
|
||||
@@ -126,13 +138,46 @@ patches:
|
||||
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
|
||||
# gitea.home.keithjackson.dev to itself and never reach a repository.
|
||||
#
|
||||
# Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
|
||||
# mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
|
||||
# sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
|
||||
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
|
||||
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
|
||||
# operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
|
||||
# reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
|
||||
# change was needed. This is the same "local bootstrap source" pattern dev itself used before
|
||||
# its Phase-6 cutover (docs/05 "Source-of-truth").
|
||||
#
|
||||
# PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
|
||||
# for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
|
||||
# the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
|
||||
# .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
|
||||
#
|
||||
# NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
|
||||
# works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
|
||||
# curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
|
||||
# git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main
|
||||
# If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
|
||||
# THAT FIRST: a private repo here fails exactly that way.
|
||||
#
|
||||
# *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
|
||||
# WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
|
||||
# through prod's own Gateway VIP with a publicly-trusted cert.
|
||||
# DO: 1. push/mirror this repo into the prod in-cluster Gitea;
|
||||
# 2. change the url below to
|
||||
# https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
||||
# and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
|
||||
# clusters/dev/kustomization.yaml, is what "done" looks like);
|
||||
# 3. confirm `flux get sources git -n flux-system` reconciles the new revision;
|
||||
# 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
|
||||
# exactly as dev retained its bootstrap Gitea.
|
||||
# UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
|
||||
# powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
|
||||
# it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
|
||||
#
|
||||
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
|
||||
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
|
||||
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
|
||||
# NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
|
||||
# Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
|
||||
# is the revision you intend prod to run (`git ls-remote` above, against local `main`).
|
||||
- patch: |-
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: GitRepository
|
||||
@@ -140,7 +185,56 @@ patches:
|
||||
name: flux-system
|
||||
namespace: flux-system
|
||||
spec:
|
||||
url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
||||
# CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles
|
||||
# from PROD'S OWN in-cluster Gitea. The workstation bootstrap container
|
||||
# (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the
|
||||
# operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a
|
||||
# laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh).
|
||||
#
|
||||
# THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization
|
||||
# renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of
|
||||
# GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves
|
||||
# too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward
|
||||
# its `main` to the exact commit prod was already running so the source swap changed nothing
|
||||
# else, (3) live-patch the url to open a window, (4) land THIS commit through that window so
|
||||
# the authored value agrees. Dev hit the identical trap on 2026-09-05
|
||||
# (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed").
|
||||
#
|
||||
# Verified before the switch: a throwaway pod INSIDE prod resolved this host through
|
||||
# in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA
|
||||
# supplied. The workstation being able to reach it proves nothing about pod egress.
|
||||
# MOVED WITH THE APEX FLIP (2026-09-13) — and it had to move in the SAME fetch.
|
||||
# This URL is a hardcoded literal, NOT templated on ${BASE_DOMAIN}, so the flip did not
|
||||
# carry it. But the flip DOES rename prod's own HTTPRoute hostnames, so:
|
||||
# - external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev, and
|
||||
# - platform/coredns/coredns-custom.yaml answers the whole ${BASE_DOMAIN} zone with
|
||||
# ${GATEWAY_VIP}, so in-cluster resolution moves from prod.home. to home.
|
||||
# Leaving this on the old name would therefore have severed prod from its OWN GitOps
|
||||
# source moments after the flip reconciled — self-reference, the same class of trap as
|
||||
# patching this url live and having the root Kustomization revert it. The fix has to land
|
||||
# THROUGH THE OLD SOURCE, which is why it is in the same push as the flip rather than
|
||||
# after it.
|
||||
# FOLLOW-UP worth doing: point this at the in-cluster Service
|
||||
# (gitea-http.gitea.svc.cluster.local) so prod's GitOps stops depending on public DNS and
|
||||
# on its own Gateway at all. Deferred: it needs the TLS/credential path rethought.
|
||||
# IN-CLUSTER SERVICE, NOT A PUBLIC HOSTNAME (2026-09-13). This is the durable fix for a
|
||||
# deadlock the apex flip walked straight into, and it is worth stating precisely.
|
||||
#
|
||||
# The url used to be a public name. The flip renamed prod's HTTPRoute hostnames, so the
|
||||
# name this GitRepository pointed at stopped serving — but the HTTPRoute that WOULD have
|
||||
# served the new name lives in a DOWNSTREAM tier, which cannot reconcile while the source
|
||||
# is broken. Source down -> routes never rename -> source stays down. Observed live:
|
||||
# `srcReady=False`, "unable to list remote for https://gitea.home.keithjackson.dev".
|
||||
#
|
||||
# Pointing at the Service breaks the cycle permanently: prod's GitOps no longer depends on
|
||||
# public DNS, on Cloudflare, on its own Gateway, or on ${BASE_DOMAIN} at all. A future
|
||||
# domain change cannot sever prod from its own source again.
|
||||
#
|
||||
# Plain http is deliberate and safe here: it is cluster-internal traffic to a ClusterIP,
|
||||
# TLS terminates at the Gateway for external clients only, the GitRepository has no
|
||||
# secretRef (the repo is cloned anonymously), and every secret in the tree is SOPS-encrypted
|
||||
# at rest regardless of transport.
|
||||
url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
|
||||
ref:
|
||||
branch: main
|
||||
- target:
|
||||
@@ -154,9 +248,97 @@ patches:
|
||||
- op: replace
|
||||
path: /data/ACME_CLUSTER_ISSUER
|
||||
value: "letsencrypt-prod"
|
||||
# ==========================================================================================
|
||||
# PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
|
||||
# ==========================================================================================
|
||||
# PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
|
||||
# home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
|
||||
# public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
|
||||
# of which already happened:
|
||||
# 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
|
||||
# default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
|
||||
# 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
|
||||
# wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
|
||||
# have raced each other's `_acme-challenge` TXT records on that one zone.
|
||||
# Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
|
||||
# record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
|
||||
#
|
||||
# dev -> home.keithjackson.dev (UNCHANGED — dev is live; phase 2 moves it)
|
||||
# prod -> prod.home.keithjackson.dev (this knob)
|
||||
#
|
||||
# CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
|
||||
# gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
|
||||
# on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
|
||||
# `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
|
||||
# would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
|
||||
#
|
||||
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
|
||||
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
|
||||
# set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
|
||||
# trusts a different Keycloak than the one prod actually runs.
|
||||
- op: replace
|
||||
path: /data/BASE_DOMAIN
|
||||
# APEX AS OF 2026-09-13. Prod now serves home.keithjackson.dev directly; dev has been
|
||||
# decommissioned and released the name. Prerequisites completed and VERIFIED before this
|
||||
# flip, in this order:
|
||||
# 1. dev's 18 A records released by its own external-dns (policy=sync, sources deleted)
|
||||
# 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand —
|
||||
# external-dns v0.21.0 removed only the `a-` prefixed form and then reported "All
|
||||
# records are already up to date", so it would NEVER have cleared them. They carried
|
||||
# `owner=default`, and prod runs `owner=prod`, so prod could not have claimed these
|
||||
# names while they existed. Confirmed gone at the AUTHORITY (betty.ns.cloudflare.com);
|
||||
# 1.1.1.1 served stale cache for minutes afterwards and must not be trusted here.
|
||||
# 3. dev's `*.home.keithjackson.dev` + apex Certificate deleted, so it stops renewing.
|
||||
# It was issued ~18d ago, so prod's request is #1 this week against LE's duplicate
|
||||
# limit of 5 per identical name set per week.
|
||||
# NOTE the zone has wildcards `*.keithjackson.dev` AND `*.home.keithjackson.dev` pointing
|
||||
# at the public IP 136.51.115.252 (legacy non-k3s prod). Dev's specific records were
|
||||
# SHADOWING that wildcard; prod's new specific records shadow it again. The legacy stack
|
||||
# lives on `*.keithjackson.dev` and is untouched by any of this.
|
||||
value: "home.keithjackson.dev"
|
||||
# SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
|
||||
# OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
|
||||
# TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
|
||||
# ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
|
||||
# the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
|
||||
# matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
|
||||
# records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
|
||||
# EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_DOMAIN_FILTER
|
||||
# MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
|
||||
# 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
|
||||
# record then skipped it with "no hosted zone matching record DNS Name", so prod would
|
||||
# publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
|
||||
# (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
|
||||
# (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
|
||||
# That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
|
||||
value: "keithjackson.dev"
|
||||
# SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
|
||||
# both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
|
||||
# its own and, under policy: sync, deleted the records behind them. This value MUST differ from
|
||||
# dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
|
||||
# cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_TXT_OWNER_ID
|
||||
value: "prod"
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
||||
value: "3"
|
||||
# Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
|
||||
# the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
|
||||
# and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
|
||||
# (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
|
||||
# measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
|
||||
# its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
|
||||
#
|
||||
# [parity] MUST equal node_baseline_longhorn_data_path in
|
||||
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
|
||||
# directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
|
||||
# path on the OS filesystem instead of the data partition and /var fills up.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DATA_PATH
|
||||
value: "/data/longhorn"
|
||||
# longer retention / larger volumes on prod NVMe.
|
||||
- op: replace
|
||||
path: /data/MONITORING_PROMETHEUS_RETENTION
|
||||
@@ -222,14 +404,32 @@ patches:
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_STOP
|
||||
value: "192.168.2.30"
|
||||
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
|
||||
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned
|
||||
# Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in
|
||||
# platform/cilium/helmrelease.yaml. Not yet decided.
|
||||
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
|
||||
#
|
||||
# *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
|
||||
# read "Not yet decided" while shipping .10 — the founder's own address — and that is what
|
||||
# the LIVE prod cluster is running: all three cilium-agents carry
|
||||
# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
|
||||
# 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
|
||||
# the datapath keeps running off Cilium's cache while no service or policy update lands, with
|
||||
# nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
|
||||
# control plane is, which is the whole reason kube-vip exists here.)
|
||||
#
|
||||
# It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
|
||||
# this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
|
||||
# equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
|
||||
# that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
|
||||
# `--server` join needs a host that is up before any VIP is floated. Two addresses, two
|
||||
# variables, deliberately.
|
||||
#
|
||||
# .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
|
||||
# LB-IPAM can never be handed the same IP — asserted by
|
||||
# conformance/tests/l0-static/ha-control-plane-as-code.
|
||||
- op: replace
|
||||
path: /data/K3S_API_HOST
|
||||
value: "192.168.2.10"
|
||||
value: "192.168.2.9"
|
||||
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
|
||||
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
|
||||
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
|
||||
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
|
||||
@@ -240,10 +440,146 @@ patches:
|
||||
- op: replace
|
||||
path: /data/CRAFTY_L4_VIP
|
||||
value: "192.168.2.22"
|
||||
# Off-cluster DR target. Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the
|
||||
# network — a prod backup written there is not off-cluster DR and would mix two environments'
|
||||
# PITR streams in one bucket. Operator supplies the prod endpoint (full URL incl. scheme+port).
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
|
||||
# radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
|
||||
#
|
||||
# THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
|
||||
# vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
|
||||
# 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
|
||||
# 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
|
||||
# outcomes give DIFFERENT values here:
|
||||
# port-forward 192.168.2.102:2049 -> 192.168.1.9:2049 => "192.168.2.102"
|
||||
# a WAN->LAN route to 192.168.1.0/24 => "192.168.1.9"
|
||||
# Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
|
||||
# ever became reachable — mounts something unintended. So it stays a placeholder.
|
||||
#
|
||||
# WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
|
||||
# NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
|
||||
# over v3 was in part this decision's cost: v3 would have needed four ports and three of them
|
||||
# on ephemeral numbers.
|
||||
#
|
||||
# WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
|
||||
# Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
|
||||
# the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
|
||||
# address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
|
||||
# 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
|
||||
# addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
|
||||
#
|
||||
# *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
|
||||
# A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
|
||||
# placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
|
||||
# failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
|
||||
# TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
|
||||
# ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
|
||||
# apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
|
||||
# a revision containing the bulk volumes; the dev overlay renders a working value, which is
|
||||
# where these manifests are meant to be proven first.
|
||||
- op: replace
|
||||
path: /data/BULK_NFS_SERVER
|
||||
# 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
|
||||
# host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
|
||||
# reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
|
||||
# keeps the base value 192.168.1.9.
|
||||
#
|
||||
# PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
|
||||
# - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
|
||||
# - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
|
||||
# carries no_root_squash, so a root write would have passed and proved nothing, while
|
||||
# LSIO images default to PUID 911 and would have gone Ready then failed every import
|
||||
# with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
|
||||
# - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
|
||||
# real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
|
||||
# pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
|
||||
# so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
|
||||
value: "192.168.2.102"
|
||||
# ==========================================================================================
|
||||
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
|
||||
# ==========================================================================================
|
||||
# Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the network and is NOT an
|
||||
# option for prod, for two independent reasons:
|
||||
# 1. UNREACHABLE. Prod is on the SOHO network 192.168.2.0/24 — pfSense's WAN — while that
|
||||
# MinIO is inside pfSense's LAN. Measured from a pod INSIDE prod (the workstation proves
|
||||
# nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` → connection
|
||||
# timed out after 15s. Reaching it would need a WAN→LAN rule that does not exist.
|
||||
# 2. NOT OFF-CLUSTER DR. It runs on the same Proxmox host as dev's VMs, so it already shares
|
||||
# a physical failure domain with dev. R2 is genuinely off-site, and its free egress means
|
||||
# restore REHEARSALS cost nothing — R-STATE-03 has no financial disincentive to stay proven.
|
||||
#
|
||||
# THIS UNBLOCKS AN ACTIVE INCIDENT, not just a roadmap item. All three prod CNPG Clusters ship
|
||||
# `archive_mode=on` with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, so barman-cloud-wal-archive
|
||||
# exits 4 on every segment and Postgres — correctly — REFUSES TO RECYCLE WAL. Verified live
|
||||
# 2026-09-07: keycloak-db / nextcloud-db / semaphore-db all report
|
||||
# ContinuousArchiving=False/ContinuousArchivingFailing, and keycloak-db's 5 GiB PVC is at
|
||||
# 579 MiB used / 4360 MiB free and climbing monotonically. A disk-full PANIC there takes the
|
||||
# cluster's SSO database down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and
|
||||
# forward-auth. Fixing the endpoint lets Postgres drain the whole backlog.
|
||||
#
|
||||
# ENDPOINT IS ACCOUNT-SCOPED, so it stays a prod-overlay value even though it is not a LAN
|
||||
# address: this is the operator's Cloudflare account id, confirmed from inside a prod pod
|
||||
# (`head-bucket` on k3s-prod-backup → BucketRegion WNAM; PUT and DELETE both succeed).
|
||||
# It is NOT put in base, because base is dev's live MinIO and dev is deliberately STAYING
|
||||
# there — moving dev to R2 would reset its PITR window and orphan the backups its PROVEN
|
||||
# restore depends on. Two environments, two targets, isolated by URL *and* by bucket.
|
||||
- op: replace
|
||||
path: /data/S3_BACKUP_ENDPOINT
|
||||
value: "TBD-PROD-S3-BACKUP-ENDPOINT"
|
||||
value: "https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com"
|
||||
# R2 requires the region string to be exactly `auto`. (`head-bucket` answers
|
||||
# `{"BucketRegion":"WNAM"}` — that is a PHYSICAL LOCATION HINT, not a signing region; using it
|
||||
# here produces AuthorizationHeaderMalformed.) Dev keeps the base `us-east-1` its live MinIO
|
||||
# has always been signed with.
|
||||
- op: replace
|
||||
path: /data/S3_BACKUP_REGION
|
||||
value: "auto"
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# OBJECT TENANCY INSIDE THE ONE BUCKET. The operator created a SINGLE R2 bucket, so velero and
|
||||
# barman cannot be separated by bucket the way they are on dev's MinIO (`velero` and `cnpg`).
|
||||
# They are separated by disjoint TOP-LEVEL PREFIXES instead:
|
||||
#
|
||||
# s3://k3s-prod-backup/velero/… Velero: kopia repo + backup metadata
|
||||
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG: WAL + base backups
|
||||
#
|
||||
# The bucket ROOT is left empty on purpose — a misrendered prefix then lands somewhere
|
||||
# obviously wrong instead of silently colliding with a live tenant. Velero's retention TTL
|
||||
# deletes objects under its own prefix only, so it can never reap a WAL segment; and barman's
|
||||
# per-cluster serverName subdirectory keeps the three Postgres servers' chains apart, which
|
||||
# matters because interleaving two servers' WAL into one prefix corrupts the chain outright.
|
||||
# Dev cannot be reached from any of these: different endpoint AND different bucket names.
|
||||
# ------------------------------------------------------------------------------------------
|
||||
- op: replace
|
||||
path: /data/VELERO_BACKUP_BUCKET
|
||||
value: "k3s-prod-backup"
|
||||
- op: replace
|
||||
path: /data/VELERO_BACKUP_PREFIX
|
||||
value: "velero"
|
||||
- op: replace
|
||||
path: /data/CNPG_BACKUP_PATH
|
||||
value: "s3://k3s-prod-backup/barman"
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# WHICH CREDENTIAL SIGNS THE REQUESTS TO THAT BUCKET (per-env credential split, 2026-09-07).
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# The knobs above moved the ENDPOINT and the OBJECT LAYOUT per environment. The credential did
|
||||
# not move with them, and it could not: the four backup-credential Secrets are SHARED BASE
|
||||
# manifests, applied to both clusters by the same Flux tiers. Writing the R2 token into them
|
||||
# gave prod a working key by TAKING DEV'S AWAY — dev would have signed MinIO requests with an
|
||||
# R2 key id, and under archive_mode=on that is a filling pgdata PVC, not a missing backup.
|
||||
#
|
||||
# So each credential is now two objects, and this suffix selects prod's:
|
||||
# velero-cloud-credentials-prod the R2 token in the AWS INI `cloud` shape
|
||||
# <db>-backup-credentials-prod the same token as ACCESS_KEY_ID / ACCESS_SECRET_KEY
|
||||
# for keycloak-db, nextcloud-db and semaphore-db. Both halves ship to both clusters (they are
|
||||
# base resources); ONLY this value decides which one each consumer reads. It is a SUFFIX rather
|
||||
# than four secret-name knobs deliberately — four could be updated three-quarters of the way,
|
||||
# leaving one consumer on dev's MinIO key with nothing rendering wrong, and strict-mode
|
||||
# substitution cannot catch a defined-but-stale variable.
|
||||
#
|
||||
# THE R2 TOKEN IS THE ONE PROVEN FROM INSIDE PROD (chore/r2-credentials 0a6a862): head-bucket on
|
||||
# k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Rotating it now
|
||||
# touches ONLY the four `-prod` files; dev's MinIO keys are untouched objects. That isolation is
|
||||
# the structural reason for the split, independent of dev's eventual decommission.
|
||||
- op: replace
|
||||
path: /data/BACKUP_CREDENTIALS_SUFFIX
|
||||
value: "-prod"
|
||||
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
|
||||
@@ -8,3 +8,22 @@ _artifacts/
|
||||
|
||||
# Python bytecode from lib/ (status.py imports coverage.py); never committed.
|
||||
__pycache__/
|
||||
|
||||
# --- THE ONE EXCEPTION: the committed run ledger (runs/) -------------------------------
|
||||
# `runs/<UTC>-<short-sha>/{report.junit.xml,run.log}` + `runs/ledger.jsonl` are the
|
||||
# machine-readable proof-of-pass record that lib/status.py's docstring asks for. They are
|
||||
# SOURCE, not regenerated artifacts: a run's evidence cannot be reconstructed after the
|
||||
# fact, so it must survive in git.
|
||||
#
|
||||
# The negations are needed because the ROOT .gitignore ignores `*.junit.xml` globally
|
||||
# (and `coverage/`). A deeper .gitignore overrides a shallower one, and nothing ignores
|
||||
# the `conformance/runs/` directory itself, so re-including the files here works (git
|
||||
# cannot re-include a file whose PARENT DIRECTORY is excluded — that is why this must not
|
||||
# live under _artifacts/).
|
||||
#
|
||||
# The run LOG is the load-bearing half: Chainsaw's JUnit carries no stdout, and the L2/L3
|
||||
# tests self-skip GREEN, so the sentinel token in run.log is the only thing separating
|
||||
# "proven" from "skipped". Never commit a report.junit.xml without its run.log.
|
||||
!runs/
|
||||
!runs/**
|
||||
!runs/**/*.junit.xml
|
||||
@@ -22,6 +22,10 @@ behavioral / L3 fault-injection** (this harness).
|
||||
./run.sh --lint # structural-validate test defs (no cluster)
|
||||
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
./run.sh --require-all # phase-exit gate: every requirement must be covered
|
||||
./run.sh --record # run AND capture the evidence (see "Proof of pass")
|
||||
./run.sh --layer l0 --include-scope whole-repo # union: the tagged subset PLUS every test
|
||||
# declaring `conformance.home.k3s/scope: whole-repo`
|
||||
./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing
|
||||
```
|
||||
|
||||
**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
|
||||
@@ -31,6 +35,29 @@ behavioral / L3 fault-injection** (this harness).
|
||||
- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
|
||||
those touched requirements are covered (a listed safety-critical one must have an L3 test).
|
||||
- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
|
||||
- `--include-scope whole-repo` — **the escape hatch scoping needs**, because scoping has one
|
||||
blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE
|
||||
tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees
|
||||
the coarse path→requirement map keys to **nothing** (`apps/`, `infra/`, most of
|
||||
`platform/`). For those, *the change most likely to break the invariant attributes no
|
||||
requirement at all*, so requirement-scoped selection filters the check out of exactly that
|
||||
change. Such a test **declares** its scope:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
```
|
||||
|
||||
and `--include-scope whole-repo` unions every declaring test into the selection. The CI
|
||||
stages (`ci/l0.sh`, `ci/l1.sh`, `ci/conformance.sh`) all pass it, so **adding the
|
||||
annotation is the only step** — there is nothing to remember and nothing to hand-write.
|
||||
Two consequences, both deliberate: a declaring test always runs, and a **SKIP of a
|
||||
declaring test is UNWIRED, not green** (the runner exits 3, which `ci/l0.sh` turns into the
|
||||
gate's own rc 3 — never a pass). `tests/l0-static/whole-repo-invariants-wired` proves the
|
||||
mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a
|
||||
hand-written "UNCONDITIONAL" stage per check in `ci/l0.sh`; see `lib/common.sh` for the
|
||||
full rationale and the incidents behind it.
|
||||
- Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
|
||||
state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
|
||||
are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
|
||||
@@ -105,13 +132,89 @@ lib/
|
||||
common.sh tagging convention + test discovery/selection helpers
|
||||
coverage.py requirement-coverage generator (stdlib only)
|
||||
status.py requirement-STATUS generator (stdlib only; --selftest)
|
||||
record_run.py run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
|
||||
emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster)
|
||||
runs/
|
||||
ledger.jsonl TRACKED append-only proof-of-pass ledger (one row per
|
||||
testcase per run) — the only source status.py trusts
|
||||
<UTC>-<short-sha>/ TRACKED per-run evidence: run.log + report.junit.xml
|
||||
tests/
|
||||
l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md)
|
||||
incluster/
|
||||
chainsaw-job.yaml rbac.yaml in-cluster L2/L3 Job + scoped test SA (ADR-0011)
|
||||
```
|
||||
|
||||
## Proof of pass — the run ledger (`runs/`)
|
||||
|
||||
`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
|
||||
proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
|
||||
root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
|
||||
only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
|
||||
asked for — **a committed, machine-readable run ledger**.
|
||||
|
||||
```bash
|
||||
./run.sh --record # runs normally, and captures the evidence
|
||||
# -> conformance/runs/<UTC>-<short-sha>/run.log (tee'd, carries the sentinels)
|
||||
# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml (that run's JUnit)
|
||||
# -> conformance/runs/ledger.jsonl (one appended row per testcase)
|
||||
python3 lib/record_run.py --run-dir runs/<id> [--dry-run] # or record an existing run dir
|
||||
python3 lib/record_run.py --selftest # deterministic, no cluster
|
||||
```
|
||||
|
||||
**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
|
||||
that silently records itself is how a bad run becomes a permanent false proof. **Commit
|
||||
the run dir and the ledger** — an uncommitted ledger proves nothing.
|
||||
|
||||
### Why a committed JUnit alone would be WORSE than nothing
|
||||
|
||||
The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
|
||||
`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
|
||||
and `exit 0`, and their `check:` accepts either outcome —
|
||||
|
||||
```yaml
|
||||
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
|
||||
|| contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
|
||||
```
|
||||
|
||||
So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
|
||||
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
|
||||
A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
|
||||
R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.
|
||||
|
||||
**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
|
||||
`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
|
||||
`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
|
||||
verdict token), so the table below is never hand-maintained:
|
||||
|
||||
| Test | pass token (proof) | skip token (never proof) |
|
||||
|---|---|---|
|
||||
| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
|
||||
| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
|
||||
| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
|
||||
| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
|
||||
| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
|
||||
| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
|
||||
| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |
|
||||
|
||||
A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
|
||||
token present in that test's log lines, and **no** skip token anywhere in them. A skip
|
||||
token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
|
||||
`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
|
||||
non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
|
||||
skips as JUnit `<skipped/>`, never as a green pass.
|
||||
|
||||
### Staleness — a proof is a claim about a tree, not a badge
|
||||
|
||||
`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
|
||||
last commit touching that requirement's implementing manifests or its verifying test**.
|
||||
Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
|
||||
dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
|
||||
it never counts. Re-run with `--record` to refresh.
|
||||
|
||||
A requirement reads `proven` only when **every** verifying test does; one skipped, failed
|
||||
or unrecorded test drags the whole requirement down. With an empty ledger `--status`
|
||||
behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.
|
||||
|
||||
## How CI invokes it
|
||||
CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
|
||||
|
||||
|
||||
@@ -57,6 +57,13 @@ rules:
|
||||
verbs: [create, delete, get, list, watch, patch]
|
||||
- apiGroups: [""]
|
||||
resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
|
||||
# LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
|
||||
# agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
|
||||
# oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
|
||||
# time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
|
||||
# chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
|
||||
# If this rule is ever narrowed to the test's own namespace, those bindings fail and the
|
||||
# tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
|
||||
verbs: [create, delete, get, list, watch, patch]
|
||||
- apiGroups: [""]
|
||||
resources: [pods/exec]
|
||||
|
||||
@@ -47,6 +47,62 @@ conf_dirs_for_requirements() {
|
||||
done < <(conf_all_test_files) | sort -u
|
||||
}
|
||||
|
||||
# --- DECLARED SCOPE: whole-repo invariants -------------------------------------------
|
||||
# A test may also declare the SCOPE of the invariant it asserts:
|
||||
#
|
||||
# metadata:
|
||||
# annotations:
|
||||
# conformance.home.k3s/scope: "whole-repo"
|
||||
#
|
||||
# WHY THIS EXISTS — the requirement-scoped-selection blind spot (R-TEST-03)
|
||||
# ------------------------------------------------------------------------
|
||||
# The merge gate runs the conformance subset tagged with the requirements a change
|
||||
# TOUCHES (ci/touched-requirements.sh -> `--requirements`). That is right for an invariant
|
||||
# whose input surface is the subtree its requirements map to. It is WRONG — silently, and in
|
||||
# the worst direction — for an invariant that quantifies over the WHOLE tracked tree, or over
|
||||
# subtrees (apps/, platform/<most>, infra/) that the coarse path->requirement map keys to
|
||||
# NOTHING. For those, the change most likely to break the invariant attributes no requirement
|
||||
# at all, so the requirement-scoped selection filters the check out of EXACTLY that change.
|
||||
# Three real incidents came out of that blind spot; each is recorded in the header of the
|
||||
# test that now guards it (flux-postbuild-substitution, keycloak-client-secret-reconciler,
|
||||
# bulk-tier-distinct-from-replicated).
|
||||
#
|
||||
# The repo's previous answer was a hand-written "UNCONDITIONAL" stage per such test in
|
||||
# ci/l0.sh — which does not scale and fails silently in the worst way: whoever adds the NEXT
|
||||
# whole-repo invariant must REMEMBER to hand-write a stage, and if they forget, the check runs
|
||||
# only when some unrelated requirement happens to be touched. The scope annotation replaces
|
||||
# that with a DECLARATION the runner enforces: `run.sh --include-scope whole-repo` unions
|
||||
# every test carrying it into the selection, so adding the annotation is the only step.
|
||||
#
|
||||
# TWO things follow from the annotation, and both are deliberate:
|
||||
# 1. the test is ALWAYS selected (union with the requirement-scoped subset), and
|
||||
# 2. a SKIP of that test is UNWIRED, NOT GREEN (lib/run_l1_offline.py --unwired-scope).
|
||||
# A whole-repo invariant that reports "nothing to audit" is reporting that the subtree
|
||||
# it audits has VANISHED, which is not a pass. This is the rc-3 semantics the bespoke
|
||||
# stages encoded ("check SKIPPED (clusters/ absent) — not a pass"), preserved verbatim.
|
||||
export CONF_SCOPE_ANNOTATION="${CONF_LABEL_PREFIX}/scope"
|
||||
|
||||
# Directories of tests DECLARING the given scope (e.g. "whole-repo"). The annotation is
|
||||
# matched on a real YAML mapping line only: a `#` comment cannot start with the key, so a
|
||||
# scope named in a test's header prose is documentation, not a declaration (the same
|
||||
# "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs).
|
||||
conf_dirs_for_scope() {
|
||||
local scope="$1" f
|
||||
[ -n "$scope" ] || return 0
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
if grep -Eq "^[[:space:]]*${CONF_SCOPE_ANNOTATION}:[[:space:]]*[\"']?${scope}[\"']?[[:space:]]*$" "$f"; then
|
||||
dirname "$f"
|
||||
fi
|
||||
done < <(conf_all_test_files) | sort -u
|
||||
}
|
||||
|
||||
# Union of conf_dirs_for_scope over a comma/space separated list of scopes.
|
||||
conf_dirs_for_scopes() {
|
||||
local scopes="$1" s
|
||||
for s in ${scopes//,/ }; do conf_dirs_for_scope "$s"; done | sort -u
|
||||
}
|
||||
|
||||
# Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
|
||||
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
|
||||
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
|
||||
|
||||
Executable
+956
@@ -0,0 +1,956 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Append a conformance RUN LEDGER row per testcase — the committed proof-of-pass record.
|
||||
|
||||
WHY THIS EXISTS
|
||||
---------------
|
||||
`lib/status.py` reports PROOF as `not-derivable` for every requirement, and names its own
|
||||
fix in its docstring: *"a committed, machine-readable run ledger"*. There was none, because
|
||||
`conformance/_artifacts/` is git-ignored and the root `.gitignore` kills `*.junit.xml` and
|
||||
`coverage/` globally, so nothing about a green run ever reached repo state. This tool is
|
||||
that ledger's writer: it reads one run's JUnit + run log out of `conformance/runs/<run-id>/`
|
||||
and appends one JSON object per testcase to `conformance/runs/ledger.jsonl`, which IS
|
||||
tracked (see conformance/.gitignore for the negations that let it survive).
|
||||
|
||||
THE TRAP THIS TOOL EXISTS TO AVOID — READ BEFORE CHANGING ANYTHING
|
||||
------------------------------------------------------------------
|
||||
The L2/L3 tests are RUNNABLE / SELF-SKIPPING: when a precondition is absent (no `longhorn`
|
||||
StorageClass, no Velero CRDs, <3 Ready nodes) they echo a skip token and `exit 0`, and their
|
||||
`check:` block is deliberately written to accept either outcome, e.g. tests/l3-fault-
|
||||
injection/dr-restore:
|
||||
|
||||
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
|
||||
|| contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
|
||||
|
||||
So a SELF-SKIP exits 0 and Chainsaw records `passed` EXACTLY as a real run does. And
|
||||
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`
|
||||
(verified against the committed reports). Therefore:
|
||||
|
||||
*** A COMMITTED JUnit ALONE WOULD RECORD `DR_RESTORE_SKIPPED_OK` AS PROOF OF R-STATE-03. ***
|
||||
|
||||
AND THE SECOND TRAP, WHICH IS THE MIRROR OF THE FIRST
|
||||
------------------------------------------------------
|
||||
Chainsaw echoes each script's SOURCE into an indented `=== COMMAND` block before running
|
||||
it, so `echo NODE_LOSS_SKIPPED_OK` — the skip branch that was NOT taken — is present in
|
||||
the log of every run of that test. A section-blind grep therefore reads a skip token out
|
||||
of the source of every skip-gated test and can never record `proven` for any of them: the
|
||||
ledger fails closed so completely that it records nothing and proves nothing. Tokens are
|
||||
consequently read from SECTIONS (see classify_sections): a pass token only from
|
||||
`=== STDOUT`, a skip token from any section that is not script source. Skip still beats
|
||||
pass — within genuine output, which is the only place either was ever evidence.
|
||||
|
||||
That is the fake-green this project refuses, and it would be worse than no ledger at all
|
||||
because it would look authoritative. THE SENTINEL TOKEN IS THE ONLY THING THAT SEPARATES
|
||||
PROVEN FROM SKIPPED, AND IT EXISTS ONLY IN THE RUN LOG. Hence: this tool requires BOTH
|
||||
files, and for the Chainsaw layers it will NEVER emit `proof: "proven"` on a JUnit `passed`
|
||||
unless the log carries that test's own pass token and carries NO skip token.
|
||||
|
||||
HOW THE TOKENS ARE OBTAINED (derived, never hand-listed)
|
||||
--------------------------------------------------------
|
||||
Hard-coding a token table would rot the moment a test is edited. Instead every token is
|
||||
parsed out of the test's own `chainsaw-test.yaml`: each `contains($stdout, 'TOKEN')` in a
|
||||
`check:` block is a declared sentinel. Classification is by the project's naming convention
|
||||
(`*_SKIP*_OK` is a skip token, anything else is a pass token), and the TERMINAL pass token
|
||||
— the last non-skip token in file order — is the test's verdict token. Edit a test's tokens
|
||||
and this tool follows automatically; delete them and the test simply becomes unprovable
|
||||
rather than silently proven.
|
||||
|
||||
node-loss pass NODE_LOSS_VOLUME_AVAILABLE_OK skip NODE_LOSS_SKIPPED_OK
|
||||
dr-restore pass DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK skip DR_RESTORE_SKIPPED_OK
|
||||
(+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK / DR_SCHEDULE_SKIPPED_OK)
|
||||
|
||||
WHY L0/L1 NEED NO SENTINEL (and why that is not a loophole)
|
||||
-----------------------------------------------------------
|
||||
The offline layers are not run by Chainsaw. `lib/run_l1_offline.py` shells out to each
|
||||
dir's `check.sh` / `render-check.sh` and maps exit 3 to a JUnit `<skipped/>` element and
|
||||
any other non-zero to `<failure>`. A self-skip there is ALREADY a JUnit skip, never a green
|
||||
pass, so the JUnit is a truthful record on its own. Only the Chainsaw layers (L2/L3/META)
|
||||
can express "skipped" as "passed", and only they require a sentinel — see
|
||||
SENTINEL_REQUIRED_LAYERS.
|
||||
|
||||
LEDGER ROW (conformance/runs/ledger.jsonl — append-only, one object per testcase per run)
|
||||
ts_utc run-dir timestamp, else the JUnit's mtime (when)
|
||||
git_sha `git rev-parse HEAD`, suffixed `-dirty` when tracked files were
|
||||
modified — a proof captured from a dirty tree is NOT honoured by
|
||||
status.py, because what ran is not what is committed (what)
|
||||
env --env / CONF_ENV (cluster overlay the run targeted)
|
||||
chainsaw_version CHAINSAW_VERSION from versions.env (the pin IS the truth)
|
||||
kube_version `kubectl version` server gitVersion, else "unknown"
|
||||
layer L0|L1|L2|L3|META, from the test's `layer` annotation / JUnit property
|
||||
test testcase name (== the test dir basename, asserted by convention)
|
||||
requirements[] the `conformance.home.k3s/R-*` labels on the test — the SAME
|
||||
authoritative source coverage.py uses, not the JUnit classname
|
||||
junit_result passed | failed | error | skipped
|
||||
sentinel the DECIDING token grepped from run.log, or null
|
||||
proof proven | skipped | failed | junit-skipped
|
||||
| unknown-no-sentinel-declared | unknown-sentinel-absent
|
||||
run_id the runs/<dir> name (provenance: which log backs this row)
|
||||
note one line saying WHY this row landed on that proof state
|
||||
|
||||
`proven` is emitted on exactly one path (see decide()); every other path is non-green.
|
||||
|
||||
RECORDING IS EXPLICIT, NEVER IMPLICIT
|
||||
-------------------------------------
|
||||
Invoke this tool by hand, or via `run.sh --record`. A run that silently records itself is
|
||||
how a bad run becomes a permanent false proof, so `run.sh` without `--record` writes no
|
||||
ledger row at all.
|
||||
|
||||
EXIT CODES (ci/lib.sh convention: 0 pass / 1 fail / 3 skipped-not-green)
|
||||
0 rows appended.
|
||||
1 a real failure: unparsable JUnit, unreadable test tree, unwritable ledger.
|
||||
3 the inputs a ledger row needs are absent (no report.junit.xml, or no run.log).
|
||||
NOTHING is recorded and this is NEVER green — a run we cannot evidence is not a run.
|
||||
|
||||
Pure standard library, matching the harness's no-pip-dependency posture (coverage.py,
|
||||
status.py). PyYAML lives in catalog/render.py; the harness does not depend on it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
|
||||
_LIB = Path(__file__).resolve().parent
|
||||
if str(_LIB) not in sys.path:
|
||||
sys.path.insert(0, str(_LIB))
|
||||
# Importing a sibling would otherwise litter conformance/lib/__pycache__/ into the working
|
||||
# tree, and the lint stage scans untracked files. Same guard as status.py.
|
||||
sys.dont_write_bytecode = True
|
||||
import coverage as cov # noqa: E402 — reuse the registry/test parsers, don't fork them
|
||||
|
||||
CONF_ROOT = _LIB.parent
|
||||
REPO_ROOT = CONF_ROOT.parent
|
||||
DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
|
||||
|
||||
# The ONLY layers exempt from needing a pass sentinel: the offline ones, whose skips
|
||||
# lib/run_l1_offline.py already renders as JUnit <skipped/> rather than as a green pass
|
||||
# (see docstring). Written as an EXEMPT set, not a REQUIRED set, so that an unrecognised
|
||||
# or new layer fails CLOSED — it needs a sentinel — instead of quietly becoming provable
|
||||
# by JUnit alone.
|
||||
SENTINEL_EXEMPT_LAYERS = frozenset({"L0", "L1"})
|
||||
|
||||
# Layer from the test's subtree. Authoritative over the annotation: coverage.py's layer
|
||||
# regex stops at the first non-[L/A/M/0-9] character, so `layer: "META"` parses as "M" —
|
||||
# harmless for its own report, but here a mis-read layer decides whether a sentinel is
|
||||
# required at all, so it is derived from the path exactly as emit_skipped_junit.py does.
|
||||
_LAYER_BY_SUBTREE = (("/l0-static/", "L0"), ("/l1-rendering/", "L1"),
|
||||
("/l2-behavioral/", "L2"), ("/l3-fault-injection/", "L3"),
|
||||
("/meta/", "META"))
|
||||
|
||||
|
||||
def layer_of(path: Path) -> str:
|
||||
rel = str(path)
|
||||
for frag, layer in _LAYER_BY_SUBTREE:
|
||||
if frag in rel:
|
||||
return layer
|
||||
return ""
|
||||
|
||||
# A declared sentinel: `contains($stdout, 'TOKEN')` inside a Chainsaw `check:` block.
|
||||
# Both quote styles are accepted; the repo uses single quotes throughout.
|
||||
_SENTINEL_RE = re.compile(r"contains\(\s*\$stdout\s*,\s*['\"]([A-Za-z0-9_]+)['\"]\s*\)")
|
||||
|
||||
# Naming convention for a SKIP token (tests/l3-fault-injection/*: NODE_LOSS_SKIPPED_OK,
|
||||
# DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK, VM_DATA_SURVIVAL_SKIPPED_OK, ...).
|
||||
# Deliberately broad: mis-classifying a PASS token as a skip only ever loses a proof;
|
||||
# mis-classifying a SKIP token as a pass would manufacture one.
|
||||
_SKIP_TOKEN_RE = re.compile(r"_SKIP")
|
||||
|
||||
# A Chainsaw log line header, as it really appears (Go `t.Log` prefixes the source site and
|
||||
# pads it with backspaces, and the fields are ANSI-coloured — both are stripped first):
|
||||
# l.go:52: | 23:51:18 | node-loss | volume-stays-available... | SCRIPT | LOG |
|
||||
# Field 2 is the test name; it is how output is attributed back to a testcase. The
|
||||
# `<file>.go:<line>:` prefix is optional so a redirected/plain log still attributes.
|
||||
_LOG_HEADER_RE = re.compile(
|
||||
r"^\s*(?:[\w./-]+\.go:\d+:\s*)?"
|
||||
r"\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
|
||||
|
||||
# ANSI colour + the backspace padding Go writes ahead of a `t.Log` continuation. Stripped
|
||||
# before ANY parsing; neither ever appears at the head of a line, so indentation survives.
|
||||
_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
|
||||
|
||||
|
||||
def _strip_noise(line: str) -> str:
|
||||
return _ANSI_RE.sub("", line).replace("\b", "")
|
||||
|
||||
|
||||
# ===================================================================== output sections
|
||||
# THE BUG THIS SECTION MACHINERY EXISTS TO KILL — READ BEFORE CHANGING ANYTHING
|
||||
# -----------------------------------------------------------------------------
|
||||
# Chainsaw echoes the script it is ABOUT TO RUN, verbatim, in an indented `=== COMMAND`
|
||||
# block, and only afterwards prints what the script actually wrote in `=== STDOUT`. Every
|
||||
# skip-gated test contains its own skip token as SOURCE TEXT (`echo NODE_LOSS_SKIPPED_OK`),
|
||||
# so a raw whole-log grep finds that token in EVERY run of that test — including runs that
|
||||
# never took the skip branch. Combined with decide()'s (correct) rule that a skip token
|
||||
# outranks a pass token, a section-blind scan can never record `proven` for ANY test that
|
||||
# has a skip gate: the ledger fails closed so hard it records nothing. Observed on
|
||||
# runs/20260901T235117Z-e9a4b66, where NODE_LOSS_SKIPPED_OK appears twice as source at
|
||||
# lines 51/62 (`=== COMMAND`) and never in output, yet the run genuinely proved R-STATE-02.
|
||||
#
|
||||
# So tokens are attributed to SECTIONS, never to raw lines:
|
||||
# l.go:52: | 23:51:18 | node-loss | step | CMD | RUN | <- header, no section
|
||||
# === COMMAND <- marker (indent 8)
|
||||
# /usr/bin/sh -c set -euo pipefail <- source, indent >= 8
|
||||
# echo NODE_LOSS_SKIPPED_OK <- SOURCE, not output
|
||||
# l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG | <- indent 4 => section ends
|
||||
# === STDOUT <- marker
|
||||
# NODE_LOSS_VOLUME_AVAILABLE_OK <- the REAL emission
|
||||
#
|
||||
# A marker is an INDENTED `=== NAME` on a line of its own. Go's own test markers
|
||||
# (`=== RUN chainsaw`, `=== PAUSE`, `=== CONT`) are at column 0 and carry a test name
|
||||
# after the verb, so they match neither the indent nor the end-of-line anchor; they are
|
||||
# additionally named in _GO_TEST_MARKERS so a future indented variant still cannot open a
|
||||
# section that would swallow the whole log.
|
||||
_SECTION_MARKER_RE = re.compile(r"^(\s+)=== ([A-Z][A-Z0-9_]*)\s*$")
|
||||
_GO_TEST_MARKERS = frozenset({"RUN", "PAUSE", "CONT"})
|
||||
|
||||
# Pseudo-section for a log that contains NO section markers at all (the plain/redirected
|
||||
# logs the earlier ledger rows were written from, and the selftest fixtures). Such a log
|
||||
# never echoes script source, so there is nothing to be confused by and the legacy
|
||||
# whole-log scan is exactly as safe as it was — narrowing is only needed where a COMMAND
|
||||
# echo exists. Lines OUTSIDE a section in a log that DOES use sections are attributed to
|
||||
# "" and are evidence for nothing.
|
||||
UNSECTIONED = "*"
|
||||
|
||||
# A pass token counts ONLY where Chainsaw's own `contains($stdout, ...)` check looks:
|
||||
# real stdout. Anything else — source echo, stderr, an unrecognised future section — can
|
||||
# never manufacture a proof.
|
||||
PASS_SECTIONS = frozenset({"STDOUT", UNSECTIONED})
|
||||
|
||||
# Sections that carry SOURCE rather than output. A skip token is honoured in every OTHER
|
||||
# section (STDOUT, STDERR, any section name added by a later Chainsaw release), because
|
||||
# over-reading a skip only ever COSTS a proof, while over-reading a pass would fabricate
|
||||
# one. Unknown section names therefore fail closed in both directions.
|
||||
SOURCE_SECTIONS = frozenset({"COMMAND"})
|
||||
|
||||
|
||||
def classify_sections(log_text: str) -> list[tuple[str, str]]:
|
||||
"""[(section, line)] for every line of a run log.
|
||||
|
||||
`section` is the enclosing `=== NAME` block, "" for a line inside no block, or
|
||||
UNSECTIONED for every line when the log declares no sections at all. A block runs from
|
||||
its marker until the first non-blank line indented LESS than the marker (blank lines do
|
||||
not end it) or until the next marker; the marker line itself is content of nothing.
|
||||
"""
|
||||
rows: list[tuple[str, str]] = []
|
||||
cur, cur_indent, sectioned = "", 0, False
|
||||
for raw in log_text.splitlines():
|
||||
line = _strip_noise(raw)
|
||||
m = _SECTION_MARKER_RE.match(line)
|
||||
if m and m.group(2) not in _GO_TEST_MARKERS:
|
||||
sectioned = True
|
||||
cur, cur_indent = m.group(2), len(m.group(1))
|
||||
rows.append(("", line))
|
||||
continue
|
||||
if cur:
|
||||
if not line.strip() or len(line) - len(line.lstrip()) >= cur_indent:
|
||||
rows.append((cur, line))
|
||||
continue
|
||||
cur = ""
|
||||
rows.append(("", line))
|
||||
if not sectioned:
|
||||
return [(UNSECTIONED, line) for _, line in rows]
|
||||
return rows
|
||||
|
||||
|
||||
def scan_texts(lines) -> tuple[str, str]:
|
||||
"""(text a PASS token may be read from, text a SKIP token may be read from).
|
||||
|
||||
Accepts either [(section, line)] pairs from classify_sections() or a plain list of
|
||||
strings, which is treated as an unsectioned log (legacy whole-log scanning).
|
||||
"""
|
||||
pass_lines, skip_lines = [], []
|
||||
for item in lines:
|
||||
section, line = item if isinstance(item, tuple) else (UNSECTIONED, item)
|
||||
if section in PASS_SECTIONS:
|
||||
pass_lines.append(line)
|
||||
if section and section not in SOURCE_SECTIONS:
|
||||
skip_lines.append(line)
|
||||
return "\n".join(pass_lines), "\n".join(skip_lines)
|
||||
|
||||
_RUN_ID_TS_RE = re.compile(r"^(\d{8}T\d{6}Z)-")
|
||||
|
||||
|
||||
# ========================================================================= sentinels
|
||||
def parse_sentinels(test_yaml: Path) -> dict:
|
||||
"""Declared sentinel tokens of one Chainsaw test, derived from the test file itself.
|
||||
|
||||
Returns {"pass": [...], "skip": [...], "terminal_pass": str|None}. `terminal_pass` is
|
||||
the LAST non-skip token in file order — the test's verdict token, i.e. the one its
|
||||
final `check:` block asserts (dr-restore -> DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK).
|
||||
A test with no `contains($stdout, ...)` assertion declares no sentinel and is reported
|
||||
unprovable rather than proven; see decide().
|
||||
"""
|
||||
try:
|
||||
text = test_yaml.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return {"pass": [], "skip": [], "terminal_pass": None}
|
||||
toks = _SENTINEL_RE.findall(text)
|
||||
passes, skips = [], []
|
||||
for t in toks:
|
||||
(skips if _SKIP_TOKEN_RE.search(t) else passes).append(t)
|
||||
# De-duplicate, preserving order (dr-restore names DR_RESTORE_SKIPPED_OK twice).
|
||||
def uniq(xs: list[str]) -> list[str]:
|
||||
return list(dict.fromkeys(xs))
|
||||
passes, skips = uniq(passes), uniq(skips)
|
||||
return {"pass": passes, "skip": skips,
|
||||
"terminal_pass": passes[-1] if passes else None}
|
||||
|
||||
|
||||
def sentinel_map(tests_dir: Path) -> dict[str, dict]:
|
||||
"""{test-name: sentinels} for every test in the tree.
|
||||
|
||||
Keyed on the Chainsaw `metadata.name`, which is what lands in the JUnit `<testcase
|
||||
name=...>`. This repo keeps that equal to the test's directory basename; the mapping
|
||||
is by name, so it stays correct if that ever drifts.
|
||||
"""
|
||||
out: dict[str, dict] = {}
|
||||
for f in sorted(tests_dir.rglob("chainsaw-test.yaml")):
|
||||
meta = cov.parse_test(f)
|
||||
name = meta["name"] or f.parent.name
|
||||
s = parse_sentinels(f)
|
||||
s["dir"] = str(f.parent)
|
||||
s["layer"] = layer_of(f) or meta["layer"]
|
||||
s["requirements"] = meta["labels"]
|
||||
out[name] = s
|
||||
return out
|
||||
|
||||
|
||||
# =============================================================================== log
|
||||
def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[tuple[str, str]]],
|
||||
bool]:
|
||||
"""Split a run log into per-testcase buckets of (section, line) pairs.
|
||||
|
||||
Chainsaw prefixes its log lines with `| HH:MM:SS | <test> | <step> | ...` and prints a
|
||||
script's sections underneath that header, so the test name is carried by a state
|
||||
machine: a header naming a KNOWN test switches the current bucket, every following line
|
||||
lands in it. Requiring the name to be known is what stops run.sh's own `[run.sh]`
|
||||
chatter (and any other pipe-shaped line) from opening a bogus bucket.
|
||||
|
||||
Section labels come from classify_sections() and travel WITH the lines, so a bucket
|
||||
still knows which of its lines were script source and which were real output.
|
||||
|
||||
Returns (buckets, attributed) where `attributed` is False when the log contained no
|
||||
recognisable header at all — the signal for the degraded global scan in decide().
|
||||
"""
|
||||
buckets: dict[str, list[tuple[str, str]]] = {}
|
||||
cur: str | None = None
|
||||
attributed = False
|
||||
for section, line in classify_sections(log_text):
|
||||
m = _LOG_HEADER_RE.match(line)
|
||||
if m and m.group(1) in known:
|
||||
cur = m.group(1)
|
||||
attributed = True
|
||||
if cur is not None:
|
||||
buckets.setdefault(cur, []).append((section, line))
|
||||
return buckets, attributed
|
||||
|
||||
|
||||
def unique_tokens(smap: dict[str, dict]) -> set[str]:
|
||||
"""Tokens claimed by EXACTLY ONE test in the tree.
|
||||
|
||||
Only these may be used by the degraded global scan: `DNS_OK` is echoed by two different
|
||||
tests, so in an unattributable log it can never be evidence for either; but
|
||||
`NODE_LOSS_VOLUME_AVAILABLE_OK` names precisely one test and still can.
|
||||
"""
|
||||
owners: dict[str, set[str]] = {}
|
||||
for name, s in smap.items():
|
||||
for t in s["pass"] + s["skip"]:
|
||||
owners.setdefault(t, set()).add(name)
|
||||
return {t for t, o in owners.items() if len(o) == 1}
|
||||
|
||||
|
||||
# ============================================================================ decide
|
||||
def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
|
||||
scope: str) -> tuple[str | None, str, str]:
|
||||
"""The ONE place a testcase becomes `proven` — or, far more often, does not.
|
||||
|
||||
Returns (sentinel, proof, note).
|
||||
|
||||
Ordering is worst-first ON PURPOSE, so that ambiguity can only ever cost a proof:
|
||||
1. JUnit failure/error -> failed (a red run is recorded as red)
|
||||
2. JUnit <skipped/> -> junit-skipped (never green; ci/lib.sh rule)
|
||||
3. ANY declared skip token seen -> skipped (BEATS a green JUnit and beats
|
||||
IN REAL OUTPUT the pass token: if any gated
|
||||
step self-skipped, the run did
|
||||
not prove the requirement)
|
||||
4. no sentinel declared -> proven only for L0/L1, whose skips are already
|
||||
JUnit skips; unknown-no-sentinel-declared for
|
||||
the Chainsaw layers
|
||||
5. EVERY declared pass token seen -> proven
|
||||
6. anything else -> unknown-sentinel-absent
|
||||
Step 5 requires ALL pass tokens, not just the terminal one: every Chainsaw step here
|
||||
asserts `pass || skip`, so a missing pass token means a step did not reach its proven
|
||||
branch, whatever the JUnit says.
|
||||
|
||||
Steps 3 and 5 read from SECTION-SCOPED text, never from raw lines: a pass token counts
|
||||
only in `=== STDOUT`, a skip token counts in any section that is not script source.
|
||||
That narrows WHERE a token is read from and changes nothing about what it means — skip
|
||||
still outranks pass, and it outranks it within genuine output, which is the only place
|
||||
either of them was ever evidence. `lines` may be [(section, line)] pairs or plain
|
||||
strings (an unsectioned log; see scan_texts / UNSECTIONED).
|
||||
"""
|
||||
if junit_result in ("failed", "error"):
|
||||
return (None, "failed", f"JUnit recorded {junit_result}")
|
||||
if junit_result == "skipped":
|
||||
return (None, "junit-skipped",
|
||||
"JUnit <skipped/> — the test did not execute; a skip is never green")
|
||||
|
||||
pass_text, skip_text = scan_texts(lines)
|
||||
seen_skip = [t for t in sent["skip"] if t in skip_text]
|
||||
if seen_skip:
|
||||
return (seen_skip[0], "skipped",
|
||||
f"run log OUTPUT carries the SELF-SKIP token {seen_skip[0]!r} — the test "
|
||||
f"exited 0 without exercising the behaviour, so this is NOT proof")
|
||||
|
||||
if not sent["pass"] and not sent["skip"]:
|
||||
if layer not in SENTINEL_EXEMPT_LAYERS:
|
||||
return (None, "unknown-no-sentinel-declared",
|
||||
f"{layer or 'unknown-layer'} test declares no contains($stdout, ...) "
|
||||
f"sentinel, so a green JUnit cannot be told apart from a self-skip — "
|
||||
f"refusing to call it proven")
|
||||
return (None, "proven",
|
||||
f"{layer} offline check exited 0; run_l1_offline.py renders its skips as "
|
||||
f"JUnit <skipped/>, so a green JUnit is truthful without a sentinel")
|
||||
|
||||
missing = [t for t in sent["pass"] if t not in pass_text]
|
||||
if not missing:
|
||||
return (sent["terminal_pass"], "proven",
|
||||
f"run log OUTPUT carries every declared pass token "
|
||||
f"({', '.join(sent['pass'])}) and no skip token [{scope}]")
|
||||
return (None, "unknown-sentinel-absent",
|
||||
f"run log OUTPUT lacks pass token(s) {', '.join(missing)} for this test "
|
||||
f"[{scope}] — refusing to infer proof from the JUnit alone")
|
||||
|
||||
|
||||
# ============================================================================= junit
|
||||
def junit_cases(path: Path) -> list[dict]:
|
||||
"""[{name, layer, requirements, result}] for every <testcase> in a JUnit report."""
|
||||
root = ET.parse(path).getroot()
|
||||
cases: list[dict] = []
|
||||
for c in root.iter("testcase"):
|
||||
if c.find("failure") is not None:
|
||||
result = "failed"
|
||||
elif c.find("error") is not None:
|
||||
result = "error"
|
||||
elif c.find("skipped") is not None:
|
||||
result = "skipped"
|
||||
else:
|
||||
result = "passed"
|
||||
props = {p.get("name"): (p.get("value") or "")
|
||||
for p in c.iter("property") if p.get("name")}
|
||||
reqs = sorted(set(cov.RID_RE.findall(props.get("requirements", ""))))
|
||||
layer = (props.get("layer") or "").upper()
|
||||
if not layer:
|
||||
layer = (c.get("classname") or "").split(":", 1)[0].upper()
|
||||
cases.append({"name": c.get("name") or "?", "layer": layer,
|
||||
"requirements": reqs, "result": result})
|
||||
return cases
|
||||
|
||||
|
||||
# =============================================================================== git
|
||||
def git_sha(repo: Path) -> str:
|
||||
"""HEAD, suffixed `-dirty` when TRACKED files differ from it.
|
||||
|
||||
Untracked files are deliberately ignored, matching status.py's `tracked_files()`
|
||||
posture (untracked scratch is not repo state) — and because a `--record` run creates
|
||||
untracked `runs/<id>/` files as it goes, which would otherwise mark every run dirty.
|
||||
A `-dirty` sha is not a valid rev, so status.py's ancestry check cannot resolve it and
|
||||
the proof degrades to `stale-proof`: correct, since what ran is not what is committed.
|
||||
"""
|
||||
try:
|
||||
sha = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
|
||||
capture_output=True, text=True, check=True).stdout.strip()
|
||||
except (subprocess.CalledProcessError, OSError):
|
||||
return "unknown"
|
||||
# EXCLUDE conformance/runs/. It is TRACKED, so `--record` writing ledger.jsonl dirtied the
|
||||
# tree and every LATER run in the same session was stamped `-dirty` — recording one test
|
||||
# poisoned the next, and check.sh refuses to honour a dirty proven row. That cost a second
|
||||
# live fault-injection run on prod to work around. The docstring above already states the
|
||||
# intent for the untracked runs/<id>/ half of this same artifact; the ledger is the tracked
|
||||
# half. Mirrors status.py's IMPL_EXCLUDE_PREFIXES, which lists this path for the same reason.
|
||||
dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--",
|
||||
".", ":(exclude)conformance/runs/"],
|
||||
capture_output=True).returncode != 0
|
||||
return f"{sha}-dirty" if dirty else sha
|
||||
|
||||
|
||||
def pinned_chainsaw_version(conf_root: Path) -> str:
|
||||
"""CHAINSAW_VERSION from versions.env — the pin is the single source of truth."""
|
||||
f = conf_root / "versions.env"
|
||||
if f.is_file():
|
||||
m = re.search(r'^CHAINSAW_VERSION="?([^"\n]+)"?', f.read_text(), re.M)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return "unknown"
|
||||
|
||||
|
||||
def server_kube_version() -> str:
|
||||
"""Server gitVersion via `kubectl version` (read-only), else "unknown"."""
|
||||
for args in (["kubectl", "version", "-o", "json"],
|
||||
["kubectl", "version", "--output=json"]):
|
||||
try:
|
||||
p = subprocess.run(args, capture_output=True, text=True, timeout=15)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
continue
|
||||
if p.returncode != 0 or not p.stdout.strip():
|
||||
continue
|
||||
try:
|
||||
v = json.loads(p.stdout).get("serverVersion") or {}
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if v.get("gitVersion"):
|
||||
return v["gitVersion"]
|
||||
return "unknown"
|
||||
|
||||
|
||||
def run_timestamp(run_dir: Path, junit: Path) -> str:
|
||||
"""UTC stamp: the run-dir name's own `<UTC>-<sha>` prefix, else the JUnit's mtime."""
|
||||
m = _RUN_ID_TS_RE.match(run_dir.name)
|
||||
if m:
|
||||
t = m.group(1)
|
||||
return f"{t[0:4]}-{t[4:6]}-{t[6:8]}T{t[9:11]}:{t[11:13]}:{t[13:15]}Z"
|
||||
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(junit.stat().st_mtime))
|
||||
|
||||
|
||||
# ============================================================================ record
|
||||
def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
|
||||
kube_version: str, sha: str) -> list[dict]:
|
||||
junit = run_dir / "report.junit.xml"
|
||||
log = run_dir / "run.log"
|
||||
smap = sentinel_map(tests_dir)
|
||||
log_text = log.read_text(errors="replace")
|
||||
buckets, attributed = attribute_log(log_text, set(smap))
|
||||
uniq = unique_tokens(smap)
|
||||
all_pairs = classify_sections(log_text)
|
||||
sectioned = any(sec not in ("", UNSECTIONED) for sec, _ in all_pairs)
|
||||
section_note = ("tokens read from real-output sections only"
|
||||
if sectioned else "log declares no === sections; whole-log scan")
|
||||
ts = run_timestamp(run_dir, junit)
|
||||
|
||||
rows: list[dict] = []
|
||||
for case in junit_cases(junit):
|
||||
name = case["name"]
|
||||
sent = smap.get(name, {"pass": [], "skip": [], "terminal_pass": None,
|
||||
"layer": case["layer"], "requirements": []})
|
||||
layer = sent.get("layer") or case["layer"] or "?"
|
||||
# Requirements come from the test's LABELS (coverage.py's authoritative source);
|
||||
# the JUnit property is the fallback for a testcase with no matching test file.
|
||||
reqs = sent.get("requirements") or case["requirements"]
|
||||
|
||||
if name in buckets:
|
||||
lines, scope = buckets[name], f"log scoped to this testcase; {section_note}"
|
||||
elif not attributed:
|
||||
# Degraded mode: the log had no recognisable Chainsaw header (a redirected or
|
||||
# reformatted log). Fall back to a whole-log scan, but ONLY for tokens owned by
|
||||
# exactly one test, so a token another test emitted can never prove this one.
|
||||
lines = all_pairs
|
||||
sent = dict(sent,
|
||||
**{"pass": [t for t in sent["pass"] if t in uniq],
|
||||
"skip": [t for t in sent["skip"] if t in uniq]})
|
||||
sent["terminal_pass"] = sent["pass"][-1] if sent["pass"] else None
|
||||
scope = ("DEGRADED: unattributable log, globally-unique tokens only; "
|
||||
+ section_note)
|
||||
else:
|
||||
lines, scope = [], "no log lines attributed to this testcase"
|
||||
|
||||
sentinel, proof, note = decide(layer, case["result"], sent, lines, scope)
|
||||
rows.append({
|
||||
"ts_utc": ts,
|
||||
"git_sha": sha,
|
||||
"env": env,
|
||||
"chainsaw_version": chainsaw_version,
|
||||
"kube_version": kube_version,
|
||||
"layer": layer,
|
||||
"test": name,
|
||||
"requirements": reqs,
|
||||
"junit_result": case["result"],
|
||||
"sentinel": sentinel,
|
||||
"proof": proof,
|
||||
"run_id": run_dir.name,
|
||||
"note": note,
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
def append_rows(ledger: Path, rows: list[dict]) -> None:
|
||||
"""Append-only, one compact JSON object per line. Never rewrites existing rows."""
|
||||
ledger.parent.mkdir(parents=True, exist_ok=True)
|
||||
with ledger.open("a", encoding="utf-8") as fh:
|
||||
for r in rows:
|
||||
fh.write(json.dumps(r, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
# ========================================================================== selftest
|
||||
def _selftest() -> int:
|
||||
"""Deterministic, no-cluster self-test of every decision rule (house pattern:
|
||||
`status.py --selftest`, `catalog/render.py --selftest`). Throwaway tree only.
|
||||
|
||||
The load-bearing case is `SKIP sentinel does NOT produce proven`: it is the single
|
||||
assertion standing between this ledger and a permanent false proof of R-STATE-03.
|
||||
"""
|
||||
import tempfile
|
||||
ok = True
|
||||
|
||||
def check(label: str, cond: bool) -> None:
|
||||
nonlocal ok
|
||||
if not cond:
|
||||
sys.stderr.write(f"record_run.py selftest: FAIL — {label}\n")
|
||||
ok = False
|
||||
|
||||
# --- token derivation, against the dr-restore/node-loss shape ---------------------
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
root = Path(td)
|
||||
tdir = root / "tests" / "l3-fault-injection" / "dr-restore"
|
||||
tdir.mkdir(parents=True)
|
||||
(tdir / "chainsaw-test.yaml").write_text(
|
||||
"apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n"
|
||||
" name: dr-restore\n labels:\n"
|
||||
" conformance.home.k3s/R-STATE-03: \"true\"\n annotations:\n"
|
||||
" conformance.home.k3s/requirements: \"R-STATE-03\"\n"
|
||||
" conformance.home.k3s/layer: \"L3\"\nspec:\n steps:\n"
|
||||
" - script:\n content: 'echo OFF_CLUSTER_TARGET_OK'\n"
|
||||
" check:\n"
|
||||
" (contains($stdout, 'OFF_CLUSTER_TARGET_OK')"
|
||||
" || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n"
|
||||
" - script:\n content: 'echo x'\n check:\n"
|
||||
" (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')"
|
||||
" || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n")
|
||||
s = parse_sentinels(tdir / "chainsaw-test.yaml")
|
||||
check("pass tokens derived from the test file",
|
||||
s["pass"] == ["OFF_CLUSTER_TARGET_OK",
|
||||
"DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"])
|
||||
check("skip token classified by the *_SKIP*_OK convention, de-duplicated",
|
||||
s["skip"] == ["DR_RESTORE_SKIPPED_OK"])
|
||||
check("terminal pass token is the LAST non-skip token",
|
||||
s["terminal_pass"] == "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK")
|
||||
|
||||
smap = sentinel_map(root / "tests")
|
||||
check("sentinel_map keys on metadata.name and carries layer + labels",
|
||||
smap["dr-restore"]["layer"] == "L3"
|
||||
and smap["dr-restore"]["requirements"] == ["R-STATE-03"])
|
||||
|
||||
# --- decide(): the SKIP-is-never-proven contract ----------------------------------
|
||||
l3 = {"pass": ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"],
|
||||
"skip": ["DR_RESTORE_SKIPPED_OK"],
|
||||
"terminal_pass": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"}
|
||||
|
||||
# *** THE CASE THIS WHOLE FILE EXISTS FOR ***
|
||||
sent, proof, _ = decide("L3", "passed", l3, ["DR_RESTORE_SKIPPED_OK"], "t")
|
||||
check("SKIP sentinel on a GREEN JUnit is 'skipped', NEVER 'proven'",
|
||||
proof == "skipped" and sent == "DR_RESTORE_SKIPPED_OK")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
|
||||
check("pass sentinel on a green JUnit is 'proven'", proof == "proven")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK",
|
||||
"DR_RESTORE_SKIPPED_OK"], "t")
|
||||
check("a skip token BEATS a pass token in the same log (worst wins)",
|
||||
proof == "skipped")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3, ["nothing relevant here"], "t")
|
||||
check("green JUnit with no sentinel in the log is unknown, not proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3, [], "t")
|
||||
check("green JUnit with NO log lines attributed is unknown, not proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
_, proof, _ = decide("L3", "failed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
|
||||
check("a failing JUnit outranks a pass token in the log", proof == "failed")
|
||||
|
||||
_, proof, _ = decide("L3", "skipped", l3, [], "t")
|
||||
check("JUnit <skipped/> records junit-skipped, never green",
|
||||
proof == "junit-skipped")
|
||||
|
||||
multi = {"pass": ["A_OK", "B_OK"], "skip": [], "terminal_pass": "B_OK"}
|
||||
_, proof, _ = decide("L3", "passed", multi, ["B_OK"], "t")
|
||||
check("every declared pass token must be present, not just the terminal one",
|
||||
proof == "unknown-sentinel-absent")
|
||||
_, proof, _ = decide("L3", "passed", multi, ["A_OK", "B_OK"], "t")
|
||||
check("all pass tokens present == proven", proof == "proven")
|
||||
|
||||
none = {"pass": [], "skip": [], "terminal_pass": None}
|
||||
_, proof, _ = decide("L2", "passed", none, ["whatever"], "t")
|
||||
check("a Chainsaw test declaring no sentinel is NOT proven by a green JUnit",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
_, proof, _ = decide("L1", "passed", none, [], "t")
|
||||
check("L0/L1 offline checks are proven by JUnit (their skips are JUnit skips)",
|
||||
proof == "proven")
|
||||
_, proof, _ = decide("L0", "skipped", none, [], "t")
|
||||
check("an L0 JUnit skip is still never green", proof == "junit-skipped")
|
||||
_, proof, _ = decide("META", "passed", none, [], "t")
|
||||
check("META (a Chainsaw layer) needs a sentinel like L2/L3",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
_, proof, _ = decide("", "passed", none, [], "t")
|
||||
check("an unknown/blank layer fails CLOSED — sentinel required",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
|
||||
# --- SECTION SCOPING: the COMMAND echo is SOURCE, never evidence -------------------
|
||||
# Regression cases for the ledger's first real L3 run, which recorded a genuine PASS as
|
||||
# `skipped` because `echo NODE_LOSS_SKIPPED_OK` — the skip branch NOT taken — is echoed
|
||||
# inside `=== COMMAND` on every run of a skip-gated test.
|
||||
nl = {"pass": ["NODE_LOSS_VOLUME_AVAILABLE_OK"],
|
||||
"skip": ["NODE_LOSS_SKIPPED_OK"],
|
||||
"terminal_pass": "NODE_LOSS_VOLUME_AVAILABLE_OK"}
|
||||
_CMD = (" l.go:52: | 23:51:18 | node-loss | step | CMD | RUN |\n"
|
||||
" === COMMAND\n"
|
||||
" /usr/bin/sh -c set -euo pipefail\n"
|
||||
" if ! kubectl get storageclass longhorn >/dev/null 2>&1; then\n"
|
||||
" echo NODE_LOSS_SKIPPED_OK\n"
|
||||
"\n"
|
||||
" exit 0\n"
|
||||
" fi\n"
|
||||
" echo NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |\n")
|
||||
|
||||
ran = classify_sections(_CMD + " === STDOUT\n"
|
||||
" ready nodes: 3\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" l.go:52: | 23:52:42 | node-loss | step | SCRIPT | DONE |\n")
|
||||
sent, proof, _ = decide("L3", "passed", nl, ran, "t")
|
||||
check("skip token ONLY in the COMMAND echo + pass token in STDOUT is PROVEN",
|
||||
proof == "proven" and sent == "NODE_LOSS_VOLUME_AVAILABLE_OK")
|
||||
|
||||
selfskipped = classify_sections(_CMD + " === STDOUT\n"
|
||||
" SKIP: 'longhorn' StorageClass absent\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
sent, proof, _ = decide("L3", "passed", nl, selfskipped, "t")
|
||||
check("skip token GENUINELY in STDOUT is still 'skipped' (the guarantee holds)",
|
||||
proof == "skipped" and sent == "NODE_LOSS_SKIPPED_OK")
|
||||
|
||||
neither = classify_sections(_CMD + " === STDOUT\n ready nodes: 3\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, neither, "t")
|
||||
check("both tokens in COMMAND and neither in STDOUT is NOT proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
both_out = classify_sections(" === STDOUT\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, both_out, "t")
|
||||
check("skip still BEATS pass when both are in genuine output", proof == "skipped")
|
||||
|
||||
stderr_skip = classify_sections(" === STDERR\n NODE_LOSS_SKIPPED_OK\n"
|
||||
" === STDOUT\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, stderr_skip, "t")
|
||||
check("a skip token in a non-source section (STDERR) still costs the proof",
|
||||
proof == "skipped")
|
||||
|
||||
stderr_pass = classify_sections(" === STDERR\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, stderr_pass, "t")
|
||||
check("a pass token OUTSIDE stdout cannot prove anything",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
# --- classify_sections: markers, extents, and Go's own `=== RUN` --------------------
|
||||
gorows = classify_sections("=== RUN chainsaw\n=== PAUSE chainsaw\n"
|
||||
"=== CONT chainsaw/node-loss\n"
|
||||
" === COMMAND\n echo NODE_LOSS_SKIPPED_OK\n"
|
||||
" l.go:52: | 1:00:00 | node-loss | s | SCRIPT | DONE |\n"
|
||||
"--- PASS: chainsaw/node-loss (131.09s)\n")
|
||||
secs = [sec for sec, _ in gorows]
|
||||
check("Go's column-0 `=== RUN/PAUSE/CONT` never open a section",
|
||||
secs[:3] == ["", "", ""])
|
||||
check("a section covers its indented body and ends at a less-indented line",
|
||||
secs[3] == "" and secs[4] == "COMMAND" and secs[5] == "" and secs[6] == "")
|
||||
check("a log with no === markers stays wholly scannable (UNSECTIONED)",
|
||||
{sec for sec, _ in classify_sections("plain line\n another\n")} == {UNSECTIONED})
|
||||
|
||||
# --- the preserved run itself, as a fixture ----------------------------------------
|
||||
# runs/20260901T235117Z-e9a4b66 IS the bug: NODE_LOSS_SKIPPED_OK twice in `=== COMMAND`
|
||||
# (source), never in output; NODE_LOSS_VOLUME_AVAILABLE_OK once in `=== STDOUT`.
|
||||
fixture = CONF_ROOT / "runs" / "20260901T235117Z-e9a4b66"
|
||||
if (fixture / "run.log").is_file() and (fixture / "report.junit.xml").is_file():
|
||||
rows = build_rows(fixture, CONF_ROOT / "tests", "dev", "v0.2.12", "v1.36.4", "sha")
|
||||
row = next((r for r in rows if r["test"] == "node-loss"), None)
|
||||
check("preserved real L3 run records node-loss/R-STATE-02 as proven",
|
||||
row is not None and row["proof"] == "proven"
|
||||
and row["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK"
|
||||
and row["requirements"] == ["R-STATE-02"])
|
||||
check("preserved real L3 run attributes (no DEGRADED fallback needed)",
|
||||
row is not None and "DEGRADED" not in row["note"])
|
||||
pairs = classify_sections((fixture / "run.log").read_text(errors="replace"))
|
||||
secs = {sec for sec, line in pairs if "NODE_LOSS_SKIPPED_OK" in line}
|
||||
check("every NODE_LOSS_SKIPPED_OK in the preserved log is COMMAND source",
|
||||
secs == {"COMMAND"})
|
||||
secs = {sec for sec, line in pairs
|
||||
if line.strip() == "NODE_LOSS_VOLUME_AVAILABLE_OK"}
|
||||
check("the real emission in the preserved log is in STDOUT", secs == {"STDOUT"})
|
||||
|
||||
# --- log attribution --------------------------------------------------------------
|
||||
log = ("[run.sh] Coverage report first\n"
|
||||
" | 20:41:32 | node-loss | step | SCRIPT | RUN |\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n"
|
||||
" | 20:44:01 | dr-restore | step | SCRIPT | RUN |\n"
|
||||
" DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK\n")
|
||||
b, attributed = attribute_log(log, {"node-loss", "dr-restore"})
|
||||
check("log lines are attributed to the testcase whose header preceded them",
|
||||
attributed
|
||||
and any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["node-loss"])
|
||||
and not any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["dr-restore"]))
|
||||
check("a pass token does NOT leak across testcase boundaries",
|
||||
not any("DR_RESTORED" in l for _, l in b["node-loss"]))
|
||||
check("an unsectioned log keeps whole-log scanning (nothing narrowed away)",
|
||||
all(sec == UNSECTIONED for sec, _ in b["node-loss"]))
|
||||
|
||||
# Chainsaw's real headers carry a Go `t.Log` source prefix and ANSI colour; the earlier
|
||||
# regex matched neither, so every genuine Chainsaw log silently fell into DEGRADED mode.
|
||||
real_hdr = (" l.go:52: \x08\x08\x08| 23:51:18 | \x1b[32;1mnode-loss\x1b[0;22m | "
|
||||
"\x1b[32;1mstep\x1b[0;22m | SCRIPT | LOG |\n"
|
||||
" === STDOUT\n NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
b2, attributed_real = attribute_log(real_hdr, {"node-loss"})
|
||||
check("a REAL Chainsaw header (go source prefix + ANSI) attributes, not degrades",
|
||||
attributed_real and "node-loss" in b2
|
||||
and ("STDOUT", " NODE_LOSS_VOLUME_AVAILABLE_OK") in b2["node-loss"])
|
||||
_, attributed2 = attribute_log("no chainsaw headers at all\n", {"node-loss"})
|
||||
check("an unattributable log is reported as such (degraded mode)", not attributed2)
|
||||
|
||||
smap2 = {"a": {"pass": ["DNS_OK", "A_ONLY_OK"], "skip": []},
|
||||
"b": {"pass": ["DNS_OK"], "skip": []}}
|
||||
check("only tokens owned by exactly one test may be used in degraded mode",
|
||||
unique_tokens(smap2) == {"A_ONLY_OK"})
|
||||
|
||||
# --- end-to-end: a self-skipped L3 run must land in the ledger as `skipped` --------
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
root = Path(td)
|
||||
tdir = root / "tests" / "l3-fault-injection" / "node-loss"
|
||||
tdir.mkdir(parents=True)
|
||||
(tdir / "chainsaw-test.yaml").write_text(
|
||||
"kind: Test\nmetadata:\n name: node-loss\n labels:\n"
|
||||
" conformance.home.k3s/R-STATE-02: \"true\"\n annotations:\n"
|
||||
" conformance.home.k3s/requirements: \"R-STATE-02\"\n"
|
||||
" conformance.home.k3s/layer: \"L3\"\nspec:\n steps:\n"
|
||||
" - script:\n content: 'echo hi'\n check:\n"
|
||||
" (contains($stdout, 'NODE_LOSS_VOLUME_AVAILABLE_OK')"
|
||||
" || contains($stdout, 'NODE_LOSS_SKIPPED_OK')): true\n")
|
||||
rd = root / "runs" / "20260901T120000Z-abc1234"
|
||||
rd.mkdir(parents=True)
|
||||
(rd / "report.junit.xml").write_text(
|
||||
'<testsuites><testsuite><testcase classname="L3:R-STATE-02" '
|
||||
'name="node-loss"><properties>'
|
||||
'<property name="requirements" value="R-STATE-02"/>'
|
||||
'<property name="layer" value="L3"/></properties></testcase>'
|
||||
"</testsuite></testsuites>")
|
||||
(rd / "run.log").write_text(
|
||||
" | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
|
||||
" SKIP: 'longhorn' StorageClass absent\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
|
||||
check("end-to-end: Chainsaw 'passed' + skip token => proof 'skipped'",
|
||||
len(rows) == 1 and rows[0]["junit_result"] == "passed"
|
||||
and rows[0]["proof"] == "skipped"
|
||||
and rows[0]["sentinel"] == "NODE_LOSS_SKIPPED_OK")
|
||||
check("end-to-end: row carries ts from the run-dir name and the R-ID label",
|
||||
rows[0]["ts_utc"] == "2026-09-01T12:00:00Z"
|
||||
and rows[0]["requirements"] == ["R-STATE-02"])
|
||||
|
||||
(rd / "run.log").write_text(
|
||||
" | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
|
||||
check("end-to-end: Chainsaw 'passed' + pass token => proof 'proven'",
|
||||
rows[0]["proof"] == "proven"
|
||||
and rows[0]["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK")
|
||||
|
||||
ledger = root / "ledger.jsonl"
|
||||
append_rows(ledger, rows)
|
||||
append_rows(ledger, rows)
|
||||
check("ledger is append-only JSONL (one parsable object per line)",
|
||||
len(ledger.read_text().strip().splitlines()) == 2
|
||||
and json.loads(ledger.read_text().splitlines()[0])["test"] == "node-loss")
|
||||
|
||||
if not ok:
|
||||
return 1
|
||||
sys.stderr.write("record_run.py selftest: PASS\n")
|
||||
return 0
|
||||
|
||||
|
||||
# ============================================================================== main
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--run-dir", type=Path,
|
||||
help="conformance/runs/<UTC>-<sha>/ holding report.junit.xml + run.log")
|
||||
ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
|
||||
help=f"append-only ledger (default {DEFAULT_LEDGER})")
|
||||
ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
|
||||
ap.add_argument("--repo", type=Path, default=REPO_ROOT)
|
||||
ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
|
||||
ap.add_argument("--chainsaw-version", default=None,
|
||||
help="override the versions.env pin (recorded verbatim)")
|
||||
ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
|
||||
help="override the `kubectl version` probe (recorded verbatim)")
|
||||
ap.add_argument("--dry-run", action="store_true",
|
||||
help="print the rows that WOULD be appended; write nothing")
|
||||
ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.selftest:
|
||||
return _selftest()
|
||||
if args.run_dir is None:
|
||||
sys.stderr.write("[record] --run-dir is required (or use --selftest)\n")
|
||||
return 1
|
||||
|
||||
run_dir = args.run_dir.resolve()
|
||||
junit, log = run_dir / "report.junit.xml", run_dir / "run.log"
|
||||
# BOTH files or nothing. The JUnit alone cannot tell a self-skip from a real pass, so
|
||||
# recording from it alone is exactly the false proof this tool exists to prevent.
|
||||
missing = [p.name for p in (junit, log) if not p.is_file()]
|
||||
if missing:
|
||||
sys.stderr.write(
|
||||
f"[record] {run_dir} is missing {', '.join(missing)} — nothing recorded. "
|
||||
f"The JUnit alone cannot distinguish a self-skip from a real pass "
|
||||
f"(see module docstring). SKIPPED (not green).\n")
|
||||
return 3
|
||||
|
||||
try:
|
||||
rows = build_rows(
|
||||
run_dir, args.tests, args.env,
|
||||
args.chainsaw_version or pinned_chainsaw_version(CONF_ROOT),
|
||||
args.kube_version or server_kube_version(),
|
||||
git_sha(args.repo))
|
||||
except ET.ParseError as e:
|
||||
sys.stderr.write(f"[record] {junit} is not parsable JUnit: {e}\n")
|
||||
return 1
|
||||
except OSError as e:
|
||||
sys.stderr.write(f"[record] cannot read the run inputs: {e}\n")
|
||||
return 1
|
||||
|
||||
if not rows:
|
||||
sys.stderr.write(f"[record] {junit} contains no <testcase> — nothing to record. "
|
||||
f"SKIPPED (not green).\n")
|
||||
return 3
|
||||
|
||||
for r in rows:
|
||||
print(f"[record] {r['layer']:4s} {r['proof']:28s} {r['test']} "
|
||||
f"({','.join(r['requirements']) or 'untagged'}) "
|
||||
f"junit={r['junit_result']} sentinel={r['sentinel'] or '—'}")
|
||||
counts: dict[str, int] = {}
|
||||
for r in rows:
|
||||
counts[r["proof"]] = counts.get(r["proof"], 0) + 1
|
||||
print(f"[record] {len(rows)} testcase(s): "
|
||||
+ ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
|
||||
|
||||
if args.dry_run:
|
||||
print("[record] --dry-run: ledger NOT written.")
|
||||
return 0
|
||||
try:
|
||||
append_rows(args.ledger, rows)
|
||||
except OSError as e:
|
||||
sys.stderr.write(f"[record] cannot append to {args.ledger}: {e}\n")
|
||||
return 1
|
||||
print(f"[record] appended {len(rows)} row(s) -> {args.ledger}")
|
||||
print("[record] COMMIT the run dir and the ledger; an uncommitted ledger proves nothing.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -19,6 +19,30 @@ Exit 0 iff every selected offline check passes.
|
||||
|
||||
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
|
||||
carries; a dir with neither is reported SKIPPED (never a pass).
|
||||
|
||||
DECLARED SCOPE, AND WHY A SKIP IS NOT ALWAYS HARMLESS (--unwired-scope)
|
||||
----------------------------------------------------------------------
|
||||
Exit 0 here means "every selected check passed"; a check that exits 3 is SKIPPED and, by
|
||||
default, does not fail the run (a scaffold with no check script, an audit whose subject
|
||||
subtree does not exist yet). That default is WRONG for a WHOLE-REPO invariant. Those are the
|
||||
checks ci/l0.sh used to run as hand-written "UNCONDITIONAL" stages, and each of those stages
|
||||
mapped rc 3 to the GATE's rc 3 with the words "not a pass":
|
||||
|
||||
3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
|
||||
summary "- subst-safety: **SKIPPED** (nothing to audit)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
|
||||
A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has
|
||||
VANISHED — which is a louder problem than the check failing, not a quieter one. So:
|
||||
|
||||
--unwired-scope SCOPE a test DECLARING `<prefix>/scope: SCOPE` whose check SKIPS (rc 3,
|
||||
or no check script at all) makes this runner exit 3 instead of 0.
|
||||
Hard failures still win (exit 1); rc 3 never masks one.
|
||||
|
||||
Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing failed but a
|
||||
scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
|
||||
into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
|
||||
instead of once per hand-written stage.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -36,6 +60,24 @@ LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
|
||||
CHECK_SCRIPTS = ("render-check.sh", "check.sh")
|
||||
|
||||
|
||||
def scope_of(test_yaml: Path, annotation: str) -> str:
|
||||
"""The scope a test DECLARES, or "" when it declares none.
|
||||
|
||||
Matched on a real YAML mapping line: a `#` comment cannot begin with the annotation key,
|
||||
so a scope named in the test's header prose is documentation, not a declaration — the
|
||||
same "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs.
|
||||
Mirrors conf_dirs_for_scope() in lib/common.sh; they must agree.
|
||||
"""
|
||||
if not test_yaml.exists():
|
||||
return ""
|
||||
rx = re.compile(r"^\s*" + re.escape(annotation) + r"\s*:\s*[\"']?([A-Za-z0-9_.-]+)[\"']?\s*$")
|
||||
for line in test_yaml.read_text().splitlines():
|
||||
m = rx.match(line)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return ""
|
||||
|
||||
|
||||
def layer_of(d: Path) -> str:
|
||||
"""Layer for the JUnit properties, from the subtree the test lives in."""
|
||||
rel = str(d)
|
||||
@@ -59,25 +101,38 @@ def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("--junit", type=Path, required=True)
|
||||
ap.add_argument("--label-prefix", default="conformance.home.k3s")
|
||||
ap.add_argument("--scope-annotation", default="conformance.home.k3s/scope",
|
||||
help="annotation key a test uses to DECLARE its invariant's scope")
|
||||
ap.add_argument("--unwired-scope", action="append", default=[], metavar="SCOPE",
|
||||
help="a test declaring this scope that SKIPS is UNWIRED, not green "
|
||||
"(makes this runner exit 3); repeatable")
|
||||
ap.add_argument("--summary-out", type=Path, default=None,
|
||||
help="write one Markdown line per case here, for the CI step summary")
|
||||
ap.add_argument("dirs", nargs="+", type=Path)
|
||||
args = ap.parse_args()
|
||||
|
||||
cases = [] # (name, reqs, dir, rc, output)
|
||||
strict_scopes = set(args.unwired_scope)
|
||||
|
||||
cases = [] # (name, reqs, dir, rc, output, scope)
|
||||
for d in args.dirs:
|
||||
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
|
||||
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
|
||||
scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
|
||||
if check is None:
|
||||
cases.append((name, reqs, d, 3,
|
||||
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
|
||||
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
|
||||
scope))
|
||||
continue
|
||||
proc = subprocess.run(
|
||||
["bash", str(check)], capture_output=True, text=True
|
||||
)
|
||||
out = (proc.stdout + proc.stderr).strip()
|
||||
cases.append((name, reqs, d, proc.returncode, out))
|
||||
cases.append((name, reqs, d, proc.returncode, out, scope))
|
||||
|
||||
failures = sum(1 for c in cases if c[3] not in (0, 3))
|
||||
skipped = sum(1 for c in cases if c[3] == 3)
|
||||
# A SKIP of a scope-declared check is UNWIRED, not green (see the module docstring).
|
||||
unwired = [c for c in cases if c[3] == 3 and c[5] in strict_scopes]
|
||||
|
||||
suite = ET.Element(
|
||||
"testsuite",
|
||||
@@ -89,7 +144,8 @@ def main() -> int:
|
||||
"skipped": str(skipped),
|
||||
},
|
||||
)
|
||||
for name, reqs, d, rc, out in cases:
|
||||
summary_lines = []
|
||||
for name, reqs, d, rc, out, scope in cases:
|
||||
tc = ET.SubElement(
|
||||
suite,
|
||||
"testcase",
|
||||
@@ -100,17 +156,33 @@ def main() -> int:
|
||||
ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
|
||||
ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
|
||||
ET.SubElement(props, "property", {"name": "path", "value": str(d)})
|
||||
ET.SubElement(props, "property", {"name": "scope", "value": scope})
|
||||
if rc == 3:
|
||||
ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
|
||||
elif rc != 0:
|
||||
fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
|
||||
fail.text = out
|
||||
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
|
||||
mark = f" scope={scope}" if scope else ""
|
||||
print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
|
||||
f"({','.join(reqs) or 'untagged'})",
|
||||
f"({','.join(reqs) or 'untagged'}){mark}",
|
||||
file=sys.stderr)
|
||||
if rc not in (0, 3):
|
||||
print(out, file=sys.stderr)
|
||||
# One step-summary line per case, naming what ran — the generated replacement for the
|
||||
# hand-written `summary "- subst-safety: **passed**"` line each deleted stage carried.
|
||||
tag = f" ({scope})" if scope else ""
|
||||
if rc == 0:
|
||||
verdict = "**passed**"
|
||||
elif rc == 3:
|
||||
verdict = ("**SKIPPED** (UNWIRED, not green)" if scope in strict_scopes
|
||||
else "**SKIPPED** (nothing to audit)")
|
||||
else:
|
||||
verdict = f"**FAILED** (exit {rc})"
|
||||
summary_lines.append(f"- {layer_of(d).lower()}/{name}{tag}: {verdict}")
|
||||
if rc == 3 and scope in strict_scopes:
|
||||
print(f"[offline] {name}: SKIPPED, and it declares scope={scope!r} — "
|
||||
f"UNWIRED, NOT GREEN. {out}", file=sys.stderr)
|
||||
|
||||
root = ET.Element(
|
||||
"testsuites",
|
||||
@@ -126,9 +198,19 @@ def main() -> int:
|
||||
args.junit.parent.mkdir(parents=True, exist_ok=True)
|
||||
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
|
||||
args.junit.write_text(xml)
|
||||
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
|
||||
f"{args.junit}", file=sys.stderr)
|
||||
return 1 if failures else 0
|
||||
if args.summary_out is not None:
|
||||
args.summary_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.summary_out.write_text("\n".join(summary_lines) + ("\n" if summary_lines else ""))
|
||||
|
||||
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s), "
|
||||
f"{skipped} skip(s), {len(unwired)} unwired) -> {args.junit}", file=sys.stderr)
|
||||
if failures:
|
||||
return 1
|
||||
if unwired:
|
||||
print("[offline] UNWIRED (declared scope, SKIPPED — not a pass): "
|
||||
+ ", ".join(c[0] for c in unwired), file=sys.stderr)
|
||||
return 3
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+668
-59
@@ -15,26 +15,42 @@ This generator derives, from the REPOSITORY ALONE, three orthogonal axes per req
|
||||
from a Flux Kustomization entrypoint (i.e. actually reconciled)?
|
||||
2. VERIFICATION — is there a requirement-tagged test, at which layer, and is it an
|
||||
EXECUTABLE test or a TODO scaffold?
|
||||
3. PROOF — has that test actually passed? *** DELIBERATELY NOT DERIVED. ***
|
||||
3. PROOF — has that test actually passed? Derived ONLY from the committed
|
||||
run ledger, and ONLY as far as that ledger's evidence reaches.
|
||||
|
||||
WHAT THIS TOOL REFUSES TO DERIVE (and why)
|
||||
------------------------------------------
|
||||
PROOF-OF-PASS IS NOT DERIVABLE FROM REPO STATE. Nothing tracked in git records a test
|
||||
result: `conformance/_artifacts/` and `results/` are git-ignored, and the only prose
|
||||
record of a green run lives in DECISIONS-LOG.md — hand-maintained, unstructured, and
|
||||
exactly the kind of source the operator has ruled out. The project's own culture is that
|
||||
"merged" is never "done" (agents/definition-of-done.md, R-TEST-03), so this report will
|
||||
NOT infer, guess, or default a requirement to green. Every requirement's proof state is
|
||||
reported as `not-derivable`, and the headline says so. If an untracked JUnit artifact
|
||||
happens to be lying in the workspace it is surfaced ONLY as a quarantined advisory with
|
||||
its mtime — never counted, never mixed into a status.
|
||||
PROOF — WHAT CHANGED, AND WHAT DID NOT
|
||||
--------------------------------------
|
||||
This report used to say `not-derivable` for every requirement, because nothing tracked in
|
||||
git recorded a test result and the only prose record of a green run lived in
|
||||
DECISIONS-LOG.md — hand-maintained, unstructured, exactly the source the operator ruled
|
||||
out. The fix this docstring asked for now exists: `conformance/runs/ledger.jsonl`, an
|
||||
append-only ledger written by `lib/record_run.py` from a run's JUnit **and its run log**
|
||||
(see conformance/.gitignore for the negations that keep both in git).
|
||||
|
||||
The honest fix is a committed, machine-readable run ledger; until that exists, the most
|
||||
this report can say is "wired and testable", never "proven".
|
||||
THE REFUSAL POSTURE IS UNCHANGED. Evidence replaced "not derivable" only where evidence
|
||||
exists; nowhere is optimism substituted for it:
|
||||
|
||||
* A ledger row reaches `proven` only if `record_run.py` saw that test's own PASS
|
||||
sentinel token in the run log. The L2/L3 tests SELF-SKIP GREEN — Chainsaw records
|
||||
`passed` on `DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries no
|
||||
stdout — so a skip token maps to `skipped` and can never become proof.
|
||||
* A proof is honoured only while it still describes the current code: if any of the
|
||||
requirement's implementing manifests or verifying tests changed in a commit that is
|
||||
NOT an ancestor of the run's `git_sha`, the row degrades to `stale-proof`, which is
|
||||
not green. A run recorded from a dirty tree (`<sha>-dirty`) is never honoured.
|
||||
* A requirement with no ledger row, or whose verifying tests did not ALL report proven,
|
||||
stays `not-derivable` / `unknown` — never rounded up.
|
||||
* The untracked `_artifacts/report.junit.xml` is STILL only a quarantined advisory with
|
||||
its mtime. It is not the ledger, it is not evidence, and it feeds no status.
|
||||
|
||||
With an empty or absent ledger this report behaves exactly as it did before: every
|
||||
requirement `not-derivable`, headline PROVEN 0, "the most this report can say is wired and
|
||||
testable, never proven".
|
||||
|
||||
READS (all read-only)
|
||||
requirements/registry.yaml the authoritative R-* mirror (via lib/coverage.py)
|
||||
tests/**/chainsaw-test.yaml requirement tags + layer + executable annotation
|
||||
runs/ledger.jsonl COMMITTED proof-of-pass rows (via lib/record_run.py)
|
||||
clusters/<env>/ Flux entrypoint -> the reconciliation graph
|
||||
`git ls-files` TRACKED files only; untracked scratch is not repo state
|
||||
../home-network-k3s/docs/00-vision-and-scope.md OPTIONAL registry<->spec parity check
|
||||
@@ -92,7 +108,14 @@ FLUX_API = "kustomize.toolkit.fluxcd.io"
|
||||
# conformance/tests — those are the TESTS; they are the VERIFICATION axis, counted there.
|
||||
# ci/tests — gate unit-test fixtures embed example R-IDs as DATA.
|
||||
# registry.yaml — the requirement mirror itself, not an implementation of anything.
|
||||
IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/")
|
||||
# `conformance/runs/` is excluded for a subtler reason than the test dirs: the ledger and the
|
||||
# preserved run logs CONTAIN the requirement IDs they are evidence FOR. Counting them as
|
||||
# implementing evidence makes `last_change` move every time proof is recorded, so a row's
|
||||
# git_sha always predates it and EVERY proof degrades to stale-proof the instant it is
|
||||
# committed — recording the proof destroys it. The ledger is the proof record; it must not
|
||||
# participate in the freshness computation about itself. (Found live 2026-09-01 on the first
|
||||
# PROVEN row.)
|
||||
IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/", "conformance/runs/")
|
||||
IMPL_EXCLUDE_PATHS = ("conformance/requirements/registry.yaml",)
|
||||
|
||||
|
||||
@@ -274,6 +297,109 @@ _EXECUTABLE_RE = re.compile(
|
||||
_BLOCKED_RE = re.compile(
|
||||
r"^\s*" + re.escape(LABEL_PREFIX) + r"/blocked-by:\s*(.+)$", re.M)
|
||||
|
||||
# ---------------------------------------------------------- scaffold-marker detection
|
||||
# WHY THIS IS NOT A WHOLE-FILE `"TODO" in text`
|
||||
# ---------------------------------------------------
|
||||
# It used to be, and that mis-reported a SAFETY-CRITICAL requirement. A test whose SUBJECT
|
||||
# is placeholder detection has to name the idioms it hunts for:
|
||||
# `l0-static/placeholder-guard-coupling/check.sh` carries
|
||||
# `SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", ...]` — its detection vocabulary, as data.
|
||||
# A whole-file substring test read that as the test declaring itself unfinished, classified
|
||||
# a 32-assertion executable check as a scaffold, raised a permanent `--strict` annotation
|
||||
# conflict, and (because the same file is tagged R-STATE-03) reported R-STATE-03 as
|
||||
# `exec=1 scaffold=1`. A previous engineer worked around it by rewording prose.
|
||||
#
|
||||
# The rule below is derived from the corpus (43 tests), not invented. Every genuine
|
||||
# scaffold in this tree — `l2-behavioral/onboarding-completeness`, `l3-fault-injection/
|
||||
# alerting`, `l3-fault-injection/gitops-drift`, and no others — declares itself in BOTH of
|
||||
# two structural positions, and no executable test does either:
|
||||
#
|
||||
# 1. a `# STATUS: ... scaffold` line in the file's documentation header. This is the
|
||||
# repo's DEDICATED declaration slot (tests/README.md): 3 of 43 tests say
|
||||
# `STATUS: scaffold` and they are exactly the three scaffolds; 19 others declare
|
||||
# `STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`; the rest declare nothing.
|
||||
# 2. a TODO/FIXME marker in the BODY that CARRIES A MESSAGE — `echo "TODO: assert PVC
|
||||
# bound ..."`, `TODO(T-4.1): hermetic SMTP capture sink`. A marker that carries a
|
||||
# message is a note about work that is missing. A marker that is the ENTIRE content of
|
||||
# a quoted literal (`"TODO",` in a vocabulary list) carries no message: it is the
|
||||
# marker as a *datum*, which is what a placeholder DETECTOR necessarily contains.
|
||||
#
|
||||
# Both are still evaluated over the body of `chainsaw-test.yaml` AND of any
|
||||
# `render-check.sh` / `check.sh`, and either one still OUTRANKS an
|
||||
# `executable: "true"` annotation (see classify_test) — the invariant that an annotation
|
||||
# must never outrank the body is unchanged. Ambiguity still only ever DOWNGRADES: an
|
||||
# unrecognised marker shape, a marker in body code outside a quoted literal, or a marker
|
||||
# carrying any trailing text all classify as scaffold.
|
||||
#
|
||||
# KNOWN, ACCEPTED IMPRECISION (stated rather than hidden): a test that embeds a marker with
|
||||
# text attached but as data — e.g. `re.compile(r"TODO|FIXME")`, or a single string holding
|
||||
# several idioms — is still called a scaffold. That is the conservative direction, and the
|
||||
# author's remedy is the same structural discipline the corpus already uses: keep each
|
||||
# idiom a standalone literal in a list.
|
||||
|
||||
# Vocabulary: only the idioms that DECLARE unfinished work. Deliberately excludes `TBD`,
|
||||
# `PLACEHOLDER`, `XXX`, `REPLACE-ME` — tests in this tree legitimately carry those in
|
||||
# executable code as data (`l0-static/egress-paths-selected-by-rules/check.sh` compares
|
||||
# authored values against `"TBD"`; `l0-static/no-plaintext-secrets/check.sh` matches
|
||||
# `placeholder.*`), and none of them is used here to mark a placeholder body.
|
||||
# Case-insensitive: no English word contains either token, and the corpus carries no
|
||||
# lower-case occurrence, so folding case only ever ADDS detection (`echo "todo: ..."`).
|
||||
_MARKER_RE = re.compile(r"\b(?:TODO|FIXME)\b", re.I)
|
||||
|
||||
# The dedicated declaration slot, on a comment line, case-insensitive. The declared state
|
||||
# must be the FIRST token after `STATUS:` — a `STATUS:` line declares exactly one state,
|
||||
# and all three real scaffolds read `# STATUS: scaffold — <what is missing>`. Anchoring it
|
||||
# there is what keeps `# STATUS: EXECUTABLE ... this is not a scaffold` from matching; a
|
||||
# loose `.*scaffold` would turn any prose that merely uses the word into a declaration.
|
||||
_STATUS_SCAFFOLD_RE = re.compile(
|
||||
r"^[ \t]*#[ \t]*STATUS:[ \t]*(?:scaffold|placeholder)\b", re.M | re.I)
|
||||
|
||||
# A quoted literal whose entire content is one marker word: the marker as data.
|
||||
_MARKER_AS_DATUM_RE = re.compile(r"(['\"])[ \t]*(?:TODO|FIXME)[ \t]*\1", re.I)
|
||||
|
||||
|
||||
def _split_doc_header(text: str) -> tuple[str, str]:
|
||||
"""Split a test file into (documentation header, body).
|
||||
|
||||
Both file kinds in this tree open with a banner comment block that is PROSE about the
|
||||
test: the Given/When/Then, the incident being closed, the mandatory "what this does NOT
|
||||
prove" list (tests/README.md, L0 authoring rule 1). Prose necessarily discusses
|
||||
placeholder idioms when the test hunts them, so an incidental mention there is
|
||||
documentation, not a placeholder body.
|
||||
|
||||
The body starts at the first line that is neither blank nor a `#` comment:
|
||||
*.sh -> `set -euo pipefail` (the shebang is a comment)
|
||||
*.yaml -> `apiVersion: chainsaw.kyverno.io/v1alpha1`
|
||||
Comments BELOW that point are body, deliberately: a scaffold marks a missing fixture
|
||||
with an inline comment (`l3-fault-injection/alerting`: `file: fixtures/smtp-sink.yaml
|
||||
# TODO(T-4.1): hermetic SMTP capture sink`), and that must keep counting.
|
||||
"""
|
||||
lines = text.splitlines(keepends=True)
|
||||
for i, ln in enumerate(lines):
|
||||
s = ln.strip()
|
||||
if s and not s.startswith("#"):
|
||||
return "".join(lines[:i]), "".join(lines[i:])
|
||||
return text, "" # no executable line at all: it is all header, body is empty
|
||||
|
||||
|
||||
def scaffold_marker(text: str) -> str:
|
||||
"""Why `text` declares itself a placeholder body, or "" if it does not.
|
||||
|
||||
Returns a short human-readable reason (used verbatim in the hygiene report) so the
|
||||
defect is actionable rather than a bare path.
|
||||
"""
|
||||
head, body = _split_doc_header(text)
|
||||
m = _STATUS_SCAFFOLD_RE.search(head) or _STATUS_SCAFFOLD_RE.search(body)
|
||||
if m:
|
||||
return m.group(0).strip().lstrip("#").strip()[:80]
|
||||
# Blank out markers held as pure data before looking for a marker that has a message.
|
||||
m2 = _MARKER_RE.search(_MARKER_AS_DATUM_RE.sub('""', body))
|
||||
if m2:
|
||||
line = body[:m2.start()].count("\n")
|
||||
snippet = body.splitlines()[line].strip()
|
||||
return f"placeholder marker in body: {snippet[:80]}"
|
||||
return ""
|
||||
|
||||
|
||||
def classify_test(t: dict) -> dict:
|
||||
"""Decide EXECUTABLE vs SCAFFOLD for one discovered test, conservatively.
|
||||
@@ -281,14 +407,18 @@ def classify_test(t: dict) -> dict:
|
||||
A tagged test is only worth counting if it can actually assert something. Rules, in
|
||||
order, chosen so ambiguity NEVER upgrades a test:
|
||||
|
||||
scaffold the body contains TODO markers (a placeholder), *even if* it
|
||||
is annotated executable — an annotation must never outrank the
|
||||
body. That mismatch is also reported as a hygiene defect.
|
||||
executable annotated `conformance.home.k3s/executable: "true"` and no TODOs.
|
||||
executable-untagged no TODOs and a real runnable body, but the annotation is
|
||||
missing (true of the L1 render tests, which execute a
|
||||
render-check.sh + golden diff, of the L0 static checks, and
|
||||
of the META self-check).
|
||||
scaffold the test DECLARES itself a placeholder — a `# STATUS: scaffold`
|
||||
header line, or a TODO/FIXME marker carrying a message in the
|
||||
body (see `scaffold_marker` for the rule and the corpus it was
|
||||
derived from) — *even if* it is annotated executable. An
|
||||
annotation must never outrank the body. That mismatch is also
|
||||
reported as a hygiene defect.
|
||||
executable annotated `conformance.home.k3s/executable: "true"` and no
|
||||
scaffold declaration.
|
||||
executable-untagged no scaffold declaration and a real runnable body, but the
|
||||
annotation is missing (true of the L1 render tests, which
|
||||
execute a render-check.sh + golden diff, of the L0 static
|
||||
checks, and of the META self-check).
|
||||
Counted as executable, and flagged as an annotation gap.
|
||||
scaffold anything else.
|
||||
"""
|
||||
@@ -299,16 +429,26 @@ def classify_test(t: dict) -> dict:
|
||||
# downgrades the test to a scaffold — an annotation must never outrank the body,
|
||||
# whichever file the body lives in.
|
||||
extra = ""
|
||||
scripts: list[tuple[str, str]] = []
|
||||
for script in ("render-check.sh", "check.sh"):
|
||||
sp = p.parent / script
|
||||
if sp.is_file():
|
||||
extra += sp.read_text()
|
||||
body = sp.read_text()
|
||||
extra += body
|
||||
scripts.append((script, body))
|
||||
declared = bool(_EXECUTABLE_RE.search(text))
|
||||
has_todo = "TODO" in text or "TODO" in extra
|
||||
# Each file is classified on ITS OWN structure — the yaml and the script have different
|
||||
# header blocks, so they must not be concatenated before the header/body split.
|
||||
reason = scaffold_marker(text)
|
||||
for script, body in scripts:
|
||||
if reason:
|
||||
break
|
||||
r = scaffold_marker(body)
|
||||
reason = f"{script}: {r}" if r else ""
|
||||
bm = _BLOCKED_RE.search(text)
|
||||
has_body = bool(extra) or bool(re.search(r"^\s+- (script|assert|apply):", text, re.M))
|
||||
|
||||
if has_todo:
|
||||
if reason:
|
||||
state = "scaffold"
|
||||
elif declared:
|
||||
state = "executable"
|
||||
@@ -325,7 +465,8 @@ def classify_test(t: dict) -> dict:
|
||||
"name": t["name"], "path": rel, "layer": t["layer"],
|
||||
"state": state,
|
||||
"declared_executable": declared,
|
||||
"annotation_conflict": bool(declared and has_todo),
|
||||
"annotation_conflict": bool(declared and reason),
|
||||
"scaffold_reason": reason,
|
||||
"blocked_by": bm.group(1).strip().strip('"') if bm else "",
|
||||
}
|
||||
|
||||
@@ -343,6 +484,169 @@ def spec_requirement_ids(spec_root: Path) -> set[str] | None:
|
||||
return set(RID_RE.findall(doc.read_text()))
|
||||
|
||||
|
||||
# ========================================================================== ledger
|
||||
# PROOF axis. Everything here reads `conformance/runs/ledger.jsonl` (written ONLY by an
|
||||
# explicit `lib/record_run.py` / `run.sh --record` invocation) and downgrades relentlessly.
|
||||
# Nothing in this section may ever turn absent, ambiguous or aged evidence into green.
|
||||
|
||||
DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
|
||||
|
||||
# Severity ranking, WORST FIRST. A requirement's proof is the MINIMUM over its verifying
|
||||
# tests, so one failed/skipped/missing test drags the whole requirement down and a single
|
||||
# proven test can never carry the others. `stale-proof` ranks below `proven` on purpose:
|
||||
# it is a real past pass that no longer describes the current code, and it is NOT green.
|
||||
PROOF_ORDER = {
|
||||
"failed": 0, # a ledger row recorded a JUnit failure/error
|
||||
"unknown": 1, # ran, but the log could not evidence pass vs self-skip
|
||||
"not-derivable": 2, # no ledger row for this test at all (the pre-ledger default)
|
||||
"junit-skipped": 3, # JUnit <skipped/> — did not execute
|
||||
"skipped": 4, # SELF-SKIPPED green: exited 0 without exercising the behaviour
|
||||
"stale-proof": 5, # proven at a sha the implementation/test has since moved past
|
||||
"proven": 6, # the ONLY green state
|
||||
}
|
||||
# record_run.py's two "we cannot tell" states collapse to one reported state here; the
|
||||
# per-test detail in status.json keeps the distinction for whoever has to fix it.
|
||||
_PROOF_ALIASES = {
|
||||
"unknown-no-sentinel-declared": "unknown",
|
||||
"unknown-sentinel-absent": "unknown",
|
||||
}
|
||||
|
||||
|
||||
def read_ledger(path: Path) -> list[dict]:
|
||||
"""Parse the append-only JSONL ledger. A malformed line is SKIPPED, never guessed at.
|
||||
|
||||
Absent or empty file -> [] , which reproduces this report's pre-ledger behaviour
|
||||
exactly (every requirement `not-derivable`, headline PROVEN 0).
|
||||
"""
|
||||
if not path.is_file():
|
||||
return []
|
||||
rows: list[dict] = []
|
||||
for line in path.read_text(errors="replace").splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(obj, dict) and obj.get("test"):
|
||||
rows.append(obj)
|
||||
return rows
|
||||
|
||||
|
||||
def latest_by_test(rows: list[dict]) -> dict[str, dict]:
|
||||
"""Most recent ledger row per test name (ts_utc, then file order as the tiebreak).
|
||||
|
||||
Latest wins rather than best-ever: a test that has since gone back to self-skipping
|
||||
must stop reading as proven.
|
||||
"""
|
||||
latest: dict[str, dict] = {}
|
||||
for i, r in enumerate(rows):
|
||||
key = r["test"]
|
||||
prev = latest.get(key)
|
||||
if prev is None or (r.get("ts_utc", ""), i) >= (prev.get("ts_utc", ""),
|
||||
prev["_i"]):
|
||||
latest[key] = {**r, "_i": i}
|
||||
return latest
|
||||
|
||||
|
||||
class _GitOracle:
|
||||
"""Small cached wrapper for the two git questions the staleness rule asks."""
|
||||
|
||||
def __init__(self, repo: Path):
|
||||
self.repo = repo
|
||||
self._last: dict[tuple[str, ...], str | None] = {}
|
||||
self._anc: dict[tuple[str, str], bool] = {}
|
||||
|
||||
def _git(self, *args: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(["git", "-C", str(self.repo), *args],
|
||||
capture_output=True, text=True)
|
||||
|
||||
def last_change(self, paths: tuple[str, ...]) -> str | None:
|
||||
"""Commit that last touched any of `paths`; None if git knows none of them."""
|
||||
if not paths:
|
||||
return None
|
||||
if paths in self._last:
|
||||
return self._last[paths]
|
||||
p = self._git("log", "-1", "--format=%H", "--", *paths)
|
||||
sha = p.stdout.strip() if p.returncode == 0 else ""
|
||||
self._last[paths] = sha or None
|
||||
return self._last[paths]
|
||||
|
||||
def is_ancestor(self, older: str, newer: str) -> bool:
|
||||
"""True iff `older` is an ancestor of (or equal to) `newer`.
|
||||
|
||||
False whenever either rev is unresolvable — an unknown or `-dirty` sha (a run
|
||||
recorded from a modified working tree) can never satisfy the freshness test, which
|
||||
is the intended fail-closed direction.
|
||||
"""
|
||||
key = (older, newer)
|
||||
if key in self._anc:
|
||||
return self._anc[key]
|
||||
ok = self._git("merge-base", "--is-ancestor", older, newer).returncode == 0
|
||||
self._anc[key] = ok
|
||||
return ok
|
||||
|
||||
|
||||
def derive_proof(git: _GitOracle, rid: str, verifiers: list[dict],
|
||||
impl_files: list[dict], latest: dict[str, dict]) -> dict:
|
||||
"""PROOF for one requirement, from the ledger, with the staleness rule applied.
|
||||
|
||||
STALENESS RULE — a proof is a claim about a specific tree, not a permanent badge:
|
||||
relevant paths = every implementing artifact this report found for the requirement
|
||||
+ every verifying test's directory
|
||||
last_change = `git log -1 -- <relevant paths>`
|
||||
A row's `proven` is honoured only if `last_change` is an ANCESTOR of the row's
|
||||
`git_sha`, i.e. the run happened at or after the newest change to the code it
|
||||
claims to prove. Otherwise it degrades to `stale-proof` (not green). A `-dirty`
|
||||
sha, a sha this clone does not have (rebased/dropped), or a path git has never
|
||||
seen (an uncommitted test) all fail the ancestry test and degrade the same way.
|
||||
|
||||
The requirement's state is the MINIMUM over its verifying tests: every verifying test
|
||||
must be proven and fresh before the requirement reads `proven`. A test with no ledger
|
||||
row contributes `not-derivable`, so a partially-recorded run cannot manufacture green.
|
||||
"""
|
||||
paths = sorted({f["path"] for f in impl_files}
|
||||
| {os.path.dirname(v["path"]) for v in verifiers if v.get("path")})
|
||||
last_change = git.last_change(tuple(paths))
|
||||
|
||||
per_test: list[dict] = []
|
||||
for v in verifiers:
|
||||
row = latest.get(v["name"])
|
||||
if row is None:
|
||||
per_test.append({"test": v["name"], "state": "not-derivable",
|
||||
"reason": "no ledger row — this test has never been "
|
||||
"recorded by run.sh --record / record_run.py"})
|
||||
continue
|
||||
base = _PROOF_ALIASES.get(row.get("proof", ""), row.get("proof", "unknown"))
|
||||
if base not in PROOF_ORDER:
|
||||
base = "unknown"
|
||||
state, reason = base, row.get("note", "")
|
||||
if base == "proven":
|
||||
sha = row.get("git_sha", "")
|
||||
if last_change and not git.is_ancestor(last_change, sha):
|
||||
state = "stale-proof"
|
||||
reason = (f"proven at {sha[:12]}, but the implementing manifests / test "
|
||||
f"changed in {last_change[:12]}, which that run predates")
|
||||
per_test.append({"test": v["name"], "state": state, "reason": reason,
|
||||
"run_id": row.get("run_id"), "ts_utc": row.get("ts_utc"),
|
||||
"git_sha": row.get("git_sha"), "env": row.get("env"),
|
||||
"sentinel": row.get("sentinel"),
|
||||
"junit_result": row.get("junit_result")})
|
||||
|
||||
if not per_test:
|
||||
state = "not-derivable"
|
||||
reason = "no requirement-tagged test exists to be proven"
|
||||
else:
|
||||
worst = min(per_test, key=lambda d: PROOF_ORDER[d["state"]])
|
||||
state, reason = worst["state"], worst["reason"]
|
||||
if state == "not-derivable" and not reason:
|
||||
reason = "no committed ledger row for this requirement's verifying test(s)"
|
||||
|
||||
return {"state": state, "reason": reason, "last_change_sha": last_change,
|
||||
"evidence_paths": paths, "per_test": per_test}
|
||||
|
||||
|
||||
# ========================================================================== build
|
||||
IMPL_ORDER = {"none": 0, "off-cluster": 1, "unwired": 2, "reconciled-partial": 3,
|
||||
"reconciled": 4}
|
||||
@@ -369,7 +673,13 @@ LABELS = {
|
||||
|
||||
def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, list[dict]],
|
||||
applied: set[str], gitops_roots: set[str], spec_ids: set[str] | None,
|
||||
env: str) -> dict:
|
||||
env: str, ledger_rows: list[dict] | None = None) -> dict:
|
||||
# PROOF axis inputs. `ledger_rows=None` / `[]` reproduces the pre-ledger behaviour:
|
||||
# `latest` is empty, so every requirement derives `not-derivable` exactly as before.
|
||||
ledger_rows = ledger_rows or []
|
||||
latest = latest_by_test(ledger_rows)
|
||||
git = _GitOracle(repo)
|
||||
|
||||
classified = [classify_test(t) for t in tests]
|
||||
by_req_tests: dict[str, list[dict]] = {r["id"]: [] for r in registry}
|
||||
for t, c in zip(tests, classified):
|
||||
@@ -439,17 +749,19 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
|
||||
"blocked_by": sorted({t["blocked_by"] for t in ts if t["blocked_by"]}),
|
||||
"tests": ts,
|
||||
},
|
||||
# NEVER derived. See the module docstring.
|
||||
"proof": {"state": "not-derivable",
|
||||
"reason": "no committed, machine-readable record of a test run "
|
||||
"exists in the repo (_artifacts/ and results/ are "
|
||||
"git-ignored); DECISIONS-LOG.md prose is not a signal"},
|
||||
# Derived ONLY from the committed ledger, and only as far as it reaches.
|
||||
# With no ledger row this stays `not-derivable`, as it always was.
|
||||
"proof": derive_proof(git, rid, ts, ev, latest),
|
||||
"status": LABELS[(impl_state, verify_state)],
|
||||
}
|
||||
|
||||
# --- hygiene defects this tool owns -------------------------------------------
|
||||
hygiene = {
|
||||
"annotation_conflicts": [t["path"] for t in classified if t["annotation_conflict"]],
|
||||
# Same set, with WHY each one was called a placeholder body, so the fix is obvious
|
||||
# from the report instead of requiring the reader to re-derive the rule.
|
||||
"annotation_conflict_reasons": {t["path"]: t["scaffold_reason"]
|
||||
for t in classified if t["annotation_conflict"]},
|
||||
"executable_untagged": [t["path"] for t in classified
|
||||
if t["state"] == "executable-untagged"],
|
||||
"registry_not_in_spec": [],
|
||||
@@ -492,8 +804,19 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
|
||||
and v["verification"]["state"] == "executable"),
|
||||
"safety_critical_total": count(lambda v: v["safety_critical"]),
|
||||
"safety_critical_without_executable_l3": len(safety_scaffold),
|
||||
"proven": 0,
|
||||
"proven_derivable": False,
|
||||
# PROVEN is now a real count — but only of requirements whose EVERY verifying
|
||||
# test has a committed ledger row that saw its pass sentinel, at a sha the
|
||||
# implementation has not moved past. `stale-proof` and `skipped` are counted
|
||||
# separately and are NOT green.
|
||||
"proven": count(lambda v: v["proof"]["state"] == "proven"),
|
||||
"proof_stale": count(lambda v: v["proof"]["state"] == "stale-proof"),
|
||||
"proof_skipped": count(lambda v: v["proof"]["state"] == "skipped"),
|
||||
"proof_failed": count(lambda v: v["proof"]["state"] == "failed"),
|
||||
"proof_unknown": count(lambda v: v["proof"]["state"] == "unknown"),
|
||||
"proof_not_derivable": count(
|
||||
lambda v: v["proof"]["state"] in ("not-derivable", "junit-skipped")),
|
||||
"ledger_rows": len(ledger_rows),
|
||||
"proven_derivable": bool(ledger_rows),
|
||||
"coverage_style_covered": count(lambda v: v["verification"]["state"] != "none"),
|
||||
}
|
||||
summary["uncited_by_any_reconciled_manifest"] = count(
|
||||
@@ -552,16 +875,39 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
|
||||
o.append(f"| Safety-critical requirements whose L3 test is a scaffold | "
|
||||
f"**{s['safety_critical_without_executable_l3']}** of "
|
||||
f"{s['safety_critical_total']} |")
|
||||
o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
|
||||
o.append("")
|
||||
o.append("> **Proof is deliberately not derived.** Nothing tracked in git records a "
|
||||
"test result: `conformance/_artifacts/` and `results/` are git-ignored, and "
|
||||
"the only record of a green run is DECISIONS-LOG.md prose — hand-maintained, "
|
||||
"which is the very thing this report replaces. Per "
|
||||
"`agents/definition-of-done.md` a requirement is done only when its tagged "
|
||||
"test **passes**, so the strongest claim this report can make is "
|
||||
"*wired and testable*. Closing the gap needs a committed, machine-readable "
|
||||
"run ledger; until then, treat every row below as unproven.\n")
|
||||
if s["proven_derivable"]:
|
||||
o.append(f"| **PROVEN (every verifying test passed, evidenced)** | "
|
||||
f"**{s['proven']}** |")
|
||||
o.append(f"| Proof exists but is STALE (code changed after the run) | "
|
||||
f"{s['proof_stale']} |")
|
||||
o.append(f"| Verifying test SELF-SKIPPED green (not proof) | {s['proof_skipped']} |")
|
||||
o.append(f"| Verifying test FAILED in the recorded run | {s['proof_failed']} |")
|
||||
o.append(f"| Ran, but the log could not evidence pass vs self-skip | "
|
||||
f"{s['proof_unknown']} |")
|
||||
o.append(f"| No committed ledger row yet | {s['proof_not_derivable']} |")
|
||||
o.append("")
|
||||
o.append(f"> **Proof comes from `conformance/runs/ledger.jsonl`** "
|
||||
f"({s['ledger_rows']} row(s)), written only by an explicit "
|
||||
f"`run.sh --record` / `lib/record_run.py` invocation. A row reaches "
|
||||
f"`proven` ONLY when the run log carried that test's own pass sentinel "
|
||||
f"token — the L2/L3 tests self-skip GREEN (Chainsaw records `passed` on "
|
||||
f"`DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries "
|
||||
f"no stdout), so a skip token maps to `skipped` and can never become "
|
||||
f"proof. A proof is honoured only while the implementing manifests and "
|
||||
f"the test itself have not changed since the recorded `git_sha`; "
|
||||
f"otherwise it degrades to `stale-proof`, which is not green.\n")
|
||||
else:
|
||||
o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
|
||||
o.append("")
|
||||
o.append("> **No proof is derivable: `conformance/runs/ledger.jsonl` is absent or "
|
||||
"empty.** The ledger is the committed, machine-readable run record; until "
|
||||
"a run is captured with `run.sh --record`, nothing tracked in git records "
|
||||
"a test result (`conformance/_artifacts/` and `results/` are git-ignored, "
|
||||
"and DECISIONS-LOG.md prose is hand-maintained — the very thing this "
|
||||
"report replaces). Per `agents/definition-of-done.md` a requirement is "
|
||||
"done only when its tagged test **passes**, so the strongest claim this "
|
||||
"report can make is *wired and testable*; treat every row below as "
|
||||
"unproven.\n")
|
||||
o.append(f"> For contrast, `coverage.md` counts **{s['coverage_style_covered']}** "
|
||||
f"requirements as covered — that count includes the "
|
||||
f"{s['verify_scaffold_only']} whose only verifying test is a TODO scaffold.\n")
|
||||
@@ -612,7 +958,15 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
|
||||
if ver["blocked_by"]:
|
||||
vcell += f" · blocked-by: {'; '.join(ver['blocked_by'])}"
|
||||
sc = " ⚠safety" if v["safety_critical"] else ""
|
||||
o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | not-derivable |")
|
||||
pr = v["proof"]
|
||||
if pr["state"] == "not-derivable":
|
||||
pcell = "not-derivable"
|
||||
else:
|
||||
runs = sorted({d["run_id"] for d in pr["per_test"] if d.get("run_id")})
|
||||
pcell = f"**{pr['state']}**" if pr["state"] == "proven" else pr["state"]
|
||||
if runs:
|
||||
pcell += f" ({', '.join(runs)})"
|
||||
o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | {pcell} |")
|
||||
|
||||
if rep["safety_critical_without_executable_l3"]:
|
||||
o.append("\n## Safety-critical requirements whose L3 fault test is a SCAFFOLD "
|
||||
@@ -678,8 +1032,10 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
|
||||
else:
|
||||
o.append("- registry ↔ spec parity: **NOT CHECKED** (spec tree not available; "
|
||||
"this is a skip, not a pass)")
|
||||
o.append(f"- tests annotated executable but containing TODO bodies: "
|
||||
f"{h['annotation_conflicts'] or 'none'}")
|
||||
o.append("- tests annotated executable but declaring a placeholder body: "
|
||||
+ (", ".join(f"`{k}` ({v})"
|
||||
for k, v in sorted(h["annotation_conflict_reasons"].items()))
|
||||
if h["annotation_conflicts"] else "none"))
|
||||
o.append(f"- executable tests missing the `{LABEL_PREFIX}/executable` annotation: "
|
||||
f"{h['executable_untagged'] or 'none'}")
|
||||
|
||||
@@ -698,25 +1054,82 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
|
||||
"reconciled |")
|
||||
o.append("| verifying test | `conformance.home.k3s/R-*` label on a Chainsaw test "
|
||||
"(the authoritative tag, same source as `coverage.md`) | exact |")
|
||||
o.append("| executable vs scaffold | TODO markers in the test body outrank the "
|
||||
"`executable` annotation; a real body without the annotation still counts "
|
||||
"and is flagged | conservative — ambiguity never upgrades |")
|
||||
o.append("| **test actually passed** | **NOT DERIVED** — nothing in git records a "
|
||||
"run result | refused |")
|
||||
o.append("| executable vs scaffold | a `# STATUS: scaffold` header line, or a "
|
||||
"TODO/FIXME marker CARRYING A MESSAGE in the body (a marker held as a bare "
|
||||
"quoted datum — a placeholder detector's own vocabulary — is not one); "
|
||||
"either outranks the `executable` annotation, and a real body without the "
|
||||
"annotation still counts and is flagged | conservative — ambiguity never "
|
||||
"upgrades |")
|
||||
o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for "
|
||||
"every verifying test: JUnit result **plus** that test's own pass sentinel "
|
||||
"token grepped from the run log | exact where a run was recorded, refused "
|
||||
"everywhere else — a self-skip token or a missing/unrecognised sentinel "
|
||||
"NEVER yields `proven` |")
|
||||
o.append("| proof still current | the row's `git_sha` must be a descendant of the "
|
||||
"last commit touching the requirement's implementing manifests or its test "
|
||||
"| conservative — an unresolvable or `-dirty` sha degrades to `stale-proof` |")
|
||||
o.append("| cluster liveness (pods/CRs healthy) | **NOT DERIVED** — no cluster is "
|
||||
"contacted; `reconciled` means Flux *would* apply it, not that it is "
|
||||
"healthy | refused |")
|
||||
o.append("| phase/board status | **NOT DERIVED** — `agents/task-board.md` is stale "
|
||||
"and is not a signal | refused |")
|
||||
|
||||
o.append("\n## Run ledger — the committed proof-of-pass evidence\n")
|
||||
if not s["proven_derivable"]:
|
||||
o.append("- `conformance/runs/ledger.jsonl` holds no rows. Capture a run with "
|
||||
"`./run.sh --record` (recording is never implicit: a run that silently "
|
||||
"records itself is how a bad run becomes a permanent false proof), then "
|
||||
"commit `conformance/runs/<run-id>/` **with its `run.log`** and the "
|
||||
"ledger. The log is the load-bearing half — Chainsaw's JUnit carries no "
|
||||
"stdout, so without it a self-skip is indistinguishable from a pass.")
|
||||
else:
|
||||
o.append(f"- {s['ledger_rows']} row(s) in `conformance/runs/ledger.jsonl`. "
|
||||
f"Per-test evidence for every non-`not-derivable` requirement below; "
|
||||
f"full rows (env, chainsaw/kube versions, sentinel, note) are in "
|
||||
f"`status.json` under `by_requirement.<R-ID>.proof.per_test`.\n")
|
||||
o.append("| Requirement | Proof | Test | Ledger state | Sentinel | Run |")
|
||||
o.append("|---|---|---|---|---|---|")
|
||||
for rid, v in rep["by_requirement"].items():
|
||||
pr = v["proof"]
|
||||
if pr["state"] == "not-derivable":
|
||||
continue
|
||||
for d in pr["per_test"]:
|
||||
o.append(f"| {rid} | {pr['state']} | `{d['test']}` | {d['state']} | "
|
||||
f"`{d.get('sentinel') or '—'}` | {d.get('run_id') or '—'} |")
|
||||
stale = [(rid, v) for rid, v in rep["by_requirement"].items()
|
||||
if v["proof"]["state"] == "stale-proof"]
|
||||
if stale:
|
||||
o.append("\n### Stale proofs — a real pass that no longer describes this tree\n")
|
||||
o.append("These requirements DID pass, but their implementing manifests or "
|
||||
"their verifying test changed in a commit the recorded run predates. "
|
||||
"A stale proof is not green; re-run with `--record`.\n")
|
||||
for rid, v in stale:
|
||||
pr = v["proof"]
|
||||
o.append(f"- **{rid}** — {pr['reason']}")
|
||||
skipped = [(rid, v) for rid, v in rep["by_requirement"].items()
|
||||
if v["proof"]["state"] == "skipped"]
|
||||
if skipped:
|
||||
o.append("\n### SELF-SKIPPED green — recorded, and deliberately NOT proof\n")
|
||||
o.append("Chainsaw reported `passed` for these, because the test exited 0 "
|
||||
"after emitting its skip token: a precondition (Longhorn "
|
||||
"StorageClass, Velero CRDs, >=3 Ready nodes) was absent, so the "
|
||||
"behaviour was never exercised. This is exactly the row a "
|
||||
"JUnit-only ledger would have recorded as green.\n")
|
||||
for rid, v in skipped:
|
||||
toks = ", ".join(sorted({d.get("sentinel") or "?"
|
||||
for d in v["proof"]["per_test"]
|
||||
if d["state"] == "skipped"}))
|
||||
o.append(f"- **{rid}** — skip token(s): `{toks}`")
|
||||
|
||||
o.append("\n## Advisory — untracked local run artifact (NOT counted)\n")
|
||||
if advisory:
|
||||
o.append(f"- `{advisory['path']}` (untracked, git-ignored), last written "
|
||||
f"{advisory['mtime']}: {advisory['tests']} testcase(s), "
|
||||
f"{advisory['failures']} failure(s), {advisory['skipped']} skipped.")
|
||||
o.append(f"- requirement IDs it names: {', '.join(advisory['requirements']) or '—'}")
|
||||
o.append("- This is one developer's local workspace, not repo state. It is shown "
|
||||
"for orientation only and contributes to NO status above.")
|
||||
o.append("- This is one developer's local workspace, not repo state, and it is NOT "
|
||||
"the ledger. It is shown for orientation only and contributes to NO "
|
||||
"status above — a JUnit alone cannot tell a self-skip from a pass.")
|
||||
else:
|
||||
o.append("- none present.")
|
||||
return "\n".join(o) + "\n"
|
||||
@@ -841,6 +1254,186 @@ def _selftest() -> int:
|
||||
classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"]
|
||||
== "scaffold")
|
||||
|
||||
# === THE BUG THIS DETECTOR WAS REWRITTEN FOR ==============================
|
||||
# A whole-file `"TODO" in text` mis-classified `l0-static/placeholder-guard-
|
||||
# coupling` — a 32-assertion executable check whose SUBJECT is placeholder
|
||||
# detection, so it must name the idioms it hunts — as a scaffold, and through it
|
||||
# reported the SAFETY-CRITICAL R-STATE-03 as `exec=1 scaffold=1`. These
|
||||
# assertions pin BOTH directions: a marker held as DATA must not downgrade, and a
|
||||
# real placeholder body must still be caught. Both halves are load-bearing; the
|
||||
# second is what stops this fix from degrading into "always executable".
|
||||
yml = ("metadata:\n annotations:\n"
|
||||
" conformance.home.k3s/executable: \"true\"\n"
|
||||
"spec:\n steps:\n - script:\n"
|
||||
" content: bash ./check.sh\n")
|
||||
f.write_text(yml)
|
||||
detector = td2 / "check.sh"
|
||||
# Shaped exactly like the real check: a prose header that discusses the idioms,
|
||||
# then executable code holding them as a vocabulary list.
|
||||
detector.write_text(
|
||||
"#!/usr/bin/env bash\n"
|
||||
"# L0 static check — placeholder vocabulary\n"
|
||||
"# * SEED — loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER)\n"
|
||||
"# STATUS: EXECUTABLE. A pure offline assertion script.\n"
|
||||
"set -euo pipefail\n"
|
||||
"python3 - <<'EOF'\n"
|
||||
"SEED = [\"TBD\", \"TODO\", \"FIXME\", \"PLACEHOLDER\"]\n"
|
||||
"assert any(w in open('/etc/hostname').read().upper() for w in SEED) is False\n"
|
||||
"EOF\n")
|
||||
c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
|
||||
check("a marker in HEADER PROSE does not downgrade an executable check",
|
||||
c["state"] == "executable" and not c["annotation_conflict"])
|
||||
check("a marker held as a bare quoted DATUM does not downgrade (the R-STATE-03 bug)",
|
||||
c["scaffold_reason"] == "")
|
||||
|
||||
# ... and the protection is intact: a real placeholder body is still caught, in
|
||||
# each of the three shapes the corpus actually uses.
|
||||
detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
|
||||
"echo \"TODO: assert the PVC is bound\"\n")
|
||||
c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
|
||||
check("a TODO carrying a message in check.sh IS still a scaffold",
|
||||
c["state"] == "scaffold" and c["annotation_conflict"])
|
||||
check("the scaffold reason names the file and the offending line",
|
||||
c["scaffold_reason"].startswith("check.sh:")
|
||||
and "TODO: assert the PVC is bound" in c["scaffold_reason"])
|
||||
detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
|
||||
"# TODO(T-4.1) write the hermetic sink fixture\ntrue\n")
|
||||
check("a TODO(T-x.y) roadmap marker in an inline body comment IS still a scaffold",
|
||||
classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
|
||||
== "scaffold")
|
||||
detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
|
||||
"echo \"FIXME the extraction before trusting this\"\n")
|
||||
check("an unpunctuated FIXME carrying trailing prose IS still a scaffold",
|
||||
classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
|
||||
== "scaffold")
|
||||
# The dedicated declaration slot outranks a body that would otherwise look real —
|
||||
# this is how all three genuine scaffolds in the tree declare themselves.
|
||||
detector.unlink()
|
||||
f.write_text("# L3 FAULT-INJECTION SCAFFOLD — x\n"
|
||||
"# STATUS: scaffold — needs T-1.4 (Flux bootstrapped).\n" + yml)
|
||||
c = classify_test({"name": "s", "path": str(f), "layer": "L3"})
|
||||
check("a `# STATUS: scaffold` header declaration IS a scaffold, body notwithstanding",
|
||||
c["state"] == "scaffold" and c["annotation_conflict"])
|
||||
check("`# STATUS: EXECUTABLE` is not read as a scaffold declaration",
|
||||
scaffold_marker("# STATUS: EXECUTABLE. `check.sh` is a pure offline "
|
||||
"assertion script; covers no scaffold.\nset -e\n") == "")
|
||||
|
||||
# ---------------------------------------------------------------- PROOF / ledger
|
||||
# These are the assertions that stand between this report and a fake green.
|
||||
check("empty ledger parses to no rows (pre-ledger behaviour is preserved)",
|
||||
read_ledger(Path("/nonexistent/ledger.jsonl")) == [])
|
||||
|
||||
def row(**kw):
|
||||
base = {"ts_utc": "2026-09-01T12:00:00Z", "git_sha": "deadbeef", "env": "dev",
|
||||
"chainsaw_version": "v0.2.12", "kube_version": "v1.36.4", "layer": "L3",
|
||||
"test": "dr-restore", "requirements": ["R-STATE-03"],
|
||||
"junit_result": "passed", "sentinel": None, "proof": "skipped",
|
||||
"run_id": "20260901T120000Z-deadbee", "note": ""}
|
||||
base.update(kw)
|
||||
return base
|
||||
|
||||
verifiers = [{"name": "dr-restore",
|
||||
"path": "conformance/tests/l3-fault-injection/dr-restore/"
|
||||
"chainsaw-test.yaml"}]
|
||||
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
gr = Path(td)
|
||||
env = {**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@t",
|
||||
"GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@t"}
|
||||
|
||||
def git(*a):
|
||||
return subprocess.run(["git", "-C", str(gr), *a], capture_output=True,
|
||||
text=True, env=env)
|
||||
|
||||
git("init", "-q", "-b", "main")
|
||||
(gr / "platform").mkdir()
|
||||
(gr / "platform" / "velero.yaml").write_text("kind: X # R-STATE-03\n")
|
||||
git("add", "-A"); git("commit", "-qm", "one")
|
||||
sha_a = git("rev-parse", "HEAD").stdout.strip()
|
||||
(gr / "platform" / "velero.yaml").write_text("kind: X # R-STATE-03 v2\n")
|
||||
git("add", "-A"); git("commit", "-qm", "two")
|
||||
sha_b = git("rev-parse", "HEAD").stdout.strip()
|
||||
|
||||
oracle = _GitOracle(gr)
|
||||
impl = [{"path": "platform/velero.yaml"}]
|
||||
|
||||
# *** THE CASE THIS WHOLE FEATURE EXISTS FOR ***
|
||||
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
|
||||
latest_by_test([row(proof="skipped", junit_result="passed",
|
||||
sentinel="DR_RESTORE_SKIPPED_OK",
|
||||
git_sha=sha_b)]))
|
||||
check("a SELF-SKIP row on a GREEN JUnit is 'skipped', NEVER 'proven'",
|
||||
pr["state"] == "skipped")
|
||||
|
||||
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
|
||||
latest_by_test([row(proof="proven", git_sha=sha_b,
|
||||
sentinel="DR_RESTORED_OK")]))
|
||||
check("a proven row at the CURRENT sha is honoured", pr["state"] == "proven")
|
||||
|
||||
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
|
||||
latest_by_test([row(proof="proven", git_sha=sha_a,
|
||||
sentinel="DR_RESTORED_OK")]))
|
||||
check("STALENESS: proven at a sha the implementation has moved past degrades "
|
||||
"to stale-proof", pr["state"] == "stale-proof")
|
||||
|
||||
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
|
||||
latest_by_test([row(proof="proven", git_sha=f"{sha_b}-dirty",
|
||||
sentinel="DR_RESTORED_OK")]))
|
||||
check("a run recorded from a DIRTY tree is never honoured as proof",
|
||||
pr["state"] == "stale-proof")
|
||||
|
||||
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
|
||||
latest_by_test([row(proof="proven", git_sha="0" * 40)]))
|
||||
check("a git_sha this clone cannot resolve degrades, never upgrades",
|
||||
pr["state"] == "stale-proof")
|
||||
|
||||
for bad, expect in (("failed", "failed"),
|
||||
("junit-skipped", "junit-skipped"),
|
||||
("unknown-sentinel-absent", "unknown"),
|
||||
("unknown-no-sentinel-declared", "unknown"),
|
||||
("something-new-from-the-future", "unknown")):
|
||||
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
|
||||
latest_by_test([row(proof=bad, git_sha=sha_b)]))
|
||||
check(f"ledger state {bad!r} maps to {expect!r}, never to proven",
|
||||
pr["state"] == expect)
|
||||
|
||||
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl, {})
|
||||
check("no ledger row at all stays not-derivable",
|
||||
pr["state"] == "not-derivable")
|
||||
|
||||
# Two verifying tests, only one proven: the requirement must NOT read proven.
|
||||
two = verifiers + [{"name": "data-survival",
|
||||
"path": "conformance/tests/l3-fault-injection/"
|
||||
"data-survival/chainsaw-test.yaml"}]
|
||||
pr = derive_proof(oracle, "R-STATE-03", two, impl,
|
||||
latest_by_test([row(proof="proven", git_sha=sha_b)]))
|
||||
check("a requirement is proven only when EVERY verifying test is",
|
||||
pr["state"] == "not-derivable")
|
||||
pr = derive_proof(oracle, "R-STATE-03", two, impl,
|
||||
latest_by_test([row(proof="proven", git_sha=sha_b),
|
||||
row(test="data-survival", proof="skipped",
|
||||
git_sha=sha_b)]))
|
||||
check("worst-of wins across verifying tests (skip drags proven down)",
|
||||
pr["state"] == "skipped")
|
||||
|
||||
# Latest row wins, not best-ever: a test that regressed to self-skipping must
|
||||
# stop reading as proven.
|
||||
pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
|
||||
latest_by_test([row(ts_utc="2026-09-01T10:00:00Z",
|
||||
proof="proven", git_sha=sha_b),
|
||||
row(ts_utc="2026-09-02T10:00:00Z",
|
||||
proof="skipped", git_sha=sha_b)]))
|
||||
check("the LATEST ledger row wins, not the best one ever recorded",
|
||||
pr["state"] == "skipped")
|
||||
|
||||
# A malformed ledger line is skipped, never guessed at.
|
||||
lf = gr / "ledger.jsonl"
|
||||
lf.write_text('{"test":"a","proof":"proven"}\nnot json at all\n\n'
|
||||
'{"proof":"proven"}\n')
|
||||
rows = read_ledger(lf)
|
||||
check("malformed / test-less ledger lines are dropped, not guessed at",
|
||||
len(rows) == 1 and rows[0]["test"] == "a")
|
||||
|
||||
if not ok:
|
||||
return 1
|
||||
sys.stderr.write("status.py selftest: PASS\n")
|
||||
@@ -865,6 +1458,10 @@ def main() -> int:
|
||||
help="exit 1 on defects this tool owns: registry/spec drift, "
|
||||
"executable-annotation conflicts, or a safety-critical "
|
||||
"requirement whose only L3 test is a scaffold")
|
||||
ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
|
||||
help=f"committed proof-of-pass ledger written by lib/record_run.py "
|
||||
f"(default {DEFAULT_LEDGER}). Absent or empty => every "
|
||||
f"requirement stays `not-derivable`, as before the ledger existed")
|
||||
ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
|
||||
args = ap.parse_args()
|
||||
|
||||
@@ -895,7 +1492,12 @@ def main() -> int:
|
||||
applied, nodes = reconciled_files(repo, entry)
|
||||
gitops_roots = {n["resolved"].split("/", 1)[0] for n in nodes if n["exists"]}
|
||||
impl = scan_implementation(repo, tracked_files(repo))
|
||||
rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env)
|
||||
ledger_rows = read_ledger(args.ledger)
|
||||
rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env,
|
||||
ledger_rows)
|
||||
rep["ledger"] = {"path": os.path.relpath(args.ledger, REPO_ROOT),
|
||||
"rows": len(ledger_rows),
|
||||
"present": args.ledger.is_file()}
|
||||
rep["flux_graph"] = nodes
|
||||
rep["gitops_roots"] = sorted(gitops_roots)
|
||||
rep["reconciled_file_count"] = len(applied)
|
||||
@@ -915,7 +1517,11 @@ def main() -> int:
|
||||
f"(+{s['wired_or_partial_and_testable'] - s['wired_and_testable']} partial) | "
|
||||
f"scaffold-only {s['verify_scaffold_only']} | "
|
||||
f"safety-critical w/o executable L3 "
|
||||
f"{s['safety_critical_without_executable_l3']} | PROVEN: not derivable")
|
||||
f"{s['safety_critical_without_executable_l3']} | "
|
||||
+ (f"PROVEN: {s['proven']} (stale {s['proof_stale']}, self-skipped "
|
||||
f"{s['proof_skipped']}, failed {s['proof_failed']}, unknown "
|
||||
f"{s['proof_unknown']}) from {s['ledger_rows']} ledger row(s)"
|
||||
if s["proven_derivable"] else "PROVEN: not derivable (empty run ledger)"))
|
||||
print(f"[status] wrote {args.out/'status.md'} and {args.out/'status.json'}")
|
||||
|
||||
rc = 0
|
||||
@@ -926,8 +1532,11 @@ def main() -> int:
|
||||
f"{h['registry_not_in_spec']} {h['spec_not_in_registry']}\n")
|
||||
rc = 1
|
||||
if h["annotation_conflicts"]:
|
||||
sys.stderr.write(f"[status] STRICT: tests annotated executable with TODO "
|
||||
f"bodies: {h['annotation_conflicts']}\n")
|
||||
sys.stderr.write(
|
||||
"[status] STRICT: tests annotated executable that declare a placeholder "
|
||||
"body: " + "; ".join(f"{k} ({v})" for k, v in
|
||||
sorted(h["annotation_conflict_reasons"].items()))
|
||||
+ "\n")
|
||||
rc = 1
|
||||
if rep["safety_critical_without_executable_l3"]:
|
||||
sys.stderr.write(f"[status] STRICT: safety-critical requirement(s) whose L3 "
|
||||
|
||||
+175
-13
@@ -10,16 +10,48 @@
|
||||
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
|
||||
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
|
||||
# ./run.sh --layer l0 --requirements R-X --include-scope whole-repo
|
||||
# # UNION: the tagged subset PLUS every test that
|
||||
# # DECLARES `conformance.home.k3s/scope: whole-repo`.
|
||||
# # A whole-repo invariant must run on EVERY gate
|
||||
# # invocation — see lib/common.sh's scope block for
|
||||
# # why, and why a SKIP of one is exit 3, not green.
|
||||
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
|
||||
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
|
||||
# # requirement, is it implemented, is it actually
|
||||
# # reconciled by Flux, is its test executable or a
|
||||
# # scaffold. Proof-of-pass is deliberately NOT
|
||||
# # derived — see lib/status.py.
|
||||
# # scaffold, and whether the committed run ledger
|
||||
# # (runs/ledger.jsonl) actually proves it — see
|
||||
# # lib/status.py. No ledger row => not-derivable.
|
||||
# ./run.sh --lint # structural-validate test defs (no cluster)
|
||||
# ./run.sh --print-selection # print the test dirs the CURRENT selection resolves
|
||||
# # to, one per line, and exit. Runs no check. This is
|
||||
# # how a test can PROVE the selection includes it
|
||||
# # (tests/l0-static/whole-repo-invariants-wired) without
|
||||
# # executing the suite and recursing into itself.
|
||||
# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate)
|
||||
# ./run.sh --require-all # phase-exit: every requirement must be covered
|
||||
# ./run.sh --record # RUN AND CAPTURE: tee the whole run to
|
||||
# # runs/<UTC>-<short-sha>/run.log, copy the JUnit
|
||||
# # beside it, and append one ledger row per
|
||||
# # testcase to runs/ledger.jsonl (lib/record_run.py)
|
||||
#
|
||||
# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
|
||||
# --------------------------------------------------------------------------
|
||||
# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
|
||||
# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
|
||||
# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
|
||||
# that — `conformance/runs/` (see conformance/.gitignore for the negations).
|
||||
#
|
||||
# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
|
||||
# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
|
||||
#
|
||||
# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
|
||||
# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
|
||||
# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
|
||||
# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
|
||||
# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
|
||||
#
|
||||
# Artifacts (always written under conformance/_artifacts/, gitignored):
|
||||
# _artifacts/report.junit.xml JUnit (Chainsaw output, or skipped-scaffold shim)
|
||||
@@ -28,6 +60,11 @@
|
||||
# _artifacts/status/status.md human requirement-status report (--status)
|
||||
# _artifacts/status/status.json machine requirement-status report (--status)
|
||||
#
|
||||
# Run ledger (TRACKED — commit these; --record only):
|
||||
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
|
||||
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
|
||||
# runs/ledger.jsonl append-only, one row per testcase per run
|
||||
#
|
||||
# Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
|
||||
# execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
|
||||
# green — the harness is proven reproducible while later phases fill in behavior.
|
||||
@@ -47,12 +84,18 @@ STATUS_OUT="${ARTIFACTS}/status"
|
||||
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
|
||||
# under `tee`. Captured BEFORE parsing so nothing is normalized away.
|
||||
ORIG_ARGS=("$@")
|
||||
|
||||
LAYER="all"
|
||||
REQS=""
|
||||
INCLUDE_SCOPES="" # --include-scope: scopes unioned into the selection (see lib/common.sh)
|
||||
MODE="run" # run | coverage-only | status | lint
|
||||
STRICT_COVERAGE=0
|
||||
REQUIRE_ALL=0
|
||||
FORCE_NO_CLUSTER=0
|
||||
RECORD=0
|
||||
|
||||
# Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
|
||||
# owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
|
||||
@@ -83,18 +126,23 @@ FORCE_NO_CLUSTER=0
|
||||
# longer a gap and has been removed from this list; the coverage report counts it as covered.
|
||||
DOCUMENTED_GAPS=""
|
||||
|
||||
usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
|
||||
# Print the header block (line 2 through the closing `####` rule, which is dropped) as
|
||||
# help. Bounded by the rule rather than a fixed line number so the header can grow.
|
||||
usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--layer) LAYER="$2"; shift 2 ;;
|
||||
--requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
|
||||
--include-scope) INCLUDE_SCOPES="${INCLUDE_SCOPES:+${INCLUDE_SCOPES},}$2"; shift 2 ;;
|
||||
--coverage-only) MODE="coverage-only"; shift ;;
|
||||
--status) MODE="status"; shift ;;
|
||||
--lint) MODE="lint"; shift ;;
|
||||
--print-selection) MODE="print-selection"; shift ;;
|
||||
--strict-coverage) STRICT_COVERAGE=1; shift ;;
|
||||
--require-all) REQUIRE_ALL=1; shift ;;
|
||||
--no-cluster) FORCE_NO_CLUSTER=1; shift ;;
|
||||
--record) RECORD=1; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "unknown arg: $1" >&2; usage; exit 2 ;;
|
||||
esac
|
||||
@@ -104,6 +152,62 @@ mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
|
||||
|
||||
log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
|
||||
|
||||
###############################################################################
|
||||
# --record: capture this run as committed evidence, then hand it to the ledger.
|
||||
#
|
||||
# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
|
||||
# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
|
||||
# operator-visible output — no second, differently-behaving "recording mode" to drift.
|
||||
# CONF_RECORD_CHILD guards against recursion.
|
||||
#
|
||||
# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
|
||||
# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
|
||||
# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
|
||||
###############################################################################
|
||||
if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
|
||||
case "${MODE}" in
|
||||
run) ;;
|
||||
# --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
|
||||
# test, so a ledger row from one of them would be a row about nothing.
|
||||
*) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
|
||||
esac
|
||||
RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
|
||||
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
|
||||
RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
|
||||
mkdir -p "${RUN_DIR}"
|
||||
# Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
|
||||
# ever capture a report THIS run produced. Copying a stale artifact would pair someone
|
||||
# else's result with this run's log — a fabricated pairing, which is precisely the class
|
||||
# of false proof the ledger exists to prevent.
|
||||
rm -f "${JUNIT_OUT}"
|
||||
# Strip --record from the argv handed to the child; everything else passes through.
|
||||
CHILD_ARGS=()
|
||||
for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
|
||||
[ "$a" = "--record" ] || CHILD_ARGS+=("$a")
|
||||
done
|
||||
log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
|
||||
set +e
|
||||
CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
|
||||
| tee "${RUN_DIR}/run.log"
|
||||
RUN_RC=${PIPESTATUS[0]}
|
||||
set -e
|
||||
if [ -f "${JUNIT_OUT}" ]; then
|
||||
cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
|
||||
else
|
||||
log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
|
||||
fi
|
||||
# Explicit, separate step: read the JUnit + the log and append one row per testcase.
|
||||
# rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
|
||||
# CONF_LEDGER exists so the harness's own tests can exercise this path without writing
|
||||
# to the real ledger; operators should never set it.
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
|
||||
--run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
|
||||
--ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
|
||||
log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
|
||||
log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
|
||||
exit "${RUN_RC}"
|
||||
fi
|
||||
|
||||
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
|
||||
gen_coverage() {
|
||||
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
|
||||
@@ -161,11 +265,38 @@ lint_tests() {
|
||||
}
|
||||
|
||||
# --- Select which test directories Chainsaw should run --------------------------------
|
||||
# Requirement-scoped selection, then the DECLARED-SCOPE union on top:
|
||||
#
|
||||
# (--requirements ∩ --layer) ∪ (--include-scope ∩ --layer)
|
||||
#
|
||||
# The left half is R-TEST-03's touched subset. The right half is the fix for its blind
|
||||
# spot: an invariant that quantifies over the whole tracked tree cannot be selected BY the
|
||||
# requirements a change touches, because the change most likely to break it touches none of
|
||||
# them. See lib/common.sh's scope block for the incidents. The union is fail-closed by
|
||||
# construction — it can only ADD tests to a run, never remove one.
|
||||
select_dirs() {
|
||||
# --requirements AND --layer together => intersection (tests carrying the tag
|
||||
# AND living in that layer subtree). Either alone keeps its prior meaning, so
|
||||
# this is backward compatible (LAYER defaults to "all"). Both helpers emit
|
||||
# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
|
||||
{
|
||||
requirement_scoped_dirs
|
||||
# The scope union is still LAYER-bounded: `--layer l0` must not pull a whole-repo-scoped
|
||||
# L2 test into an offline run. With no --layer (all), nothing is bounded away.
|
||||
if [ -n "${INCLUDE_SCOPES}" ]; then
|
||||
if [ "${LAYER}" != all ]; then
|
||||
comm -12 \
|
||||
<(conf_dirs_for_scopes "${INCLUDE_SCOPES}") \
|
||||
<(conf_dirs_for_layer "${LAYER}")
|
||||
else
|
||||
conf_dirs_for_scopes "${INCLUDE_SCOPES}"
|
||||
fi
|
||||
fi
|
||||
} | sort -u
|
||||
}
|
||||
|
||||
# The pre-existing requirement/layer selection, unchanged:
|
||||
# --requirements AND --layer together => intersection (tests carrying the tag
|
||||
# AND living in that layer subtree). Either alone keeps its prior meaning, so
|
||||
# this is backward compatible (LAYER defaults to "all"). Both helpers emit
|
||||
# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
|
||||
requirement_scoped_dirs() {
|
||||
if [ -n "${REQS}" ] && [ "${LAYER}" != all ]; then
|
||||
comm -12 \
|
||||
<(conf_dirs_for_requirements "${REQS}") \
|
||||
@@ -306,6 +437,12 @@ layers_are_offline_only() {
|
||||
return 0
|
||||
}
|
||||
|
||||
# Markdown one-liner per executed offline check, for the CI step summary. Written next to
|
||||
# the JUnit; ci/l0.sh + ci/l1.sh splice it into $GITHUB_STEP_SUMMARY so the gate output still
|
||||
# NAMES every check that ran (that is what the deleted per-check gate stages did, one hard-coded
|
||||
# `summary` line each — now generated for all of them, whole-repo-scoped ones marked as such).
|
||||
: "${OFFLINE_SUMMARY_OUT:=${ARTIFACTS}/offline-summary.md}"
|
||||
|
||||
run_offline_checks() {
|
||||
local -a dirs
|
||||
mapfile -t dirs < <(offline_selected_dirs)
|
||||
@@ -313,14 +450,30 @@ run_offline_checks() {
|
||||
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
|
||||
return 0
|
||||
fi
|
||||
# Declared scopes are passed through so the runner knows WHICH cases must not be allowed to
|
||||
# report a SKIP as green (rc 3). See lib/common.sh's scope block and run_l1_offline.py.
|
||||
local -a scope_args=() s
|
||||
for s in ${INCLUDE_SCOPES//,/ }; do scope_args+=(--unwired-scope "$s"); done
|
||||
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
|
||||
[ -n "${INCLUDE_SCOPES}" ] && log " unconditional (declared scope ${INCLUDE_SCOPES}): $(conf_dirs_for_scopes "${INCLUDE_SCOPES}" | xargs -n1 basename 2>/dev/null | tr '\n' ' ')"
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
|
||||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
|
||||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" \
|
||||
--scope-annotation "${CONF_SCOPE_ANNOTATION}" \
|
||||
--summary-out "${OFFLINE_SUMMARY_OUT}" \
|
||||
${scope_args[@]+"${scope_args[@]}"} "${dirs[@]}"
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
# Main
|
||||
###############################################################################
|
||||
# --print-selection is deliberately handled FIRST: it resolves the selection and exits,
|
||||
# writing nothing and running no check. A test that asserts "the gate's selection cannot
|
||||
# filter me out" needs to ask that question without executing the suite it is part of.
|
||||
if [ "${MODE}" = "print-selection" ]; then
|
||||
select_dirs
|
||||
exit 0
|
||||
fi
|
||||
|
||||
log "Coverage report first (always)"
|
||||
gen_coverage
|
||||
|
||||
@@ -350,17 +503,26 @@ esac
|
||||
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
|
||||
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
|
||||
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
||||
# rc 3 is the harness-wide SKIPPED/not-run code (ci/README.md): NOT a pass, NOT a fail. The
|
||||
# offline runner returns it when a test carrying a --unwired-scope scope SKIPPED — a whole-repo
|
||||
# invariant reporting "nothing to audit" means the subtree it audits is gone, which the gate
|
||||
# must see as UNWIRED. It is propagated verbatim so ci/l0.sh can set the gate's own rc=3,
|
||||
# exactly as the deleted per-check stages did.
|
||||
OFFLINE_RC=0
|
||||
run_offline_checks || OFFLINE_RC=$?
|
||||
if layers_are_offline_only; then
|
||||
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
|
||||
if [ "${OFFLINE_RC}" -eq 0 ]; then
|
||||
log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||||
else
|
||||
log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
|
||||
fi
|
||||
case "${OFFLINE_RC}" in
|
||||
0) log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" ;;
|
||||
3) log "Offline ${LAYER}: a scope-declared check SKIPPED — UNWIRED, not green (rc=3). JUnit: ${JUNIT_OUT}" ;;
|
||||
*) log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}" ;;
|
||||
esac
|
||||
exit "${OFFLINE_RC}"
|
||||
fi
|
||||
if [ "${OFFLINE_RC}" -eq 3 ]; then
|
||||
log "Offline L0/L1: a scope-declared check SKIPPED — UNWIRED, not green (rc=3)."
|
||||
exit 3
|
||||
fi
|
||||
if [ "${OFFLINE_RC}" -ne 0 ]; then
|
||||
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
|
||||
exit "${OFFLINE_RC}"
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="node-loss" classname="" time="131.090121" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,232 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/node-loss
|
||||
=== PAUSE chainsaw/node-loss
|
||||
=== CONT chainsaw/node-loss
|
||||
l.go:52: | 23:51:18 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-fond-marlin
|
||||
l.go:52: | 23:51:18 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 23:51:18 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -euo pipefail
|
||||
NS="$NAMESPACE"
|
||||
|
||||
# -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
|
||||
READY="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
| grep -c '^True$' || true)"
|
||||
echo "ready nodes: ${READY}"
|
||||
if [ "${READY:-0}" -lt 3 ]; then
|
||||
echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
CANARY="node-loss-canary-$(date +%s%N)"
|
||||
|
||||
# -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
|
||||
# writer pod that writes the canary and flushes it to the replicated volume. --
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: survivor
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
apply_pod() {
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: $1
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 3600"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: survivor
|
||||
EOF
|
||||
}
|
||||
|
||||
apply_pod writer
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
|
||||
printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
|
||||
echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
|
||||
|
||||
# The victim node is the one the attached volume currently lives on.
|
||||
VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
|
||||
[ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
|
||||
echo "victim node (currently hosts the attached volume): $VICTIM"
|
||||
|
||||
# Best-effort precondition log: if the Longhorn Volume CR is readable, show the
|
||||
# volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
|
||||
# runner is least-privilege and may not have longhorn.io read rights) — the
|
||||
# data-availability assertion below is the authoritative proof, not this.
|
||||
PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
if [ -n "$PVNAME" ]; then
|
||||
kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
|
||||
2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
|
||||
fi
|
||||
|
||||
# -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
|
||||
# Label it with THIS test's unique namespace so `finally` can always find and
|
||||
# uncordon exactly this node; then cordon it; then evict ONLY our own writer
|
||||
# pod, forcing the volume to detach from the victim. (No `drain` — unrelated
|
||||
# pods on the node are never touched.)
|
||||
kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
|
||||
kubectl cordon "$VICTIM"
|
||||
echo "cordoned + labelled victim node: $VICTIM"
|
||||
kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
|
||||
echo "evicted writer off the victim node; volume detached from $VICTIM"
|
||||
|
||||
# -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
|
||||
# so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
|
||||
# the volume there from its replicas on the other nodes. --
|
||||
apply_pod reader
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
|
||||
READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
|
||||
echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
|
||||
if [ "$READER_NODE" = "$VICTIM" ]; then
|
||||
echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
[ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
|
||||
echo "volume available + data intact from a surviving node"
|
||||
|
||||
# Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
|
||||
# (bounded) for robustness to settle back to available (healthy|degraded — both
|
||||
# serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
|
||||
# Non-fatal for the same least-privilege reason as the pre-fault log.
|
||||
if [ -n "$PVNAME" ]; then
|
||||
for i in $(seq 1 30); do
|
||||
ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='{.status.robustness}' 2>/dev/null || true)"
|
||||
[ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
|
||||
echo "post-fault robustness[$i]=$ROB"
|
||||
case "$ROB" in healthy|degraded) break ;; esac
|
||||
sleep 10
|
||||
done
|
||||
fi
|
||||
|
||||
echo NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
ready nodes: 3
|
||||
persistentvolumeclaim/survivor created
|
||||
pod/writer created
|
||||
pod/writer condition met
|
||||
wrote canary: node-loss-canary-1788306678898809444
|
||||
victim node (currently hosts the attached volume): k3s-dev-2
|
||||
pre-fault robustness=healthy replicas=3
|
||||
node/k3s-dev-2 labeled
|
||||
node/k3s-dev-2 cordoned
|
||||
cordoned + labelled victim node: k3s-dev-2
|
||||
pod "writer" deleted from chainsaw-fond-marlin namespace
|
||||
evicted writer off the victim node; volume detached from k3s-dev-2
|
||||
pod/reader created
|
||||
pod/reader condition met
|
||||
reader scheduled on surviving node: k3s-dev-1 (victim was k3s-dev-2)
|
||||
recovered='node-loss-canary-1788306678898809444' expected='node-loss-canary-1788306678898809444'
|
||||
volume available + data intact from a surviving node
|
||||
post-fault robustness[1]=healthy
|
||||
NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 23:52:42 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -uo pipefail
|
||||
for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
|
||||
kubectl uncordon "${n#node/}" || true
|
||||
kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
|
||||
echo "restored node to service: ${n#node/}"
|
||||
done
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
node/k3s-dev-2 uncordoned
|
||||
node/k3s-dev-2 unlabeled
|
||||
restored node to service: k3s-dev-2
|
||||
l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
|
||||
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
pod "reader" deleted from chainsaw-fond-marlin namespace
|
||||
persistentvolumeclaim "survivor" deleted from chainsaw-fond-marlin namespace
|
||||
cleanup best-effort; chainsaw namespace teardown finalizes
|
||||
l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-fond-marlin
|
||||
l.go:52: | 23:53:29 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/node-loss (131.09s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -0,0 +1,4 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="dr-restore" classname="" time="149.053701" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,671 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/dr-restore
|
||||
=== PAUSE chainsaw/dr-restore
|
||||
=== CONT chainsaw/dr-restore
|
||||
l.go:52: | 00:19:07 | [32;1mdr-restore[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-cunning-hen
|
||||
l.go:52: | 00:19:07 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 00:19:07 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
|
||||
# ---- helpers -------------------------------------------------------
|
||||
# Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
|
||||
# fail loudly; an absent API is a genuine precondition gap and may skip.
|
||||
die_or_skip() {
|
||||
case "$2" in
|
||||
*orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
|
||||
echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
|
||||
echo "$2" >&2
|
||||
exit 1 ;;
|
||||
*)
|
||||
echo "SKIP: $1 not available on this cluster -- $2"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
ip2int() {
|
||||
_q="$(echo "$1" | tr '.' ' ')"
|
||||
set -- $_q
|
||||
[ $# -eq 4 ] || { echo -1; return 0; }
|
||||
echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
|
||||
}
|
||||
|
||||
# in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
|
||||
in_cidr() {
|
||||
case "$2" in *"/"*) ;; *) return 1 ;; esac
|
||||
_net="${2%/*}"; _len="${2#*/}"
|
||||
case "$_len" in ''|*[!0-9]*) return 1 ;; esac
|
||||
[ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
|
||||
_i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
|
||||
[ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
|
||||
[ "$_len" -eq 0 ] && return 0
|
||||
_d=1; _s=0
|
||||
while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
|
||||
[ $(( _i / _d )) -eq $(( _n / _d )) ]
|
||||
}
|
||||
|
||||
# ---- Gate 1: the replicated storage tier the DR-critical data lives on --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
|
||||
2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
|
||||
if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
|
||||
echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "backup storage locations:"
|
||||
echo "$BSL_LINES"
|
||||
|
||||
VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
|
||||
|
||||
# ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
|
||||
# File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
|
||||
# deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
|
||||
# ready node-agent no volume data can be captured, so the drill is not runnable.
|
||||
VREADY=0; NAREADY=0
|
||||
for i in $(seq 1 12); do
|
||||
VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
|
||||
NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
|
||||
echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
|
||||
if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
|
||||
sleep 10
|
||||
done
|
||||
if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
|
||||
echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- Pick the BSL this drill will use: prefer the default, and it MUST be
|
||||
# Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
|
||||
# gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
|
||||
# wait for first-validation on a cold start), it does not skip. --------
|
||||
BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
|
||||
for i in $(seq 1 12); do
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
|
||||
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
|
||||
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
|
||||
if [ -n "$CAND" ]; then
|
||||
BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
|
||||
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
|
||||
BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
|
||||
BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
|
||||
break
|
||||
fi
|
||||
echo "poll $i: no Available BackupStorageLocation yet"
|
||||
sleep 10
|
||||
done
|
||||
if [ -z "$BSL_NAME" ]; then
|
||||
echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
|
||||
echo " The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
|
||||
echo "$BSL_LINES" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "using BSL ${BSL_NS}/${BSL_NAME} bucket=${BSL_BUCKET} endpoint=${BSL_URL}"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
|
||||
if [ -z "$BSL_URL" ]; then
|
||||
echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
|
||||
echo " the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
|
||||
echo " test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
|
||||
exit 1
|
||||
fi
|
||||
HOST="${BSL_URL#*://}" # strip scheme
|
||||
HOST="${HOST%%/*}" # strip path
|
||||
HOST="${HOST%%:*}" # strip port
|
||||
echo "backup endpoint host: $HOST"
|
||||
|
||||
case "$HOST" in
|
||||
localhost|*.svc|*.svc.cluster.local|*.cluster.local)
|
||||
echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
|
||||
echo " An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
|
||||
exit 1 ;;
|
||||
esac
|
||||
|
||||
# Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
|
||||
# off-cluster property -- that is a failure, never a silent pass.
|
||||
case "$HOST" in
|
||||
*[!0-9.]*)
|
||||
# Endpoint given as a name. Try the resolvers that may exist in the
|
||||
# runner image, in order; if none is available or none resolves, the
|
||||
# test FAILS below rather than guessing.
|
||||
EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
|
||||
[ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
|
||||
[ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
|
||||
*) EPIP="$HOST" ;;
|
||||
esac
|
||||
if [ -z "$EPIP" ]; then
|
||||
echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
|
||||
echo " the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "backup endpoint resolves to: $EPIP"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
|
||||
case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
|
||||
|
||||
NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
for a in $NODE_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
|
||||
|
||||
SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
|
||||
for a in $SVC_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
|
||||
|
||||
POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
|
||||
for a in $POD_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
|
||||
|
||||
POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
for c in $POD_CIDRS; do
|
||||
case "$c" in *:*) continue ;; esac # IPv6 prefixes are out of scope here
|
||||
if in_cidr "$EPIP" "$c"; then
|
||||
echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
|
||||
|
||||
# Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
|
||||
# runner may not read): the checks above are the authoritative ones.
|
||||
SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
if [ -n "$SVC_CIDRS" ]; then
|
||||
for c in $SVC_CIDRS; do
|
||||
case "$c" in *:*) continue ;; esac
|
||||
if in_cidr "$EPIP" "$c"; then
|
||||
echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
|
||||
else
|
||||
echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
|
||||
fi
|
||||
|
||||
echo "OFF_CLUSTER_TARGET_OK bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
backup storage locations:
|
||||
velero default Available aws true http://192.168.1.9:9000 velero
|
||||
poll 1: velero readyReplicas=1 node-agent numberReady=3
|
||||
using BSL velero/default bucket=velero endpoint=http://192.168.1.9:9000
|
||||
backup endpoint host: 192.168.1.9
|
||||
backup endpoint resolves to: 192.168.1.9
|
||||
endpoint is not any node address (checked 6 node addresses)
|
||||
endpoint is not any Service ClusterIP (checked 81)
|
||||
endpoint is not any Pod IP (checked 186)
|
||||
endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
|
||||
endpoint is outside every ServiceCIDR (10.43.0.0/16 )
|
||||
OFF_CLUSTER_TARGET_OK bsl=velero/default endpoint=192.168.1.9 bucket=velero
|
||||
l.go:52: | 00:19:11 | [32;1mdr-restore[0;22m | [32;1massert-backup-target-is-off-cluster [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
SCHED="$(kubectl get schedules.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
|
||||
if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
|
||||
# Distinguish "Velero absent" (skip, in lock-step with step 1) from
|
||||
# "Velero present but nothing scheduled" (a real R-STATE-03 defect).
|
||||
if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
|
||||
if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
|
||||
echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
|
||||
echo DR_SCHEDULE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
|
||||
echo " R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
|
||||
echo DR_SCHEDULE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "velero schedules:"
|
||||
echo "$SCHED"
|
||||
# A Schedule that exists but is paused/disabled backs nothing up.
|
||||
if echo "$SCHED" | grep -q 'phase=Enabled'; then
|
||||
echo BACKUPS_ARE_SCHEDULED_OK
|
||||
else
|
||||
echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
|
||||
exit 1
|
||||
fi
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
velero schedules:
|
||||
velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
|
||||
velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
|
||||
BACKUPS_ARE_SCHEDULED_OK
|
||||
l.go:52: | 00:19:11 | [32;1mdr-restore[0;22m | [32;1massert-backups-are-scheduled [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
NS="$NAMESPACE"
|
||||
SRC="${NS}-dr-src" # scratch source namespace (destroyed by design)
|
||||
DST="${NS}-dr-dst" # clean restore target (must not pre-exist)
|
||||
BK="${NS}-dr-backup"
|
||||
RS="${NS}-dr-restore"
|
||||
OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
|
||||
|
||||
# ---- Re-derive the preconditions (steps do not share shell state). Every
|
||||
# skip below mirrors step 1 exactly, so the whole test skips or runs
|
||||
# as a unit -- it can never half-run. -----------------------------
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
|
||||
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
|
||||
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
|
||||
if [ -z "$CAND" ]; then
|
||||
echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
|
||||
echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
VNS="$(echo "$CAND" | cut -d' ' -f1)"
|
||||
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
|
||||
if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
|
||||
echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
|
||||
echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
|
||||
|
||||
# A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
|
||||
# be free. If either is lying around from an aborted run, remove it before we
|
||||
# claim "clean" -- and fail loudly if it will not go, because a half-deleted
|
||||
# namespace would silently invalidate the destroy proof below.
|
||||
kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
|
||||
|
||||
# ============================ GIVEN ==============================
|
||||
# A DR-critical stateful workload with known data on the replicated tier.
|
||||
CANARY="dr-canary-$(date +%s%N)"
|
||||
kubectl create namespace "$SRC"
|
||||
kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
|
||||
|
||||
kubectl -n "$SRC" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: dr-data
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
kubectl -n "$SRC" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: dr-workload
|
||||
labels:
|
||||
app: dr-workload
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 7200"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: dr-data
|
||||
EOF
|
||||
|
||||
kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
|
||||
printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
|
||||
echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
|
||||
|
||||
ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
|
||||
[ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
|
||||
echo "original in-cluster PersistentVolume: $ORIG_PV"
|
||||
|
||||
# ======================= WHEN (part 1): back up off-cluster =======
|
||||
# Created as a Velero Backup CR -- exactly what `velero backup create` does.
|
||||
# defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
|
||||
# the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
|
||||
# ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
|
||||
# own GC still removes this drill's objects from the bucket.
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: velero.io/v1
|
||||
kind: Backup
|
||||
metadata:
|
||||
name: ${BK}
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
includedNamespaces: [${SRC}]
|
||||
storageLocation: ${BSL_NAME}
|
||||
snapshotVolumes: false
|
||||
defaultVolumesToFsBackup: true
|
||||
ttl: 2h0m0s
|
||||
EOF
|
||||
|
||||
BPHASE=""
|
||||
for i in $(seq 1 90); do
|
||||
BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
echo "poll $i: backup phase=${BPHASE:-<pending>}"
|
||||
case "${BPHASE:-}" in
|
||||
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
|
||||
esac
|
||||
sleep 10
|
||||
done
|
||||
if [ "${BPHASE:-}" != "Completed" ]; then
|
||||
echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
|
||||
kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
echo "backup $BK Completed to ${BSL_NAME}"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
|
||||
# kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
|
||||
PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
|
||||
-l velero.io/backup-name="$BK" \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
|
||||
echo "pod volume backups:"
|
||||
echo "$PVB"
|
||||
if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
|
||||
echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
|
||||
echo " Resource-only backups cannot restore DR-critical DATA." >&2
|
||||
exit 1
|
||||
fi
|
||||
# A line is "<name> <phase> <snapshotID>"; require Completed AND a
|
||||
# non-empty snapshot ID (an empty ID means nothing reached the repository).
|
||||
GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
|
||||
if [ "${GOOD_PVB:-0}" -lt 1 ]; then
|
||||
echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
|
||||
echo " the volume data did not reach the off-cluster repository." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
|
||||
|
||||
# ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
|
||||
# Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
|
||||
# underlying PersistentVolume (and with it the Longhorn replicas). After this
|
||||
# there is no in-cluster source the restore could quietly read from.
|
||||
kubectl delete namespace "$SRC" --wait=true --timeout=420s
|
||||
for i in $(seq 1 30); do
|
||||
if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
|
||||
echo "poll $i: waiting for namespace $SRC to disappear"
|
||||
sleep 10
|
||||
done
|
||||
if kubectl get namespace "$SRC" >/dev/null 2>&1; then
|
||||
echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "source namespace destroyed"
|
||||
|
||||
# The longhorn StorageClass reclaims Delete, so the PV should already be gone;
|
||||
# force it if a Retain policy (or a stuck finalizer) left it behind, because a
|
||||
# surviving PV would make the restore prove nothing about off-cluster recovery.
|
||||
kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
|
||||
for i in $(seq 1 30); do
|
||||
if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
|
||||
echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
|
||||
sleep 10
|
||||
done
|
||||
if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
|
||||
echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
|
||||
echo " An in-cluster copy of the data still exists, so a successful restore" >&2
|
||||
echo " would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
|
||||
|
||||
# Best-effort corroboration (the least-privilege runner may not read longhorn.io):
|
||||
# the Longhorn Volume CR and its replicas should be gone too.
|
||||
if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
|
||||
echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
|
||||
|
||||
# ============================ RESTORE ============================
|
||||
# Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
|
||||
# the catalog's "restored into a clean state".
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: velero.io/v1
|
||||
kind: Restore
|
||||
metadata:
|
||||
name: ${RS}
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
backupName: ${BK}
|
||||
includedNamespaces: [${SRC}]
|
||||
namespaceMapping:
|
||||
${SRC}: ${DST}
|
||||
existingResourcePolicy: none
|
||||
EOF
|
||||
|
||||
RPHASE=""
|
||||
for i in $(seq 1 90); do
|
||||
RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
echo "poll $i: restore phase=${RPHASE:-<pending>}"
|
||||
case "${RPHASE:-}" in
|
||||
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
|
||||
esac
|
||||
sleep 10
|
||||
done
|
||||
if [ "${RPHASE:-}" != "Completed" ]; then
|
||||
echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
|
||||
kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
|
||||
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
echo "restore $RS Completed into $DST"
|
||||
|
||||
# ============================= THEN ==============================
|
||||
# The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
|
||||
# `restore-wait` init container that blocks until the node-agent has written the
|
||||
# volume contents back, so Ready here already implies the data landed.
|
||||
if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
|
||||
echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
|
||||
kubectl -n "$DST" describe pod dr-workload >&2 || true
|
||||
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
|
||||
[ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
|
||||
if [ "$NEW_PV" = "$ORIG_PV" ]; then
|
||||
echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
|
||||
echo " That is an in-cluster survival, not an off-cluster restore." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
|
||||
|
||||
GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
if [ "$GOT" != "$CANARY" ]; then
|
||||
echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
|
||||
l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
velero ns=velero bsl=default src=chainsaw-cunning-hen-dr-src dst=chainsaw-cunning-hen-dr-dst
|
||||
namespace/chainsaw-cunning-hen-dr-src created
|
||||
namespace/chainsaw-cunning-hen-dr-src labeled
|
||||
persistentvolumeclaim/dr-data created
|
||||
pod/dr-workload created
|
||||
pod/dr-workload condition met
|
||||
seeded canary: dr-canary-1788308352223263220
|
||||
original in-cluster PersistentVolume: pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745
|
||||
backup.velero.io/chainsaw-cunning-hen-dr-backup created
|
||||
poll 1: backup phase=InProgress
|
||||
poll 2: backup phase=InProgress
|
||||
poll 3: backup phase=Completed
|
||||
backup chainsaw-cunning-hen-dr-backup Completed to default
|
||||
pod volume backups:
|
||||
chainsaw-cunning-hen-dr-backup-l95lw Completed 47fa2737115e17142fb41857cef4e779
|
||||
volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
|
||||
namespace "chainsaw-cunning-hen-dr-src" deleted
|
||||
source namespace destroyed
|
||||
original PersistentVolume pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 is GONE -- no in-cluster copy of the data remains
|
||||
(longhorn Volume CR pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 not present -- either deleted or not readable by this identity)
|
||||
restore.velero.io/chainsaw-cunning-hen-dr-restore created
|
||||
poll 1: restore phase=InProgress
|
||||
poll 2: restore phase=InProgress
|
||||
poll 3: restore phase=InProgress
|
||||
poll 4: restore phase=Completed
|
||||
restore chainsaw-cunning-hen-dr-restore Completed into chainsaw-cunning-hen-dr-dst
|
||||
pod/dr-workload condition met
|
||||
restored PVC bound to PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
|
||||
restored onto a NEW volume (pvc-63c47973-9c7f-4423-a27e-caa008026e49 != pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745) -- the data came from outside the cluster
|
||||
recovered='dr-canary-1788308352223263220' expected='dr-canary-1788308352223263220'
|
||||
DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
|
||||
=== STDERR
|
||||
Defaulted container "app" out of: app, restore-wait (init)
|
||||
l.go:52: | 00:21:16 | [32;1mdr-restore[0;22m | [32;1mdata-restores-from-the-off-cluster-backup[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -u
|
||||
NS="$NAMESPACE"
|
||||
SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
|
||||
|
||||
VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
|
||||
|
||||
# Velero objects first: a DeleteBackupRequest makes Velero remove the backup
|
||||
# DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
|
||||
if [ -n "$VNS" ]; then
|
||||
kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
|
||||
if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
|
||||
kubectl apply -f - <<EOF || true
|
||||
apiVersion: velero.io/v1
|
||||
kind: DeleteBackupRequest
|
||||
metadata:
|
||||
name: ${BK}-delete
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
backupName: ${BK}
|
||||
EOF
|
||||
for i in $(seq 1 18); do
|
||||
kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
|
||||
sleep 10
|
||||
done
|
||||
# Backstop if Velero could not process the request (the 2h ttl also GCs it).
|
||||
kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
|
||||
kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
|
||||
fi
|
||||
echo "velero objects for $NS cleaned up (ns $VNS)"
|
||||
fi
|
||||
|
||||
# Scratch namespaces (both, in case the run aborted at any point).
|
||||
for n in "$DST" "$SRC"; do
|
||||
kubectl delete namespace "$n" --ignore-not-found --wait=false || true
|
||||
done
|
||||
# Anything left labelled as ours (defensive; the two names above are the only
|
||||
# namespaces this test ever creates).
|
||||
kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
|
||||
|
||||
# Released PVs from either namespace's PVCs would otherwise leak storage.
|
||||
for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
|
||||
kubectl delete pv "$pv" --ignore-not-found --wait=false || true
|
||||
echo "released leftover PersistentVolume: $pv"
|
||||
done
|
||||
|
||||
echo "dr-restore teardown complete (best-effort; every object was created by this test)"
|
||||
l.go:52: | 00:21:28 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
restore.velero.io "chainsaw-cunning-hen-dr-restore" deleted from velero namespace
|
||||
deletebackuprequest.velero.io/chainsaw-cunning-hen-dr-backup-delete created
|
||||
velero objects for chainsaw-cunning-hen cleaned up (ns velero)
|
||||
namespace "chainsaw-cunning-hen-dr-dst" deleted
|
||||
namespace "chainsaw-cunning-hen-dr-dst" deleted
|
||||
persistentvolume "pvc-63c47973-9c7f-4423-a27e-caa008026e49" deleted
|
||||
released leftover PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
|
||||
dr-restore teardown complete (best-effort; every object was created by this test)
|
||||
l.go:52: | 00:21:28 | [32;1mdr-restore[0;22m | [32;1mdata-restores-from-the-off-cluster-backup[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 00:21:28 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 00:21:28 | [36;1mdr-restore[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 00:21:28 | [32;1mdr-restore[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-cunning-hen
|
||||
l.go:52: | 00:21:36 | [36;1mdr-restore[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/dr-restore (149.05s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -0,0 +1,12 @@
|
||||
<?xml version="1.0" ?>
|
||||
<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
|
||||
<testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
|
||||
<testcase classname="L1:R-NET-05,R-NET-06,R-NET-07" name="egress-class-render">
|
||||
<properties>
|
||||
<property name="requirements" value="R-NET-05,R-NET-06,R-NET-07"/>
|
||||
<property name="layer" value="L1"/>
|
||||
<property name="path" value="/work/.claude/conf-clone/conformance/tests/l1-rendering/egress-class-render"/>
|
||||
</properties>
|
||||
</testcase>
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,7 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m Running 1 offline L0/L1 check(s) (no cluster; docs/14).
|
||||
[offline] L1 PASS egress-class-render (R-NET-05,R-NET-06,R-NET-07)
|
||||
[offline] wrote JUnit (1 case(s), 0 failure(s)) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml
|
||||
[1m[run.sh][0m Done (offline l1). JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -0,0 +1,4 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="vpn-killswitch" classname="" time="174.632779" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,237 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-NET-06').
|
||||
[1m[run.sh][0m LIGHT pass: 1 test(s), parallel=2 (pinned v0.2.12)
|
||||
Version: 0.2.12
|
||||
Loading config (/work/.claude/conf-clone/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-light'
|
||||
- ReportPath '/work/.claude/conf-clone/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 2
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- vpn-killswitch (/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/vpn-killswitch
|
||||
=== PAUSE chainsaw/vpn-killswitch
|
||||
=== CONT chainsaw/vpn-killswitch
|
||||
l.go:52: | 01:44:38 | [32;1mvpn-killswitch[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-credible-bird
|
||||
l.go:52: | 01:44:38 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:44:38 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
kubectl label namespace "$NAMESPACE" \
|
||||
pod-security.kubernetes.io/enforce=privileged \
|
||||
pod-security.kubernetes.io/warn=privileged \
|
||||
pod-security.kubernetes.io/audit=privileged \
|
||||
--overwrite
|
||||
echo "NS_PRIVILEGED_OK"
|
||||
l.go:52: | 01:44:39 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
namespace/chainsaw-credible-bird labeled
|
||||
NS_PRIVILEGED_OK
|
||||
l.go:52: | 01:44:39 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:44:39 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mAPPLY[0;22m | [36;1mRUN[0;22m | v1/Pod @ chainsaw-credible-bird/egress-internet
|
||||
l.go:52: | 01:44:39 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Pod @ chainsaw-credible-bird/egress-internet
|
||||
l.go:52: | 01:44:39 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mAPPLY[0;22m | [32;1mDONE[0;22m | v1/Pod @ chainsaw-credible-bird/egress-internet
|
||||
l.go:52: | 01:44:39 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mAPPLY[0;22m | [36;1mRUN[0;22m | v1/Pod @ chainsaw-credible-bird/egress-vpn
|
||||
l.go:52: | 01:44:39 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Pod @ chainsaw-credible-bird/egress-vpn
|
||||
l.go:52: | 01:44:39 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mAPPLY[0;22m | [32;1mDONE[0;22m | v1/Pod @ chainsaw-credible-bird/egress-vpn
|
||||
l.go:52: | 01:44:39 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mASSERT[0;22m | [36;1mRUN[0;22m | v1/Pod @ chainsaw-credible-bird/egress-internet
|
||||
l.go:52: | 01:44:41 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mASSERT[0;22m | [32;1mDONE[0;22m | v1/Pod @ chainsaw-credible-bird/egress-internet
|
||||
l.go:52: | 01:44:41 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mASSERT[0;22m | [36;1mRUN[0;22m | v1/Pod @ chainsaw-credible-bird/egress-vpn
|
||||
l.go:52: | 01:44:41 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mASSERT[0;22m | [32;1mDONE[0;22m | v1/Pod @ chainsaw-credible-bird/egress-vpn
|
||||
l.go:52: | 01:44:41 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
ns="$NAMESPACE"
|
||||
get_ip() {
|
||||
kubectl -n "$ns" exec "$1" -c curl -- \
|
||||
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
|
||||
}
|
||||
wan="" vpn=""
|
||||
for i in $(seq 1 12); do
|
||||
[ -n "$wan" ] || wan="$(get_ip egress-internet)"
|
||||
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
|
||||
if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
|
||||
[ "$wan" = "$vpn" ] && vpn="" # not steered yet; re-poll
|
||||
sleep 5
|
||||
done
|
||||
echo "wan=$wan vpn=$vpn"
|
||||
[ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
|
||||
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
|
||||
[ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
|
||||
echo "TUNNEL_HEALTHY_OK"
|
||||
l.go:52: | 01:44:43 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
wan=136.51.115.252 vpn=158.173.241.11
|
||||
TUNNEL_HEALTHY_OK
|
||||
l.go:52: | 01:44:43 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:44:43 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:44:43 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:44:43 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mAPPLY[0;22m | [36;1mRUN[0;22m | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
|
||||
l.go:52: | 01:44:43 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
|
||||
l.go:52: | 01:44:43 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mAPPLY[0;22m | [32;1mDONE[0;22m | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
|
||||
l.go:52: | 01:44:43 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mASSERT[0;22m | [36;1mRUN[0;22m | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
|
||||
l.go:52: | 01:44:48 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mASSERT[0;22m | [32;1mDONE[0;22m | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
|
||||
l.go:52: | 01:44:48 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -u
|
||||
ns="$NAMESPACE"
|
||||
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
||||
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
||||
for i in $(seq 1 8); do
|
||||
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
||||
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
|
||||
if [ -n "$out" ]; then
|
||||
if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
|
||||
echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
|
||||
fi
|
||||
echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
|
||||
fi
|
||||
echo "attempt $i: vpn-class blackholed (no response), good"
|
||||
sleep 3
|
||||
done
|
||||
echo "NO_LEAK_OK"
|
||||
l.go:52: | 01:46:18 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
attempt 1: vpn-class blackholed (no response), good
|
||||
attempt 2: vpn-class blackholed (no response), good
|
||||
attempt 3: vpn-class blackholed (no response), good
|
||||
attempt 4: vpn-class blackholed (no response), good
|
||||
attempt 5: vpn-class blackholed (no response), good
|
||||
attempt 6: vpn-class blackholed (no response), good
|
||||
attempt 7: vpn-class blackholed (no response), good
|
||||
attempt 8: vpn-class blackholed (no response), good
|
||||
NO_LEAK_OK
|
||||
l.go:52: | 01:46:18 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:46:18 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
ns="$NAMESPACE"
|
||||
out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
||||
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
||||
echo "internet_ip=$out"
|
||||
[ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
|
||||
echo "INTERNET_UNAFFECTED_OK"
|
||||
l.go:52: | 01:46:18 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
internet_ip=136.51.115.252
|
||||
INTERNET_UNAFFECTED_OK
|
||||
l.go:52: | 01:46:18 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:46:18 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:46:18 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:46:18 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
ns="$NAMESPACE"
|
||||
# Distinct Job name (tunnel-restart) so it never collides with the
|
||||
# tunnel-stop Job. Left-aligned manifest so the block scalar yields
|
||||
# valid YAML at column 0 at runtime.
|
||||
cat <<'JOB' | kubectl -n "$ns" apply -f -
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: tunnel-restart
|
||||
labels: { app: vpn-killswitch-fault }
|
||||
spec:
|
||||
backoffLimit: 3
|
||||
activeDeadlineSeconds: 90
|
||||
ttlSecondsAfterFinished: 600
|
||||
template:
|
||||
metadata:
|
||||
labels: { app: vpn-killswitch-fault }
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
homelab.keithjackson.dev/egress-gateway: "true"
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
containers:
|
||||
- name: nsenter
|
||||
image: busybox:1.37.0
|
||||
securityContext:
|
||||
privileged: true
|
||||
command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
|
||||
JOB
|
||||
# Block teardown until the restart actually completed on the host — a
|
||||
# leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
|
||||
# or timeout, which (set -e) fails the step and surfaces the problem.
|
||||
kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
|
||||
echo "TUNNEL_SELF_HEAL_OK"
|
||||
l.go:52: | 01:46:21 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
job.batch/tunnel-restart created
|
||||
job.batch/tunnel-restart condition met
|
||||
TUNNEL_SELF_HEAL_OK
|
||||
l.go:52: | 01:46:21 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:46:21 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:46:21 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-tunnel-recovers-connectivity-returns[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:46:21 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-tunnel-recovers-connectivity-returns[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
ns="$NAMESPACE"
|
||||
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
||||
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
||||
vpn=""
|
||||
for i in $(seq 1 24); do
|
||||
vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
||||
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
|
||||
if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
|
||||
vpn=""
|
||||
sleep 5
|
||||
done
|
||||
echo "recovered_vpn=$vpn wan=$wan"
|
||||
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
|
||||
[ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
|
||||
echo "TUNNEL_RECOVERED_OK"
|
||||
l.go:52: | 01:46:25 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-tunnel-recovers-connectivity-returns[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
recovered_vpn=158.173.241.11 wan=136.51.115.252
|
||||
TUNNEL_RECOVERED_OK
|
||||
l.go:52: | 01:46:25 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-tunnel-recovers-connectivity-returns[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:46:25 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-tunnel-recovers-connectivity-returns[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:46:25 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:46:25 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
|
||||
l.go:52: | 01:46:25 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:46:25 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:46:25 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Pod @ chainsaw-credible-bird/egress-vpn
|
||||
l.go:52: | 01:46:56 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Pod @ chainsaw-credible-bird/egress-internet
|
||||
l.go:52: | 01:47:26 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:47:26 | [36;1mvpn-killswitch[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:47:26 | [32;1mvpn-killswitch[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-credible-bird
|
||||
l.go:52: | 01:47:33 | [36;1mvpn-killswitch[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/vpn-killswitch (174.63s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
[1m[run.sh][0m HEAVY pass: no heavy tests selected.
|
||||
merge_junit: merged 1 report(s) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -0,0 +1,4 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="vpn-killswitch" classname="" time="173.473422" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -0,0 +1,237 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-NET-06').
|
||||
[1m[run.sh][0m LIGHT pass: 1 test(s), parallel=2 (pinned v0.2.12)
|
||||
Version: 0.2.12
|
||||
Loading config (/work/.claude/conf-clone/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-light'
|
||||
- ReportPath '/work/.claude/conf-clone/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 2
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- vpn-killswitch (/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/vpn-killswitch
|
||||
=== PAUSE chainsaw/vpn-killswitch
|
||||
=== CONT chainsaw/vpn-killswitch
|
||||
l.go:52: | 01:49:24 | [32;1mvpn-killswitch[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-beloved-toad
|
||||
l.go:52: | 01:49:24 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:49:24 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
kubectl label namespace "$NAMESPACE" \
|
||||
pod-security.kubernetes.io/enforce=privileged \
|
||||
pod-security.kubernetes.io/warn=privileged \
|
||||
pod-security.kubernetes.io/audit=privileged \
|
||||
--overwrite
|
||||
echo "NS_PRIVILEGED_OK"
|
||||
l.go:52: | 01:49:24 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
namespace/chainsaw-beloved-toad labeled
|
||||
NS_PRIVILEGED_OK
|
||||
l.go:52: | 01:49:24 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:49:24 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mAPPLY[0;22m | [36;1mRUN[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-internet
|
||||
l.go:52: | 01:49:24 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-internet
|
||||
l.go:52: | 01:49:24 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mAPPLY[0;22m | [32;1mDONE[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-internet
|
||||
l.go:52: | 01:49:24 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mAPPLY[0;22m | [36;1mRUN[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-vpn
|
||||
l.go:52: | 01:49:25 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-vpn
|
||||
l.go:52: | 01:49:25 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mAPPLY[0;22m | [32;1mDONE[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-vpn
|
||||
l.go:52: | 01:49:25 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mASSERT[0;22m | [36;1mRUN[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-internet
|
||||
l.go:52: | 01:49:26 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mASSERT[0;22m | [32;1mDONE[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-internet
|
||||
l.go:52: | 01:49:26 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mASSERT[0;22m | [36;1mRUN[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-vpn
|
||||
l.go:52: | 01:49:26 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mASSERT[0;22m | [32;1mDONE[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-vpn
|
||||
l.go:52: | 01:49:26 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
ns="$NAMESPACE"
|
||||
get_ip() {
|
||||
kubectl -n "$ns" exec "$1" -c curl -- \
|
||||
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
|
||||
}
|
||||
wan="" vpn=""
|
||||
for i in $(seq 1 12); do
|
||||
[ -n "$wan" ] || wan="$(get_ip egress-internet)"
|
||||
[ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
|
||||
if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
|
||||
[ "$wan" = "$vpn" ] && vpn="" # not steered yet; re-poll
|
||||
sleep 5
|
||||
done
|
||||
echo "wan=$wan vpn=$vpn"
|
||||
[ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
|
||||
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
|
||||
[ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
|
||||
echo "TUNNEL_HEALTHY_OK"
|
||||
l.go:52: | 01:49:27 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
wan=136.51.115.252 vpn=158.173.241.11
|
||||
TUNNEL_HEALTHY_OK
|
||||
l.go:52: | 01:49:27 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:49:27 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:49:27 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:49:27 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mAPPLY[0;22m | [36;1mRUN[0;22m | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
|
||||
l.go:52: | 01:49:27 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
|
||||
l.go:52: | 01:49:27 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mAPPLY[0;22m | [32;1mDONE[0;22m | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
|
||||
l.go:52: | 01:49:27 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mASSERT[0;22m | [36;1mRUN[0;22m | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
|
||||
l.go:52: | 01:49:31 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mASSERT[0;22m | [32;1mDONE[0;22m | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
|
||||
l.go:52: | 01:49:31 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -u
|
||||
ns="$NAMESPACE"
|
||||
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
||||
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
||||
for i in $(seq 1 8); do
|
||||
out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
||||
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
|
||||
if [ -n "$out" ]; then
|
||||
if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
|
||||
echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
|
||||
fi
|
||||
echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
|
||||
fi
|
||||
echo "attempt $i: vpn-class blackholed (no response), good"
|
||||
sleep 3
|
||||
done
|
||||
echo "NO_LEAK_OK"
|
||||
l.go:52: | 01:51:01 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
attempt 1: vpn-class blackholed (no response), good
|
||||
attempt 2: vpn-class blackholed (no response), good
|
||||
attempt 3: vpn-class blackholed (no response), good
|
||||
attempt 4: vpn-class blackholed (no response), good
|
||||
attempt 5: vpn-class blackholed (no response), good
|
||||
attempt 6: vpn-class blackholed (no response), good
|
||||
attempt 7: vpn-class blackholed (no response), good
|
||||
attempt 8: vpn-class blackholed (no response), good
|
||||
NO_LEAK_OK
|
||||
l.go:52: | 01:51:01 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:51:01 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
ns="$NAMESPACE"
|
||||
out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
||||
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
||||
echo "internet_ip=$out"
|
||||
[ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
|
||||
echo "INTERNET_UNAFFECTED_OK"
|
||||
l.go:52: | 01:51:02 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
internet_ip=136.51.115.252
|
||||
INTERNET_UNAFFECTED_OK
|
||||
l.go:52: | 01:51:02 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:51:02 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:51:02 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:51:02 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
ns="$NAMESPACE"
|
||||
# Distinct Job name (tunnel-restart) so it never collides with the
|
||||
# tunnel-stop Job. Left-aligned manifest so the block scalar yields
|
||||
# valid YAML at column 0 at runtime.
|
||||
cat <<'JOB' | kubectl -n "$ns" apply -f -
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: tunnel-restart
|
||||
labels: { app: vpn-killswitch-fault }
|
||||
spec:
|
||||
backoffLimit: 3
|
||||
activeDeadlineSeconds: 90
|
||||
ttlSecondsAfterFinished: 600
|
||||
template:
|
||||
metadata:
|
||||
labels: { app: vpn-killswitch-fault }
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
hostPID: true
|
||||
nodeSelector:
|
||||
homelab.keithjackson.dev/egress-gateway: "true"
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
containers:
|
||||
- name: nsenter
|
||||
image: busybox:1.37.0
|
||||
securityContext:
|
||||
privileged: true
|
||||
command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
|
||||
JOB
|
||||
# Block teardown until the restart actually completed on the host — a
|
||||
# leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
|
||||
# or timeout, which (set -e) fails the step and surfaces the problem.
|
||||
kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
|
||||
echo "TUNNEL_SELF_HEAL_OK"
|
||||
l.go:52: | 01:51:05 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
job.batch/tunnel-restart created
|
||||
job.batch/tunnel-restart condition met
|
||||
TUNNEL_SELF_HEAL_OK
|
||||
l.go:52: | 01:51:05 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:51:05 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:51:05 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-tunnel-recovers-connectivity-returns[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:51:05 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-tunnel-recovers-connectivity-returns[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
ns="$NAMESPACE"
|
||||
wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
|
||||
curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
|
||||
vpn=""
|
||||
for i in $(seq 1 24); do
|
||||
vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
|
||||
curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
|
||||
if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
|
||||
vpn=""
|
||||
sleep 5
|
||||
done
|
||||
echo "recovered_vpn=$vpn wan=$wan"
|
||||
[ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
|
||||
[ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
|
||||
echo "TUNNEL_RECOVERED_OK"
|
||||
l.go:52: | 01:51:08 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-tunnel-recovers-connectivity-returns[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
recovered_vpn=158.173.241.11 wan=136.51.115.252
|
||||
TUNNEL_RECOVERED_OK
|
||||
l.go:52: | 01:51:08 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-tunnel-recovers-connectivity-returns[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 01:51:08 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-tunnel-recovers-connectivity-returns[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:51:08 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:51:08 | [32;1mvpn-killswitch[0;22m | [32;1mwhen-force-tunnel-down-then-no-leak [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
|
||||
l.go:52: | 01:51:08 | [36;1mvpn-killswitch[0;22m | [36;1mwhen-force-tunnel-down-then-no-leak [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:51:08 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:51:08 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-vpn
|
||||
l.go:52: | 01:51:39 | [32;1mvpn-killswitch[0;22m | [32;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Pod @ chainsaw-beloved-toad/egress-internet
|
||||
l.go:52: | 01:52:10 | [36;1mvpn-killswitch[0;22m | [36;1mgiven-healthy-tunnel-and-two-workloads [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 01:52:10 | [36;1mvpn-killswitch[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 01:52:10 | [32;1mvpn-killswitch[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-beloved-toad
|
||||
l.go:52: | 01:52:18 | [36;1mvpn-killswitch[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/vpn-killswitch (173.47s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
[1m[run.sh][0m HEAVY pass: no heavy tests selected.
|
||||
merge_junit: merged 1 report(s) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -0,0 +1,5 @@
|
||||
{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "57d4df8c4f992732a49f3e2daddcace866778ab0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260901T235117Z-e9a4b66", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-01T23:51:17Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "fdc3265585f4b775c0a6ae3fb33c837b0c643655", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260902T001907Z-fdc3265", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-02T00:19:07Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L1", "note": "L1 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-NET-05", "R-NET-06", "R-NET-07"], "run_id": "20260913T014321Z-19fc8bc", "sentinel": null, "test": "egress-class-render", "ts_utc": "2026-09-13T01:43:21Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6-dirty", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014437Z-19fc8bc", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:44:37Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
|
||||
@@ -93,9 +93,52 @@ and coverage extraction exact and robust.
|
||||
break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
|
||||
- **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
|
||||
the PM — do not relax the assertion or the requirement.
|
||||
- **Never write a per-environment value into a test (R-ENV-02).** No hostname suffix, no
|
||||
Gateway VIP, no LB-pool range. One authored suite must run green against dev *and* prod,
|
||||
so a test **asks the cluster it is pointed at** what it is. The mechanism is a chainsaw
|
||||
binding over the same ConfigMap Flux substitutes from:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
bindings:
|
||||
- name: baseDomain
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
||||
```
|
||||
|
||||
Declare it at **test level** (step `outputs`/`bindings` are step-scoped), then consume it:
|
||||
`($baseDomain)` for a pod `env:` value, `(join('.', ['myhost', $baseDomain]))` where a
|
||||
full hostname is needed — including inside a `fixtures/*.yaml` applied with `apply: file:`,
|
||||
which chainsaw templates with the same bindings. Prefer reading the *actual* observed value
|
||||
over restating a range (e.g. assert the DNS answer equals `GATEWAY_VIP`, or equals the LB IP
|
||||
the Service was assigned) — it is both stricter and environment-independent. Never take the
|
||||
environment from a CLI flag or an exported variable: a flag someone forgets is how a suite
|
||||
ends up silently asserting the other cluster's expectations. L1 goldens are the one
|
||||
exception: they are rendering fixtures and carry the literal `${BASE_DOMAIN}` token.
|
||||
|
||||
### How a scaffold declares itself (and how `status.py` tells one from a verifier)
|
||||
|
||||
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
|
||||
later roadmap phase delivers the capability under test. `lib/status.py` treats such a
|
||||
marker — in the `chainsaw-test.yaml`, in `render-check.sh`, or in `check.sh` — as
|
||||
outranking any `conformance.home.k3s/executable: "true"` annotation, so a placeholder body
|
||||
can never be reported as an executable test.
|
||||
later roadmap phase delivers the capability under test. `lib/status.py` classifies each
|
||||
test EXECUTABLE or SCAFFOLD from the test's own files — `chainsaw-test.yaml`,
|
||||
`render-check.sh`, `check.sh` — and either signal below **outranks**
|
||||
`conformance.home.k3s/executable: "true"`, so a placeholder body can never be reported as
|
||||
an executable test. Declare a scaffold in **both** positions:
|
||||
|
||||
1. **`# STATUS: scaffold — <what is missing>`** in the file's header comment block. The
|
||||
declared state must be the first token after `STATUS:` (executable tests say
|
||||
`# STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`).
|
||||
2. **A `TODO`/`FIXME` marker carrying a message, in the body** — the file from its first
|
||||
non-comment line onward. `echo "TODO(T-4.2): inject a fault to fire a rule"`,
|
||||
or an inline comment on a missing fixture:
|
||||
`file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink`.
|
||||
|
||||
What is **not** a scaffold marker, and why the distinction is load-bearing: a marker
|
||||
mentioned in the **header prose** (the Given/When/Then and the mandatory "what this does
|
||||
NOT prove" list), and a marker that is the **entire content of a quoted literal** — a
|
||||
placeholder *detector*'s own vocabulary, e.g. `SEED = ["TBD", "TODO", "FIXME", ...]` in
|
||||
`l0-static/placeholder-guard-coupling/check.sh`. A whole-file `"TODO" in text` test used to
|
||||
classify that 32-assertion check as a scaffold and, through it, report the safety-critical
|
||||
R-STATE-03 as `exec=1 scaffold=1`. If your check must hold a marker as data, keep each
|
||||
idiom a **standalone quoted literal in a list**; a marker with text attached (a regex
|
||||
alternation, several idioms in one string) still classifies as a scaffold, because
|
||||
ambiguity here only ever downgrades.
|
||||
@@ -0,0 +1,119 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every metric an alert rule queries is CLAIMED by a scrape target this repo configures
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE DEFECT THIS CLOSES. An alert rule whose metric is never scraped is not a broken alert — it is
|
||||
# an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no
|
||||
# longhorn_* series returns no vector, so the rule is permanently quiet, permanently "healthy", and
|
||||
# indistinguishable from a correct rule on a healthy cluster. It survives review (the PromQL is
|
||||
# right), it survives `promtool test rules` (the unit test supplies the series itself), and it
|
||||
# survives every dashboard (nobody looks for a panel that was never built).
|
||||
#
|
||||
# MEASURED ON PROD, 2026-09-13, before this change — 13 rules across three families were inert:
|
||||
# count(longhorn_volume_robustness) => (empty) 4 rules
|
||||
# count({__name__=~"certmanager.*"}) => (empty) 3 rules
|
||||
# count(hubble_drop_total) => (empty) 1 rule
|
||||
# count by (job) (up) => 20 jobs, none Longhorn/cert-manager/Hubble
|
||||
# So storage health was unmonitored on a cluster whose prod-1 had just had a failing SSD
|
||||
# (DECISIONS-LOG), TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from
|
||||
# expiry, and the platform had no policy-drop visibility at all. The Flux family (gotk_*, 5 rules)
|
||||
# was the same defect and was fixed separately in a79693d.
|
||||
#
|
||||
# GIVEN every alerting rule in every PrometheusRule in the repo (platform/**, apps/wave-*),
|
||||
# WHEN each rule's PromQL is parsed for the metric names it selects,
|
||||
# THEN (1) every metric name is CLAIMED by exactly one declared scrape owner, and each owner's
|
||||
# claim is PROVEN AGAINST THE REPO — the file that enables that scrape must still contain
|
||||
# the enabling declaration, so deleting `metrics.serviceMonitor.enabled` from
|
||||
# platform/longhorn/helmrelease.yaml fails this check instead of silently re-inerting four
|
||||
# storage alerts;
|
||||
# (2) an owner's NEGATIVE conditions hold too, where the enablement can be defeated without
|
||||
# being deleted — cert-manager's chart calls `fail` if podmonitor and servicemonitor are
|
||||
# both set, Longhorn's own NetworkPolicies drop the scrape unless
|
||||
# networkpolicy-metrics.yaml is wired, and kube_node_labels does not exist at all unless
|
||||
# kube-state-metrics' metricLabelsAllowlist names the label;
|
||||
# (3) a metric with NO owner FAILS, unless it is declared in KNOWN_UNWIRED_METRICS — a
|
||||
# shrink-only register (hubble_drop_total today, with the CNI-rollout reason recorded);
|
||||
# (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
|
||||
# check above can never see this failure: where the repo authors a Service and a
|
||||
# ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
|
||||
# actually defines. All 12 apps/* `<app>-down` SLI rules fail this today (the monitors name
|
||||
# port `metrics`; no app Service defines one), so they are registered in
|
||||
# KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series do not exist;
|
||||
# AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
|
||||
# invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
|
||||
#
|
||||
# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
|
||||
# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would make the gate red until
|
||||
# exporters exist for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends —
|
||||
# none of which speak Prometheus natively — so the gate would be disabled, and a disabled gate
|
||||
# protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
|
||||
# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
|
||||
# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
|
||||
# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot
|
||||
# into fiction.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, the same dependency as
|
||||
# catalog/render.py and dev-nodes-provisioned-as-code); conformance/run.sh --layer l0 and ci/l0.sh
|
||||
# run it directly, and this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven (all named in check.sh's header, which carries the full list). The one worth stating
|
||||
# here because it bit this very repo: an owner claims metrics by PREFIX, so this proves the SCRAPE
|
||||
# TARGET exists, never that the exporter emits that exact SERIES NAME.
|
||||
# `kubevirt_vmi_phase_count{phase="Failed"}` sat inside the scraped kubevirt_ family on an UP target
|
||||
# and did not exist at all on the deployed KubeVirt (corrected here to
|
||||
# `kubevirt_vmi_info{phase="failed"}`, lowercase — a capital-F matcher is equally inert). Catching
|
||||
# that needs the exporter's own metric list, i.e. a live read; it belongs to a future L1/L3 check
|
||||
# that diffs alert metrics against /api/v1/label/__name__/values.
|
||||
#
|
||||
# Also not proven: that a target is UP. This is static — it asserts the
|
||||
# enabling declaration is in git, never that the endpoint answers, that the chart still honours the
|
||||
# value key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
|
||||
# recorded in the commit message (longhorn_volume_robustness 44 series, certmanager_* 104 series,
|
||||
# all six new targets up==1). It also cannot check a monitor whose Service comes from a Helm chart
|
||||
# rather than from this repo — the port name lives outside the tree — which is why (4) is scoped to
|
||||
# same-file Service+ServiceMonitor pairs. And it does not judge thresholds or reachability-to-human
|
||||
# (R-OBS-02 / the L3 alerting test).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: alert-metrics-have-scrape-targets
|
||||
labels:
|
||||
conformance.home.k3s/R-OBS-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-OBS-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree AND over
|
||||
# every scrape-enabling declaration that feeds them, which live in eight different components
|
||||
# (platform/monitoring, platform/longhorn, platform/cert-manager, platform/velero,
|
||||
# platform/kubevirt, platform/keycloak, platform/cnpg, apps/wave-*). No single requirement
|
||||
# path-maps to that set.
|
||||
#
|
||||
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh maps apps/* to NOTHING and
|
||||
# platform/monitoring/* to R-OBS-01/R-OBS-02 — not R-OBS-04. A change that adds an alert under
|
||||
# apps/ or platform/kubevirt/, or one that flips a chart's `serviceMonitor.enabled` back to
|
||||
# false under platform/longhorn/, therefore attributes a requirement set that would FILTER THIS
|
||||
# CHECK OUT of exactly the change that re-inerts the alerts. Declaring the scope makes
|
||||
# ci/l0.sh Stage 3b union it into every run instead, and makes an honest SKIP loud (rc 3 =
|
||||
# UNWIRED, never green) rather than silently green.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every metric name queried by every alert rule is claimed by a scrape target whose enabling
|
||||
declaration is still present in this repo, every `up{job=...}` selector names a job a repo
|
||||
monitor can actually produce, and the two gaps that remain are enumerated in registers that
|
||||
can only shrink — so an alert can never again be authored, reviewed and merged against a
|
||||
metric nothing scrapes.
|
||||
spec:
|
||||
steps:
|
||||
- name: audit-every-alert-metric-against-the-configured-scrape-targets
|
||||
description: >-
|
||||
Parse every metric name out of every alerting rule, resolve each against the declared
|
||||
scrape-owner table, prove each owner's enabling declaration (and its negative conditions)
|
||||
still hold in git, audit `up{job=...}` against same-file Service/ServiceMonitor port
|
||||
agreement, and ratchet both registers of declared gaps.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,702 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — alert-metrics-have-scrape-targets
|
||||
# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
|
||||
# VPN)." — spec docs/10-observability-and-alerting.md)
|
||||
#
|
||||
# WHY THIS EXISTS — AN ALERT ON AN UNSCRAPED METRIC IS AN ABSENT ALERT THAT READS AS A PRESENT ONE
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# `longhorn_volume_robustness == 2` evaluated against a TSDB that holds no longhorn_* series
|
||||
# returns no vector. The rule is therefore permanently quiet, permanently "healthy", and
|
||||
# INDISTINGUISHABLE from a correct rule on a healthy cluster. It passes review (the PromQL is
|
||||
# right). It passes `promtool test rules` (the unit test supplies the series itself, so a unit
|
||||
# test proves nothing about whether anything scrapes them). It shows up on no dashboard, because
|
||||
# nobody looks for a panel that was never built.
|
||||
#
|
||||
# Measured on prod 2026-09-13, BEFORE the change that added this check — 13 rules were inert:
|
||||
# count(longhorn_volume_robustness) => (empty) LonghornVolumeDegraded/Faulted/
|
||||
# CapacityHigh, LonghornNodeStorageHigh
|
||||
# count({__name__=~"certmanager.*"}) => (empty) CertificateExpiringSoon/
|
||||
# ExpiryImminent/NotReady
|
||||
# count(hubble_drop_total) => (empty) HubbleDeniedFlowSpike
|
||||
# count by (job) (up) => 20 jobs, none Longhorn, cert-manager or Hubble
|
||||
# Storage health was unmonitored on the cluster whose prod-1 had just had a failing SSD; TLS
|
||||
# expiry was unmonitored with the live `wildcard-home` wildcard 89 days out; there was no
|
||||
# policy-drop visibility at all. The gotk_* family (5 more rules) was the same defect, fixed
|
||||
# separately in a79693d.
|
||||
#
|
||||
# WHAT MAKES THIS CLASS DIFFERENT from the two failure modes alert-rules-unit-tested catches: a
|
||||
# permanently-firing rule is loud, and a rule that behaves wrongly against synthetic series is
|
||||
# caught by promtool. This one is caught by NEITHER, because the missing thing is not in the rule
|
||||
# file at all — it is a `serviceMonitor.enabled` that was never set in a completely different
|
||||
# component's HelmRelease. So the audit has to cross from the alert to the scrape config, which is
|
||||
# what the owner table below does.
|
||||
#
|
||||
# GIVEN every alerting rule in every PrometheusRule in the repo,
|
||||
# WHEN each rule's PromQL is parsed for the metric names it selects,
|
||||
# THEN (1) every metric name is CLAIMED by exactly one declared owner in SCRAPE_OWNERS, and the
|
||||
# owner's `require` regexes still match its file — so deleting the enablement from
|
||||
# platform/longhorn/helmrelease.yaml FAILS here instead of silently re-inerting four
|
||||
# storage alerts;
|
||||
# (2) each owner's `forbid` regexes do NOT match, for the cases where the enablement can be
|
||||
# defeated without being deleted (cert-manager's chart `fail`s if podmonitor and
|
||||
# servicemonitor are both enabled; an `enabled: false` inside the exporter's own block);
|
||||
# (3) a metric with no owner FAILS unless declared in KNOWN_UNWIRED_METRICS (shrink-only);
|
||||
# (4) `up{job="X"}` is audited separately — `up` itself always exists, so (1) is blind to
|
||||
# this — by requiring that where this repo authors a Service and a ServiceMonitor in the
|
||||
# SAME file, the monitor's endpoint port is a port that Service actually defines;
|
||||
# violations must be declared in KNOWN_UNWIRED_JOBS (shrink-only);
|
||||
# AND (5) the check is LOAD-BEARING: this test declares `conformance.home.k3s/scope: whole-repo`
|
||||
# so ci/l0.sh unions it into every gate run, and ci/gate.sh blocks on an l0 failure.
|
||||
#
|
||||
# WHY THE REGISTERS ARE RATCHETS (the decision and its reasons) — the same three options as
|
||||
# alert-rules-unit-tested faced:
|
||||
# * FAIL outright on all 13: correct in principle, but 12 of them are app SLI rules that need
|
||||
# Prometheus exporters for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr and Seerr —
|
||||
# none of which speak Prometheus natively. The gate would be red for every change until that
|
||||
# work lands, so it would get disabled, and a disabled gate protects nothing.
|
||||
# * merely REPORT: that is precisely how these rules stayed inert across phases 4-6.
|
||||
# * RATCHET (chosen): each gap is declared here, printed with counts on every run, and CANNOT
|
||||
# GROW. A new alert on an unclaimed metric FAILS. A new same-file Service+ServiceMonitor pair
|
||||
# with a mismatched port FAILS. A STALE entry — the gap was closed, or the alert deleted —
|
||||
# also FAILS, so neither register can rot into fiction.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
|
||||
# - It does not prove a target is UP, that an endpoint answers, that the chart still honours the
|
||||
# values key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
|
||||
# recorded in the commit message: with the enablements applied, longhorn_volume_robustness
|
||||
# returned 44 series, {__name__=~"certmanager.*"} returned 104, and all six new targets read
|
||||
# up==1. A live probe needs a cluster, which L0 by definition does not have.
|
||||
# - It cannot audit a ServiceMonitor whose Service is created by a Helm chart rather than by this
|
||||
# repo: the port name lives outside the tree. That is why (4) is scoped to same-file
|
||||
# Service+ServiceMonitor pairs — the apps/* ServiceDefinition pattern — and why the
|
||||
# platform/* monitors (oauth2-proxy, kubevirt, keycloak, flux) are out of its reach. Their
|
||||
# liveness is covered by the fact that their metrics have owners under (1).
|
||||
# - IT PROVES THE SCRAPE TARGET, NOT THE SERIES NAME. An owner claims a metric by PREFIX, so a
|
||||
# metric in a scraped family whose exact name the exporter does not emit still passes. This is
|
||||
# a real residual hole and it was live in this repo: `kubevirt_vmi_phase_count{phase="Failed"}`
|
||||
# (KubeVirtVMLaunchFailing) sat inside the scraped `kubevirt_` family on an UP target, and the
|
||||
# series simply does not exist on the deployed KubeVirt — verified on prod 2026-09-13, where the
|
||||
# only kubevirt_vmi_* names are `info` and `number_of_outdated`, and the only phase-bearing
|
||||
# names are the phase_transition_time histograms. Corrected in the same change to
|
||||
# `kubevirt_vmi_info{phase="failed"}` (note the LOWERCASE value KubeVirt writes: a capital-F
|
||||
# matcher selects nothing and is just as inert). Catching that class needs the exporter's own
|
||||
# metric list, which is a live read, not a static one — it belongs to a future L1/L3 check that
|
||||
# diffs every alert metric against `/api/v1/label/__name__/values`. The same limit applies to
|
||||
# every LABEL VALUE an alert matches on.
|
||||
# - The metric-name parser is a PromQL approximation, not a parser: it strips string literals,
|
||||
# label-matcher bodies and aggregation label lists, then treats remaining bare identifiers as
|
||||
# metric selectors. A metric referenced ONLY through a `{__name__=~"..."}` regex selector would
|
||||
# be missed. No rule in this repo does that, and assertion (0) fails if the parser stops
|
||||
# finding metrics at all.
|
||||
# - It does not judge thresholds, and does not prove an alert reaches a human (R-OBS-02 and the
|
||||
# L3 alerting test own that).
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, and because this test
|
||||
# declares the whole-repo scope the runner turns it into rc 3 = UNWIRED, never green) only if the
|
||||
# repo contains no PrometheusRule at all — i.e. there is genuinely nothing to audit.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
|
||||
if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
|
||||
echo "check SKIPPED: no PrometheusRule found under platform/ or apps/ — no alert rules to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit: every alert rule's metrics are claimed by a scrape target this repo configures.
|
||||
|
||||
PyYAML is used deliberately rather than the line-based parsing the neighbouring checks favour:
|
||||
alert `expr:` values are multi-line block scalars, and a line-based reader of PromQL would be the
|
||||
kind of approximate parser that silently stops matching — the exact failure class this check
|
||||
exists to prevent. Same dependency as catalog/render.py and dev-nodes-provisioned-as-code.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def read(p: Path) -> str:
|
||||
try:
|
||||
return p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
|
||||
|
||||
# =============================================================================================
|
||||
# THE DECLARED SCRAPE-OWNER TABLE (assertions 1 and 2).
|
||||
#
|
||||
# Each entry claims a set of metric names (exact names and/or prefixes) and names the file whose
|
||||
# declaration is what makes those metrics exist in the TSDB, plus the regexes that prove it. This
|
||||
# is the crossing from "an alert queries X" to "something in this repo scrapes X", and it is the
|
||||
# part that makes the check a REGRESSION guard rather than a one-time audit: every `require`
|
||||
# regex is re-evaluated against the working tree on every gate run.
|
||||
#
|
||||
# `require` — ALL must match the file. The enablement itself.
|
||||
# `forbid` — NONE may match. For enablements that can be defeated without being deleted.
|
||||
#
|
||||
# Longest prefix wins, so kube_velero_schedule_ is resolved before kube_.
|
||||
# =============================================================================================
|
||||
KPS = "platform/monitoring/kube-prometheus-stack.yaml"
|
||||
|
||||
SCRAPE_OWNERS: list[dict] = [
|
||||
{
|
||||
"owner": "kube-prometheus-stack: bundled node-exporter DaemonSet + its ServiceMonitor",
|
||||
"prefixes": ("node_",),
|
||||
"file": KPS,
|
||||
# The subchart block must be present and configured here (it is where the exporter's
|
||||
# resources live). The chart default is enabled:true, so the regression to guard is a
|
||||
# DISABLE, not a missing enable.
|
||||
"require": [r"^\s*prometheus-node-exporter:\s*$"],
|
||||
"forbid": [r"prometheus-node-exporter:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
|
||||
"why": "node_* powers all seven platform-health.nodes rules including the two IO-pressure "
|
||||
"rules that exist because of prod-1's etcd WAL starvation",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)",
|
||||
"prefixes": ("kubelet_",),
|
||||
"file": KPS,
|
||||
# Chart default kubelet.enabled:true. There is no explicit enable to require, so the only
|
||||
# honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were.
|
||||
"require": [r"kubeProxy:"], # sentinel: the exporter-toggle block is still authored here
|
||||
"forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
|
||||
"why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three "
|
||||
"pvc-capacity-alerts rules",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules",
|
||||
"prefixes": ("kube_velero_schedule_",),
|
||||
"file": KPS,
|
||||
"require": [
|
||||
r"customResourceState:\s*\n\s*enabled:\s*true",
|
||||
r"metricNamePrefix:\s*kube_velero_schedule",
|
||||
# The RBAC is MANDATORY, not cosmetic: without velero.io read access KSM emits the
|
||||
# custom-resource-state metrics as an EMPTY SET and logs a forbidden error — silent,
|
||||
# and the inert-alert failure mode all over again (measured; see the block's header).
|
||||
r'apiGroups:\s*\["velero\.io"\]',
|
||||
r'resources:\s*\["schedules"\]',
|
||||
],
|
||||
"forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
|
||||
"why": "kube_velero_schedule_* is the only thing that can see a Schedule that has NEVER "
|
||||
"backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
|
||||
"Velero itself publishes nothing until a backup is attempted",
|
||||
},
|
||||
{
|
||||
"owner": "kube-prometheus-stack: bundled kube-state-metrics",
|
||||
"prefixes": ("kube_",),
|
||||
"file": KPS,
|
||||
"require": [r"^\s*kube-state-metrics:\s*$"],
|
||||
"forbid": [r"kube-state-metrics:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
|
||||
# kube_node_labels is a SPECIAL CASE and is checked on its own below: KSM exposes no
|
||||
# custom node labels by default, so the metric does not exist unless metricLabelsAllowlist
|
||||
# names the label. The prefix claim alone would pass while VPNTunnelDown's join matched
|
||||
# nothing.
|
||||
"why": "kube_* powers the Keycloak, KubeVirt-crashloop, CNPG-colocation and Velero-schedule "
|
||||
"rules",
|
||||
},
|
||||
{
|
||||
"owner": "kube-state-metrics metricLabelsAllowlist (the egress-gateway node label)",
|
||||
"exact": ("kube_node_labels",),
|
||||
"file": KPS,
|
||||
"require": [
|
||||
r"metricLabelsAllowlist:",
|
||||
r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
|
||||
],
|
||||
"forbid": [],
|
||||
"why": "KSM exposes NO custom labels by default. Without the allowlist entry "
|
||||
"kube_node_labels carries no egress-gateway label and VPNTunnelDown's join silently "
|
||||
"matches nothing — a rule that is present, evaluating, and structurally inert",
|
||||
},
|
||||
{
|
||||
"owner": "CNPG: enablePodMonitor on the Cluster CRs (the operator renders one PodMonitor each)",
|
||||
"prefixes": ("cnpg_",),
|
||||
"file": "platform/keycloak/postgres-cluster.yaml",
|
||||
"require": [r"^\s*enablePodMonitor:\s*true\s*$"],
|
||||
"forbid": [],
|
||||
"why": "cnpg_* powers all six platform-health.cnpg rules, including the WAL-archive backlog "
|
||||
"and no-recoverability-point rules that guard R-STATE-03",
|
||||
},
|
||||
{
|
||||
"owner": "Flux controllers: platform/monitoring/flux-podmonitor.yaml (merged a79693d)",
|
||||
"prefixes": ("gotk_",),
|
||||
"file": "platform/monitoring/flux-podmonitor.yaml",
|
||||
"require": [r"kind:\s*PodMonitor", r"port:\s*http-prom"],
|
||||
"forbid": [],
|
||||
"why": "gotk_* powers FluxReconciliationFailure, FluxResourceSuspended and the three "
|
||||
"automated-update rules — on a platform whose entire delivery model is Flux",
|
||||
},
|
||||
{
|
||||
"owner": "cert-manager: prometheus.servicemonitor.enabled in the HelmRelease",
|
||||
"prefixes": ("certmanager_",),
|
||||
"file": "platform/cert-manager/helmrelease.yaml",
|
||||
"require": [r"prometheus:\s*\n\s*servicemonitor:\s*\n\s*enabled:\s*true"],
|
||||
# The chart calls `fail` when both monitor kinds are enabled, which would break the whole
|
||||
# HelmRelease rather than just the scrape. Measured against v1.21.2.
|
||||
"forbid": [r"podmonitor:\s*\n\s*enabled:\s*true"],
|
||||
"why": "certmanager_* powers CertificateExpiringSoon / ExpiryImminent / NotReady — the only "
|
||||
"warning that an ACME DNS-01 renewal of the live wildcard has quietly failed",
|
||||
},
|
||||
{
|
||||
"owner": "Longhorn: metrics.serviceMonitor.enabled in the HelmRelease",
|
||||
"prefixes": ("longhorn_",),
|
||||
"file": "platform/longhorn/helmrelease.yaml",
|
||||
"require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
|
||||
"forbid": [],
|
||||
"why": "longhorn_* powers the four storage rules on a cluster whose recurring failure mode "
|
||||
"is a disk",
|
||||
# The ServiceMonitor ALONE is not sufficient here — see the companion assertion below.
|
||||
"companion": {
|
||||
"file": "platform/longhorn/kustomization.yaml",
|
||||
"require": [r"^\s*-\s*networkpolicy-metrics\.yaml\s*$"],
|
||||
"why": "Longhorn 1.12.1 ships its own effectively-unconditional NetworkPolicies "
|
||||
"(guarded on networkPolicies.restrictInternalTraffic, default true — NOT on "
|
||||
"networkPolicies.enabled) whose longhorn-manager policy admits only other "
|
||||
"Longhorn components in the same namespace. MEASURED on prod: with the "
|
||||
"ServiceMonitor applied and no allowance, all three targets were DISCOVERED and "
|
||||
"every scrape failed with 'context deadline exceeded' after 10.0s — a drop, not "
|
||||
"a refusal — and the TSDB stayed empty. So without networkpolicy-metrics.yaml "
|
||||
"the four storage alerts are inert even though the monitor exists",
|
||||
},
|
||||
},
|
||||
{
|
||||
"owner": "KubeVirt/CDI: platform/kubevirt/servicemonitor.yaml",
|
||||
"prefixes": ("kubevirt_",),
|
||||
"file": "platform/kubevirt/servicemonitor.yaml",
|
||||
"require": [r"kind:\s*ServiceMonitor", r"port:\s*metrics"],
|
||||
"forbid": [],
|
||||
"why": "kubevirt_* powers the four platform-health.kubevirt rules (note the CDI rules use "
|
||||
"the kubevirt_cdi_* names, not bare cdi_*)",
|
||||
},
|
||||
{
|
||||
"owner": "Velero: metrics.serviceMonitor.enabled in the HelmRelease",
|
||||
"prefixes": ("velero_",),
|
||||
"file": "platform/velero/helmrelease.yaml",
|
||||
"require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
|
||||
"forbid": [],
|
||||
"why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
|
||||
"that failed to notice prod running with zero volume backups",
|
||||
},
|
||||
]
|
||||
|
||||
# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
|
||||
# is structurally blind to `up{job="X"}` naming a job nothing produces. That axis is assertion (4).
|
||||
UNIVERSAL = {"up"}
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER 1 — KNOWN_UNWIRED_METRICS. Shrink-only. A metric here is queried by an alert and
|
||||
# claimed by NOTHING, so that alert is inert and the platform is blind to what it describes.
|
||||
# Removing an entry means wiring the scrape. An entry that stops being referenced by any alert,
|
||||
# or that gains an owner, FAILS as stale.
|
||||
# =============================================================================================
|
||||
KNOWN_UNWIRED_METRICS: dict[str, str] = {
|
||||
"hubble_drop_total":
|
||||
"HubbleDeniedFlowSpike — the platform has NO policy-drop / NetworkPolicy-regression "
|
||||
"visibility. Unlike Longhorn and cert-manager (whose charts already serve /metrics and "
|
||||
"only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is switched on "
|
||||
"in Cilium's own config: platform/cilium/helmrelease.yaml sets hubble.{enabled,relay,ui} "
|
||||
"but not hubble.metrics.enabled, and not prometheus.enabled for the agent. Turning them on "
|
||||
"OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all three nodes "
|
||||
"— the single most dangerous rollout on this cluster (the 2026-09-01 k8sServiceHost outage; "
|
||||
"cilium was moved to its own Flux Kustomization precisely to isolate that blast radius) — "
|
||||
"and requires a matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same "
|
||||
"commit or a rebuilt node's bootstrap CNI config silently disagrees with the reconciled "
|
||||
"one. Deliberately DEFERRED to its own change with a one-node canary; see the rule's "
|
||||
"header in platform/monitoring/alerts.yaml.",
|
||||
}
|
||||
|
||||
# =============================================================================================
|
||||
# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only. Each is a `up{job="X"}` selector whose monitor
|
||||
# names an endpoint port that the Service it selects does not define, so the monitor discovers
|
||||
# ZERO targets, no `up{job="X"}` series exists, and `up{job="X"} == 0` can never fire. This is the
|
||||
# nastiest variant of the class: `up` exists in abundance, so nothing about the metric name looks
|
||||
# wrong, and `PrometheusTargetsMissing` cannot see it either — a target that was never discovered
|
||||
# has no up series to be 0.
|
||||
#
|
||||
# All twelve are the apps/* ServiceDefinition SLI pattern: the rendered ServiceMonitor names
|
||||
# `port: metrics`, and no app Service defines a port of that name (verified: zero occurrences of
|
||||
# `name: metrics` across all twelve manifests). VERIFIED on prod 2026-09-13: `count by (job) (up)`
|
||||
# listed 20 jobs and none of these twelve appeared; all twelve `<app>-down` rules returned no
|
||||
# series. Closing this needs a Prometheus exporter per app — Plex, Pi-hole, Jackett, qBittorrent,
|
||||
# Sonarr, Radarr and Seerr do not speak Prometheus natively — so it is its own piece of work, and
|
||||
# for keycloak specifically the job labels that DO exist are `keycloak-service`/`keycloak-discovery`
|
||||
# (platform/keycloak/servicemonitor.yaml), so that one is a rename away rather than an exporter.
|
||||
# =============================================================================================
|
||||
KNOWN_UNWIRED_JOBS: dict[str, str] = {
|
||||
"gitea": "apps/wave-0/gitea.yaml — Service defines only port-3000",
|
||||
"jackett": "apps/wave-0/jackett.yaml — no exporter",
|
||||
"pihole": "apps/wave-0/pihole.yaml — needs pihole-exporter",
|
||||
"plex": "apps/wave-0/plex.yaml — no native Prometheus endpoint",
|
||||
"qbittorrent": "apps/wave-0/qbittorrent.yaml — needs qbittorrent-exporter",
|
||||
"vm-demo": "apps/wave-0/vm-demo.yaml — a KubeVirt demo VM, no exporter",
|
||||
"keycloak": "apps/wave-1/keycloak.yaml — the live jobs are keycloak-service / "
|
||||
"keycloak-discovery (platform/keycloak/servicemonitor.yaml); this SLI rule names "
|
||||
"neither, so it is a rename away from being live rather than an exporter away",
|
||||
"nextcloud": "apps/wave-1/nextcloud.yaml — needs nextcloud-exporter (serverinfo token)",
|
||||
"radarr": "apps/wave-1/radarr.yaml — needs exportarr",
|
||||
"semaphore": "apps/wave-1/semaphore.yaml — no exporter",
|
||||
"sonarr": "apps/wave-1/sonarr.yaml — needs exportarr",
|
||||
"seerr": "apps/wave-2/seerr.yaml — no exporter",
|
||||
}
|
||||
|
||||
# =============================================================================================
|
||||
# (0) PARSE EVERY ALERT RULE
|
||||
# =============================================================================================
|
||||
# PromQL metric-name approximation. Documented as a gap in the header: string literals,
|
||||
# label-matcher bodies and aggregation/vector-matching label lists are stripped first, then every
|
||||
# remaining bare identifier that is not a PromQL keyword or function is treated as a metric
|
||||
# selector. Assertion (0b) fails loudly if this stops finding metrics at all, which is the way an
|
||||
# approximate parser silently stops guarding.
|
||||
PROMQL_KEYWORDS = set("""
|
||||
abs absent absent_over_time ceil changes clamp clamp_max clamp_min day_of_month day_of_week
|
||||
day_of_year days_in_month delta deriv exp floor histogram_quantile histogram_count histogram_sum
|
||||
histogram_fraction histogram_stddev histogram_stdvar holt_winters hour idelta increase irate
|
||||
label_join label_replace ln log2 log10 minute month predict_linear rate resets round scalar sgn
|
||||
sort sort_desc sort_by_label sort_by_label_desc sqrt time timestamp vector year avg count
|
||||
count_values group max min quantile stddev stdvar sum topk bottomk limitk limit_ratio
|
||||
last_over_time avg_over_time min_over_time max_over_time sum_over_time count_over_time
|
||||
quantile_over_time stddev_over_time stdvar_over_time present_over_time mad_over_time
|
||||
double_exponential_smoothing info ts_of_max_over_time ts_of_min_over_time ts_of_last_over_time
|
||||
by without on ignoring group_left group_right offset bool and or unless if atan2 start end
|
||||
inf nan NaN Inf
|
||||
""".split())
|
||||
|
||||
IDENT_RE = re.compile(r"(?<![\w:.])([a-zA-Z_:][a-zA-Z0-9_:]*)")
|
||||
UP_JOB_RE = re.compile(r"""\bup\s*\{[^}]*\bjob\s*=\s*["']([^"']+)["']""")
|
||||
|
||||
|
||||
def metric_names(expr: str) -> list[str]:
|
||||
e = re.sub(r'"(?:[^"\\]|\\.)*"', '""', expr)
|
||||
e = re.sub(r"'(?:[^'\\]|\\.)*'", "''", e)
|
||||
e = re.sub(r"`(?:[^`\\]|\\.)*`", "``", e)
|
||||
e = re.sub(r"\{[^{}]*\}", " ", e) # label matchers
|
||||
for _ in range(6): # nested aggregations
|
||||
e = re.sub(r"\b(?:by|without|on|ignoring|group_left|group_right)\s*\([^()]*\)", " ", e)
|
||||
out, seen = [], set()
|
||||
for m in IDENT_RE.finditer(e):
|
||||
n = m.group(1)
|
||||
if n in PROMQL_KEYWORDS or re.fullmatch(r"\d+[smhdwy]?", n) or n in seen:
|
||||
continue
|
||||
seen.add(n)
|
||||
out.append(n)
|
||||
return out
|
||||
|
||||
|
||||
def is_golden_fixture(p: Path) -> bool:
|
||||
"""conformance/tests/**/golden/** holds EXPECTED-OUTPUT fixtures, not deployed manifests.
|
||||
|
||||
conformance/tests/l1-rendering/egress-class-render/golden/*.expected.yaml each contain a
|
||||
rendered `<app>-down` PrometheusRule. They are the assertion data of another test, never
|
||||
reconciled into a cluster, so auditing their scrape targets would be auditing a copy — the
|
||||
same "reference, not declaration" distinction ci/touched-requirements.sh draws for ci/tests/
|
||||
fixture paths. The templates they are rendered FROM are audited through apps/ and catalog/.
|
||||
"""
|
||||
parts = p.parts
|
||||
return "conformance" in parts and "golden" in parts
|
||||
|
||||
|
||||
rule_files = sorted(
|
||||
p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
|
||||
)
|
||||
check(len(rule_files) > 0,
|
||||
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
|
||||
"this check would silently audit nothing")
|
||||
|
||||
# metric -> [ "<relpath>:<alert>", ... ] ; and the same shape for up{job="X"} selectors
|
||||
refs: dict[str, list[str]] = {}
|
||||
job_refs: dict[str, list[str]] = {}
|
||||
n_rules = 0
|
||||
for f in rule_files:
|
||||
rel = str(f.relative_to(ROOT))
|
||||
try:
|
||||
docs = list(yaml.safe_load_all(read(f)))
|
||||
except yaml.YAMLError as exc:
|
||||
check(False, f"{rel} is not parseable YAML ({exc}); every alert in it is unaudited")
|
||||
continue
|
||||
for d in docs:
|
||||
if not isinstance(d, dict) or d.get("kind") != "PrometheusRule":
|
||||
continue
|
||||
for grp in (d.get("spec") or {}).get("groups") or []:
|
||||
for r in grp.get("rules") or []:
|
||||
if "alert" not in r:
|
||||
continue # recording rules: not this check's subject
|
||||
n_rules += 1
|
||||
site = f"{rel}:{r['alert']}"
|
||||
expr = str(r.get("expr", ""))
|
||||
for n in metric_names(expr):
|
||||
refs.setdefault(n, []).append(site)
|
||||
for j in UP_JOB_RE.findall(expr):
|
||||
job_refs.setdefault(j, []).append(site)
|
||||
|
||||
check(n_rules > 0,
|
||||
"no alerting rule parsed out of any PrometheusRule — the parse contract broke and this "
|
||||
"check would pass vacuously while every alert's metric went unaudited")
|
||||
check(len(refs) >= 20,
|
||||
f"only {len(refs)} distinct metric name(s) parsed out of {n_rules} alerting rule(s). The "
|
||||
f"PromQL approximation has stopped matching — which is exactly how an approximate parser "
|
||||
f"turns into a guard that passes while checking nothing. Fix metric_names() or replace it "
|
||||
f"with a real parser")
|
||||
|
||||
# =============================================================================================
|
||||
# (1)(2) RESOLVE EVERY METRIC TO AN OWNER, AND PROVE THE OWNER'S CLAIM AGAINST THE REPO
|
||||
# =============================================================================================
|
||||
def resolve(metric: str) -> dict | None:
|
||||
"""Longest-prefix / exact-name match against SCRAPE_OWNERS."""
|
||||
best, best_len = None, -1
|
||||
for o in SCRAPE_OWNERS:
|
||||
if metric in o.get("exact", ()): # exact names win outright
|
||||
return o
|
||||
for pfx in o.get("prefixes", ()):
|
||||
if metric.startswith(pfx) and len(pfx) > best_len:
|
||||
best, best_len = o, len(pfx)
|
||||
return best
|
||||
|
||||
|
||||
# Every owner's FILE must exist, whether or not a metric currently lands on it: an owner pointing
|
||||
# at a deleted file is a claim nothing backs.
|
||||
for o in SCRAPE_OWNERS:
|
||||
check((ROOT / o["file"]).is_file(),
|
||||
f"SCRAPE_OWNERS names {o['file']} as the scrape owner for "
|
||||
f"{o.get('prefixes') or o.get('exact')}, and that file does not exist in this repo — the "
|
||||
f"claim is unbacked and the alerts that depend on it are inert")
|
||||
|
||||
claimed: dict[str, dict] = {}
|
||||
unowned: list[str] = []
|
||||
for metric in sorted(refs):
|
||||
if metric in UNIVERSAL:
|
||||
continue
|
||||
o = resolve(metric)
|
||||
if o is None:
|
||||
if metric in KNOWN_UNWIRED_METRICS:
|
||||
continue # declared gap; ratcheted below
|
||||
unowned.append(metric)
|
||||
check(False,
|
||||
f"alert metric `{metric}` (queried by {', '.join(sorted(set(refs[metric])))}) is "
|
||||
f"claimed by NO entry in SCRAPE_OWNERS. Nothing in this repo configures a scrape "
|
||||
f"that produces it, so every rule above is STRUCTURALLY INERT: it evaluates to no "
|
||||
f"vector, stays silent forever, and is indistinguishable from a healthy result. "
|
||||
f"Either wire the scrape and add the owner entry, or — if it cannot be wired yet — "
|
||||
f"declare it in KNOWN_UNWIRED_METRICS with the reason, so the gap is reported "
|
||||
f"instead of silent")
|
||||
continue
|
||||
check(metric not in KNOWN_UNWIRED_METRICS,
|
||||
f"`{metric}` is listed in KNOWN_UNWIRED_METRICS but is now claimed by "
|
||||
f"{o['owner']} — the register is shrink-only: delete the entry so the next real gap is "
|
||||
f"the one that shows up")
|
||||
claimed[metric] = o
|
||||
|
||||
# The owners that are actually load-bearing right now get their declarations verified.
|
||||
for o in SCRAPE_OWNERS:
|
||||
if o not in claimed.values():
|
||||
continue # nothing queries it today; nothing to guard
|
||||
body = read(ROOT / o["file"])
|
||||
for rx in o.get("require", []):
|
||||
check(re.search(rx, body, re.M) is not None,
|
||||
f"{o['file']} no longer matches /{rx}/, which is what makes {o['owner']} scrape. "
|
||||
f"Removing it re-inerts every alert on "
|
||||
f"{o.get('prefixes') or o.get('exact')} — {o['why']}")
|
||||
for rx in o.get("forbid", []):
|
||||
check(re.search(rx, body, re.M) is None,
|
||||
f"{o['file']} matches the FORBIDDEN pattern /{rx}/, which defeats {o['owner']} "
|
||||
f"even though the enablement is still authored. {o['why']}")
|
||||
comp = o.get("companion")
|
||||
if comp:
|
||||
cbody = read(ROOT / comp["file"])
|
||||
check((ROOT / comp["file"]).is_file(),
|
||||
f"{comp['file']} is missing — {comp['why']}")
|
||||
for rx in comp.get("require", []):
|
||||
check(re.search(rx, cbody, re.M) is not None,
|
||||
f"{comp['file']} no longer matches /{rx}/. The monitor alone is NOT enough here: "
|
||||
f"{comp['why']}")
|
||||
|
||||
# The Longhorn NetworkPolicy must actually grant the path it exists to grant, and no more: the
|
||||
# whole point of splitting it out instead of flipping networkPolicies.restrictInternalTraffic off.
|
||||
NP = ROOT / "platform/longhorn/networkpolicy-metrics.yaml"
|
||||
if any(m.startswith("longhorn_") for m in refs):
|
||||
np = read(NP)
|
||||
check(np != "", f"{NP.relative_to(ROOT)} is missing; the Longhorn scrape is dropped and its "
|
||||
f"four alerts are inert")
|
||||
check(re.search(r"kind:\s*NetworkPolicy", np) is not None,
|
||||
"platform/longhorn/networkpolicy-metrics.yaml is not a NetworkPolicy")
|
||||
check(re.search(r"app:\s*longhorn-manager", np) is not None,
|
||||
"the Longhorn metrics NetworkPolicy does not select app=longhorn-manager, so it does not "
|
||||
"open the pods that serve /metrics")
|
||||
check(re.search(r"kubernetes\.io/metadata\.name:\s*monitoring", np) is not None,
|
||||
"the Longhorn metrics NetworkPolicy does not admit namespace monitoring, so Prometheus "
|
||||
"still cannot reach longhorn-manager and the storage alerts stay inert")
|
||||
check(re.search(r"port:\s*9500", np) is not None,
|
||||
"the Longhorn metrics NetworkPolicy does not name port 9500 (longhorn-manager's `manager` "
|
||||
"port, which serves /metrics)")
|
||||
# LEAST PRIVILEGE (R-SEC-02): ingress only, one port, and NOT the webhook/recovery ports.
|
||||
check(re.search(r"policyTypes:\s*\n\s*-\s*Ingress\s*$", np, re.M) is not None
|
||||
and "Egress" not in np,
|
||||
"the Longhorn metrics NetworkPolicy is not Ingress-only — it must grant exactly the "
|
||||
"scrape path and nothing else (R-SEC-02)")
|
||||
# Comments are stripped first: the file's header deliberately NAMES 9502/9503 to record that
|
||||
# they stay closed, and a substring search over prose would fail on the documentation of the
|
||||
# very property it is asserting.
|
||||
np_spec = "\n".join(re.sub(r"#.*$", "", ln) for ln in np.splitlines())
|
||||
for closed in ("9502", "9503"):
|
||||
check(closed not in np_spec,
|
||||
f"the Longhorn metrics NetworkPolicy names port {closed} (admission webhook / "
|
||||
f"recovery backend). Only 9500 serves metrics; opening the others to ns monitoring "
|
||||
f"is reach this observability path does not need (R-SEC-02)")
|
||||
|
||||
# stale KNOWN_UNWIRED_METRICS entries
|
||||
for m in sorted(KNOWN_UNWIRED_METRICS):
|
||||
check(m in refs,
|
||||
f"KNOWN_UNWIRED_METRICS declares `{m}`, which NO alert rule in this repo queries any "
|
||||
f"more — a stale exemption hides the next real gap; delete the entry")
|
||||
|
||||
# =============================================================================================
|
||||
# (4) up{job="X"} — the axis assertion (1) is structurally blind to.
|
||||
#
|
||||
# Scoped to files where THIS REPO authors both the Service and the ServiceMonitor (the apps/*
|
||||
# ServiceDefinition pattern), because only then is the port name inside the tree. A monitor whose
|
||||
# Service comes from a Helm chart cannot be audited statically and is out of scope (header gap).
|
||||
# =============================================================================================
|
||||
def yaml_docs(path: Path) -> list[dict]:
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(read(path)) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
|
||||
same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
|
||||
for p in sorted(ROOT.rglob("*.yaml")):
|
||||
if ".git" in p.parts:
|
||||
continue
|
||||
txt = read(p)
|
||||
if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:
|
||||
continue
|
||||
docs = yaml_docs(p)
|
||||
monitors = [d for d in docs if d.get("kind") == "ServiceMonitor"]
|
||||
services = [d for d in docs if d.get("kind") == "Service"]
|
||||
if not monitors or not services:
|
||||
continue
|
||||
svc_ports: set[str] = set()
|
||||
for s in services:
|
||||
for port in (s.get("spec") or {}).get("ports") or []:
|
||||
if isinstance(port, dict) and port.get("name"):
|
||||
svc_ports.add(str(port["name"]))
|
||||
for m in monitors:
|
||||
ep_ports = {str(e["port"]) for e in (m.get("spec") or {}).get("endpoints") or []
|
||||
if isinstance(e, dict) and e.get("port")}
|
||||
name = (m.get("metadata") or {}).get("name")
|
||||
if name:
|
||||
same_file_monitors[str(name)] = (str(p.relative_to(ROOT)), ep_ports, svc_ports)
|
||||
|
||||
for job in sorted(job_refs):
|
||||
sites = ", ".join(sorted(set(job_refs[job])))
|
||||
mon = same_file_monitors.get(job)
|
||||
if mon is None:
|
||||
# No same-file Service+ServiceMonitor pair is named for this job. It may well be produced
|
||||
# by a platform monitor against a chart Service (oauth2-proxy, kubevirt, velero, flux all
|
||||
# are), which this check cannot see — so this is NOT a failure, only unauditable.
|
||||
continue
|
||||
rel, ep_ports, svc_ports = mon
|
||||
missing = sorted(ep_ports - svc_ports)
|
||||
if missing:
|
||||
check(job in KNOWN_UNWIRED_JOBS,
|
||||
f"`up{{job=\"{job}\"}}` ({sites}) can never produce a series: {rel}'s ServiceMonitor "
|
||||
f"scrapes port(s) {missing}, and the Service(s) in that same file define only "
|
||||
f"{sorted(svc_ports) or ['(no named ports)']}. The monitor therefore discovers ZERO "
|
||||
f"targets, there is no `up{{job=\"{job}\"}}` to compare against 0, and the rule is "
|
||||
f"structurally inert — note PrometheusTargetsMissing cannot see this either, because "
|
||||
f"a target that was never discovered has no up series to be 0. Give the Service a "
|
||||
f"named port the workload actually serves metrics on (and an exporter if it has "
|
||||
f"none), or declare it in KNOWN_UNWIRED_JOBS")
|
||||
else:
|
||||
check(job not in KNOWN_UNWIRED_JOBS,
|
||||
f"`{job}` is listed in KNOWN_UNWIRED_JOBS but {rel}'s ServiceMonitor port(s) "
|
||||
f"{sorted(ep_ports)} now all exist on its Service — the register is shrink-only: "
|
||||
f"delete the entry")
|
||||
|
||||
# stale KNOWN_UNWIRED_JOBS entries
|
||||
for job in sorted(KNOWN_UNWIRED_JOBS):
|
||||
check(job in job_refs,
|
||||
f"KNOWN_UNWIRED_JOBS declares `{job}`, which no alert rule selects via up{{job=...}} any "
|
||||
f"more — a stale exemption hides the next real gap; delete the entry")
|
||||
|
||||
# =============================================================================================
|
||||
# (5) THE CHECK IS LOAD-BEARING. Asserted against the DECLARATIVE mechanism, in both halves —
|
||||
# neither alone is sufficient. The per-test hand-written ci/l0.sh stage is deliberately NOT used
|
||||
# (whole-repo-invariants-wired FAILS the gate if one reappears); the property is DECLARED here and
|
||||
# HONOURED by ci/l0.sh's --include-scope union.
|
||||
# =============================================================================================
|
||||
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo"
|
||||
SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "alert-metrics-have-scrape-targets"
|
||||
/ "chainsaw-test.yaml")
|
||||
check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
|
||||
"alert-metrics-have-scrape-targets/chainsaw-test.yaml does not declare "
|
||||
"conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
|
||||
"change happens to touch R-OBS-04. ci/touched-requirements.sh maps apps/* to NOTHING and "
|
||||
"platform/monitoring/* to R-OBS-01/R-OBS-02 — so adding an alert under apps/ or flipping a "
|
||||
"chart's serviceMonitor.enabled back to false would be filtered out of exactly the change "
|
||||
"that re-inerts the alerts")
|
||||
L0 = read(ROOT / "ci" / "l0.sh")
|
||||
GATE = read(ROOT / "ci" / "gate.sh")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
|
||||
"purely requirement-scoped run again and the scope annotation above is inert decoration")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
|
||||
and re.search(r"fail\).*BLOCK=1", GATE) is not None,
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
# ======================================================================== report
|
||||
owners_used = {id(o): o for o in claimed.values()}
|
||||
print(f"Alert metric inventory: {n_rules} alerting rule(s) across {len(rule_files)} "
|
||||
f"PrometheusRule file(s) -> {len(refs)} distinct metric name(s)")
|
||||
print(f" claimed by a declared scrape owner: {len(claimed)}/{len(refs) - len(UNIVERSAL & set(refs))}"
|
||||
f" owners exercised: {len(owners_used)}/{len(SCRAPE_OWNERS)}")
|
||||
for o in SCRAPE_OWNERS:
|
||||
ms = sorted(m for m, oo in claimed.items() if oo is o)
|
||||
mark = " " if ms else "· "
|
||||
print(f" {mark}{o['file']:<46} {len(ms):>2} metric(s) {o['owner'][:64]}")
|
||||
print(f"\n DECLARED GAPS (shrink-only registers):")
|
||||
print(f" KNOWN_UNWIRED_METRICS: {len(KNOWN_UNWIRED_METRICS)} — "
|
||||
f"{', '.join(sorted(KNOWN_UNWIRED_METRICS))}")
|
||||
print(f" KNOWN_UNWIRED_JOBS: {len(KNOWN_UNWIRED_JOBS)} inert up{{job=...}} SLI rule(s) — "
|
||||
f"{', '.join(sorted(KNOWN_UNWIRED_JOBS))}")
|
||||
if unowned:
|
||||
print(f"\n UNCLAIMED (inert, undeclared): {', '.join(unowned)}", file=sys.stderr)
|
||||
|
||||
print()
|
||||
if not FAILURES:
|
||||
print(f"check OK: alert-metrics-have-scrape-targets — {CHECKS} assertions (R-OBS-04)")
|
||||
raise SystemExit(0)
|
||||
print(f"CHECK FAIL (alert-metrics-have-scrape-targets): {len(FAILURES)} failed, "
|
||||
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" FAIL {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
@@ -0,0 +1,95 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the alert rules are UNIT-TESTED, by a pinned engine, and the gaps are declared
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# R-OBS-04 ("the platform alerts on its own health") is implemented by PrometheusRule CRs, and
|
||||
# this repo has shipped both ways they fail SILENTLY:
|
||||
# * PERMANENTLY FIRING — the CNPG primaries alert (fixed 2026-09-12), VPNTunnelDown before it;
|
||||
# * STRUCTURALLY INERT — VeleroNoRecentBackup, whose `time() - <absent series>` yields no
|
||||
# series, so a schedule that had NEVER backed up could not raise it. Prod ran with zero
|
||||
# volume backups and nothing alerted, masked by a resources-only schedule completing
|
||||
# cleanly every night (2026-09-12).
|
||||
# An inert rule and a correct rule are indistinguishable while the platform is healthy. Only
|
||||
# `promtool test rules` separates them, by constructing the bad state on purpose, offline.
|
||||
#
|
||||
# The unit tests were then written — and RUN BY NOTHING. promtool was not in the pinned
|
||||
# toolchain, no gate stage evaluated the files, and the two harnesses that grew up independently
|
||||
# each fetched an engine off the network mid-run if promtool was missing (a GitHub release
|
||||
# download in one, a prometheus container image in the other). Coverage that looks present and
|
||||
# is enforced nowhere is the failure mode this test ends.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# WHEN the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
|
||||
# COMMITTED PrometheusRule CRs into the rules file promtool wants — never a copy) is run
|
||||
# over every test file in the repo with the PINNED promtool,
|
||||
# THEN promtool is pinned in BOTH versions.env files in lockstep at the Prometheus version prod
|
||||
# actually runs, baked into the image and installed on the CI runner; the promtool in use
|
||||
# reports exactly that version; no harness acquires an engine at run time; there is
|
||||
# exactly ONE harness; every rule-unit test file PASSES; the harness is proven able to
|
||||
# FAIL by a mutated synthetic test on every run; and every alert rule with NO unit test is
|
||||
# ENUMERATED in a shrink-only register, so the gap is reported rather than hidden.
|
||||
#
|
||||
# MISSING TESTS: RATCHET, NOT SKIP, NOT BLANKET-FAIL. 48 of 55 alert rules have no unit test
|
||||
# today. Failing on all of them would make the gate red until ~48 tests exist (so it would be
|
||||
# disabled, and a disabled gate protects nothing); merely reporting is precisely how the CNPG
|
||||
# tests came to be enforced by nothing. So the gap is DECLARED in check.sh, printed with counts
|
||||
# every run, and CANNOT GROW: a new PrometheusRule, or a new alert in an already-tested file,
|
||||
# FAILS without a test, and a stale register entry FAILS too. A missing test is never an exit-3
|
||||
# skip — "SKIPPED is not green" is preserved for the one thing that is genuinely an environment
|
||||
# gap: promtool being absent.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (standard library + the harness
|
||||
# it delegates to); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
||||
# Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven (named in check.sh's header): that an alert reaches a human (Alertmanager routing
|
||||
# and SMTP are L3 / R-OBS-02), that a threshold is the right one, that a rule file is actually
|
||||
# reconciled into a cluster, or that an author's assertions are meaningful — coverage of one
|
||||
# alert is detected by its name appearing in a covering test file, and only the mutation
|
||||
# self-test proves the harness CAN fail. R-OBS-04 is tagged for the static slice only: the rules
|
||||
# the repo ships behave, under the engine prod runs, as their own unit tests say.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: alert-rules-unit-tested
|
||||
labels:
|
||||
conformance.home.k3s/R-OBS-04: "true"
|
||||
conformance.home.k3s/R-TEST-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-OBS-04, R-TEST-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree
|
||||
# (apps/wave-0..2, platform/cnpg, platform/kubevirt, platform/monitoring), and the register
|
||||
# it maintains is a property of the WHOLE repo, not of any one path.
|
||||
#
|
||||
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map keys
|
||||
# apps/* to NOTHING and platform/monitoring/* to R-OBS-01/02 — not R-OBS-04. So adding a new
|
||||
# untested PrometheusRule under apps/, or editing one under platform/kubevirt/, attributes
|
||||
# no requirement this test carries, and a requirement-scoped selection would filter the
|
||||
# check out of exactly the change that adds the untested alert. The same blind spot that let
|
||||
# `prod-baremetal-not-dev-surface` be filtered out of the run that gated its own breakage
|
||||
# (DECISIONS-LOG 2026-09-12).
|
||||
#
|
||||
# Declaring the scope also makes an honest SKIP loud: lib/run_l1_offline.py turns a rc-3
|
||||
# skip of a scope-declaring check into the runner's rc 3, and ci/l0.sh into the gate's rc 3.
|
||||
# So a gate run on a toolchain WITHOUT promtool reports UNWIRED instead of green — which is
|
||||
# the whole reason this check exists rather than another hand-written ci/l0.sh stage.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every committed alert rule that has unit tests passes them under the pinned promtool that
|
||||
matches prod's Prometheus, exactly one harness runs them, nothing fetches a rule engine at
|
||||
run time, the harness is proven able to fail, and every alert rule without a unit test is
|
||||
enumerated in a register that can only shrink.
|
||||
spec:
|
||||
steps:
|
||||
- name: run-every-alert-rule-unit-test-and-report-the-gaps
|
||||
description: >-
|
||||
Assert the promtool pin is in lockstep and baked, run the single rule-unit harness over
|
||||
every test file in the repo, prove it can fail via a mutated synthetic test, and
|
||||
enumerate every PrometheusRule with no unit tests against the shrink-only register.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,550 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — alert-rules-unit-tested
|
||||
# Serves: R-OBS-04 ("Platform alerts on its own health (nodes, storage, GitOps, certs,
|
||||
# VPN)." — spec docs/10-observability-and-alerting.md)
|
||||
# R-TEST-02 ("Tests are reproducible: deterministic, hermetic, single-command."
|
||||
# — spec docs/14-testing-and-conformance.md)
|
||||
#
|
||||
# WHY THIS EXISTS — AN UNTESTED ALERT IS NOT A MONITORING CONTROL
|
||||
# --------------------------------------------------------------
|
||||
# This repo has shipped BOTH halves of the alerting failure space to prod:
|
||||
# * a PERMANENTLY-FIRING `critical` (the CNPG alert fixed 2026-09-12; VPNTunnelDown before
|
||||
# it), and
|
||||
# * a STRUCTURALLY INERT rule — VeleroNoRecentBackup, whose `time() - <absent series>`
|
||||
# yields no series, so a schedule that had NEVER backed up could not raise it. Prod ran
|
||||
# with no volume backups at all and nothing alerted (2026-09-12).
|
||||
# Neither is visible from the expression, from `kubectl apply`, or from a healthy cluster: an
|
||||
# inert rule and a correct rule are indistinguishable while everything is fine. `promtool test
|
||||
# rules` is the only tool that separates them, because the BAD state can be constructed on
|
||||
# purpose. Unit tests for those rules were then WRITTEN — and evaluated by nothing: promtool
|
||||
# was not in the pinned toolchain, no gate stage ran the files, and each of the two harnesses
|
||||
# that existed acquired an engine off the network mid-run if one was missing. Coverage that
|
||||
# looks present and is never enforced is the failure mode this check ends.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# WHEN the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
|
||||
# COMMITTED PrometheusRule CRs — never a copy) is asked which manifests are under test
|
||||
# and then run over every test file in the repo,
|
||||
# THEN (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
|
||||
# infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
|
||||
# lockstep rule), the Containerfile takes it as a build ARG, installs it from the
|
||||
# pinned prometheus release and records it for provenance, dev.sh passes it, and the
|
||||
# CI runner installs it for the l0 stage;
|
||||
# (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
|
||||
# evidence only if the engine matches the one prod evaluates rules with;
|
||||
# (3) NO harness acquires a rule engine at RUN TIME — neither `curl | tar -xz` of a
|
||||
# GitHub release nor a `podman/docker run` of a prometheus image. Both existed on
|
||||
# 2026-09-12 and both are removed; this is the regression guard;
|
||||
# (4) there is exactly ONE harness. Two were written independently within an hour of
|
||||
# each other; the one nothing runs is the one that rots;
|
||||
# (5) every rule-unit test file in the repo PASSES, run through that harness (which
|
||||
# `promtool check rules` first, so a non-parsing rules file cannot make unit tests
|
||||
# pass vacuously by finding no rules at all);
|
||||
# (6) the harness is NOT VACUOUS: a synthetic PrometheusRule + companion test is run
|
||||
# through the SAME harness twice — intact (must PASS) and with one expectation
|
||||
# mutated (must FAIL) — so a harness that has silently stopped evaluating anything
|
||||
# is caught on every run, not only when someone thinks to try it;
|
||||
# AND (7) the alert rules with NO unit tests are ENUMERATED and RATCHETED: each is declared
|
||||
# below, printed with counts on every run, and the register can only SHRINK.
|
||||
#
|
||||
# WHY A MISSING TEST IS A RATCHET AND NOT AN OUTRIGHT FAIL (the decision, and its reasons)
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# 48 of the repo's 55 alert rules have no unit test today. Three options were on the table:
|
||||
# * FAIL on any missing test — correct in principle, but it makes the merge gate red for
|
||||
# every change until ~48 unit tests exist, so in practice the gate gets disabled, and a
|
||||
# disabled gate protects nothing.
|
||||
# * merely REPORT — the gap prints and nothing holds it. That is exactly how the CNPG unit
|
||||
# tests came to exist and be enforced by nothing; "it is in the output somewhere" is not
|
||||
# enforcement.
|
||||
# * RATCHET (chosen) — the gap is DECLARED IN THE REPO as a reviewable register, printed with
|
||||
# counts on every gate run, and it CANNOT GROW: a new PrometheusRule, or a new alert in a
|
||||
# file that already has tests, FAILS unless it ships a unit test. Removing an entry requires
|
||||
# writing the test, and a STALE entry (file gone, or now covered) also FAILS, so the
|
||||
# register cannot rot into fiction.
|
||||
# "SKIPPED is not green" is honoured: a missing test is NEVER an exit-3 skip here. This check
|
||||
# exits 0 only with the gap fully enumerated, and exits 1 the moment the debt grows, the
|
||||
# register drifts, or a test that exists stops passing.
|
||||
# The one exit-3 path is promtool itself being ABSENT — an environment gap, not a code property
|
||||
# (the same convention as ci/l0.sh's kyverno handling), and this test DECLARES
|
||||
# `conformance.home.k3s/scope: whole-repo`, which makes lib/run_l1_offline.py turn that skip
|
||||
# into the runner's rc 3 and ci/l0.sh into the gate's rc 3: UNWIRED, never green. promtool
|
||||
# PRESENT AT THE WRONG VERSION is a FAILURE, not a skip: that case still prints SUCCESS while
|
||||
# evaluating the rules with an engine prod does not run.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps):
|
||||
# - It does not prove an alert reaches a human. promtool evaluates rules against synthetic
|
||||
# series in-process; Alertmanager routing, SMTP delivery and inhibition are L3
|
||||
# (tests/l3-fault-injection/alerting) and R-OBS-02, and are not claimed here.
|
||||
# - It does not prove the THRESHOLDS are right, only that a rule behaves as its own unit
|
||||
# test says.
|
||||
# - It does not prove a rule file is RECONCILED into a cluster. An alert file no
|
||||
# kustomization includes is a separate "looks applied and isn't" failure, not audited here.
|
||||
# - Coverage of an individual alert is detected by its NAME appearing in a test file that
|
||||
# covers its manifest. A test file that merely mentions a name without asserting anything
|
||||
# counts as covered; only the mutation self-test (6) proves the harness CAN fail, not that
|
||||
# every author wrote a meaningful assertion. Review still owns test quality.
|
||||
# - It does not read the live cluster, so "the pin matches prod's Prometheus" is asserted
|
||||
# against the repo's pin, which was established by a live read recorded in
|
||||
# infra/devcontainer/versions.env (2026-09-12: v3.14.0-distroless).
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
HARNESS="${ALERT_RULE_HARNESS:-${ROOT}/platform/monitoring/tests/run.sh}"
|
||||
|
||||
# Honest SKIP (never green) if the thing under test is not present at all. Because this test
|
||||
# declares scope: whole-repo, the runner turns this into rc 3 for the whole L0 stage — a
|
||||
# PrometheusRule-less repo is a vanished tree, not a pass.
|
||||
if ! grep -rqls --include='*.yaml' -e '^[[:space:]]*kind:[[:space:]]*PrometheusRule[[:space:]]*$' \
|
||||
"${ROOT}/platform" "${ROOT}/apps" 2>/dev/null; then
|
||||
echo "check SKIPPED: no PrometheusRule under platform/ or apps/ — nothing to test" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# promtool resolution. PROMTOOL= overrides (the same override the harness honours); otherwise
|
||||
# PATH. Absent -> every OFFLINE assertion still runs and the check exits 3, so a toolchain gap
|
||||
# is reported as a gap and never as a pass.
|
||||
PROMTOOL_BIN="${PROMTOOL:-$(command -v promtool || true)}"
|
||||
if [ -n "${PROMTOOL_BIN}" ] && [ -x "${PROMTOOL_BIN}" ]; then
|
||||
PROMTOOL_BIN="$(cd "$(dirname "${PROMTOOL_BIN}")" && pwd)/$(basename "${PROMTOOL_BIN}")"
|
||||
else
|
||||
PROMTOOL_BIN=""
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" "${HARNESS}" "${PROMTOOL_BIN}" <<'PY'
|
||||
"""Assert the alert-rule unit tests are pinned, enforced, non-vacuous, and that the alert
|
||||
rules with no tests are enumerated and ratcheted. Standard library + the harness itself."""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
HARNESS = Path(sys.argv[2])
|
||||
PROMTOOL = sys.argv[3]
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
|
||||
#
|
||||
# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. The twelve apps/* entries
|
||||
# are single `<app>-down` liveness rules generated from one ServiceDefinition template;
|
||||
# platform/kubevirt/alerts.yaml (4 rules) is the substantial one and is where the next tests
|
||||
# belong. A file listed here that later gains coverage, or stops carrying a PrometheusRule,
|
||||
# FAILS as stale.
|
||||
# =============================================================================
|
||||
UNTESTED_FILES = {
|
||||
"apps/wave-0/gitea.yaml",
|
||||
"apps/wave-0/jackett.yaml",
|
||||
"apps/wave-0/pihole.yaml",
|
||||
"apps/wave-0/plex.yaml",
|
||||
"apps/wave-0/qbittorrent.yaml",
|
||||
"apps/wave-0/vm-demo.yaml",
|
||||
"apps/wave-1/keycloak.yaml",
|
||||
"apps/wave-1/nextcloud.yaml",
|
||||
"apps/wave-1/radarr.yaml",
|
||||
"apps/wave-1/semaphore.yaml",
|
||||
"apps/wave-1/sonarr.yaml",
|
||||
"apps/wave-2/seerr.yaml",
|
||||
"platform/kubevirt/alerts.yaml",
|
||||
}
|
||||
|
||||
# UNTESTED_ALERTS — individual alerts inside a file that IS covered by a test file, but whose
|
||||
# own rule nothing asserts on. This is the half that stops a new alert being slipped into an
|
||||
# already-"tested" file untested: anything not listed here must appear in a covering test.
|
||||
# The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
|
||||
# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
|
||||
# rules file by the suite convention but has no assertions of its own yet.
|
||||
UNTESTED_ALERTS = {
|
||||
("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
|
||||
("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
|
||||
("platform/cnpg/alerts.yaml", "CNPGNoRecentBaseBackup"),
|
||||
("platform/cnpg/alerts.yaml", "CNPGWALVolumeNotRecycling"),
|
||||
("platform/cnpg/alerts.yaml", "CNPGMetricsCollectorErroring"),
|
||||
("platform/monitoring/alerts.yaml", "NodeExporterDown"),
|
||||
("platform/monitoring/alerts.yaml", "NodeMemoryPressureHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
|
||||
("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
|
||||
("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
|
||||
("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
|
||||
("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
|
||||
("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
|
||||
("platform/monitoring/alerts.yaml", "CertificateNotReady"),
|
||||
("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
|
||||
("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDown"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
|
||||
("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
|
||||
("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
|
||||
("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
|
||||
("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
|
||||
("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
|
||||
("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
|
||||
("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
|
||||
}
|
||||
|
||||
# --------------------------------------------------------------- (1) the pins, in lockstep
|
||||
def env_pin(rel: str, key: str) -> str | None:
|
||||
f = ROOT / rel
|
||||
if not f.is_file():
|
||||
return None
|
||||
m = re.search(rf'^{re.escape(key)}=["\']?([^"\'\s#]+)', f.read_text(), re.M)
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
DEV_PIN = env_pin("infra/devcontainer/versions.env", "PROMTOOL_VERSION")
|
||||
CI_PIN = env_pin("ci/versions.env", "PROMTOOL_VERSION")
|
||||
check(DEV_PIN is not None,
|
||||
"infra/devcontainer/versions.env declares no PROMTOOL_VERSION — promtool is not part of the "
|
||||
"baked toolchain, so the alert unit tests are evaluated by whatever the host happens to "
|
||||
"have, or by nothing at all (which is where this repo was on 2026-09-12)")
|
||||
check(CI_PIN is not None,
|
||||
"ci/versions.env declares no PROMTOOL_VERSION — the gate's own version file does not pin the "
|
||||
"rule engine, so CI and the devcontainer can silently diverge")
|
||||
check(DEV_PIN is None or DEV_PIN == CI_PIN,
|
||||
f"PROMTOOL_VERSION is NOT in lockstep: infra/devcontainer/versions.env={DEV_PIN!r} vs "
|
||||
f"ci/versions.env={CI_PIN!r}. The repo's rule is that the conformance-tool versions in the "
|
||||
f"two files stay identical, so a laptop and CI run the same engine")
|
||||
|
||||
CF = ROOT / "infra/devcontainer/Containerfile"
|
||||
cf_text = CF.read_text() if CF.is_file() else ""
|
||||
check(re.search(r"^ARG PROMTOOL_VERSION\s*$", cf_text, re.M),
|
||||
"infra/devcontainer/Containerfile has no `ARG PROMTOOL_VERSION` — the pin is declared but "
|
||||
"the image build does not consume it")
|
||||
check("prometheus/prometheus/releases/download" in cf_text and "promtool" in cf_text,
|
||||
"infra/devcontainer/Containerfile does not install promtool from the pinned prometheus "
|
||||
"release tarball (promtool ships inside it)")
|
||||
check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
|
||||
"infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
|
||||
"that provenance line is how an operator tells a stale image from a current one")
|
||||
|
||||
DEVSH = ROOT / "infra/devcontainer/dev.sh"
|
||||
check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
|
||||
"infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
|
||||
"empty and the build would fetch a bogus URL")
|
||||
|
||||
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
|
||||
if INSTALL.is_file():
|
||||
itext = INSTALL.read_text()
|
||||
check("PROMTOOL_VERSION" in itext
|
||||
and re.search(r"^\s*l0\)[^\n]*install_promtool", itext, re.M),
|
||||
".gitea/workflows/scripts/install-tools.sh does not install promtool for the l0 stage — "
|
||||
"on the CI runner this check would exit 3 (UNWIRED) forever and the alert unit tests "
|
||||
"would be enforced nowhere, which is the exact gap it exists to close")
|
||||
|
||||
# --------------------------------------------- (3)(4) one harness, and it fetches no engine
|
||||
check(HARNESS.is_file() and os.access(HARNESS, os.X_OK),
|
||||
f"the rule-unit harness is missing or not executable at {HARNESS.relative_to(ROOT)} — "
|
||||
f"nothing would run the alert unit tests")
|
||||
|
||||
# Any script that invokes `promtool test rules` is a rule-unit harness. There must be exactly
|
||||
# one: two harnesses were written independently on 2026-09-12 (a cnpg-only script and this one),
|
||||
# and the one nothing runs is the one that rots out of sync with the rules.
|
||||
harnesses: list[str] = []
|
||||
fetchers: list[str] = []
|
||||
FETCH_RE = re.compile(
|
||||
r"(curl|wget)[^\n]*prometheus/prometheus/releases" # GitHub release download
|
||||
r"|(podman|docker)\s+run[^\n]*prometheus", # container-image pull
|
||||
re.I)
|
||||
# Two exclusions, both the repo's existing "reference, not declaration" rule (see
|
||||
# ci/touched-requirements.sh changed_diff() rules 2-3):
|
||||
# * THIS CHECK'S OWN DIRECTORY. It names every forbidden pattern in order to forbid it, so
|
||||
# scanning itself would make both guards permanently red — the self-referential trap.
|
||||
# * THE PINNED TOOLCHAIN INSTALLERS. Downloading the pinned promtool is their JOB, at a
|
||||
# reviewed seam (an image build, or the runner's tool-install step), from a version in
|
||||
# versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
|
||||
# version, the network and the machine are decided silently at test time, and a missing
|
||||
# toolchain becomes a green run instead of an exit 3.
|
||||
SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
|
||||
INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
|
||||
for p in sorted(ROOT.rglob("*.sh")):
|
||||
rel = str(p.relative_to(ROOT))
|
||||
if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
|
||||
continue
|
||||
try:
|
||||
text = p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
continue
|
||||
body = "\n".join(l for l in text.splitlines() if not l.lstrip().startswith("#"))
|
||||
if "promtool" in body and re.search(r"test\s+rules", body):
|
||||
harnesses.append(rel)
|
||||
if FETCH_RE.search(body) and not rel.startswith(INSTALLERS):
|
||||
fetchers.append(rel)
|
||||
|
||||
check(harnesses == [str(HARNESS.relative_to(ROOT))],
|
||||
f"expected exactly ONE rule-unit harness ({HARNESS.relative_to(ROOT)}) but found "
|
||||
f"{harnesses}. A second harness means two behaviours and two things to keep in step; the "
|
||||
f"one the gate does not run is the one that rots. Fold it in instead")
|
||||
check(not fetchers,
|
||||
f"{fetchers} acquire a rule ENGINE at run time (a prometheus release download or a "
|
||||
f"prometheus container image). promtool must come from the pinned toolchain "
|
||||
f"(PROMTOOL_VERSION): a run-time fetch decides the version, the network and the machine "
|
||||
f"silently at test time, makes the tests non-hermetic (R-TEST-02), and quietly turns 'the "
|
||||
f"toolchain has no promtool' into a green run — which is how it stayed missing")
|
||||
|
||||
# ------------------------------------------------------------------ discover the alert files
|
||||
# Alert files: tracked YAML carrying a PrometheusRule. conformance/ is excluded deliberately —
|
||||
# the PrometheusRules under tests/l1-rendering/*/golden/ are EXPECTED-RENDER fixtures for the
|
||||
# render checks, not rules any cluster evaluates. Everything a cluster actually gets lives under
|
||||
# apps/, platform/, platform-config/ or clusters/.
|
||||
def tracked_yaml() -> list[Path]:
|
||||
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
|
||||
capture_output=True, text=True)
|
||||
if out.returncode == 0 and out.stdout:
|
||||
return [ROOT / p for p in out.stdout.split("\0") if p]
|
||||
# No usable index (the gate container mounts a worktree whose .git points outside the
|
||||
# mount). Fall back to a filesystem walk rather than auditing nothing.
|
||||
return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
|
||||
|
||||
|
||||
KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
|
||||
ALERT_RE = re.compile(r"^\s*-\s*alert:\s*(\S+)", re.M)
|
||||
|
||||
alert_files: dict[str, list[str]] = {}
|
||||
for p in tracked_yaml():
|
||||
rel = str(p.relative_to(ROOT))
|
||||
if rel.startswith("conformance/") or not p.is_file():
|
||||
continue
|
||||
try:
|
||||
text = p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
continue
|
||||
if KIND_RE.search(text):
|
||||
alert_files[rel] = ALERT_RE.findall(text)
|
||||
|
||||
check(len(alert_files) > 0,
|
||||
"no PrometheusRule found outside conformance/ — discovery or the repo layout changed, and "
|
||||
"this check would silently audit nothing")
|
||||
|
||||
# ------------------- ask the HARNESS which manifests are under test (never re-derive it here)
|
||||
# Two implementations of "which rules does this test cover" would eventually disagree, and the
|
||||
# disagreement shows up as a file the gate believes is tested and the harness never touches.
|
||||
listing = subprocess.run(["bash", str(HARNESS), "--list"], capture_output=True, text=True,
|
||||
cwd=str(ROOT))
|
||||
check(listing.returncode == 0,
|
||||
f"`{HARNESS.relative_to(ROOT)} --list` failed (rc={listing.returncode}): "
|
||||
f"{(listing.stderr or listing.stdout).strip()[-600:]}")
|
||||
covering: dict[str, list[str]] = defaultdict(list) # manifest -> [test files]
|
||||
for line in listing.stdout.splitlines():
|
||||
if "\t" in line:
|
||||
tfile, manifest = line.split("\t", 1)
|
||||
covering[manifest.strip()].append(tfile.strip())
|
||||
|
||||
for manifest in sorted(covering):
|
||||
check(manifest in alert_files,
|
||||
f"the harness reports {manifest} as under test, but it is not one of the "
|
||||
f"PrometheusRule files found in the repo — the mapping and the tree disagree")
|
||||
|
||||
# --------------------------------------------------------- (2) the engine is the pinned one
|
||||
version_line = ""
|
||||
if PROMTOOL:
|
||||
vp = subprocess.run([PROMTOOL, "--version"], capture_output=True, text=True)
|
||||
out = (vp.stdout + vp.stderr).strip()
|
||||
version_line = out.splitlines()[0] if out else ""
|
||||
m = re.search(r"version\s+(\d+\.\d+\.\d+)", version_line)
|
||||
found = m.group(1) if m else None
|
||||
check(found is not None,
|
||||
f"could not parse a version from `promtool --version`: {version_line!r}")
|
||||
check(found is None or DEV_PIN is None or found == DEV_PIN,
|
||||
f"promtool reports {found}, but the toolchain pins PROMTOOL_VERSION={DEV_PIN} (the "
|
||||
f"Prometheus version prod runs). promtool IS the rule engine: a unit test green under "
|
||||
f"one version and red under another is worse than no test. Run the gate in the pinned "
|
||||
f"devcontainer (rebuild it: ./infra/devcontainer/dev.sh build), or set PROMTOOL= to a "
|
||||
f"v{DEV_PIN} binary. A FAILURE, not a skip, because the wrong engine still prints "
|
||||
f"SUCCESS")
|
||||
|
||||
# ------------------------------------------------- (5) every rule-unit test file must pass
|
||||
if PROMTOOL:
|
||||
env = dict(os.environ, PROMTOOL=PROMTOOL)
|
||||
run = subprocess.run(["bash", str(HARNESS)], capture_output=True, text=True,
|
||||
cwd=str(ROOT), env=env)
|
||||
tailed = "\n ".join((run.stdout + run.stderr).strip().splitlines()[-40:])
|
||||
check(run.returncode == 0,
|
||||
f"`{HARNESS.relative_to(ROOT)}` FAILED (rc={run.returncode}) — the committed alert "
|
||||
f"rules do not behave as their own unit tests say:\n {tailed}")
|
||||
|
||||
# ------------------------------------------- (6) the harness is proven non-vacuous
|
||||
# A synthetic PrometheusRule + companion test through the SAME harness, outside the repo so
|
||||
# discovery never picks it up. Intact must PASS; a mutated expectation must FAIL. Without
|
||||
# this, a harness that stopped evaluating anything (an empty extraction, a rule file written
|
||||
# under the wrong name) would report green forever — which is the disease being treated.
|
||||
work = tempfile.mkdtemp(prefix="alert-rules-selftest.")
|
||||
try:
|
||||
(Path(work) / "synth.yaml").write_text("""apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: selftest
|
||||
spec:
|
||||
groups:
|
||||
- name: selftest
|
||||
rules:
|
||||
- alert: SelfTestCanary
|
||||
expr: selftest_metric > 0
|
||||
for: 5m
|
||||
annotations:
|
||||
runbook_url: https://${BASE_DOMAIN}/runbooks/selftest
|
||||
""")
|
||||
test_yaml = """rule_files:
|
||||
- generated.rules.yaml
|
||||
evaluation_interval: 1m
|
||||
tests:
|
||||
- interval: 1m
|
||||
name: canary fires only after `for` elapses
|
||||
input_series:
|
||||
- series: 'selftest_metric'
|
||||
values: '1x10'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
|
||||
eval_time: EVAL_TIME
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
|
||||
value: 1
|
||||
"""
|
||||
synth_test = Path(work) / "synth.promtool-test.yaml"
|
||||
synth_test.write_text(test_yaml.replace("EVAL_TIME", "6m"))
|
||||
good = subprocess.run(["bash", str(HARNESS), str(synth_test)],
|
||||
capture_output=True, text=True, cwd=str(ROOT), env=env)
|
||||
check(good.returncode == 0,
|
||||
"self-test: the INTACT synthetic unit test did not pass, so the harness itself is "
|
||||
"broken and every green result above is meaningless:\n "
|
||||
+ "\n ".join((good.stdout + good.stderr).strip().splitlines()[-20:]))
|
||||
# Mutation: assert firing at 2m, which `for: 5m` forbids.
|
||||
synth_test.write_text(test_yaml.replace("EVAL_TIME", "2m"))
|
||||
bad = subprocess.run(["bash", str(HARNESS), str(synth_test)],
|
||||
capture_output=True, text=True, cwd=str(ROOT), env=env)
|
||||
check(bad.returncode != 0,
|
||||
"self-test: a MUTATED expectation (firing before `for:` elapsed) still PASSED — "
|
||||
"the harness is not actually evaluating anything, so every green result above is "
|
||||
"vacuous")
|
||||
finally:
|
||||
shutil.rmtree(work, ignore_errors=True)
|
||||
|
||||
# --------------------------------------------------------- (7) the gap inventory + ratchet
|
||||
INVENTORY: list[str] = []
|
||||
GAP_FILES: list[str] = []
|
||||
alerts_total = covered_total = 0
|
||||
|
||||
for rel in sorted(alert_files):
|
||||
names = alert_files[rel]
|
||||
alerts_total += len(names)
|
||||
check(len(names) > 0,
|
||||
f"{rel} declares a PrometheusRule but no `alert:` rule was parsed out of it — the "
|
||||
f"parser or the file shape changed, and any unit test over it would pass vacuously")
|
||||
tests = covering.get(rel, [])
|
||||
if not tests:
|
||||
GAP_FILES.append(rel)
|
||||
INVENTORY.append(f" {rel:<46} alerts={len(names):<3} tested=0/{len(names):<3} NO TEST FILE")
|
||||
check(rel in UNTESTED_FILES,
|
||||
f"{rel} carries {len(names)} alert rule(s) and NO test file covers it. A new "
|
||||
f"PrometheusRule must ship promtool unit tests: this repo has shipped both a "
|
||||
f"permanently-firing critical and a structurally inert alert, and neither was "
|
||||
f"visible without them. Add `{rel[:-5]}.promtool-test.yaml` (companion convention) "
|
||||
f"or a <component>/tests/*.test.yaml suite. If this is deliberate debt it must be "
|
||||
f"DECLARED in UNTESTED_FILES in this check — the register is what keeps the gap "
|
||||
f"visible instead of silent")
|
||||
continue
|
||||
|
||||
check(rel not in UNTESTED_FILES,
|
||||
f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
|
||||
f"shrink-only: delete the line so the next missing test is caught")
|
||||
text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
|
||||
covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
|
||||
covered_total += len(covered)
|
||||
for n in names:
|
||||
if n in covered:
|
||||
check((rel, n) not in UNTESTED_ALERTS,
|
||||
f"{rel}: alert {n} is listed in UNTESTED_ALERTS but IS asserted on now — "
|
||||
f"remove the stale entry (the register is shrink-only)")
|
||||
else:
|
||||
check((rel, n) in UNTESTED_ALERTS,
|
||||
f"{rel}: alert {n} has NO unit test in {tests}. This file is already covered, "
|
||||
f"so a new alert in it must be tested too — otherwise a 'tested' alert file "
|
||||
f"quietly grows untested rules")
|
||||
INVENTORY.append(f" {rel:<46} alerts={len(names):<3} tested={len(covered)}/{len(names):<3} "
|
||||
f"{','.join(tests)}")
|
||||
|
||||
# stale register entries
|
||||
for rel in sorted(UNTESTED_FILES - set(alert_files)):
|
||||
check(False, f"UNTESTED_FILES lists {rel}, which carries no PrometheusRule in this repo any "
|
||||
f"more — a stale exemption hides the next real gap; delete the line")
|
||||
for rel in sorted({f for f, _ in UNTESTED_ALERTS} - set(alert_files)):
|
||||
check(False, f"UNTESTED_ALERTS lists alerts in {rel}, which carries no PrometheusRule in this "
|
||||
f"repo any more — delete those lines")
|
||||
|
||||
# --------------------------------------------------------------------------------- report
|
||||
print("Alert-rule unit-test inventory (PrometheusRule -> covering test file(s)):")
|
||||
for line in INVENTORY:
|
||||
print(line)
|
||||
print(f"\nharness: {HARNESS.relative_to(ROOT)} promtool: {version_line or 'NOT FOUND'}")
|
||||
print(f"pinned: PROMTOOL_VERSION={DEV_PIN} (devcontainer) / {CI_PIN} (ci) — prod runs Prometheus "
|
||||
f"v{DEV_PIN}")
|
||||
print(f"alert files: {len(alert_files)} covered by a test file: "
|
||||
f"{len(alert_files) - len(GAP_FILES)} with NO test file: {len(GAP_FILES)}")
|
||||
print(f"alert rules: {alerts_total} unit-tested: {covered_total} "
|
||||
f"NOT unit-tested: {alerts_total - covered_total}")
|
||||
|
||||
if GAP_FILES:
|
||||
print("\nDECLARED GAPS — alert files nothing tests at all (shrink-only register; each is a")
|
||||
print("rule nothing has ever evaluated, NOT a rule known to work):")
|
||||
for g in GAP_FILES:
|
||||
print(f" - {g} ({len(alert_files[g])} alert(s))")
|
||||
if UNTESTED_ALERTS:
|
||||
print("\nDECLARED GAPS — individual alerts inside a covered file that nothing asserts on:")
|
||||
for f, n in sorted(UNTESTED_ALERTS):
|
||||
print(f" - {f} {n}")
|
||||
|
||||
print()
|
||||
if not PROMTOOL and not FAILURES:
|
||||
print("check SKIPPED (exit 3, NOT a pass): promtool is not on PATH, so NO alert rule was "
|
||||
"parsed and NO unit test was evaluated.", file=sys.stderr)
|
||||
print(f" promtool is pinned at v{DEV_PIN} in infra/devcontainer/versions.env + "
|
||||
f"ci/versions.env.", file=sys.stderr)
|
||||
print(" Run the gate in the pinned toolbox (./infra/devcontainer/dev.sh build, then "
|
||||
"make -C ci gate), or set PROMTOOL=/path/to/promtool.", file=sys.stderr)
|
||||
print(f" The {CHECKS} offline assertions above (pins, lockstep, one harness, no run-time "
|
||||
f"engine fetch, the gap register) DID run and passed.", file=sys.stderr)
|
||||
print(" This test declares scope: whole-repo, so the runner reports rc 3 — UNWIRED, never "
|
||||
"green.", file=sys.stderr)
|
||||
raise SystemExit(3)
|
||||
|
||||
if not FAILURES:
|
||||
print(f"check OK: alert-rules-unit-tested — {CHECKS} assertions "
|
||||
f"({covered_total}/{alerts_total} alert rules unit-tested, "
|
||||
f"{len(GAP_FILES)} files with no test file, all declared) (R-OBS-04, R-TEST-02)")
|
||||
raise SystemExit(0)
|
||||
print(f"CHECK FAIL (alert-rules-unit-tested): {len(FAILURES)} failed, "
|
||||
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" FAIL {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
@@ -0,0 +1,89 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — every alert's runbook_url resolves to a runbook that is actually served
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# Every platform-health alert rule in this repo carries
|
||||
# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations across four files, over 34
|
||||
# distinct names. For the whole of phases 4-6 there was NO runbooks/ directory and NO HTTPRoute serving
|
||||
# that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404.
|
||||
# So an alert reaching a human handed them a dead link at the one moment documentation is read, while the
|
||||
# annotation made the problem look solved — the same green-surface-over-absent-mechanism class as the
|
||||
# import Job that reported success while storing placeholder secrets, and the guard whose glob silently
|
||||
# stopped matching.
|
||||
#
|
||||
# The link is served now (runbooks/, clusters/base/runbooks.yaml). This test is the guard that keeps it
|
||||
# true, and it asserts BOTH halves of the claim because the regression is silent in both directions: an
|
||||
# alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its
|
||||
# configMapGenerator entry is in git, reviewed, merged — and still 404s in the cluster. "The file exists"
|
||||
# and "the URL resolves" are different claims; only the second one helps an operator.
|
||||
#
|
||||
# GIVEN every `runbook_url` annotation on every PrometheusRule in the repo,
|
||||
# WHEN each is parsed out of git and resolved against runbooks/,
|
||||
# THEN (1) each URL has EXACTLY the shape the server resolves — https://${BASE_DOMAIN}/runbooks/<name>,
|
||||
# the ${BASE_DOMAIN} token (never a literal hostname), no `.md`, no trailing slash, no query;
|
||||
# (2) every referenced <name> has a runbooks/<name>.md in git;
|
||||
# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
|
||||
# dev-container kubectl invocation with --context k3s-prod — so a stub cannot satisfy (2) and
|
||||
# make the link look fixed while still helping nobody;
|
||||
# (4) every runbook is PROJECTED INTO THE SERVER by runbooks/kustomization.yaml's
|
||||
# configMapGenerator under the extensionless key the URL asks for;
|
||||
# (5) route, listener and server agree with the annotations (PathPrefix == the annotations' prefix,
|
||||
# a sectionName that platform/gateway/gateway.yaml actually defines, the namespace opted in to
|
||||
# the Gateway, and an nginx location for that same prefix with a text/plain default_type);
|
||||
# (6) the Flux tier is wired into clusters/base AND declares NO postBuild.substituteFrom —
|
||||
# substitution would run envsubst over the runbook prose and silently empty every `${...}`
|
||||
# shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium
|
||||
# k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check
|
||||
# scans files reachable through `resources:`, not configMapGenerator inputs;
|
||||
# AND (7) the check is LOAD-BEARING: this Test DECLARES
|
||||
# `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's
|
||||
# `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs
|
||||
# whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block
|
||||
# when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only);
|
||||
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME
|
||||
# script.
|
||||
#
|
||||
# NOT proven (named in check.sh's header): this does not FETCH the URL. That it returns HTTP 200 with
|
||||
# real content was proven by an in-cluster probe recorded in the commit message; a live probe needs a
|
||||
# cluster, which L0 by definition does not have. What is asserted here is every static precondition for
|
||||
# that 200. It also does not judge whether a runbook is correct or current — (3) is a floor, not a
|
||||
# review — and it does not prove Alertmanager delivers the annotation to a human (that is the L3
|
||||
# `alerting` test). R-OBS-04 is tagged for the narrow slice only: the platform's own health alerts carry
|
||||
# runbook links that resolve to real, served procedure rather than to a 404.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: alert-runbooks-resolvable
|
||||
labels:
|
||||
conformance.home.k3s/R-OBS-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-OBS-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs
|
||||
# `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this
|
||||
# annotation into the selection regardless of which requirements the change touched. That union is
|
||||
# load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" —
|
||||
# NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most
|
||||
# likely breaking change (adding an alert with a runbook_url, or adding a runbook without its
|
||||
# configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT
|
||||
# of exactly the change that introduces the bug. Declaring the scope replaces the hand-written
|
||||
# per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add).
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every runbook_url on every alert rule resolves to a runbook that exists in git, clears a quality
|
||||
floor, and is actually projected into the server that answers that path — so an alert can never
|
||||
hand an operator a dead link.
|
||||
spec:
|
||||
steps:
|
||||
- name: audit-every-alert-runbook-link
|
||||
description: >-
|
||||
Parse every runbook_url out of every PrometheusRule and assert the runbook exists, is real, is
|
||||
served at that exact path, and that the serving tier is wired and unsubstituted.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,410 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — alert-runbooks-resolvable
|
||||
# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
|
||||
# VPN)." — spec docs/10-observability-and-alerting.md)
|
||||
#
|
||||
# WHY THIS EXISTS — A DEAD LINK IN AN ALERT IS WORSE THAN NO LINK
|
||||
# --------------------------------------------------------------
|
||||
# Every platform-health alert rule in this repo carries
|
||||
# runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
|
||||
# 42 such annotations across four files, over 34 distinct names. For the whole of phases 4-6
|
||||
# there was NO runbooks/ directory and NO HTTPRoute serving that path: a live in-cluster probe
|
||||
# of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. So an alert reaching a
|
||||
# human at 3am handed them a link that fails, at the one moment the documentation is read. The
|
||||
# annotation LOOKED like the problem was solved, which is the specific failure this repo keeps
|
||||
# finding (an import Job that reports success while storing placeholder secrets; a guard whose
|
||||
# glob silently stopped matching) — a green-looking surface over an absent mechanism.
|
||||
#
|
||||
# The link is now served (runbooks/, clusters/base/runbooks.yaml). This check is the guard that
|
||||
# keeps it true, because the regression is SILENT IN BOTH DIRECTIONS and nothing else catches it:
|
||||
# * add an alert with a new runbook_url and forget the runbook -> that alert links to a 404,
|
||||
# and no test, lint, render or policy notices;
|
||||
# * write runbooks/<name>.md but forget to add it to the ConfigMap generator -> the file is in
|
||||
# git, reviewable, apparently done, and the URL still 404s in the cluster. "The file exists"
|
||||
# and "the URL resolves" are different claims and only the second one helps an operator.
|
||||
# Both halves are asserted below, which is why this check reaches past the markdown into the
|
||||
# kustomization, the HTTPRoute, the nginx config and the Flux tier.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# WHEN every `runbook_url` annotation on every PrometheusRule is parsed out of git,
|
||||
# THEN (1) each URL has EXACTLY the shape the server resolves —
|
||||
# https://${BASE_DOMAIN}/runbooks/<name>, `${BASE_DOMAIN}` as the token (never a literal
|
||||
# hostname: a literal is how prod was authored to serve dev's hostnames, see the
|
||||
# ServiceDefinition schema's note), no `.md` suffix, no trailing slash, no query;
|
||||
# (2) every referenced <name> has a runbooks/<name>.md in git;
|
||||
# (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
|
||||
# dev-container kubectl invocation — so a one-line stub cannot satisfy (2) and make the
|
||||
# link look fixed while still helping nobody (the failure mode this check is most likely
|
||||
# to be defeated by is a placeholder file, not a missing one);
|
||||
# (4) every runbook markdown file is actually PROJECTED INTO THE SERVER by
|
||||
# runbooks/kustomization.yaml's configMapGenerator, under the EXTENSIONLESS key the URL
|
||||
# asks for — the "exists in git but 404s in the cluster" half;
|
||||
# (5) the route and the server agree with the annotations: the HTTPRoute's PathPrefix is the
|
||||
# prefix the URLs use, it attaches to a Gateway listener that actually exists, its
|
||||
# namespace is opted in to that Gateway, and the nginx config answers that same prefix;
|
||||
# (6) the Flux tier that reconciles all of it is wired into clusters/base AND declares NO
|
||||
# postBuild.substituteFrom — substitution here would run envsubst over the runbook prose
|
||||
# and silently empty every `${...}` shell example in it (it fails open: DECISIONS-LOG
|
||||
# 2026-09-01, the emptied Cilium k8sServiceHost);
|
||||
# AND (7) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and ci/gate.sh sets the
|
||||
# merge block when the l0 stage fails.
|
||||
#
|
||||
# WHY (7) IS AN UNCONDITIONAL ci/l0.sh STAGE and not left to the requirement-scoped run: Stage 3b
|
||||
# scopes `conformance/run.sh --layer l0` to the TOUCHED requirements (R-TEST-03), and
|
||||
# ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and
|
||||
# maps platform/kubevirt/* and runbooks/* to nothing at all. So the single most likely breaking
|
||||
# change (adding an alert with a runbook_url to platform/monitoring/alerts.yaml, or adding a
|
||||
# runbook and forgetting the generator entry) attributes a requirement set that would FILTER THIS
|
||||
# CHECK OUT. Declaring `conformance.home.k3s/scope: whole-repo` makes ci/l0.sh Stage 3b union it into
|
||||
# EVERY gate invocation instead; a SKIP of a scope-declaring check is UNWIRED, never green.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
|
||||
# - It does not fetch the URL. That it returns HTTP 200 with real content was proven by an
|
||||
# in-cluster probe (recorded in the commit message and DECISIONS-LOG); a live probe needs a
|
||||
# cluster, which L0 by definition does not have. What is asserted here is every static
|
||||
# precondition for that 200: the file, its projection, the key name, the route, the prefix.
|
||||
# - It does not judge whether a runbook is CORRECT or current. (3) is a floor, not a review: it
|
||||
# can tell a stub from a runbook, not a good runbook from a stale one. Nothing automatable can.
|
||||
# - It does not prove Alertmanager actually delivers the annotation to a human. That is the L3
|
||||
# `alerting` test.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and is not consulted.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green) if there are no alert rules at all — nothing to audit yet.
|
||||
if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" >/dev/null 2>&1; then
|
||||
echo "check SKIPPED: no PrometheusRule found under platform/ — no alert rules to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit: every alert runbook_url resolves to a runbook that is actually served.
|
||||
|
||||
Pure standard library, no PyYAML — the same constraint and the same style as the neighbouring
|
||||
L0 checks (lib/coverage.py, lib/status.py). Every parse contract is documented at its parser;
|
||||
all of them are line-constrained shapes in hand-authored manifests.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
ADVISORIES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def read(p: Path) -> str:
|
||||
try:
|
||||
return p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
|
||||
|
||||
# =============================================================== (1) parse the annotations
|
||||
# Parse contract: `runbook_url:` is an annotation on a Prometheus alerting rule, authored one per
|
||||
# line as `runbook_url: "<url>"` (single or double quoted, or bare). Only files that declare
|
||||
# `kind: PrometheusRule` are scanned, so a runbook_url quoted inside this check or inside prose
|
||||
# elsewhere in the repo is not mistaken for an alert annotation. Commented-out lines are skipped:
|
||||
# platform/monitoring/alerts.yaml carries a `# runbook_url points at a docs base` header comment.
|
||||
SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
|
||||
RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
|
||||
|
||||
rule_files = sorted(
|
||||
p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
|
||||
)
|
||||
check(len(rule_files) > 0,
|
||||
"no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
|
||||
"this check would silently audit nothing")
|
||||
|
||||
# url -> [ "<relpath>:<lineno>", ... ]
|
||||
refs: dict[str, list[str]] = {}
|
||||
for f in rule_files:
|
||||
rel = f.relative_to(ROOT)
|
||||
for lineno, line in enumerate(read(f).splitlines(), 1):
|
||||
if line.lstrip().startswith("#"):
|
||||
continue
|
||||
m = RUNBOOK_URL_RE.match(line)
|
||||
if m:
|
||||
refs.setdefault(m.group(1), []).append(f"{rel}:{lineno}")
|
||||
|
||||
n_annotations = sum(len(v) for v in refs.values())
|
||||
check(n_annotations > 0,
|
||||
"no runbook_url annotation parsed out of any PrometheusRule — the parser contract broke; "
|
||||
"this check would pass vacuously while every alert link rots")
|
||||
|
||||
# ---- URL SHAPE. The server resolves exactly one shape. Anything else is a link that 404s even
|
||||
# though a file with a plausible name exists, so shape is a failure, not a style note.
|
||||
URL_RE = re.compile(r"^https://\$\{BASE_DOMAIN\}/(runbooks)/([a-z0-9][a-z0-9-]*)$")
|
||||
names: dict[str, list[str]] = {} # runbook name -> annotation sites
|
||||
prefixes: set[str] = set()
|
||||
for url, sites in sorted(refs.items()):
|
||||
m = URL_RE.match(url)
|
||||
check(m is not None,
|
||||
f"runbook_url '{url}' ({', '.join(sites)}) is not the shape this platform serves. "
|
||||
f"Required: https://${{BASE_DOMAIN}}/runbooks/<name> — the ${{BASE_DOMAIN}} TOKEN (never a "
|
||||
f"literal hostname: a literal is how prod got authored to serve dev's public names), a "
|
||||
f"lowercase-hyphen <name>, and NO '.md' suffix, trailing slash or query. The runbook "
|
||||
f"server maps the URL path straight onto an extensionless ConfigMap key, so any other "
|
||||
f"shape is a 404 no matter which files exist")
|
||||
if m:
|
||||
prefixes.add(m.group(1))
|
||||
names.setdefault(m.group(2), []).extend(sites)
|
||||
|
||||
check(len(prefixes) <= 1,
|
||||
f"alert annotations use more than one runbook path prefix ({sorted(prefixes)}); the single "
|
||||
f"HTTPRoute + nginx location can only serve one, so the others are dead links")
|
||||
PREFIX = (sorted(prefixes) or ["runbooks"])[0]
|
||||
|
||||
# =================================================== (2)(3) the runbooks exist and are real
|
||||
RUNBOOK_DIR = ROOT / PREFIX
|
||||
check(RUNBOOK_DIR.is_dir(),
|
||||
f"{PREFIX}/ is not a directory in this repo, so every one of the {n_annotations} "
|
||||
f"runbook_url annotations is a dead link")
|
||||
|
||||
# The quality floor. (2) alone is satisfiable by `touch runbooks/node-down.md`, which makes the
|
||||
# link resolve and still abandons the operator — a worse outcome than the 404, because the 404 at
|
||||
# least told the truth. These four sections are the operator's path through an incident: what the
|
||||
# alert means HERE, what to type first, what is probably wrong, and when to stop. `dev.sh` +
|
||||
# `--context k3s-prod` are required because kubectl does not work on the workstation at all
|
||||
# (the toolchain lives only in the pinned dev container) and because dev.sh defaults its
|
||||
# current-context to k3s-DEV — a runbook that omits the context teaches the reader to answer
|
||||
# questions about the wrong cluster during an incident on this one.
|
||||
REQUIRED_SECTIONS = (
|
||||
"## What this means here",
|
||||
"## First commands",
|
||||
"## Likely causes, most likely first",
|
||||
"## When to escalate or stop",
|
||||
)
|
||||
MIN_BYTES = 1500
|
||||
|
||||
for name, sites in sorted(names.items()):
|
||||
md = RUNBOOK_DIR / f"{name}.md"
|
||||
if not md.is_file():
|
||||
check(False,
|
||||
f"runbook '{name}' does not exist: {md.relative_to(ROOT)} is missing, but "
|
||||
f"{len(sites)} alert annotation(s) link to it ({', '.join(sites[:3])}"
|
||||
f"{', ...' if len(sites) > 3 else ''}). An alert firing now hands the operator a 404. "
|
||||
f"Write {PREFIX}/{name}.md (see {PREFIX}/README.md for the required sections)")
|
||||
continue
|
||||
body = read(md)
|
||||
check(len(body) >= MIN_BYTES,
|
||||
f"{md.relative_to(ROOT)} is {len(body)} bytes, under the {MIN_BYTES}-byte floor — a stub "
|
||||
f"that makes the link resolve while still helping nobody is worse than the 404 it "
|
||||
f"replaced, because it looks fixed")
|
||||
for sec in REQUIRED_SECTIONS:
|
||||
check(sec in body,
|
||||
f"{md.relative_to(ROOT)} has no '{sec}' section — the four operator sections are what "
|
||||
f"make this a runbook rather than a restatement of the alert name")
|
||||
check("dev.sh" in body,
|
||||
f"{md.relative_to(ROOT)} never mentions dev.sh — kubectl does not exist on the "
|
||||
f"workstation; every command in a runbook here must be runnable as "
|
||||
f"`infra/devcontainer/dev.sh run bash -c '...'` and the runbook must say so")
|
||||
check("--context k3s-prod" in body,
|
||||
f"{md.relative_to(ROOT)} never passes --context k3s-prod — dev.sh merges both clusters "
|
||||
f"and picks a current-context PER INVOCATION — a default that has already changed once "
|
||||
f"(k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now). A context-less command "
|
||||
f"silently answers about whichever cluster the default happens to be, in the middle of an "
|
||||
f"incident")
|
||||
|
||||
# ADVISORY, not a failure: a runbook nothing links to yet. Writing the runbook BEFORE the alert is
|
||||
# the right order, and the repo's precedent for "authored but unreferenced" is advisory (see the
|
||||
# dead-knob advisory in flux-postbuild-substitution/check.sh).
|
||||
served_extra = {"README"}
|
||||
if RUNBOOK_DIR.is_dir():
|
||||
for md in sorted(RUNBOOK_DIR.glob("*.md")):
|
||||
stem = md.stem
|
||||
if stem in served_extra or stem in names:
|
||||
continue
|
||||
ADVISORIES.append(f"{md.relative_to(ROOT)} is a runbook no alert links to (written ahead of "
|
||||
f"its alert, or the alert was removed — advisory, not a failure)")
|
||||
|
||||
# ================================= (4) every runbook is PROJECTED INTO THE SERVER, not just in git
|
||||
# THIS IS THE HALF A FILE-EXISTENCE CHECK WOULD MISS. The pod serves a ConfigMap, not the git tree.
|
||||
# Parse contract: runbooks/kustomization.yaml declares one configMapGenerator whose `files:` entries
|
||||
# are `- <key>=<file>.md`, one per line. The key is EXTENSIONLESS on purpose (the URL carries no
|
||||
# `.md`, and the ConfigMap volume mount turns each key into a file the nginx root resolves directly).
|
||||
KUST = RUNBOOK_DIR / "kustomization.yaml"
|
||||
check(KUST.is_file(), f"{PREFIX}/kustomization.yaml is missing — the Flux tier renders nothing and "
|
||||
f"every runbook 404s no matter what is in git")
|
||||
GEN_FILE_RE = re.compile(r"^\s*-\s*([A-Za-z0-9._-]+)=([A-Za-z0-9._/-]+\.md)\s*(?:#.*)?$")
|
||||
generated: dict[str, str] = {}
|
||||
kust_text = read(KUST)
|
||||
if kust_text:
|
||||
check("configMapGenerator:" in kust_text,
|
||||
f"{PREFIX}/kustomization.yaml has no configMapGenerator — the runbook markdown is not "
|
||||
f"rendered into the ConfigMap the server mounts")
|
||||
for line in kust_text.splitlines():
|
||||
if line.lstrip().startswith("#"):
|
||||
continue
|
||||
m = GEN_FILE_RE.match(line)
|
||||
if m:
|
||||
generated[m.group(1)] = m.group(2)
|
||||
check(len(generated) > 0,
|
||||
f"{PREFIX}/kustomization.yaml declares no `<key>=<file>.md` generator entries — the "
|
||||
f"parser contract broke, or the markdown is being added with implicit keys (which would "
|
||||
f"carry the '.md' suffix the alert URLs do NOT have, so every link would 404)")
|
||||
|
||||
for name in sorted(names):
|
||||
check(generated.get(name) == f"{name}.md",
|
||||
f"runbook '{name}' is not projected into the server: {PREFIX}/kustomization.yaml has no "
|
||||
f"configMapGenerator entry `- {name}={name}.md`. The file can be in git, reviewed and "
|
||||
f"merged and the URL still returns 404 — 'the file exists' and 'the URL resolves' are "
|
||||
f"different claims")
|
||||
|
||||
# And the reverse, so a generator entry cannot point at a file that is not there.
|
||||
for key, fname in sorted(generated.items()):
|
||||
check((RUNBOOK_DIR / fname).is_file(),
|
||||
f"{PREFIX}/kustomization.yaml generates key '{key}' from {PREFIX}/{fname}, which does not "
|
||||
f"exist — `kustomize build` fails, so the WHOLE tier stops reconciling and every runbook "
|
||||
f"goes stale at once")
|
||||
|
||||
# ============================== (5) the route, the listener and the server agree with the URLs
|
||||
ROUTE = RUNBOOK_DIR / "serving" / "httproute.yaml"
|
||||
NGINX = RUNBOOK_DIR / "serving" / "nginx-config.yaml"
|
||||
NS = RUNBOOK_DIR / "serving" / "namespace.yaml"
|
||||
GATEWAY = ROOT / "platform" / "gateway" / "gateway.yaml"
|
||||
|
||||
route_text = read(ROUTE)
|
||||
check(bool(route_text), f"{PREFIX}/serving/httproute.yaml is missing — nothing routes the "
|
||||
f"/{PREFIX} prefix to the server")
|
||||
# Parse contract: a single HTTPRoute with one PathPrefix match and one parentRefs entry naming a
|
||||
# listener via sectionName; hand-authored, one key per line.
|
||||
m_path = re.search(r"^\s*type:\s*PathPrefix\s*$\n\s*value:\s*(\S+)\s*$", route_text, re.M)
|
||||
check(m_path is not None and m_path.group(1) == f"/{PREFIX}",
|
||||
f"{PREFIX}/serving/httproute.yaml does not match PathPrefix /{PREFIX} "
|
||||
f"(found {m_path.group(1) if m_path else 'no PathPrefix match'}). The alert annotations point "
|
||||
f"at /{PREFIX}/<name>; if the route's prefix drifts from the annotations the links 404 while "
|
||||
f"both files look individually correct")
|
||||
m_sec = re.search(r"^\s*sectionName:\s*(\S+)", route_text, re.M)
|
||||
check(m_sec is not None,
|
||||
f"{PREFIX}/serving/httproute.yaml declares no parentRefs sectionName. The apex host needs the "
|
||||
f"dedicated apex listener: the wildcard `*.${{BASE_DOMAIN}}` listener does NOT match the bare "
|
||||
f"apex, so a route without sectionName binds nothing and serves nothing")
|
||||
if m_sec:
|
||||
listener = m_sec.group(1)
|
||||
gw = read(GATEWAY)
|
||||
check(re.search(rf"^\s*-\s*name:\s*{re.escape(listener)}\s*$", gw, re.M) is not None,
|
||||
f"{PREFIX}/serving/httproute.yaml attaches to Gateway listener '{listener}', which "
|
||||
f"platform/gateway/gateway.yaml does not define — the route would report NotAllowed and "
|
||||
f"never serve")
|
||||
# An HTTPRoute with no `hostnames:` inherits the listener's hostname, which is how this tier avoids
|
||||
# needing Flux substitution (see (6)). Either form is routable; a LITERAL hostname is not acceptable,
|
||||
# for the same reason the schema forbids it in a ServiceDefinition.
|
||||
m_host = re.search(r"^\s*hostnames:\s*$\n\s*-\s*(\S+)", route_text, re.M)
|
||||
check(m_host is None or "${BASE_DOMAIN}" in m_host.group(1),
|
||||
f"{PREFIX}/serving/httproute.yaml hardcodes hostname '{m_host.group(1) if m_host else ''}' — "
|
||||
f"use the ${{BASE_DOMAIN}} token, or omit `hostnames:` entirely and inherit the listener's")
|
||||
|
||||
ns_text = read(NS)
|
||||
check('gateway-access: "shared"' in ns_text or "gateway-access: shared" in ns_text,
|
||||
f"{PREFIX}/serving/namespace.yaml does not carry gateway-access=shared — the shared Gateway "
|
||||
f"selects namespaces by that label, so without it the HTTPRoute is refused attachment")
|
||||
|
||||
nginx_text = read(NGINX)
|
||||
check(f"location /{PREFIX}/" in nginx_text,
|
||||
f"{PREFIX}/serving/nginx-config.yaml has no `location /{PREFIX}/` block — the route would "
|
||||
f"deliver the request and the server would answer 404 for every runbook")
|
||||
check("default_type" in nginx_text and "text/plain" in nginx_text,
|
||||
f"{PREFIX}/serving/nginx-config.yaml does not force a text/plain default_type. The ConfigMap "
|
||||
f"keys are EXTENSIONLESS, so with the stock MIME table nginx serves them as "
|
||||
f"application/octet-stream and a browser DOWNLOADS the runbook instead of showing it")
|
||||
|
||||
# ====================================== (6) the Flux tier is wired, and is NOT substituted
|
||||
FLUX = ROOT / "clusters" / "base" / f"{PREFIX}.yaml"
|
||||
BASE_KUST = ROOT / "clusters" / "base" / "kustomization.yaml"
|
||||
flux_text = read(FLUX)
|
||||
check(bool(flux_text),
|
||||
f"clusters/base/{PREFIX}.yaml is missing — nothing reconciles {PREFIX}/ into the cluster, so "
|
||||
f"the runbooks exist in git and the URL still 404s")
|
||||
check(re.search(rf"^\s*path:\s*\./{re.escape(PREFIX)}\s*$", flux_text, re.M) is not None,
|
||||
f"clusters/base/{PREFIX}.yaml does not set `path: ./{PREFIX}`")
|
||||
check(re.search(r"^\s*wait:\s*false\s*$", flux_text, re.M) is not None,
|
||||
f"clusters/base/{PREFIX}.yaml is not wait:false — a leaf web server that can be legitimately "
|
||||
f"unhealthy must never gate the reconcile chain (DECISIONS-LOG: 'Headlamp gating stall')")
|
||||
# Strip comments before testing: this file's own comment block EXPLAINS at length why
|
||||
# substituteFrom must never be added here, and a naive substring test would fire on the warning.
|
||||
flux_code = "\n".join(l for l in flux_text.splitlines() if not l.lstrip().startswith("#"))
|
||||
check("substituteFrom" not in flux_code,
|
||||
f"clusters/base/{PREFIX}.yaml declares postBuild.substituteFrom. It MUST NOT: this tier "
|
||||
f"renders documentation into a ConfigMap, and Flux's envsubst would run over the prose and "
|
||||
f"silently EMPTY every `${{...}}` shell example in every runbook — it fails open, so the "
|
||||
f"Kustomization would still report Ready (DECISIONS-LOG 2026-09-01, the emptied Cilium "
|
||||
f"k8sServiceHost). The flux-postbuild-substitution check cannot see this: it scans files "
|
||||
f"reachable through `resources:`, not configMapGenerator inputs")
|
||||
check(re.search(rf"^\s*-\s*{re.escape(PREFIX)}\.yaml\b", read(BASE_KUST), re.M) is not None,
|
||||
f"clusters/base/kustomization.yaml does not list {PREFIX}.yaml — the Flux Kustomization is "
|
||||
f"authored and never applied, which is the 'looks done, is absent' class this repo has been "
|
||||
f"bitten by repeatedly")
|
||||
|
||||
# ============================================ (7) this check is load-bearing in the merge gate
|
||||
L0 = read(ROOT / "ci" / "l0.sh")
|
||||
GATE = read(ROOT / "ci" / "gate.sh")
|
||||
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism, in two halves.
|
||||
# It is not enough that check.sh exists: the merge gate must SELECT it on every run. The repo used to
|
||||
# get that with a hand-written "UNCONDITIONAL" stage per whole-repo check in ci/l0.sh, hardcoding the
|
||||
# path; those stages are gone, because the next whole-repo invariant only got one if somebody
|
||||
# REMEMBERED to write it. The property is now DECLARED by the test and HONOURED by the gate, so both
|
||||
# halves are asserted — neither alone is sufficient. The mechanism itself (that the union really
|
||||
# selects, and that a scope-declared SKIP exits 3 rather than passing green) is proven end-to-end and
|
||||
# falsifiably by conformance/tests/l0-static/whole-repo-invariants-wired.
|
||||
SELF_YAML = (ROOT / "conformance" / "tests" / "l0-static" / "alert-runbooks-resolvable"
|
||||
/ "chainsaw-test.yaml")
|
||||
SELF = read(SELF_YAML)
|
||||
check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
|
||||
"alert-runbooks-resolvable/chainsaw-test.yaml does not declare "
|
||||
"conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
|
||||
"change happens to touch a requirement it is labelled with. ci/touched-requirements.sh maps "
|
||||
"platform/monitoring/* to 'R-OBS-01 R-OBS-02' (NOT R-OBS-04) and maps platform/kubevirt/* and "
|
||||
"runbooks/* to nothing at all — so adding an alert with a new runbook_url, or adding a runbook "
|
||||
"without its configMapGenerator entry, would be filtered out of exactly the change that "
|
||||
"introduces the dead link")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
|
||||
"purely requirement-scoped run again and the scope annotation above is inert decoration — this "
|
||||
"check would then run only by coincidence")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
# ======================================================================== report
|
||||
print(f"Alert runbook inventory: {n_annotations} runbook_url annotation(s) across "
|
||||
f"{len(rule_files)} PrometheusRule file(s) -> {len(names)} distinct runbook(s), "
|
||||
f"prefix /{PREFIX}")
|
||||
missing = sorted(n for n in names if not (RUNBOOK_DIR / f"{n}.md").is_file())
|
||||
print(f" runbooks present: {len(names) - len(missing)}/{len(names)} "
|
||||
f"projected into the ConfigMap: {sum(1 for n in names if generated.get(n) == f'{n}.md')}/{len(names)}")
|
||||
if missing:
|
||||
print(f" MISSING: {', '.join(missing)}", file=sys.stderr)
|
||||
|
||||
if ADVISORIES:
|
||||
print("\nADVISORIES (not failures):")
|
||||
for a in ADVISORIES:
|
||||
print(f" - {a}")
|
||||
|
||||
print()
|
||||
if not FAILURES:
|
||||
print(f"check OK: alert-runbooks-resolvable — {CHECKS} assertions (R-OBS-04)")
|
||||
raise SystemExit(0)
|
||||
print(f"CHECK FAIL (alert-runbooks-resolvable): {len(FAILURES)} failed, "
|
||||
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" FAIL {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
@@ -0,0 +1,114 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
|
||||
# operator can prove isolation, egress path, cert validity, and backup integrity)"),
|
||||
# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
|
||||
# must be able to prove": Isolation (denied-probe test), Egress path (egress IP
|
||||
# check), Kill-switch (fault-injection test), TLS (cert status + external check),
|
||||
# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
|
||||
# one-time checks", with the acceptance gate "The five auditability proofs above can
|
||||
# be produced on demand."
|
||||
#
|
||||
# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
|
||||
# -----------------------------------------------------------------------------------
|
||||
# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
|
||||
# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
|
||||
# failure modes are asserted instead, each by DRIVING the real machinery:
|
||||
# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
|
||||
# the spec's own words require (fault injection / a DR drill => L3), taken from
|
||||
# lib/status.py's own verdict so this check cannot disagree with the status report;
|
||||
# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
|
||||
# must resolve to real test dirs AND pass the merge gate's coverage assertion;
|
||||
# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
|
||||
# attribute — a proof that passes but records nothing leaves the operator with a green
|
||||
# terminal and no evidence;
|
||||
# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
|
||||
# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
|
||||
# token, and a ledger row claiming `proven` at the repository's ROOT commit — and
|
||||
# requires record_run.py to report not-proven and status.py to report `stale-proof`. It
|
||||
# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
|
||||
# ci/ or run.sh ran before this test existed.
|
||||
# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
|
||||
# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
|
||||
# so a sixth proof added to the spec fails here instead of being silently unverified.
|
||||
#
|
||||
# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
|
||||
# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
|
||||
# anyone needs it.
|
||||
#
|
||||
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
||||
# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
|
||||
# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
|
||||
# and two synthetic forgeries are presented to them,
|
||||
# THEN each of docs/07's five proofs maps to an executable test at the required layer, the
|
||||
# documented one-command invocation selects and asserts them, each is capturable as a
|
||||
# ledger row, and neither a self-skip nor a stale row can be recorded as proof.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
|
||||
# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
|
||||
# it directly, and this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
|
||||
# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
|
||||
# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
|
||||
# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
|
||||
# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
|
||||
# assertion here was NOT relaxed to accommodate them (R-TEST-06).
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * That the five proofs PASS. This asserts they are present, strong, runnable by one
|
||||
# command, capturable, and that the chain refuses forgeries — not that isolation holds
|
||||
# today (R-NET-01's test) or that the restore works (R-STATE-03's).
|
||||
# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
|
||||
# suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
|
||||
# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
|
||||
# would need signed history.
|
||||
# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
|
||||
# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
|
||||
# not re-asserted here.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: auditability-proofs-producible
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. This is a META-invariant over the ENTIRE conformance suite: the five
|
||||
# spec-named proofs must each still have an EXECUTABLE test at the right layer, with a
|
||||
# pass sentinel, resolvable by the documented command. Every one of its inputs lives under
|
||||
# conformance/, which ci/touched-requirements.sh keys to R-TEST-01..04 — never to R-SEC-03.
|
||||
# So downgrading conformance/tests/l3-fault-injection/dr-restore to a scaffold attributes
|
||||
# R-STATE-03/R-TEST-*, and requirement-scoped selection drops the one check that notices
|
||||
# the audit evidence just stopped being producible.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree and the harness's own tooling, when status.py, the requirement
|
||||
resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
|
||||
tools' self-tests are driven and two synthetic forgeries are presented to them, then each
|
||||
of docs/07's five auditability proofs has an executable test at the layer the spec
|
||||
requires, is selectable and asserted by the documented one-command invocation, is
|
||||
capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
|
||||
evidence.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
|
||||
# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
|
||||
# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
|
||||
# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
|
||||
# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
|
||||
# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
|
||||
# the wrong trade (R-TEST-06).
|
||||
@@ -0,0 +1,454 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — auditability-proofs-producible
|
||||
# Serves: R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an operator can
|
||||
# prove isolation, egress path, cert validity, and backup integrity)."
|
||||
# — spec docs/00-vision-and-scope.md).
|
||||
#
|
||||
# The spec makes this meta-requirement concrete. docs/07-security-and-secrets.md
|
||||
# §"Auditability requirements (R-SEC-03) — what an operator must be able to prove":
|
||||
# - **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a
|
||||
# denied-probe test.
|
||||
# - **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
|
||||
# - **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
|
||||
# - **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert
|
||||
# status + external check.
|
||||
# - **Backups**: "irreplaceable data is restorable" — via a DR drill.
|
||||
# "These proofs are part of the platform acceptance suite, not one-time checks."
|
||||
# and its acceptance gate: "The five auditability proofs above can be produced on demand."
|
||||
#
|
||||
# HOW A META-REQUIREMENT IS TESTED WITHOUT BECOMING A TAUTOLOGY
|
||||
# ------------------------------------------------------------
|
||||
# The trap is obvious and this check is built to avoid it: "a test exists for R-NET-01,
|
||||
# therefore isolation is auditable" is true of any suite that compiles, proves nothing, and
|
||||
# would keep passing while every proof rotted into a scaffold. Four separate failure modes
|
||||
# have to be caught for "an operator can PROVE it" to mean anything, and each one is asserted
|
||||
# by DRIVING the real machinery rather than by reading files:
|
||||
#
|
||||
# (a) THE PROOF EXISTS, AT THE RIGHT STRENGTH. Each of the five named proofs maps to a
|
||||
# requirement that must have an EXECUTABLE verifying test — not a placeholder scaffold — and,
|
||||
# where the spec's own words name fault injection ("with VPN down"; "a DR drill"), an
|
||||
# executable L3 test specifically. Executable-vs-scaffold is not re-derived here: this
|
||||
# check RUNS lib/status.py and reads its verdict, so it cannot disagree with the
|
||||
# repository's own status report.
|
||||
# (b) THE PROOF IS PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND. `run.sh
|
||||
# --requirements <the five IDs>` must resolve to a non-empty test set and must ASSERT
|
||||
# their coverage (that flag is the merge gate's own path: it passes --require to
|
||||
# coverage.py). This check drives run.sh itself, so if the resolver, the labels or the
|
||||
# gate wiring break, R-SEC-03 fails.
|
||||
# (c) THE PROOF IS CAPTURABLE. A test that passes but records nothing leaves an operator
|
||||
# with a green terminal and no evidence. For each Chainsaw-layer proof, the test must
|
||||
# declare a PASS SENTINEL that lib/record_run.py can actually attribute — computed with
|
||||
# record_run.py's OWN parser, so deleting a test's sentinel makes this check fail
|
||||
# instead of silently making the requirement unprovable.
|
||||
# (d) THE EVIDENCE CANNOT BE FORGED, AND IS NOT HONOURED WHEN STALE. This is the clause
|
||||
# that makes the check bite. Two INDEPENDENT forgery attempts are mounted against the
|
||||
# real tools, in a throwaway directory:
|
||||
# * a Chainsaw run in which the JUnit says a real proof test PASSED while the run log
|
||||
# carries only that test's SKIP token — the exact shape of a self-skipping L2/L3
|
||||
# test. record_run.py must refuse to call it `proven`.
|
||||
# * a ledger row claiming `proven` at a commit that PREDATES the requirement's
|
||||
# implementing code — status.py must degrade it to `stale-proof`, never green.
|
||||
# Both tools' own built-in self-tests (`--selftest`) are also driven, because nothing
|
||||
# in ci/ or run.sh runs them today: the guards that keep a skip from becoming a proof
|
||||
# were themselves unexercised in the merge gate until this check.
|
||||
#
|
||||
# The FIVE PROOFS ARE NOT HAND-LISTED where the spec can be read: when the read-only spec
|
||||
# repo is mounted, the bullet labels are parsed out of docs/07 and must match this check's
|
||||
# mapping EXACTLY. Add a sixth proof to the spec and this check fails as unmapped rather
|
||||
# than quietly ignoring it.
|
||||
#
|
||||
# WHY L0: every clause above is a property of the repository and its own tooling, provable
|
||||
# offline, and it belongs in the merge gate — the point of an audit chain is that it is intact
|
||||
# BEFORE anyone needs it. Nothing here needs a cluster, and nothing here would be more true
|
||||
# with one.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
|
||||
# - THAT THE FIVE PROOFS PASS. This asserts the proofs are present, strong, runnable by one
|
||||
# documented command, capturable as evidence, and that the evidence chain refuses
|
||||
# forgeries. Whether isolation actually holds today is R-NET-01's test; whether the DR
|
||||
# restore works is R-STATE-03's. R-SEC-03 is about the operator's ABILITY to produce the
|
||||
# proof, which is what is asserted.
|
||||
# - HUBBLE. docs/07 names "Hubble flow logs" as part of the isolation proof. No test in this
|
||||
# suite queries Hubble, and this check does not pretend otherwise: it asserts the
|
||||
# denied-probe half, which is the half that is automated. The Hubble half remains an
|
||||
# operator-interactive capability — a real, named gap in R-SEC-03's coverage.
|
||||
# - THE OPERATOR-FACING NARRATIVE. That a human can find and read the evidence (the status
|
||||
# report's prose, the ledger's provenance fields) is a usability property, not a testable
|
||||
# one.
|
||||
# - LEDGER COMPLETENESS OVER TIME. The ledger is append-only by convention and by review;
|
||||
# nothing here can prove a row was never removed from it (that would need signed history).
|
||||
# - THAT THE PROOFS COVER "ALL THE ABOVE PROPERTIES". R-SEC-03's own wording is broader than
|
||||
# the five bullets docs/07 enumerates. The five are what the spec made concrete and are
|
||||
# what is asserted; suite-wide coverage of every requirement is R-TEST-01/04's claim,
|
||||
# tested by tests/meta/harness-selfcheck, and is deliberately NOT re-asserted here.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
|
||||
export SPEC_REPO
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
CONF = ROOT / "conformance"
|
||||
LIB = CONF / "lib"
|
||||
FAILURES: list[str] = []
|
||||
NOTES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def note(msg):
|
||||
NOTES.append(msg)
|
||||
|
||||
|
||||
def die():
|
||||
print(f"CHECK FAIL (auditability-proofs-producible): {len(FAILURES)} of {CHECKS} "
|
||||
"assertions failed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def run(cmd, **kw) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(cmd, capture_output=True, text=True, cwd=kw.pop("cwd", CONF), **kw)
|
||||
|
||||
|
||||
# ================================================= THE FIVE PROOFS docs/07 ENUMERATES
|
||||
# key = the bullet label docs/07 uses (verified against the spec below when it is mounted)
|
||||
# reqs = the requirement(s) whose verifying test IS that proof
|
||||
# layer= the WEAKEST layer the spec's own words allow. Where docs/07 says "a fault-injection
|
||||
# test" / "a DR drill", an L2 happy-path test does not discharge the proof.
|
||||
PROOFS = {
|
||||
"Isolation": {"reqs": ["R-NET-01"], "layers": {"L2", "L3"},
|
||||
"why": '"service X cannot reach service Y" — via Hubble flow logs + a '
|
||||
"denied-probe test"},
|
||||
"Egress path": {"reqs": ["R-NET-05"], "layers": {"L2", "L3"},
|
||||
"why": '"service X\'s public IP is the VPN\'s" — via an egress IP check'},
|
||||
"Kill-switch": {"reqs": ["R-NET-06"], "layers": {"L3"},
|
||||
"why": '"with VPN down, X leaks nothing" — via a fault-injection test'},
|
||||
"TLS": {"reqs": ["R-TLS-01"], "layers": {"L2", "L3"},
|
||||
"why": '"every endpoint serves a valid, non-self-signed, unexpired cert" '
|
||||
"— via cert status + external check"},
|
||||
"Backups": {"reqs": ["R-STATE-03"], "layers": {"L3"},
|
||||
"why": '"irreplaceable data is restorable" — via a DR drill'},
|
||||
}
|
||||
PROOF_REQS = sorted({r for p in PROOFS.values() for r in p["reqs"]})
|
||||
|
||||
# ---- SELF-POLICING: the proof list must still be the SPEC's proof list ---------------
|
||||
spec = Path(os.environ.get("SPEC_REPO", ""))
|
||||
sec_doc = spec / "docs" / "07-security-and-secrets.md"
|
||||
if sec_doc.is_file():
|
||||
text = sec_doc.read_text()
|
||||
m = re.search(r"^##\s+Auditability requirements \(R-SEC-03\).*?$(.*?)^##\s", text,
|
||||
re.M | re.S)
|
||||
check(m is not None,
|
||||
"docs/07 no longer carries an '## Auditability requirements (R-SEC-03)' section — "
|
||||
"the proof list this check is coupled to has moved or been removed, so the mapping "
|
||||
"below can no longer be trusted (R-TEST-06: follow the spec, never the other way)")
|
||||
if m:
|
||||
labels = re.findall(r"^\s*-\s+\*\*([^*]+)\*\*:", m.group(1), re.M)
|
||||
check(set(labels) == set(PROOFS),
|
||||
f"docs/07 enumerates the auditability proofs {sorted(labels)} but this check "
|
||||
f"maps {sorted(PROOFS)} — every proof the spec names must map to a verifying "
|
||||
"test, and a proof this check does not know about would be silently unverified")
|
||||
check(len(labels) == 5,
|
||||
f"docs/07 enumerates {len(labels)} auditability proof(s); its own acceptance "
|
||||
"gate says 'The five auditability proofs above can be produced on demand' — the "
|
||||
"prose and the list have drifted apart")
|
||||
else:
|
||||
note(f"the read-only spec repo is not mounted at {spec} — the proof list could not be "
|
||||
"re-derived from docs/07, so the mapping below was used as written. Every other "
|
||||
"assertion still ran. (ci/l0.sh mounts the spec; a bare local run may not.)")
|
||||
|
||||
# ============================================ (a) EACH PROOF EXISTS, AT THE RIGHT STRENGTH
|
||||
# Driven through lib/status.py so this check and the repository's own status report cannot
|
||||
# disagree about what is executable and what is a scaffold.
|
||||
tmp = Path(tempfile.mkdtemp(prefix="r-sec-03-"))
|
||||
try:
|
||||
out = tmp / "status"
|
||||
st = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
|
||||
"--registry", str(CONF / "requirements" / "registry.yaml"),
|
||||
"--tests", str(CONF / "tests"), "--out", str(out)])
|
||||
check(st.returncode == 0,
|
||||
f"lib/status.py exited {st.returncode} — the report an operator would use to find "
|
||||
f"the audit evidence does not generate: {(st.stderr or st.stdout).strip()[:400]}")
|
||||
status = {}
|
||||
sj = out / "status.json"
|
||||
if sj.is_file():
|
||||
status = json.loads(sj.read_text())
|
||||
check(bool(status.get("by_requirement")),
|
||||
"lib/status.py produced no per-requirement report — the audit posture of each "
|
||||
"requirement is not derivable, so R-SEC-03 cannot be assessed at all")
|
||||
by_req = status.get("by_requirement", {})
|
||||
|
||||
for label, spec_proof in sorted(PROOFS.items()):
|
||||
for rid in spec_proof["reqs"]:
|
||||
rec = by_req.get(rid)
|
||||
check(rec is not None,
|
||||
f"proof '{label}' maps to {rid}, which the status report does not know — "
|
||||
"either the registry lost the requirement or this mapping is stale")
|
||||
if not rec:
|
||||
continue
|
||||
tests = (rec.get("verification") or {}).get("tests") or []
|
||||
executable = [t for t in tests if t.get("state", "").startswith("executable")]
|
||||
check(bool(executable),
|
||||
f"proof '{label}' ({rid}) has no EXECUTABLE verifying test — its tests are "
|
||||
f"{[t.get('name') + ':' + t.get('state', '?') for t in tests]}. docs/07: "
|
||||
f"{spec_proof['why']}. A placeholder scaffold cannot be produced on demand, so the "
|
||||
"operator cannot prove this property")
|
||||
at_layer = [t for t in executable if t.get("layer") in spec_proof["layers"]]
|
||||
check(bool(at_layer),
|
||||
f"proof '{label}' ({rid}) has no executable test at layer(s) "
|
||||
f"{sorted(spec_proof['layers'])} — it has "
|
||||
f"{[t.get('name') + '@' + str(t.get('layer')) for t in executable]}. "
|
||||
f"docs/07 requires this proof to be produced {spec_proof['why']}, and a "
|
||||
"weaker layer does not discharge it (a rendering check is not a drill)")
|
||||
|
||||
# ==================================== (b) PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND
|
||||
# The resolver: --requirements must translate the five R-IDs into real test dirs.
|
||||
resolver = run(["bash", "-c",
|
||||
f'export CONF_ROOT="{CONF}"; . "{LIB}/common.sh"; '
|
||||
f'conf_dirs_for_requirements "{" ".join(PROOF_REQS)}"'])
|
||||
dirs = [l for l in resolver.stdout.splitlines() if l.strip()]
|
||||
check(resolver.returncode == 0 and len(dirs) >= len(PROOF_REQS),
|
||||
f"`run.sh --requirements {','.join(PROOF_REQS)}` resolves to {len(dirs)} test dir(s) "
|
||||
f"for {len(PROOF_REQS)} proof requirement(s) — the single command docs/07's "
|
||||
"acceptance gate depends on ('produced on demand') cannot select the proofs")
|
||||
|
||||
# The gate: the same flag ASSERTS coverage (run.sh passes --require to coverage.py), so a
|
||||
# zero exit here is the repository's own statement that all five proofs are covered.
|
||||
gate = run(["./run.sh", "--coverage-only", "--requirements", ",".join(PROOF_REQS)])
|
||||
check(gate.returncode == 0,
|
||||
f"`./run.sh --coverage-only --requirements {','.join(PROOF_REQS)}` exited "
|
||||
f"{gate.returncode} — the merge gate's own coverage assertion rejects the audit "
|
||||
f"proof set: {(gate.stderr or gate.stdout).strip()[:400]}")
|
||||
|
||||
# ================================================ (c) THE PROOF IS CAPTURABLE AS EVIDENCE
|
||||
# Use record_run.py's OWN sentinel parser: a Chainsaw-layer proof with no declared pass
|
||||
# token can pass forever and never be recorded as proof of anything.
|
||||
sys.path.insert(0, str(LIB))
|
||||
os.environ.setdefault("PYTHONDONTWRITEBYTECODE", "1")
|
||||
sys.dont_write_bytecode = True
|
||||
import record_run as rr # noqa: E402
|
||||
|
||||
# status.py owns the latest-wins rule; import it rather than reimplementing it, so the two
|
||||
# cannot disagree about which rows are live evidence.
|
||||
import status as _st # noqa: E402
|
||||
rr_latest_by_test = _st.latest_by_test
|
||||
|
||||
smap = rr.sentinel_map(CONF / "tests")
|
||||
check(len(smap) >= 10,
|
||||
f"record_run.py's sentinel map contains {len(smap)} test(s) — the parser lost its "
|
||||
"grip on the test tree, so the assertions below would assert nothing")
|
||||
for label, spec_proof in sorted(PROOFS.items()):
|
||||
for rid in spec_proof["reqs"]:
|
||||
rec = by_req.get(rid) or {}
|
||||
tests = (rec.get("verification") or {}).get("tests") or []
|
||||
names = [t.get("name") for t in tests
|
||||
if t.get("state", "").startswith("executable")
|
||||
and t.get("layer") in spec_proof["layers"]]
|
||||
for name in names:
|
||||
ent = smap.get(name)
|
||||
check(ent is not None,
|
||||
f"proof '{label}': record_run.py does not know the test {name!r} — its "
|
||||
"result could never be attributed to a ledger row")
|
||||
if not ent:
|
||||
continue
|
||||
if ent.get("layer") in rr.SENTINEL_EXEMPT_LAYERS:
|
||||
continue # offline layers are truthful by JUnit alone
|
||||
check(bool(ent.get("pass")),
|
||||
f"proof '{label}': the Chainsaw test {name!r} declares NO pass sentinel, "
|
||||
"so lib/record_run.py can never record it as proven. It would go green "
|
||||
"in the terminal and leave the operator with no evidence — exactly the "
|
||||
"gap the ledger exists to close (docs/07: the proofs are 'part of the "
|
||||
"platform acceptance suite, not one-time checks')")
|
||||
check(bool(ent.get("terminal_pass")),
|
||||
f"proof '{label}': the test {name!r} has no TERMINAL pass token, so "
|
||||
"record_run.py cannot decide its verdict")
|
||||
|
||||
# =============================== (d) THE EVIDENCE CANNOT BE FORGED, NOR HONOURED WHEN STALE
|
||||
# Both tools' own guards, which NOTHING in ci/ or run.sh drives today.
|
||||
for tool in ("record_run.py", "status.py"):
|
||||
r = run([sys.executable, str(LIB / tool), "--selftest"])
|
||||
check(r.returncode == 0,
|
||||
f"`lib/{tool} --selftest` exited {r.returncode} — the guards that keep a "
|
||||
"self-skipped or stale run from being recorded as proof are broken, so nothing "
|
||||
f"in the evidence chain can be trusted: {(r.stderr or r.stdout).strip()[:500]}")
|
||||
|
||||
# ---- FORGERY 1: a green JUnit for a REAL proof test whose log holds only a SKIP token.
|
||||
# This is the precise shape of a self-skipping L2/L3 test, and it must NOT become proof.
|
||||
forge_target, forge_skip = None, None
|
||||
for label, spec_proof in sorted(PROOFS.items()):
|
||||
for rid in spec_proof["reqs"]:
|
||||
for name, ent in smap.items():
|
||||
if rid in (ent.get("requirements") or []) and ent.get("skip") \
|
||||
and ent.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS:
|
||||
forge_target, forge_skip = name, ent["skip"][0]
|
||||
break
|
||||
if forge_target:
|
||||
break
|
||||
if forge_target:
|
||||
break
|
||||
check(forge_target is not None,
|
||||
"no Chainsaw-layer proof test declares a skip token, so the forgery attempt below "
|
||||
"could not be mounted. That is not a pass: it means this check could not verify "
|
||||
"that a self-skip is refused as proof, which is the whole point of the ledger")
|
||||
if forge_target:
|
||||
run_dir = tmp / "20260101T000000Z-forged"
|
||||
run_dir.mkdir(parents=True)
|
||||
(run_dir / "report.junit.xml").write_text(
|
||||
'<?xml version="1.0" encoding="UTF-8"?>\n'
|
||||
'<testsuites><testsuite name="chainsaw" tests="1" failures="0" errors="0">'
|
||||
f'<testcase name="{forge_target}" classname="forged"/>'
|
||||
"</testsuite></testsuites>\n")
|
||||
(run_dir / "run.log").write_text(
|
||||
f"=== RUN chainsaw/{forge_target}\n"
|
||||
f" === STDOUT\n {forge_skip}\n")
|
||||
forged = run([sys.executable, str(LIB / "record_run.py"), "--run-dir", str(run_dir),
|
||||
"--dry-run", "--ledger", str(tmp / "never-written.jsonl"),
|
||||
"--tests", str(CONF / "tests"), "--repo", str(ROOT)])
|
||||
verdicts = re.findall(rf"^\[record\]\s+\S+\s+(\S+)\s+{re.escape(forge_target)}\b",
|
||||
forged.stdout, re.M)
|
||||
check(verdicts and all(v != "proven" for v in verdicts),
|
||||
f"lib/record_run.py recorded a FORGED run as {verdicts!r} for {forge_target!r}: "
|
||||
"a JUnit saying `passed` with nothing in the log but that test's own SKIP token "
|
||||
f"({forge_skip}). A self-skip must never become proof — with this hole every "
|
||||
"audit proof in docs/07 could be produced by a cluster that skipped it")
|
||||
check(not (tmp / "never-written.jsonl").exists(),
|
||||
"record_run.py --dry-run WROTE to the ledger — a tool that records while "
|
||||
"claiming not to cannot be trusted to record only real runs")
|
||||
|
||||
# ---- FORGERY 2: a ledger row claiming `proven` at a commit that predates the code.
|
||||
# status.py must degrade it to stale-proof (not green), or an old green run would keep
|
||||
# vouching for code it never executed.
|
||||
root_sha = run(["git", "-C", str(ROOT), "rev-list", "--max-parents=0", "HEAD"])
|
||||
old_sha = (root_sha.stdout.strip().splitlines() or [""])[0]
|
||||
check(len(old_sha) == 40,
|
||||
"the repository's root commit could not be resolved, so the stale-proof guard could "
|
||||
"not be exercised (that is a gap in this check, not a pass)")
|
||||
if len(old_sha) == 40 and forge_target:
|
||||
stale_ledger = tmp / "stale.jsonl"
|
||||
rid = sorted(smap[forge_target].get("requirements") or ["R-NET-06"])[0]
|
||||
stale_ledger.write_text(json.dumps({
|
||||
"ts_utc": "2026-01-01T00:00:00Z", "git_sha": old_sha, "env": "dev",
|
||||
"chainsaw_version": "v0.0.0", "kube_version": "unknown",
|
||||
"layer": smap[forge_target].get("layer", "L3"), "test": forge_target,
|
||||
"requirements": [rid], "junit_result": "passed",
|
||||
"sentinel": (smap[forge_target].get("terminal_pass") or "X"),
|
||||
"proof": "proven", "run_id": "20260101T000000Z-forged",
|
||||
"note": "synthetic row written by the R-SEC-03 conformance check",
|
||||
}) + "\n")
|
||||
out2 = tmp / "status-stale"
|
||||
st2 = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
|
||||
"--registry", str(CONF / "requirements" / "registry.yaml"),
|
||||
"--tests", str(CONF / "tests"), "--out", str(out2),
|
||||
"--ledger", str(stale_ledger)])
|
||||
check(st2.returncode == 0,
|
||||
f"lib/status.py failed against a synthetic ledger: "
|
||||
f"{(st2.stderr or st2.stdout).strip()[:300]}")
|
||||
s2 = json.loads((out2 / "status.json").read_text()) if (out2 / "status.json").is_file() else {}
|
||||
rec2 = (s2.get("by_requirement") or {}).get(rid) or {}
|
||||
per = {p.get("test"): p.get("state") for p in ((rec2.get("proof") or {}).get("per_test") or [])}
|
||||
check(per.get(forge_target) == "stale-proof",
|
||||
f"a ledger row claiming `proven` for {forge_target!r} at the repository's ROOT "
|
||||
f"commit was reported as {per.get(forge_target)!r} instead of 'stale-proof'. A "
|
||||
"proof from a run that predates the code it vouches for is not evidence, and "
|
||||
"honouring it would let one old green run certify every later change")
|
||||
check((rec2.get("proof") or {}).get("state") != "proven",
|
||||
f"{rid} is reported PROVEN on the strength of a row from the root commit — the "
|
||||
"freshness guard is not holding")
|
||||
|
||||
# ---- the ledger itself is machine-readable, and no row proves a test that is gone ----
|
||||
ledger = CONF / "runs" / "ledger.jsonl"
|
||||
if ledger.is_file():
|
||||
rows, bad = [], []
|
||||
for i, line in enumerate(ledger.read_text().splitlines(), 1):
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
rows.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
bad.append(i)
|
||||
check(not bad,
|
||||
f"conformance/runs/ledger.jsonl has unparsable line(s) {bad} — the audit record "
|
||||
"is not machine-readable, so no proof can be derived from it")
|
||||
# Which rows are HONOURED: status.py resolves duplicates latest-wins (latest_by_test),
|
||||
# so a superseded row is not evidence of anything. The dirty-tree assertion below is
|
||||
# therefore scoped to honoured rows. Without that scoping, ONE historical dirty row reds
|
||||
# this gate permanently even after a clean re-run of the same test — and a dirty-then-
|
||||
# clean sequence is the normal shape of "noticed the tree was dirty, fixed it, re-ran".
|
||||
# The row is still kept in the ledger and its run.log stays committed: nothing is erased,
|
||||
# it simply stops being treated as a live claim. Structural assertions (provenance keys,
|
||||
# sentinel, test-still-exists) still apply to EVERY row.
|
||||
honoured_ids = {id(v) for v in rr_latest_by_test(rows).values()}
|
||||
for r in rows:
|
||||
for key in ("ts_utc", "git_sha", "test", "requirements", "junit_result", "proof",
|
||||
"run_id"):
|
||||
check(key in r,
|
||||
f"a ledger row for {r.get('test', '?')!r} lacks {key!r} — a row without "
|
||||
"provenance cannot be audited")
|
||||
if r.get("proof") == "proven":
|
||||
# L0/L1 rows are recorded proven WITHOUT a sentinel, deliberately, by
|
||||
# record_run.py's own SENTINEL_EXEMPT_LAYERS: run_l1_offline.py renders skips as
|
||||
# JUnit <skipped/>, so those layers have no pass token to attribute. This
|
||||
# assertion had NO layer exemption, so the two halves of the harness contradicted
|
||||
# each other — ANY `--record` run touching L0 or L1 wrote a row this gate then
|
||||
# rejected, which meant a requirement whose verifier set includes an L1 check
|
||||
# (R-NET-06 is one) could never be both PROVEN and L0-green. Honour the same
|
||||
# constant rather than restating the policy here, so the two cannot drift apart.
|
||||
if r.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS:
|
||||
check(bool(r.get("sentinel")),
|
||||
f"a ledger row claims `proven` for {r.get('test')!r} with NO sentinel — "
|
||||
"the sentinel is the only thing separating a real pass from a self-skip")
|
||||
check(r.get("test") in smap,
|
||||
f"a ledger row claims `proven` for {r.get('test')!r}, which is not a "
|
||||
"test in the suite any more — a proof of something that no longer exists")
|
||||
check(id(r) not in honoured_ids or "-dirty" not in str(r.get("git_sha", "")),
|
||||
f"a ledger row claims `proven` for {r.get('test')!r} from a DIRTY tree "
|
||||
f"({r.get('git_sha')}) — what ran is not what is committed")
|
||||
else:
|
||||
note("conformance/runs/ledger.jsonl does not exist yet — the ledger-shape assertions "
|
||||
"were skipped; every other assertion ran, and the forgery probes above use "
|
||||
"synthetic ledgers, not this file")
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
||||
check(CHECKS >= 30,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
die()
|
||||
for n in NOTES:
|
||||
print(f" note: {n}")
|
||||
print(f"check OK: auditability-proofs-producible — {CHECKS} assertions; the five docs/07 "
|
||||
f"proofs ({', '.join(sorted(PROOFS))}) each map to an executable test at the layer the "
|
||||
f"spec's own wording requires, `run.sh --requirements {','.join(PROOF_REQS)}` resolves "
|
||||
"and asserts them, each Chainsaw-layer proof declares a sentinel record_run.py can "
|
||||
"attribute, both evidence tools' self-tests pass, and two independent forgeries — a "
|
||||
"self-skip presented as a pass, and a `proven` row from the root commit — were REFUSED. "
|
||||
"R-SEC-03's producibility half only: whether each proof PASSES is its own "
|
||||
"requirement's test, and the Hubble half of the isolation proof is automated nowhere "
|
||||
"(see the header's gap list).")
|
||||
PY
|
||||
@@ -0,0 +1,114 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the bulk tier's capacity strategy is DISTINCT from the replicated tier's
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE BUG CLASS THIS CLOSES. The `bulk` StorageClass existed from Phase 0 as a stub:
|
||||
# `provisioner: nfs.csi.k8s.io`, `server: REPLACE-ME`, `share: REPLACE-ME`, TODO(OQ-2).
|
||||
# Four ServiceDefinitions carried a comment promising to attach their media once OQ-2 was
|
||||
# resolved, and the ONLY thing keeping a multi-terabyte library off the 3-replica
|
||||
# `longhorn` class was those four comments. A bulk volume that lands on the replicated
|
||||
# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
|
||||
# three mini-PCs cannot hold. R-STATE-04 is precisely that the two tiers are distinct, so
|
||||
# the distinctness is asserted rather than reviewed.
|
||||
#
|
||||
# It also pins the four decisions that make the STATIC form of this tier safe, each with a
|
||||
# plausible "simplification" that breaks something silently: `Retain` (bulk data pre-exists
|
||||
# the cluster and has consumers outside it, so anything else makes `delete pvc` a data-loss
|
||||
# operation); NO dynamic provisioning (a dynamic bulk class hands out a FRESH EMPTY
|
||||
# directory that mounts and Binds perfectly while the real 8 TiB is never attached — the
|
||||
# most deceptive failure available here); `hard` not `soft` (a soft mount hands a server
|
||||
# hiccup to the app as a short read, which for a media library is silent corruption); and
|
||||
# NFSv4 (mount and lock in-protocol ⇒ ONE reachable port, tcp/2049, which is what the
|
||||
# operator must open in pfSense for prod to reach this server at all).
|
||||
#
|
||||
# AND the EXDEV trap, which is not about storage. Sonarr/Radarr import by hardlinking or
|
||||
# atomically renaming, both same-filesystem operations, and to the kernel "same filesystem"
|
||||
# means SAME MOUNT. Splitting one export into two subPath mounts looks TIGHTER and breaks
|
||||
# link() with EXDEV; the arrs then degrade to copy-then-delete, rewriting every import over
|
||||
# NFS and destroying the seeding copy. It surfaces weeks later as "imports are slow and my
|
||||
# ratio is gone". A reviewer cannot see that; this check can.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# WHEN every `class: bulk` storage entry in catalog/ is compared against the objects the
|
||||
# COMMITTED apps/ goldens carry for it (what Flux reconciles — deliberately not the
|
||||
# renderer, which would only prove it agrees with itself),
|
||||
# THEN (1) no bulk volume is on a Longhorn-provisioned class, and its class is explicitly
|
||||
# EMPTY so nothing can dynamically provision an empty stand-in;
|
||||
# (2) its capacity is declared and is at least an order of magnitude above the same
|
||||
# service's replicated volume — the requirement stated as arithmetic;
|
||||
# (3) PV and PVC are pre-bound from BOTH ends (claimRef + volumeName) and agree;
|
||||
# (4) the PV reclaim policy is Retain;
|
||||
# (5) mountOptions carry `hard` and exactly one `nfsvers=4.x`, and never `soft`;
|
||||
# (6) no workload mounts one export more than once (the EXDEV trap);
|
||||
# (7) a readOnly entry renders nfs.readOnly AND a readOnly volumeMount AND
|
||||
# ReadOnlyMany; a writable one renders neither and is ReadWriteMany; every bulk
|
||||
# access mode is a `...Many`, since one export serves several namespaces and
|
||||
# several nodes;
|
||||
# (8) the PV's nfs.server is a BARE cluster-vars token (never a literal — dev and
|
||||
# prod reach this host by different addresses), that knob has a base default, and
|
||||
# EVERY overlay supplies it;
|
||||
# AND (9) the check is LOAD-BEARING (ci/l0.sh runs it unconditionally, ci/gate.sh blocks
|
||||
# on the l0 stage) and NON-VACUOUS (an empty bulk corpus FAILS rather than
|
||||
# passing trivially, and a floor on the assertion count keeps a pass from
|
||||
# degenerating into a no-op).
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
|
||||
# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
|
||||
# SAME script.
|
||||
#
|
||||
# NOT proven — the runtime half, which needs a cluster: that the server is reachable, that
|
||||
# the export admits these clients, that the PVC binds, that a write succeeds, or that a
|
||||
# hardlink between the library and the download path actually works. The uid analysis is
|
||||
# reasoning (docs/BULK-NFS-STORAGE.md §6) and the in-cluster probes that would settle it
|
||||
# are written down in §9 of that document, unrun. Nor can this prove prod's address is
|
||||
# CORRECT — only that it is a supplied knob; while it is a placeholder, prod's pods will
|
||||
# not mount, and catching that statically needs the placeholder guard, which is absent from
|
||||
# main (§7). Read-only on the media tree is a CLIENT-side guarantee only: the export serves
|
||||
# one reader and two writers from the same node addresses, so the server cannot tell them
|
||||
# apart (§5).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: bulk-tier-distinct-from-replicated
|
||||
labels:
|
||||
conformance.home.k3s/R-STATE-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-STATE-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3e
|
||||
# (UNCONDITIONAL)", which this annotation replaces:
|
||||
#
|
||||
# The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only
|
||||
# thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four
|
||||
# hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated
|
||||
# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
|
||||
# three mini-PCs cannot hold.
|
||||
#
|
||||
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's path map keys
|
||||
# NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most
|
||||
# likely to break this — moving a bulk volume onto the replicated tier, or narrowing one
|
||||
# export into two mounts — could attribute no requirement at all and be gated by nothing.
|
||||
# The invariant spans catalog/, apps/ and clusters/.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Bulk media storage is provably NOT on the 3-replica Longhorn tier, is sized for a
|
||||
different order of magnitude, cannot be silently replaced by a dynamically
|
||||
provisioned empty volume, is never reclaimable by the cluster, mounts hard over
|
||||
NFSv4, and is never split into two mounts of one export — so hardlinked imports keep
|
||||
working. Its server is a per-environment knob, not a literal, because the two
|
||||
clusters reach the same storage host by different addresses.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-bulk-tier-is-distinct-and-statically-bound
|
||||
description: >-
|
||||
Read every `class: bulk` storage entry from catalog/, pair it with the
|
||||
PersistentVolume, PersistentVolumeClaim and volumeMount the committed apps/ goldens
|
||||
carry for it, and assert the tier separation, the static pre-binding, the reclaim
|
||||
and mount-option decisions, the read-only coupling, the one-export-one-mount
|
||||
invariant, and that the server address is supplied by every environment overlay.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,496 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — bulk-tier-distinct-from-replicated
|
||||
# Serves: R-STATE-04 ("Bulk media storage has a capacity strategy distinct from the
|
||||
# replicated tier." — spec docs/04-storage-and-state.md)
|
||||
#
|
||||
# WHY THIS EXISTS — THE BULK TIER WAS A STUB FOR SIX ROADMAP PHASES, AND STUBS ROT QUIETLY
|
||||
# ----------------------------------------------------------------------------------------
|
||||
# `platform/longhorn/storageclasses.yaml` declared a `bulk` StorageClass from Phase 0 with
|
||||
# `provisioner: nfs.csi.k8s.io`, `server: REPLACE-ME`, `share: REPLACE-ME` and a TODO(OQ-2).
|
||||
# Four ServiceDefinitions carried a comment promising to attach their media "once OQ-2 is
|
||||
# resolved", and each of them said, correctly, that adding `class: bulk` before then would
|
||||
# leave the PVC unbound and the pod Pending. So the tier existed as a NAME with nothing
|
||||
# behind it, and the only thing preventing a multi-terabyte media library from being
|
||||
# authored onto the 3-replica `longhorn` class was four hand-written comments.
|
||||
#
|
||||
# That is the failure this check makes impossible to reintroduce. R-STATE-04 is not "bulk
|
||||
# storage exists" — it is that bulk storage's capacity strategy is DISTINCT from the
|
||||
# replicated tier's. Three mini-PCs cannot hold 3x of 8.33 TiB. A bulk volume that lands on
|
||||
# `longhorn` does not fail loudly: Longhorn accepts the PVC and starts replicating, and the
|
||||
# tier fills up. Distinctness is therefore a property worth asserting statically, on every
|
||||
# gate run, rather than trusting to review.
|
||||
#
|
||||
# It also pins the four decisions that make the STATIC form of this tier safe, each of
|
||||
# which has a plausible-looking "simplification" that silently breaks something:
|
||||
#
|
||||
# * RECLAIM POLICY. Bulk data pre-exists the cluster, is larger than the cluster, and has
|
||||
# consumers OUTSIDE it. `Retain` is the whole reason a PV is used here instead of an
|
||||
# inline volume. `Delete` on a dynamically-provisioned bulk class would make `kubectl
|
||||
# delete pvc` a data-loss operation.
|
||||
# * NO DYNAMIC PROVISIONING. A dynamic bulk class hands out FRESH EMPTY directories. The
|
||||
# pod mounts successfully, the PVC is Bound, everything is green — and the 8 TiB of
|
||||
# existing data was never attached. This is the single most deceptive failure available
|
||||
# in this tier, so `storageClassName` must be explicitly empty and binding must be
|
||||
# pinned from BOTH ends (claimRef + volumeName).
|
||||
# * `hard`, NOT `soft`. A soft NFS mount converts a server hiccup into a short read or
|
||||
# EIO handed to the application. For a media library that is silent corruption. `hard`
|
||||
# hangs instead — recoverable, where truncated is not.
|
||||
# * NFSv4. v4 carries mount and lock in-protocol, so the tier needs ONE reachable port
|
||||
# (tcp/2049) instead of v3's rpcbind/111 + mountd + statd on ephemeral numbers. That is
|
||||
# a FIREWALL fact as much as a storage one: prod reaches this server through pfSense,
|
||||
# and the number of ports the operator must open is a consequence of this field.
|
||||
#
|
||||
# AND ONE INVARIANT THAT IS NOT ABOUT STORAGE AT ALL — the EXDEV trap (assertion 6).
|
||||
# Sonarr and Radarr import by hardlinking or atomically renaming a finished download into
|
||||
# the library. Both are same-filesystem operations, and to the kernel "same filesystem"
|
||||
# means SAME MOUNT. Mounting one export twice (say `subPath: tv` and `subPath: downloads`)
|
||||
# looks TIGHTER — less of the tree exposed per mount — and breaks `link()` with EXDEV no
|
||||
# matter that the server knows they are one dataset. The arrs do not fail loudly on that:
|
||||
# they degrade to copy-then-delete, rewriting every imported file over NFS and destroying
|
||||
# the seeding copy. The symptom appears weeks later as "imports are slow and my ratio is
|
||||
# gone". A reviewer cannot see it, so this check does: one export, at most one mount, per
|
||||
# workload.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# WHEN every `class: bulk` storage entry in catalog/ is compared against the manifests
|
||||
# render.py produces for it (read from the committed apps/ goldens, i.e. what Flux
|
||||
# actually reconciles — not from the renderer, which would only agree with itself),
|
||||
# THEN (1) DISTINCTNESS: no bulk volume is on the replicated tier. Its PV/PVC
|
||||
# storageClassName is explicitly empty — never `longhorn`, never any class the
|
||||
# repo declares with the Longhorn provisioner — and its PV's source is an
|
||||
# EXTERNAL one (nfs), not a Longhorn volume;
|
||||
# (2) CAPACITY STRATEGY: every bulk PV declares a capacity, and for each service the
|
||||
# bulk volume is at least an ORDER OF MAGNITUDE larger than that service's own
|
||||
# replicated volume. This is the requirement stated as arithmetic: the two tiers
|
||||
# exist because the data has two magnitudes, so a "bulk" volume sized like a
|
||||
# config PVC means the strategy was named and not actually applied;
|
||||
# (3) NOT DYNAMICALLY PROVISIONED, AND PRE-BOUND BOTH WAYS: every bulk PVC names its
|
||||
# PV by volumeName, every bulk PV names its PVC by claimRef (namespace+name), and
|
||||
# the two agree — so no other claim can win a race, and nothing can provision an
|
||||
# empty replacement;
|
||||
# (4) RECLAIM: every bulk PV is `Retain`;
|
||||
# (5) MOUNT SAFETY: every bulk PV declares mountOptions carrying `hard` and an
|
||||
# `nfsvers=4.x`, and carrying NEITHER `soft` NOR an `nfsvers=3`;
|
||||
# (6) ONE EXPORT, ONE MOUNT PER WORKLOAD (the EXDEV trap above);
|
||||
# (7) READ-ONLY IS REAL AND CONSISTENT: a bulk entry declared readOnly renders BOTH
|
||||
# `spec.nfs.readOnly: true` on the PV AND `readOnly: true` on the volumeMount,
|
||||
# and its access mode is ReadOnlyMany; a writable one renders NEITHER and is
|
||||
# ReadWriteMany. Every bulk access mode is a `...Many` — one export is consumed
|
||||
# by several namespaces and several nodes, which ReadWriteOnce cannot express;
|
||||
# (8) AUTHORED ONCE, PROMOTED (the R-ENV-02 seam this tier depends on): every bulk
|
||||
# PV's `nfs.server` is the BARE cluster-vars token, never a literal address; that
|
||||
# knob has a base default; and EVERY environment overlay supplies it — so the two
|
||||
# clusters, which reach this storage by different addresses, cannot be served by
|
||||
# one hardcoded value;
|
||||
# AND (9) LOAD-BEARING + NON-VACUOUS: this check DECLARES `conformance.home.k3s/scope:
|
||||
# whole-repo`, so ci/l0.sh runs it unconditionally via --include-scope, and
|
||||
# ci/gate.sh blocks the merge when the l0 stage fails; and the corpus is NOT
|
||||
# empty — a repo with zero bulk volumes fails here rather than passing trivially,
|
||||
# because a check that is satisfiable by deleting the feature proves nothing.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps)
|
||||
# - Nothing about RUNTIME. Not that the server is reachable, not that the export permits
|
||||
# these clients, not that the PVC binds, not that a write succeeds. The uid analysis in
|
||||
# docs/BULK-NFS-STORAGE.md §6 is reasoning, and the write/hardlink probes in §9 are the
|
||||
# in-cluster proof — they need a cluster and are not run here.
|
||||
# - It cannot prove the prod address is CORRECT, only that it is a supplied knob. While
|
||||
# prod's value is a placeholder, prod's pods will not mount (docs/BULK-NFS-STORAGE.md
|
||||
# §7); catching that needs the placeholder guard, which is absent from main.
|
||||
# - Read-only for the media tree is a CLIENT-side guarantee, and this check can only
|
||||
# assert the client side. The export serves a reader and two writers from the same node
|
||||
# addresses, so the server cannot distinguish them (docs/BULK-NFS-STORAGE.md §5).
|
||||
# - Assertion (2)'s magnitude ratio is a heuristic with a deliberately generous floor. It
|
||||
# catches "bulk volume sized like a config volume", not a subtly wrong capacity — and a
|
||||
# static NFS PV's capacity is advisory anyway, since the ZFS pool is the real limit.
|
||||
# - Assertion (6) reasons over ONE workload's mounts. It cannot see two different
|
||||
# services hardlinking between each other's mounts, which is not a thing that happens.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
|
||||
if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ]; then
|
||||
echo "check SKIPPED (bulk-tier-distinct-from-replicated): catalog/ or apps/ absent — nothing to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def bail(msg):
|
||||
"""A failure that makes the rest of the check meaningless: report and stop."""
|
||||
print(f"CHECK FAIL (bulk-tier-distinct-from-replicated): {msg}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def qty_to_bytes(q):
|
||||
"""Kubernetes quantity -> bytes. Only the binary suffixes the schema permits."""
|
||||
m = re.fullmatch(r"([0-9]+)(Mi|Gi|Ti)", str(q).strip())
|
||||
if not m:
|
||||
return None
|
||||
return int(m.group(1)) * {"Mi": 2 ** 20, "Gi": 2 ** 30, "Ti": 2 ** 40}[m.group(2)]
|
||||
|
||||
|
||||
# =====================================================================================
|
||||
# The replicated tier, named by the repo rather than by this file. Any StorageClass the
|
||||
# repo declares with the Longhorn provisioner is "the replicated tier" for assertion (1);
|
||||
# deriving it means adding a new Longhorn class cannot create a blind spot here.
|
||||
# =====================================================================================
|
||||
REPLICATED_CLASSES: set[str] = set()
|
||||
LONGHORN_DIR = ROOT / "platform" / "longhorn"
|
||||
if LONGHORN_DIR.is_dir():
|
||||
for f in sorted(LONGHORN_DIR.rglob("*.yaml")):
|
||||
for doc in yaml.safe_load_all(f.read_text()):
|
||||
if isinstance(doc, dict) and doc.get("kind") == "StorageClass":
|
||||
if "longhorn" in str(doc.get("provisioner", "")):
|
||||
REPLICATED_CLASSES.add(doc.get("metadata", {}).get("name", ""))
|
||||
# `longhorn` is the name every ServiceDefinition uses for the replicated tier, so it is in
|
||||
# the set even if platform/longhorn/ is not present in this checkout.
|
||||
REPLICATED_CLASSES.add("longhorn")
|
||||
REPLICATED_CLASSES.discard("")
|
||||
|
||||
# =====================================================================================
|
||||
# The bulk corpus: every `class: bulk` storage entry in every ServiceDefinition, paired
|
||||
# with the objects the COMMITTED goldens carry for it. Reading the goldens (not calling
|
||||
# the renderer) is deliberate: the goldens are what Flux reconciles, and render==golden is
|
||||
# a separate gate stage. Asking the renderer would only prove it agrees with itself.
|
||||
# =====================================================================================
|
||||
sds: dict[str, dict] = {}
|
||||
for f in sorted((ROOT / "catalog").glob("*.yaml")):
|
||||
doc = yaml.safe_load(f.read_text())
|
||||
if isinstance(doc, dict) and doc.get("name"):
|
||||
sds[doc["name"]] = doc
|
||||
|
||||
if not sds:
|
||||
bail("no ServiceDefinitions under catalog/ — cannot audit the bulk tier")
|
||||
|
||||
# Every rendered app-tier document, indexed by (kind, name).
|
||||
rendered: dict[tuple, dict] = {}
|
||||
for f in sorted((ROOT / "apps").rglob("*.yaml")):
|
||||
if f.name == "kustomization.yaml" or ".sops." in f.name:
|
||||
continue
|
||||
try:
|
||||
docs = list(yaml.safe_load_all(f.read_text()))
|
||||
except yaml.YAMLError:
|
||||
continue
|
||||
for doc in docs:
|
||||
if isinstance(doc, dict) and doc.get("kind") and doc.get("metadata", {}).get("name"):
|
||||
rendered[(doc["kind"], doc["metadata"]["name"])] = doc
|
||||
|
||||
# (svc, vol_entry) for every bulk entry, and the replicated sizes alongside for (2).
|
||||
corpus: list[tuple[str, dict]] = []
|
||||
replicated_bytes: dict[str, list[int]] = {}
|
||||
for svc, sd in sds.items():
|
||||
for vol in sd.get("storage") or []:
|
||||
cls = vol.get("class", "longhorn")
|
||||
if cls == "bulk":
|
||||
corpus.append((svc, vol))
|
||||
else:
|
||||
b = qty_to_bytes(vol.get("size"))
|
||||
if b:
|
||||
replicated_bytes.setdefault(svc, []).append(b)
|
||||
|
||||
# ------------------------------------------------------------------- (9) NON-VACUOUS
|
||||
# A corpus of zero would make every loop below trivially satisfied. The bulk tier is
|
||||
# DELIVERED; a change that removes the last bulk volume must fail here and be argued for,
|
||||
# not sail through on an empty iteration.
|
||||
check(len(corpus) > 0,
|
||||
"NO `class: bulk` storage entry exists in catalog/ — this check would then pass "
|
||||
"vacuously, which is exactly the posture that let the bulk tier sit as a REPLACE-ME "
|
||||
"stub for six roadmap phases. If the tier is genuinely being retired, retire this "
|
||||
"check in the same change and say so; do not let it pass on an empty corpus")
|
||||
if not corpus:
|
||||
bail("empty bulk corpus — refusing to report a vacuous pass (see assertion 9)")
|
||||
|
||||
inventory: list[str] = []
|
||||
|
||||
for svc, vol in corpus:
|
||||
vname = vol.get("name", "<unnamed>")
|
||||
label = f"{svc}/{vname}"
|
||||
obj = f"{svc}-{vname}"
|
||||
ro = bool(vol.get("readOnly"))
|
||||
|
||||
# `export` is what the schema requires of a bulk entry; without it there is nothing to
|
||||
# attach and the rest of the assertions have no subject.
|
||||
export = vol.get("export")
|
||||
check(bool(export) and str(export).startswith("/"),
|
||||
f"{label}: bulk entry has no absolute `export` path — a bulk volume attaches an "
|
||||
f"EXISTING dataset, so it must say which one (schema requires it when class is bulk)")
|
||||
|
||||
pv = rendered.get(("PersistentVolume", obj))
|
||||
pvc = rendered.get(("PersistentVolumeClaim", obj))
|
||||
check(pv is not None,
|
||||
f"{label}: no PersistentVolume '{obj}' in the committed apps/ goldens. A bulk "
|
||||
f"entry that renders no PV has NO external storage behind it — the pod would get "
|
||||
f"an unbound PVC, or (worse, with a dynamic class) a fresh EMPTY directory")
|
||||
check(pvc is not None,
|
||||
f"{label}: no PersistentVolumeClaim '{obj}' in the committed apps/ goldens")
|
||||
if pv is None or pvc is None:
|
||||
continue
|
||||
|
||||
pvs, pvcs = pv.get("spec", {}) or {}, pvc.get("spec", {}) or {}
|
||||
|
||||
# ------------------------------------------------------ (1) DISTINCT FROM REPLICATED
|
||||
for what, spec in (("PV", pvs), ("PVC", pvcs)):
|
||||
sc = spec.get("storageClassName")
|
||||
check(sc not in REPLICATED_CLASSES,
|
||||
f"{label}: bulk {what} storageClassName is '{sc}', which the repo declares "
|
||||
f"with the Longhorn provisioner. That puts BULK data on the REPLICATED tier "
|
||||
f"— 3x of a multi-terabyte library across three mini-PCs. Longhorn will not "
|
||||
f"refuse it; it will accept the PVC and fill the tier (R-STATE-04)")
|
||||
check(sc == "",
|
||||
f"{label}: bulk {what} storageClassName is {sc!r}, not the explicitly EMPTY "
|
||||
f"string. Empty is what disables dynamic provisioning AND the default-class "
|
||||
f"annotation. Any named class risks a provisioner handing out a fresh EMPTY "
|
||||
f"directory that mounts and binds cleanly while the real data is never attached")
|
||||
check("nfs" in pvs,
|
||||
f"{label}: bulk PV declares no external `nfs` source — the bulk tier's whole "
|
||||
f"premise is that durability lives OUTSIDE the cluster (docs/04)")
|
||||
|
||||
# ------------------------------------------------------------ (4) RECLAIM POLICY
|
||||
check(pvs.get("persistentVolumeReclaimPolicy") == "Retain",
|
||||
f"{label}: bulk PV reclaim policy is "
|
||||
f"{pvs.get('persistentVolumeReclaimPolicy')!r}, not Retain. This data pre-exists "
|
||||
f"the cluster and has consumers outside it; anything but Retain makes `kubectl "
|
||||
f"delete pvc` a data-loss operation")
|
||||
|
||||
# ------------------------------------------------- (3) PRE-BOUND FROM BOTH ENDS
|
||||
check(pvcs.get("volumeName") == obj,
|
||||
f"{label}: bulk PVC does not name its PV via volumeName (got "
|
||||
f"{pvcs.get('volumeName')!r}, expected {obj!r}) — without it the claim can be "
|
||||
f"satisfied by some OTHER volume of matching size and mode")
|
||||
cref = pvs.get("claimRef") or {}
|
||||
check(cref.get("name") == obj and cref.get("namespace") == svc,
|
||||
f"{label}: bulk PV claimRef is {cref!r}, expected namespace={svc!r} name={obj!r} "
|
||||
f"— pre-binding must be pinned from BOTH ends or another PVC can win the race")
|
||||
|
||||
# ------------------------------------------------------------- (5) MOUNT SAFETY
|
||||
mopts = [str(o) for o in (pvs.get("mountOptions") or [])]
|
||||
check(bool(mopts),
|
||||
f"{label}: bulk PV declares no mountOptions — then `hard` vs `soft` and the NFS "
|
||||
f"version are whatever the client happens to default to, and both are decisions "
|
||||
f"with consequences (silent short reads; how many firewall ports must open)")
|
||||
check("hard" in mopts,
|
||||
f"{label}: bulk PV mountOptions do not include `hard` ({mopts}). A soft mount "
|
||||
f"hands a server hiccup to the application as a short read or EIO — silent "
|
||||
f"corruption for a media library. `hard` hangs instead, which is recoverable")
|
||||
check("soft" not in mopts,
|
||||
f"{label}: bulk PV mountOptions include `soft` ({mopts}) — see above; this is the "
|
||||
f"one option that turns an outage into damaged data")
|
||||
vers = [o for o in mopts if o.startswith("nfsvers=")]
|
||||
check(len(vers) == 1 and re.fullmatch(r"nfsvers=4(\.[0-9]+)?", vers[0]),
|
||||
f"{label}: bulk PV must pin exactly one NFSv4 `nfsvers=` option (got {vers}). "
|
||||
f"NFSv4 carries mount and lock in-protocol, so the tier needs ONE reachable port "
|
||||
f"(tcp/2049); v3 also needs rpcbind/111, mountd and statd, which changes what the "
|
||||
f"operator has to open in pfSense for prod to reach this server at all")
|
||||
|
||||
# ---------------------------------------------- (7) READ-ONLY IS REAL + CONSISTENT
|
||||
want_mode = "ReadOnlyMany" if ro else "ReadWriteMany"
|
||||
for what, spec in (("PV", pvs), ("PVC", pvcs)):
|
||||
modes = spec.get("accessModes") or []
|
||||
check(modes == [want_mode],
|
||||
f"{label}: bulk {what} accessModes {modes} != ['{want_mode}']. Every bulk "
|
||||
f"volume is a `...Many` (one export, several namespaces, several nodes — "
|
||||
f"ReadWriteOnce cannot express that), and WHICH `Many` is how the read/write "
|
||||
f"posture stays legible in `kubectl get pv`")
|
||||
nfs_ro = bool((pvs.get("nfs") or {}).get("readOnly"))
|
||||
check(nfs_ro == ro,
|
||||
f"{label}: ServiceDefinition readOnly={ro} but the PV's nfs.readOnly={nfs_ro}. "
|
||||
f"The PV field is what makes the kernel mount `ro`; a mismatch means the declared "
|
||||
f"posture is not the one the kernel enforces")
|
||||
|
||||
# The volumeMount half, read out of the rendered Deployment.
|
||||
dep = rendered.get(("Deployment", svc))
|
||||
check(dep is not None, f"{label}: no rendered Deployment '{svc}' to check the mount against")
|
||||
if dep is not None:
|
||||
containers = (((dep.get("spec") or {}).get("template") or {}).get("spec") or {}).get("containers") or []
|
||||
# The app container is containers[0] by the renderer's contract (a forward-auth
|
||||
# sidecar is appended AFTER it).
|
||||
mounts = [m for m in (containers[0].get("volumeMounts") or []) if m.get("name") == vname] if containers else []
|
||||
check(len(mounts) == 1,
|
||||
f"{label}: the app container has {len(mounts)} volumeMount(s) named {vname!r}, "
|
||||
f"expected exactly 1 — a bulk volume that is declared and never mounted is a "
|
||||
f"PV/PVC pair doing nothing, and two mounts of one volume is assertion (6)")
|
||||
if len(mounts) == 1:
|
||||
m = mounts[0]
|
||||
check(bool(m.get("readOnly")) == ro,
|
||||
f"{label}: volumeMount readOnly={bool(m.get('readOnly'))} but the "
|
||||
f"ServiceDefinition says readOnly={ro} — the PV and the mount must agree, "
|
||||
f"so that dropping one still leaves the other")
|
||||
check(m.get("subPath", vol.get("subPath")) == vol.get("subPath"),
|
||||
f"{label}: volumeMount subPath {m.get('subPath')!r} does not match the "
|
||||
f"declared subPath {vol.get('subPath')!r}")
|
||||
|
||||
# ------------------------------------------ (8) AUTHORED ONCE, SUPPLIED PER ENV
|
||||
server = (pvs.get("nfs") or {}).get("server")
|
||||
m = re.fullmatch(r"\$\{([A-Z0-9_]+)\}", str(server or ""))
|
||||
check(m is not None,
|
||||
f"{label}: bulk PV nfs.server is {server!r} — it must be a BARE cluster-vars "
|
||||
f"token, not a literal address. dev and prod reach this storage host by DIFFERENT "
|
||||
f"addresses (dev is on its LAN; prod is across pfSense), so a literal is correct "
|
||||
f"in at most one environment and silently wrong in the other (R-ENV-02)")
|
||||
|
||||
cap = (pvs.get("capacity") or {}).get("storage")
|
||||
req = ((pvcs.get("resources") or {}).get("requests") or {}).get("storage")
|
||||
check(cap is not None and qty_to_bytes(cap) is not None,
|
||||
f"{label}: bulk PV declares no parseable capacity ({cap!r}) — the API requires it, "
|
||||
f"and it is what makes `kubectl get pv` report something true about this tier")
|
||||
check(cap == req,
|
||||
f"{label}: PV capacity {cap!r} != PVC request {req!r}; a request above the "
|
||||
f"capacity never binds")
|
||||
|
||||
inventory.append(
|
||||
f" {label:<28} export={export} {'ro' if ro else 'rw'} {want_mode:<14} "
|
||||
f"cap={cap} server={server} opts={','.join(mopts)}"
|
||||
)
|
||||
|
||||
# ---------------------------------- (2) CAPACITY STRATEGY: two tiers, two magnitudes
|
||||
cap_b = qty_to_bytes(cap)
|
||||
reps = replicated_bytes.get(svc) or []
|
||||
if cap_b and reps:
|
||||
biggest = max(reps)
|
||||
# ONE ORDER OF MAGNITUDE. Calibrated against the real corpus rather than guessed:
|
||||
# the media library is ~460x its service's config volume, and the Nextcloud archive
|
||||
# ~13x its 20Gi Longhorn volume — so a floor above ~13x would fire on a legitimate
|
||||
# bulk volume, and this check would be teaching people to weaken it. 10x cannot be
|
||||
# met by accident and still catches the regression that matters: a "bulk" volume
|
||||
# sized like a config PVC, i.e. the strategy named but never applied.
|
||||
check(cap_b >= 10 * biggest,
|
||||
f"{label}: bulk capacity {cap} is less than 10x this service's largest "
|
||||
f"replicated volume ({biggest // 2 ** 30}Gi). R-STATE-04 exists because the "
|
||||
f"two tiers hold data of two different MAGNITUDES — the replicated tier keeps "
|
||||
f"the small config/DB and the bulk tier keeps what cannot be replicated 3x. A "
|
||||
f"bulk volume sized like a config volume means the tier was named, not applied")
|
||||
|
||||
# ------------------------------------------------- (6) ONE EXPORT, AT MOST ONE MOUNT
|
||||
# The EXDEV trap. Grouped per SERVICE, because the hazard is two mounts inside ONE pod.
|
||||
by_service: dict[str, list[dict]] = {}
|
||||
for svc, vol in corpus:
|
||||
by_service.setdefault(svc, []).append(vol)
|
||||
for svc, vols in sorted(by_service.items()):
|
||||
seen: dict[str, str] = {}
|
||||
for vol in vols:
|
||||
exp = vol.get("export")
|
||||
prev = seen.get(exp)
|
||||
check(prev is None,
|
||||
f"{svc}: mounts the export {exp} TWICE (volumes {prev!r} and "
|
||||
f"{vol.get('name')!r}). Two mounts of one export are two FILESYSTEMS to the "
|
||||
f"kernel, so link() between them fails EXDEV even though the server knows "
|
||||
f"they are one dataset. Sonarr/Radarr do not fail loudly on that — they "
|
||||
f"degrade to copy-then-delete, rewriting every imported file over NFS and "
|
||||
f"destroying the seeding copy. Mount the common ancestor ONCE instead "
|
||||
f"(docs/BULK-NFS-STORAGE.md §4)")
|
||||
seen[exp] = vol.get("name")
|
||||
|
||||
# ------------------------------------- (8b) the knob has a base default AND every overlay
|
||||
tokens = set()
|
||||
for _svc, vol in corpus:
|
||||
pv = rendered.get(("PersistentVolume", f"{_svc}-{vol.get('name')}"))
|
||||
if pv:
|
||||
mm = re.fullmatch(r"\$\{([A-Z0-9_]+)\}", str(((pv.get("spec") or {}).get("nfs") or {}).get("server") or ""))
|
||||
if mm:
|
||||
tokens.add(mm.group(1))
|
||||
|
||||
cv = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
||||
base_keys: set[str] = set()
|
||||
if cv.is_file():
|
||||
doc = yaml.safe_load(cv.read_text()) or {}
|
||||
base_keys = set((doc.get("data") or {}).keys())
|
||||
|
||||
overlays = sorted(
|
||||
d for d in (ROOT / "clusters").iterdir()
|
||||
if cv.is_file() and d.is_dir() and d.name != "base" and (d / "kustomization.yaml").is_file()
|
||||
) if (ROOT / "clusters").is_dir() else []
|
||||
|
||||
for tok in sorted(tokens):
|
||||
check(tok in base_keys,
|
||||
f"bulk PVs reference ${{{tok}}} but clusters/base/cluster-vars.yaml declares no "
|
||||
f"such knob. Under StrictPostBuildSubstitutions that fails the whole tier; without "
|
||||
f"it the token ships into the cluster as a literal server name")
|
||||
for ov in overlays:
|
||||
text = (ov / "kustomization.yaml").read_text()
|
||||
check(f"/data/{tok}" in text,
|
||||
f"clusters/{ov.name} does not supply ${{{tok}}}. The base default is the DEV "
|
||||
f"value by this repo's convention, so an overlay that omits it silently "
|
||||
f"inherits an address on the WRONG SIDE of pfSense — a mount that either "
|
||||
f"never connects or connects to something unintended (R-ENV-02)")
|
||||
|
||||
# --------------------------------------------------------------- (9) LOAD-BEARING
|
||||
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
|
||||
# It used to assert `"bulk-tier-distinct-from-replicated/check.sh" in L0`: ci/l0.sh carried a hand-written
|
||||
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
|
||||
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
|
||||
# to write it). The same property is now asserted two ways: this test DECLARES
|
||||
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
|
||||
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
|
||||
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
|
||||
# passing) is proven end-to-end and falsifiably by
|
||||
# conformance/tests/l0-static/whole-repo-invariants-wired.
|
||||
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "bulk-tier-distinct-from-replicated" / "chainsaw-test.yaml"
|
||||
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
|
||||
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
|
||||
"bulk-tier-distinct-from-replicated/chainsaw-test.yaml does not declare "
|
||||
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
|
||||
"when a change happens to touch one of the requirements it is labelled with. "
|
||||
"The path map keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement, so moving a multi-terabyte bulk volume onto the 3-replica tier could attribute no requirement at all and be gated by nothing")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
|
||||
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
|
||||
"this check would then run only by coincidence")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
# ------------------------------------------------------------------------- report
|
||||
print("Bulk tier inventory (from catalog/ + the committed apps/ goldens):")
|
||||
for line in inventory:
|
||||
print(line)
|
||||
print(f"\nreplicated tier StorageClasses (derived from platform/longhorn/): "
|
||||
f"{', '.join(sorted(REPLICATED_CLASSES))}")
|
||||
print(f"bulk volumes: {len(corpus)} distinct exports: "
|
||||
f"{len({v.get('export') for _s, v in corpus})} "
|
||||
f"server knob(s): {', '.join(sorted(tokens)) or 'NONE'}")
|
||||
|
||||
# A floor on the assertion count, for the same reason the corpus may not be empty: a pass
|
||||
# must not be able to degenerate into "nothing was actually asked".
|
||||
MIN_CHECKS = 40
|
||||
check(CHECKS >= MIN_CHECKS,
|
||||
f"only {CHECKS} assertions ran (floor {MIN_CHECKS}) — a pass this thin is not "
|
||||
f"evidence; the corpus or the extraction has degraded")
|
||||
|
||||
print()
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (bulk-tier-distinct-from-replicated): {len(FAILURES)} finding(s)",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
print(f"check OK: bulk-tier-distinct-from-replicated — {CHECKS} assertions (R-STATE-04)")
|
||||
PY
|
||||
@@ -0,0 +1,83 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — dev/test nodes are provisioned as code, end to end (R-ENV-01) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC:
|
||||
# created, baselined, cluster-joined, and managed as code"); the four stages
|
||||
# are enumerated in docs/02 §"Node lifecycle stages".
|
||||
#
|
||||
# WHY L0: R-ENV-01 is a requirement about HOW the dev nodes came to exist, and a live
|
||||
# Node object cannot answer it — a node built by OpenTofu and one built by hand in a
|
||||
# hypervisor UI are indistinguishable through the Kubernetes API. What is provable, with
|
||||
# no cluster, is that a COMPLETE and EXECUTABLE provisioning path exists for all four
|
||||
# stages and covers EVERY dev node. For the Ansible half that is proven by DRIVING the
|
||||
# real toolchain offline (`--syntax-check`, `--list-hosts` — neither opens a connection
|
||||
# to any node), not by reading files: a chain that does not resolve fails the check.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count appears in the test.
|
||||
# The dev group names are read from the playbooks' own default host patterns, the node
|
||||
# names are generated from the IaC module's prefix/count variables, and the two sets are
|
||||
# compared against each other — so the assertion survives a rename or a resize.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the provisioning chain is parsed and (where offline execution is possible) driven,
|
||||
# THEN the IaC module creates the dev nodes on a LOCAL KVM hypervisor (allowlisted
|
||||
# provider — a cloud provider fails), from a for_each map rather than a count index,
|
||||
# each with CPU/RAM/OS disk/dedicated Longhorn data disk/NIC/cloud-init seed; the
|
||||
# node set it creates is EXACTLY the set Ansible manages; the baseline stage applies
|
||||
# the package floor, kernel modules, sysctls, data-disk mount, swap-off and iscsid
|
||||
# declaratively; one k3s role founds the cluster with --cluster-init and joins the
|
||||
# rest with --server while disabling flannel/kube-proxy/traefik/servicelb at
|
||||
# install; and site.yml's chain (baseline -> k3s -> Cilium -> verify -> Flux)
|
||||
# resolves and targets every dev node at every stage.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
|
||||
# where present — ansible-playbook in no-connection modes; no cluster, no network);
|
||||
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
|
||||
# wraps the SAME script.
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * That the RUNNING dev VMs were created by this module — needs the OpenTofu state or
|
||||
# hypervisor access, neither available offline.
|
||||
# * `tofu validate` — needs `tofu init` (provider registry), i.e. network. The module is
|
||||
# parsed, not evaluated; a network-allowed CI stage would close this.
|
||||
# * "Never hand-edited on the box" (docs/02 stage 4's second half) — a live drift check.
|
||||
# * REBUILD/re-converge (docs/02's own proof line) — exercised at L3 by
|
||||
# tests/l3-fault-injection/node-loss (R-STATE-02).
|
||||
# * docs/02 prose puts the baseline in cloud-init; the implementation moved it into the
|
||||
# Ansible node_baseline role. R-ENV-01 requires the node to be "baselined ... as code",
|
||||
# not that cloud-init be the tool, so the check asserts the STAGE and RECORDS the
|
||||
# divergence rather than hiding it.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: dev-nodes-provisioned-as-code
|
||||
labels:
|
||||
conformance.home.k3s/R-ENV-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as
|
||||
# such (2026-09-12). Its entire input surface is infra/ (OpenTofu + the Ansible roles that
|
||||
# provision the dev nodes); it reads nothing else. The gap was that
|
||||
# ci/touched-requirements.sh keyed infra/* to NOTHING, so an infra/ edit attributed no
|
||||
# requirement and this check was filtered out. The right fix for a path-scoped invariant is
|
||||
# a path-map entry, not an unconditional run: `infra/*) echo "R-ENV-01"` was added there in
|
||||
# the same change. Keep the two in sync — if this check ever grows an assertion outside
|
||||
# infra/, either widen that map entry or give this test a whole-repo scope.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree, when the dev provisioning chain is parsed and driven
|
||||
offline, then every dev node is created by local-KVM IaC, baselined, cluster-joined
|
||||
and handed to Flux by one code path that covers all of them — with no node outside
|
||||
it and no stage missing.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-every-dev-node-is-created-baselined-joined-and-managed-as-code
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,442 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — dev-nodes-provisioned-as-code
|
||||
# Serves: R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: created,
|
||||
# baselined, cluster-joined, and managed as code." — spec
|
||||
# docs/00-vision-and-scope.md; the four stages are enumerated in
|
||||
# docs/02-environments-and-provisioning.md §"Node lifecycle stages
|
||||
# (R-ENV-01 — 'created, baselined, cluster-joined, managed as code')":
|
||||
# 1. Create — OpenTofu instantiates the VM
|
||||
# 2. Baseline — OS baseline and prerequisites
|
||||
# 3. Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables
|
||||
# k3s bundled flannel/traefik/servicelb/kube-proxy
|
||||
# 4. Manage — GitOps-managed by Flux; node config changes are made via IaC
|
||||
# and re-applied, never hand-edited on the box.)
|
||||
#
|
||||
# WHY L0 IS THE RIGHT LAYER, AND WHAT IT MEANS TO TEST THIS BEHAVIOURALLY
|
||||
# ----------------------------------------------------------------------
|
||||
# R-ENV-01 is a requirement about HOW the dev nodes come to exist. The live cluster
|
||||
# cannot answer it: a Kubernetes node object looks identical whether it was built by
|
||||
# OpenTofu or by hand in a hypervisor UI (there is no API field for "was I created by
|
||||
# IaC"). What CAN be proven, with no cluster, is that a COMPLETE, EXECUTABLE
|
||||
# provisioning path exists for all four stages and covers every dev node — and, for the
|
||||
# Ansible half, that is proven by RUNNING the real toolchain in a no-connection mode
|
||||
# (`--syntax-check`, `--list-hosts`) rather than by reading the files, so a chain that
|
||||
# does not actually resolve fails here. The IaC half is asserted by parsing the module:
|
||||
# `tofu init/validate` needs the provider registry, and L0 is offline by contract.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the provisioning chain is parsed and (where offline execution is possible)
|
||||
# actually driven,
|
||||
# THEN CREATE: the IaC module targets a LOCAL KVM hypervisor (every declared provider
|
||||
# is a KVM-family hypervisor provider — no cloud provider may appear),
|
||||
# instantiates its nodes from a for_each map (never a count index, so one
|
||||
# node can be replaced without disturbing its siblings), and gives each
|
||||
# node CPU, RAM, an OS disk, a DEDICATED Longhorn data disk, a NIC and a
|
||||
# cloud-init seed carrying the login user + SSH key;
|
||||
# AND the node set it creates is EXACTLY the dev node set Ansible manages —
|
||||
# names derived from the module's own prefix/count, so no node can be
|
||||
# hand-added to the inventory or hand-created in the hypervisor;
|
||||
# BASELINE: a baseline stage exists as declarative Ansible (package floor, kernel
|
||||
# modules persisted AND loaded, sysctls, the Longhorn data disk formatted
|
||||
# and mounted, swap off, iscsid enabled), driven by variables with
|
||||
# defaults rather than per-host special cases;
|
||||
# JOIN: one role installs k3s for both roles of every node — `--cluster-init`
|
||||
# on the primary, `--server` join on the secondaries — with flannel,
|
||||
# kube-proxy, traefik and servicelb disabled AT INSTALL, and Cilium
|
||||
# installed as the CNI afterwards;
|
||||
# MANAGE: site.yml chains baseline -> k3s -> Cilium -> verify -> Flux bootstrap;
|
||||
# `ansible-playbook --syntax-check` on that chain SUCCEEDS; and
|
||||
# `--list-hosts` shows every one of those stages targeting EVERY dev
|
||||
# node, so no node is outside the code path.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count is written down here.
|
||||
# The dev group names come from the playbooks' own default host patterns, the node names
|
||||
# from the IaC module's prefix/count variables, and the two sets are compared to each
|
||||
# other. This check therefore keeps working when the estate is renamed or resized.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
|
||||
# - That the RUNNING dev VMs were in fact created by this module. Nothing offline can:
|
||||
# it would need the OpenTofu state (a remote R2 backend + credentials) or hypervisor
|
||||
# access. `tofu plan` against the live estate is the operator's own check.
|
||||
# - That the module APPLIES cleanly. `tofu validate` needs `tofu init`, which downloads
|
||||
# the provider from the registry; L0 is offline by contract, so the module is parsed,
|
||||
# not evaluated. ci/lint.sh does not cover .tf either — a gap worth closing in CI
|
||||
# with a network-allowed stage, not here.
|
||||
# - "Never hand-edited on the box" (stage 4's second half) — a drift check against the
|
||||
# live nodes, which is an operator/Ansible-check-mode exercise, not a repo property.
|
||||
# - REBUILD/re-converge ("destroy a dev node VM and rebuild it", docs/02's own proof
|
||||
# line) — that is exercised at L3 by tests/l3-fault-injection/node-loss (R-STATE-02).
|
||||
# - That the baseline actually ran on a node (cluster/host state), and that the
|
||||
# hypervisor is on the operator's own LAN rather than remote — "local" is asserted as
|
||||
# "a KVM hypervisor provider driving a self-hosted endpoint variable", not as a
|
||||
# network topology fact.
|
||||
# - DEVIATION FROM docs/02 PROSE, ASSERTED DELIBERATELY: docs/02 describes stage 2 as
|
||||
# cloud-init applying the OS baseline. The implementation moved the rich baseline into
|
||||
# the Ansible `node_baseline` role (cloud-init keeps hostname/user/SSH/network),
|
||||
# because Proxmox provisioning is API-only and a custom cloud-init snippet needs
|
||||
# SSH/SFTP to the hypervisor. R-ENV-01's own wording requires the node to be
|
||||
# "baselined ... as code", not that cloud-init be the tool, so this check asserts the
|
||||
# STAGE, wherever it lives, and records the divergence rather than hiding it.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
NOTES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def note(msg):
|
||||
NOTES.append(msg)
|
||||
|
||||
|
||||
# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
|
||||
# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
|
||||
# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
|
||||
# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
|
||||
# narrow: every other non-zero exit is a real failure. The parse assertions always run.
|
||||
ENV_ONLY_FAILURE = re.compile(
|
||||
r"vault password|vault-id|Attempting to decrypt"
|
||||
r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
|
||||
|
||||
TOFU = ROOT / "infra" / "tofu"
|
||||
ANS = ROOT / "infra" / "ansible"
|
||||
check(TOFU.is_dir(), "infra/tofu/ is missing — the CREATE stage has no IaC at all (R-ENV-01)")
|
||||
check(ANS.is_dir(), "infra/ansible/ is missing — the BASELINE/JOIN/MANAGE stages have no code")
|
||||
if not TOFU.is_dir() or not ANS.is_dir():
|
||||
print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
TF = "\n".join((p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))))
|
||||
check(len(TF) > 500, "infra/tofu/*.tf is empty or a stub — nothing creates the dev nodes")
|
||||
|
||||
# ============================================================ STAGE 1 — CREATE (IaC)
|
||||
# A LOCAL KVM hypervisor, by allowlist: any provider outside this set fails, so moving
|
||||
# dev onto a cloud (or anything that is not a KVM hypervisor the operator runs) is caught
|
||||
# here rather than being noticed later. Each entry is justified, not just listed:
|
||||
# bpg/proxmox Proxmox VE — a KVM/QEMU hypervisor the operator hosts.
|
||||
# dmacvicar/libvirt libvirt/KVM directly (the pre-Proxmox substrate).
|
||||
KVM_PROVIDERS = {"bpg/proxmox", "dmacvicar/libvirt"}
|
||||
providers = dict(re.findall(r'source\s*=\s*"([^"]+)"', TF) and
|
||||
[(m.group(1), m.group(2)) for m in
|
||||
re.finditer(r'(\w[\w-]*)\s*=\s*\{[^}]*?source\s*=\s*"([^"]+)"', TF, re.S)])
|
||||
check(len(providers) > 0,
|
||||
"infra/tofu declares no required_providers — the substrate is unpinned, so 'via IaC' "
|
||||
"cannot be checked at all")
|
||||
for alias, source in sorted(providers.items()):
|
||||
check(source.lower() in KVM_PROVIDERS,
|
||||
f"infra/tofu declares provider {alias}={source!r}, which is not a local KVM hypervisor "
|
||||
f"provider (allowed: {sorted(KVM_PROVIDERS)}) — R-ENV-01 requires dev/test to be LOCAL KVMs")
|
||||
|
||||
# The VM resource: for_each-keyed, and carrying the hardware + seed docs/02 enumerates.
|
||||
vm_blocks = re.findall(r'resource\s+"([\w]*(?:_vm|_domain))"\s+"(\w+)"\s*\{(.*?)\n\}',
|
||||
TF, re.S)
|
||||
check(len(vm_blocks) >= 1,
|
||||
"no VM/domain resource found in infra/tofu — nothing in IaC creates a dev node")
|
||||
for rtype, rname, body in vm_blocks:
|
||||
check("for_each" in body,
|
||||
f'resource "{rtype}" "{rname}" is not for_each-keyed. A count index makes single-node '
|
||||
"replace unsafe (a -target'ed destroy cascaded to sibling nodes — DECISIONS-LOG F1), so "
|
||||
"'created ... as code' would not survive a rebuild")
|
||||
check(not re.search(r"^\s*count\s*=", body, re.M),
|
||||
f'resource "{rtype}" "{rname}" uses a positional count index')
|
||||
for want, why in (
|
||||
(r"\bcpu\b", "CPU"),
|
||||
(r"\bmemory\b", "RAM"),
|
||||
(r"network_device|network_interface", "a NIC"),
|
||||
(r"initialization|cloud_init|cloudinit", "a cloud-init seed"),
|
||||
):
|
||||
check(re.search(want, body) is not None,
|
||||
f'resource "{rtype}" "{rname}" declares no {why} — docs/02 stage 1 requires CPU, RAM, '
|
||||
"disks, NICs and a cloud-init seed to come from IaC")
|
||||
disks = len(re.findall(r"^\s*disk\s*\{", body, re.M))
|
||||
check(disks >= 2,
|
||||
f'resource "{rtype}" "{rname}" declares {disks} disk block(s); docs/02 requires the OS '
|
||||
"disk PLUS a dedicated data disk for Longhorn")
|
||||
check(re.search(r"user_account|ssh_authorized_key|ssh_keys", body) is not None,
|
||||
f'resource "{rtype}" "{rname}" seeds no login user / SSH key — the node could not be '
|
||||
"reached by the BASELINE stage without a hand step")
|
||||
|
||||
# Node identity comes from variables, and those variables define the whole node set.
|
||||
def tf_default(var: str):
|
||||
m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
|
||||
if not m:
|
||||
return None
|
||||
d = re.search(r'default\s*=\s*(.+)', m.group(1))
|
||||
return d.group(1).strip() if d else None
|
||||
|
||||
|
||||
prefix_raw = tf_default("node_name_prefix")
|
||||
count_raw = tf_default("node_count")
|
||||
ips_raw = tf_default("node_ips")
|
||||
check(prefix_raw is not None and count_raw is not None,
|
||||
"infra/tofu has no node_name_prefix/node_count variables — the node set is not derived "
|
||||
"from code, so the created nodes cannot be compared with the managed ones")
|
||||
prefix = (prefix_raw or "").strip('"')
|
||||
try:
|
||||
node_count = int((count_raw or "0").strip())
|
||||
except ValueError:
|
||||
node_count = 0
|
||||
check(node_count >= 1, f"node_count default is {count_raw!r} — no node would be created")
|
||||
if ips_raw is not None:
|
||||
n_ips = len(re.findall(r'"[^"]+"', ips_raw))
|
||||
check(n_ips >= node_count,
|
||||
f"node_ips default has {n_ips} address(es) for node_count={node_count} — the module "
|
||||
"cannot address every node it creates")
|
||||
check((TOFU / "outputs.tf").is_file() and
|
||||
re.search(r'output\s+"(ansible_inventory|nodes)"', (TOFU / "outputs.tf").read_text()) is not None,
|
||||
"infra/tofu/outputs.tf surfaces no node inventory output — the JOIN stage would have to be "
|
||||
"told the node facts by hand instead of consuming them from IaC")
|
||||
|
||||
# ------------------------------------------------- the dev node set, derived twice over
|
||||
# Group names are taken from the PLAYBOOKS' own default host patterns, so nothing here
|
||||
# restates the estate's naming.
|
||||
PLAYBOOKS = ANS / "playbooks"
|
||||
|
||||
|
||||
def default_patterns(playbook: str) -> list[str]:
|
||||
p = PLAYBOOKS / playbook
|
||||
if not p.is_file():
|
||||
return []
|
||||
return re.findall(r"default\('([^']+)'\)", p.read_text())
|
||||
|
||||
|
||||
dev_groups = set(default_patterns("node-baseline.yml") + default_patterns("k3s.yml"))
|
||||
check(len(dev_groups) > 0,
|
||||
"no playbook declares a default host pattern — the dev target is not expressed in code")
|
||||
|
||||
INV = ANS / "inventory" / "hosts.ini"
|
||||
check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — no node is managed as code")
|
||||
ini_groups: dict[str, list[str]] = {}
|
||||
children: dict[str, list[str]] = {}
|
||||
cur = None
|
||||
for line in (INV.read_text().splitlines() if INV.is_file() else []):
|
||||
s = line.strip()
|
||||
if not s or s.startswith("#"):
|
||||
continue
|
||||
m = re.match(r"^\[([^\]]+)\]$", s)
|
||||
if m:
|
||||
cur = m.group(1)
|
||||
(children if cur.endswith(":children") else ini_groups).setdefault(
|
||||
cur.split(":")[0] if cur.endswith(":children") else cur, [])
|
||||
continue
|
||||
if cur is None or cur.endswith(":vars"):
|
||||
continue
|
||||
host = s.split()[0]
|
||||
if cur.endswith(":children"):
|
||||
children.setdefault(cur.split(":")[0], []).append(host)
|
||||
else:
|
||||
ini_groups.setdefault(cur, []).append(host)
|
||||
|
||||
|
||||
def hosts_in(group: str, seen=None) -> set[str]:
|
||||
seen = seen or set()
|
||||
if group in seen:
|
||||
return set()
|
||||
seen.add(group)
|
||||
out = set(ini_groups.get(group, []))
|
||||
for child in children.get(group, []):
|
||||
out |= hosts_in(child, seen)
|
||||
return out
|
||||
|
||||
|
||||
dev_hosts: set[str] = set()
|
||||
for g in sorted(dev_groups):
|
||||
dev_hosts |= hosts_in(g)
|
||||
check(len(dev_hosts) > 0,
|
||||
f"the dev groups {sorted(dev_groups)} resolve to NO hosts in hosts.ini — nothing is managed")
|
||||
expected = {f"{prefix}{i + 1}" for i in range(node_count)}
|
||||
check(dev_hosts == expected,
|
||||
f"the dev nodes Ansible manages ({sorted(dev_hosts)}) are not exactly the nodes the IaC "
|
||||
f"module creates ({sorted(expected)}) — a node outside IaC is a hand-built node (R-ENV-01)")
|
||||
|
||||
# ============================================================ STAGE 2 — BASELINE (code)
|
||||
BASE_ROLE = ANS / "roles" / "node_baseline"
|
||||
check(BASE_ROLE.is_dir(),
|
||||
"no node_baseline role — the BASELINE stage has no code (docs/02 stage 2)")
|
||||
base_tasks = "\n".join(p.read_text(errors="replace")
|
||||
for p in sorted((BASE_ROLE / "tasks").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
|
||||
base_defaults = "\n".join(p.read_text(errors="replace")
|
||||
for p in sorted((BASE_ROLE / "defaults").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
|
||||
# Each item docs/02 names for stage 2, matched on the ANSIBLE MODULE that would do it —
|
||||
# a declarative, idempotent task, not a shell one-liner.
|
||||
for pattern, what in (
|
||||
(r"ansible\.builtin\.apt:", "a package baseline (apt module)"),
|
||||
(r"modprobe:", "kernel modules loaded (modprobe module)"),
|
||||
(r"modules-load|/etc/modules", "kernel modules PERSISTED across reboot"),
|
||||
(r"sysctl", "sysctl tuning"),
|
||||
(r"filesystem:|mkfs", "a filesystem on the Longhorn data disk"),
|
||||
(r"mount:|ansible\.posix\.mount", "the Longhorn data disk MOUNTED"),
|
||||
(r"swap", "swap disabled"),
|
||||
(r"iscsid", "iscsid enabled (Longhorn prerequisite)"),
|
||||
):
|
||||
check(re.search(pattern, base_tasks) is not None,
|
||||
f"node_baseline does not declare {what} — docs/02 stage 2 requires the OS baseline and "
|
||||
"k3s/Longhorn/KubeVirt prerequisites to be applied as code")
|
||||
check(re.search(r"^node_baseline_packages:", base_defaults, re.M) is not None and
|
||||
re.search(r"^node_baseline_kernel_modules:", base_defaults, re.M) is not None,
|
||||
"node_baseline's package/module sets are not variables with defaults — a per-host special "
|
||||
"case is not 'managed as code' (and could not be scoped per environment, R-ENV-02)")
|
||||
|
||||
# ========================================================== STAGE 3 — CLUSTER-JOIN (k3s)
|
||||
K3S_ROLE = ANS / "roles" / "k3s_server"
|
||||
k3s_text = "\n".join(p.read_text(errors="replace")
|
||||
for p in sorted((K3S_ROLE).rglob("*.yml"))) if K3S_ROLE.is_dir() else ""
|
||||
check(K3S_ROLE.is_dir() and len(k3s_text) > 200,
|
||||
"no k3s_server role — nothing joins the nodes into a cluster (docs/02 stage 3)")
|
||||
check("--cluster-init" in k3s_text,
|
||||
"the k3s role never passes --cluster-init — the HA embedded-etcd cluster is never founded")
|
||||
check(re.search(r"--server\s+https", k3s_text) is not None,
|
||||
"the k3s role never joins secondaries with --server — nodes would not form one cluster")
|
||||
check(re.search(r"K3S_TOKEN|--token", k3s_text) is not None,
|
||||
"the k3s role never supplies a join token — the join could only be completed by hand")
|
||||
# The bundled components docs/02 requires to be disabled AT INSTALL, each checked by the
|
||||
# flag that actually disables it.
|
||||
for pattern, what in (
|
||||
(r"--flannel-backend=none", "flannel (Cilium is the CNI)"),
|
||||
(r"--disable-kube-proxy", "kube-proxy (Cilium kube-proxy replacement)"),
|
||||
(r"traefik", "traefik (Gateway API via Cilium)"),
|
||||
(r"servicelb", "servicelb"),
|
||||
):
|
||||
check(re.search(pattern, k3s_text) is not None,
|
||||
f"the k3s install does not disable {what} — docs/02 stage 3 requires it disabled at "
|
||||
"install, or the bundled component fights the platform's own")
|
||||
check((ANS / "roles" / "cilium").is_dir(),
|
||||
"no cilium role — stage 3 requires Cilium to be installed as the CNI at bootstrap")
|
||||
check((ANS / "roles" / "flux_bootstrap").is_dir(),
|
||||
"no flux_bootstrap role — stage 4's hand-off to GitOps has no code")
|
||||
|
||||
# ============================================== STAGE 4 — MANAGE + the chain is REAL
|
||||
SITE = PLAYBOOKS / "site.yml"
|
||||
check(SITE.is_file(), "no playbooks/site.yml — the provisioning chain is not expressed as one path")
|
||||
site_txt = SITE.read_text() if SITE.is_file() else ""
|
||||
order = [m for m in re.findall(r"import_playbook:\s*(\S+)", site_txt)]
|
||||
check(order[:1] == ["node-baseline.yml"],
|
||||
f"site.yml imports {order} — the BASELINE stage must come first (docs/02 sequencing)")
|
||||
for want in ("k3s.yml", "cilium.yml", "flux-bootstrap.yml"):
|
||||
check(want in order, f"site.yml does not import {want} — a lifecycle stage is missing from the chain")
|
||||
if "k3s.yml" in order and "cilium.yml" in order:
|
||||
check(order.index("k3s.yml") < order.index("cilium.yml"),
|
||||
"site.yml installs Cilium before k3s — the CNI cannot precede the cluster")
|
||||
|
||||
# --- DRIVE THE REAL TOOLCHAIN, OFFLINE ------------------------------------------------
|
||||
# --syntax-check and --list-hosts open NO connection to any node: they resolve the
|
||||
# playbook, roles, collections and inventory only. Running them is the difference between
|
||||
# "the chain is written down" and "the chain resolves".
|
||||
if shutil.which("ansible-playbook") is None:
|
||||
note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
|
||||
"chain was PARSED but not DRIVEN; the merge gate's container has it and does drive it")
|
||||
else:
|
||||
# cwd=infra/ansible so the tracked ansible.cfg (roles_path, collections_path, inventory)
|
||||
# is the one in force — the same resolution a real run gets.
|
||||
def ansible(*args) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(["ansible-playbook", *args], cwd=ANS,
|
||||
capture_output=True, text=True)
|
||||
|
||||
syn = ansible("--syntax-check", "playbooks/site.yml")
|
||||
syn_out = (syn.stderr or "") + (syn.stdout or "")
|
||||
lh = None
|
||||
if syn.returncode != 0 and ENV_ONLY_FAILURE.search(syn_out):
|
||||
# NARROW, SELF-POLICING SKIP — see ENV_ONLY_FAILURE above.
|
||||
note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
|
||||
"passphrase and/or the gitignored Galaxy collections), so --syntax-check/--list-hosts "
|
||||
"were not driven; every parse assertion above still ran")
|
||||
else:
|
||||
check(syn.returncode == 0,
|
||||
"`ansible-playbook --syntax-check playbooks/site.yml` FAILED — the provisioning chain "
|
||||
f"does not even load: {syn_out.strip()[:400]}")
|
||||
lh = ansible("--list-hosts", "playbooks/site.yml")
|
||||
check(lh.returncode == 0,
|
||||
"`ansible-playbook --list-hosts playbooks/site.yml` FAILED — the chain does not "
|
||||
f"resolve against the tracked inventory: "
|
||||
f"{((lh.stderr or '') + (lh.stdout or '')).strip()[:400]}")
|
||||
if lh is not None and lh.returncode == 0:
|
||||
# Parse plays -> hosts. NEVER `ansible-inventory --list`: it prints DECRYPTED vault values.
|
||||
plays: list[tuple[str, set[str]]] = []
|
||||
name, hosts, in_hosts = None, set(), False
|
||||
for line in lh.stdout.splitlines():
|
||||
m = re.match(r"\s*play #\d+ \([^)]*\):\s*(.*?)\s*TAGS:", line)
|
||||
if m:
|
||||
if name is not None:
|
||||
plays.append((name, hosts))
|
||||
name, hosts, in_hosts = m.group(1), set(), False
|
||||
continue
|
||||
if re.match(r"\s*hosts \(\d+\):", line):
|
||||
in_hosts = True
|
||||
continue
|
||||
if in_hosts and line.strip() and not line.strip().startswith("pattern:"):
|
||||
hosts.add(line.strip())
|
||||
if name is not None:
|
||||
plays.append((name, hosts))
|
||||
check(len(plays) >= 4,
|
||||
f"--list-hosts resolved {len(plays)} play(s) — the four-stage chain did not expand")
|
||||
# Every lifecycle stage must cover EVERY dev node. A stage that reaches two of three
|
||||
# nodes means the third was finished by hand.
|
||||
STAGES = {
|
||||
"baseline": r"baseline",
|
||||
"cluster-join": r"^k3s (primary|secondar)",
|
||||
"cni": r"cilium",
|
||||
"manage (flux hand-off)": r"flux",
|
||||
}
|
||||
for stage, rx in STAGES.items():
|
||||
covered: set[str] = set()
|
||||
matched = 0
|
||||
for pname, phosts in plays:
|
||||
if re.search(rx, pname, re.I):
|
||||
matched += 1
|
||||
covered |= phosts
|
||||
check(matched >= 1, f"no play in site.yml implements the {stage} stage")
|
||||
check(covered == dev_hosts,
|
||||
f"the {stage} stage targets {sorted(covered)}, not every dev node "
|
||||
f"{sorted(dev_hosts)} — a node outside the code path is a hand-provisioned "
|
||||
"node (R-ENV-01)")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
||||
check(CHECKS >= 30,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
for n in NOTES:
|
||||
print(f" note: {n}")
|
||||
print(f"check OK: dev-nodes-provisioned-as-code — {CHECKS} assertions; "
|
||||
f"{len(dev_hosts)} dev node(s) {sorted(dev_hosts)} created by IaC ({sorted(providers.values())}), "
|
||||
f"baselined, cluster-joined and Flux-managed by the same chain. R-ENV-01's authored-path "
|
||||
"clause only — the live VMs' provenance, `tofu validate`, and rebuild/re-converge are NOT "
|
||||
"proven here (see the header's gap list; rebuild is L3 node-loss).")
|
||||
PY
|
||||
@@ -0,0 +1,99 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — two egress paths, and the rule set that selects between them (R-NET-04) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
|
||||
# routing rules ... realized as two paths — the local `internet` path and the
|
||||
# in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress
|
||||
# routing: two paths, one physical WAN": "the `internet` path is simply the default
|
||||
# route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and
|
||||
# the nodes themselves — egress the local/private WAN by default and **never** the
|
||||
# VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the
|
||||
# kill-switch blackholes **only** that class."
|
||||
#
|
||||
# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here)
|
||||
# ------------------------------------------------------------------------
|
||||
# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class
|
||||
# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and
|
||||
# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES
|
||||
# that select among them, which nothing else asserts:
|
||||
# * exactly TWO paths, the same closed set in the schema, the admission policy and the
|
||||
# renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope);
|
||||
# * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one
|
||||
# Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one
|
||||
# `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because
|
||||
# absence of a matching rule IS the default path;
|
||||
# * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a
|
||||
# vpn-class pod's cluster/LAN traffic still takes the local path;
|
||||
# * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing
|
||||
# table, `route-noexec` is set and a pushed `redirect-gateway` is ignored;
|
||||
# * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every
|
||||
# DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN.
|
||||
# (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.)
|
||||
#
|
||||
# WHY L0: every clause above is a statement about ALL rules — including rules that must be
|
||||
# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default
|
||||
# route in the main table). A cluster can show which path two pods took; it cannot show that
|
||||
# no third rule exists or that the vocabulary is closed.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are
|
||||
# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod
|
||||
# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked
|
||||
# against its own values.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy
|
||||
# routing / OpenVPN / nftables templates, and every environment's overlay addresses are
|
||||
# parsed and cross-checked,
|
||||
# THEN there are exactly two paths with `internet` as the default; exactly one steering rule
|
||||
# selects exactly the vpn class by label; the gateway is selected by node label with a
|
||||
# named egress interface; all RFC1918 space and every address any environment assigns
|
||||
# itself is excluded from the tunnel; the tunnel's default route and its fail-closed
|
||||
# blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN
|
||||
# cannot take the main table; and no nftables DROP is unscoped.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
||||
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
||||
# Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by
|
||||
# tests/l2-behavioral/egress-default-path (same requirement label); the two labelled
|
||||
# classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test.
|
||||
# * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates
|
||||
# that generate them; a hand-added rule on the box is invisible here.
|
||||
# * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no
|
||||
# honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default".
|
||||
# * IPv6 egress.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: egress-paths-selected-by-rules
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE. Universally quantified over tracked YAML: every manifest that carries
|
||||
# an egress-class label/annotation must use a value the policy set actually steers. The
|
||||
# file most likely to break it is a NEW workload under apps/ — which attributes no
|
||||
# requirement — not platform/vpn-egress/ (the only path the map keys to R-NET-04).
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree, when the egress vocabulary, every steering rule and the gateway
|
||||
node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers
|
||||
exactly two egress paths with the WAN as the default, exactly one rule selects exactly
|
||||
the vpn class, local destinations stay on the local path in every environment, the
|
||||
tunnel never becomes the default route, and nothing unscoped can blackhole the default
|
||||
path.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,521 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — egress-paths-selected-by-rules
|
||||
# Serves: R-NET-04 ("The platform MUST support **multiple egress paths** selected by routing
|
||||
# rules. Per OQ-1 the environment has **one physical commercial WAN and no
|
||||
# multi-ISP plans**, so this is realized as two paths — the local `internet` path and
|
||||
# the in-cluster `vpn` path — not multi-ISP routing. Physical multi-WAN (multiple ISP
|
||||
# uplinks) is **out of scope** unless a future ADR reopens it."
|
||||
# — spec docs/00-vision-and-scope.md).
|
||||
#
|
||||
# The elaborating spec text this check quotes and asserts against — docs/03
|
||||
# §"Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)":
|
||||
# | `internet` | Out the single commercial WAN | Normal pod egress via node -> SOHO edge
|
||||
# router -> internet |
|
||||
# | `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN
|
||||
# egress gateway (ADR-0013) |
|
||||
# "**Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing
|
||||
# to policy-route across; the `internet` path is simply the default route out."
|
||||
# "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes
|
||||
# themselves — egress the local/private WAN by default and **never** the VPN. Only
|
||||
# services labeled `egress-class=vpn` are steered to the tunnel, and the kill-switch
|
||||
# blackholes **only** that class."
|
||||
# "It is **split-tunnel**: it creates `tun0` but does **not** set the node's default
|
||||
# route (`route-noexec` + policy routing), so the node's own traffic stays on the WAN —
|
||||
# only what Cilium points at `tun0` uses the VPN."
|
||||
#
|
||||
# WHAT R-NET-04 ADDS OVER R-NET-05/07 (which are ALREADY covered) — READ THIS FIRST
|
||||
# ---------------------------------------------------------------------------------
|
||||
# R-NET-05/07 are about a SERVICE: it can declare its class, enforcement happens outside it,
|
||||
# and the class can be flipped via GitOps. Those are proven by
|
||||
# tests/l1-rendering/egress-class-render (the render) and tests/l2-behavioral/egress-class
|
||||
# (two labelled pods observe two different public IPs, and a relabel re-routes the same pod).
|
||||
# Re-tagging either of those with R-NET-04 would be a duplicate wearing a new label.
|
||||
#
|
||||
# R-NET-04 is about the PLATFORM's PATH SET and the RULES THAT SELECT AMONG THEM. Its
|
||||
# distinctive content — none of which any existing test asserts — is:
|
||||
#
|
||||
# (1) THERE ARE EXACTLY TWO PATHS, and the same two everywhere. The class vocabulary is
|
||||
# one closed set of two values agreed by the schema, the admission policy and the
|
||||
# renderer. A third value appearing in one of them but not the others would be a path
|
||||
# nothing routes (or a path nothing validates); "multi-ISP" reappearing as a third
|
||||
# class would contradict OQ-1's resolution.
|
||||
# (2) SELECTION IS BY A ROUTING RULE, and the rule set is TOTAL AND DISJOINT. The `vpn`
|
||||
# path is entered only via rules that select the vpn class — a Cilium
|
||||
# CiliumEgressGatewayPolicy pod-selector, and on the gateway node an `ip rule` scoped
|
||||
# to the tunnel source address. Nothing selects the `internet` class, because
|
||||
# "the `internet` path is simply the default route out": ABSENCE of a matching rule IS
|
||||
# the default path. So an unlabelled pod, and an `internet`-class pod, take the WAN.
|
||||
# (3) THE PATHS COEXIST — the choice is per-destination, not per-pod-lifetime. The vpn
|
||||
# rule tunnels public destinations while EXCLUDING all private space, so a vpn-class
|
||||
# pod's cluster/LAN traffic still takes the local path at the same time as its
|
||||
# internet traffic takes the tunnel.
|
||||
# (4) THE VPN PATH NEVER BECOMES THE DEFAULT. Split tunnel: the tunnel's default route
|
||||
# lives in a DEDICATED routing table reached only through the scoped rule, the
|
||||
# provider's pushed `redirect-gateway` is ignored, and `route-noexec` keeps OpenVPN
|
||||
# from touching the main table. Otherwise there would be one path (the VPN), not two.
|
||||
# (5) THE KILL-SWITCH BLACKHOLES ONLY THAT CLASS. The blackhole default is inside the
|
||||
# dedicated table; the nftables chain's policy is ACCEPT and every DROP is qualified
|
||||
# by the vpn class's own source set / fwmark. The `internet` path therefore fails
|
||||
# OPEN and can never be blackholed. (R-NET-06's opposite claim — that `vpn` fails
|
||||
# CLOSED — is proven at L3 by tests/l3-fault-injection/vpn-killswitch. This check
|
||||
# asserts the OTHER direction, which that test does not.)
|
||||
#
|
||||
# WHY L0. Every clause above is a property of the RULE SET, i.e. of authored configuration:
|
||||
# totality ("nothing selects the default path"), disjointness ("exactly one rule, selecting
|
||||
# exactly one class"), and scoping ("no unqualified drop, no default route in the main
|
||||
# table") are statements about ALL rules, including the ones that would have to be absent.
|
||||
# A running cluster can show you which path two pods took; it cannot show you that no third
|
||||
# rule exists, that no rule is unscoped, or that the vocabulary is closed. The behavioural
|
||||
# complement — that an UNLABELLED pod really does take the WAN path — is at L2 in
|
||||
# tests/l2-behavioral/egress-default-path (same requirement label).
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no address is written down. The private ranges the vpn rule must
|
||||
# exclude are checked as CONTAINMENT of the addresses each environment's own Flux overlay
|
||||
# declares (pod CIDR, Gateway VIP, LB pool, API host) — so a new environment on a different
|
||||
# LAN is checked against its own addresses, and the check keeps working when they change.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
|
||||
# - Behaviour. That a vpn-class pod's packets really leave via tun0, that an unlabelled
|
||||
# pod's really do not, and that the tunnel is up at all. L2 covers the two labelled
|
||||
# classes (tests/l2-behavioral/egress-class, R-NET-05) and the unlabelled default
|
||||
# (tests/l2-behavioral/egress-default-path, R-NET-04); the kill-switch firing is L3.
|
||||
# - THE RUNNING NODE's routing tables and nftables ruleset. This reads the templates that
|
||||
# generate them, driven by no node. A hand-added `ip rule` on the gateway node is
|
||||
# invisible here (and to every other test); drift on the box is an Ansible check-mode
|
||||
# exercise for the operator.
|
||||
# - PHYSICAL MULTI-WAN. Out of scope per OQ-1, so its absence is not asserted: there is no
|
||||
# honest repository signal that would distinguish "one WAN, correctly" from "one WAN,
|
||||
# because nobody tried" — and inventing one would be a hollow assertion.
|
||||
# - IPv6. The policy's CIDRs and the routing rules are IPv4-only, matching the platform.
|
||||
# If IPv6 egress is ever enabled, the `internet` path would need the same analysis and
|
||||
# this check would NOT catch its absence.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import ipaddress
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
LBL = "homelab.keithjackson.dev"
|
||||
CLASS_LABEL = f"{LBL}/egress-class"
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def die():
|
||||
print(f"CHECK FAIL (egress-paths-selected-by-rules): {len(FAILURES)} of {CHECKS} "
|
||||
"assertions failed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def docs(path: Path):
|
||||
"""Every YAML document in a file, skipping empties."""
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
# Skip dot-DIRECTORIES (.git, .claude, .cache) while still matching dot-FILES such as
|
||||
# .sops.yaml. A bare rglob from ROOT also descends into NESTED WORKTREES —
|
||||
# .claude/worktrees/agent-*/ created for isolated agent runs — which made this check see
|
||||
# 5 CiliumEgressGatewayPolicy objects instead of the 1 authored, failing R-NET-04 for a
|
||||
# reason that has nothing to do with egress. .claude/ is gitignored (.gitignore:43), so
|
||||
# nothing under it is part of the authored surface by definition.
|
||||
TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml")
|
||||
if "_artifacts" not in p.parts
|
||||
and not any(part.startswith(".")
|
||||
for part in p.relative_to(ROOT).parent.parts))
|
||||
SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json"
|
||||
RENDER = ROOT / "catalog" / "render.py"
|
||||
VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway"
|
||||
CLUSTERS = ROOT / "clusters"
|
||||
|
||||
for p in (SCHEMA, RENDER, VPN_ROLE, CLUSTERS):
|
||||
check(p.exists(), f"{p.relative_to(ROOT)} is missing — the egress path model cannot be "
|
||||
"read at all (R-NET-04)")
|
||||
if FAILURES:
|
||||
die()
|
||||
|
||||
|
||||
# ======================================================= (1) EXACTLY TWO PATHS, AGREED
|
||||
import json # noqa: E402 — only needed for the schema
|
||||
|
||||
schema = json.loads(SCHEMA.read_text())
|
||||
egress_prop = ((schema.get("properties") or {}).get("egress") or {})
|
||||
enum = egress_prop.get("enum")
|
||||
check(isinstance(enum, list) and len(enum) == 2,
|
||||
f"schemas/servicedefinition.schema.json declares egress enum {enum!r} — R-NET-04 is "
|
||||
"realized as EXACTLY TWO paths (OQ-1: one WAN, no multi-ISP). A third value is a path "
|
||||
"nothing routes; one value is not 'multiple egress paths'")
|
||||
PATHS = set(enum or [])
|
||||
check(PATHS == {"internet", "vpn"},
|
||||
f"the declared egress paths are {sorted(PATHS)} — docs/03's table fixes them as "
|
||||
"`internet` (out the single commercial WAN) and `vpn` (the in-cluster tunnel)")
|
||||
check(egress_prop.get("default") == "internet",
|
||||
f"the schema's default egress class is {egress_prop.get('default')!r} — docs/03: "
|
||||
"'Default is the WAN; the VPN is strictly opt-in', and 'the `internet` path is simply "
|
||||
"the default route out'. A service that says nothing must get the WAN")
|
||||
|
||||
# The admission policy must constrain the pod label to the SAME closed set, or a workload
|
||||
# could carry a class the schema never allowed (and that no rule routes).
|
||||
pol_files = sorted((ROOT / "platform" / "kyverno").rglob("*egress*.yaml"))
|
||||
check(len(pol_files) >= 1,
|
||||
"no Kyverno policy file mentions egress under platform/kyverno — nothing stops a pod "
|
||||
"carrying an egress class outside the declared path set")
|
||||
pol_text = "\n".join(p.read_text(errors="replace") for p in pol_files)
|
||||
for cls in sorted(PATHS):
|
||||
check(re.search(rf"\b{re.escape(cls)}\b", pol_text) is not None,
|
||||
f"the egress admission policy never names the {cls!r} path — a class the policy "
|
||||
"does not know cannot be validated against the path set")
|
||||
# No tracked object may carry a class outside the closed set — with ONE self-policing
|
||||
# exemption. A Kyverno policy-test fixture deliberately carries an INVALID class so the
|
||||
# admission test can assert the policy REJECTS it; that pod is exempt only while a
|
||||
# kyverno-test.yaml in the same tests tree still expects `result: fail` for it. Delete the
|
||||
# expectation (or flip it to pass) and the exemption evaporates — never a bare path allowlist.
|
||||
def rejected_by_a_policy_test(resource_name: str, fixture: Path) -> bool:
|
||||
tests_root = fixture.parent
|
||||
while tests_root != ROOT and tests_root.name != "tests":
|
||||
tests_root = tests_root.parent
|
||||
if tests_root == ROOT:
|
||||
return False
|
||||
for tf in sorted(tests_root.rglob("kyverno-test.yaml")):
|
||||
for d in docs(tf):
|
||||
for case in (d.get("results") or []):
|
||||
if case.get("result") == "fail" and resource_name in (case.get("resources") or []):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
stray: list[str] = []
|
||||
_STRAY_RE = re.compile(rf"{re.escape(CLASS_LABEL)}:\s*[\"']?([A-Za-z][A-Za-z0-9-]*)")
|
||||
for f in TRACKED_YAML:
|
||||
for d in docs(f):
|
||||
labels = ((d.get("metadata") or {}).get("labels") or {})
|
||||
cls = labels.get(CLASS_LABEL)
|
||||
# Also catch pod-template labels (Deployments/DaemonSets/Jobs/VMs).
|
||||
tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}).get("labels") or {}
|
||||
for value, owner in ((cls, d), (tmpl.get(CLASS_LABEL), d)):
|
||||
if value is None or value in PATHS or value == "CLASS":
|
||||
continue
|
||||
name = ((owner.get("metadata") or {}).get("name") or "?")
|
||||
if rejected_by_a_policy_test(name, f):
|
||||
continue
|
||||
stray.append(f"{f.relative_to(ROOT)}:{name}={value}")
|
||||
# Values written as raw text (templates, docs-as-code) are caught too.
|
||||
for value in set(_STRAY_RE.findall(f.read_text(errors="replace"))):
|
||||
if value in PATHS or value == "CLASS":
|
||||
continue
|
||||
if any(value in s for s in stray):
|
||||
continue
|
||||
# Raw-text hit with no parsed owner: exempt only if EVERY object in the file that
|
||||
# carries this value is a policy-test fixture proven to be rejected.
|
||||
owners = [((d.get("metadata") or {}).get("name") or "?") for d in docs(f)
|
||||
if ((d.get("metadata") or {}).get("labels") or {}).get(CLASS_LABEL) == value]
|
||||
if owners and all(rejected_by_a_policy_test(o, f) for o in owners):
|
||||
continue
|
||||
stray.append(f"{f.relative_to(ROOT)}:<text>={value}")
|
||||
check(not stray,
|
||||
f"tracked YAML assigns egress-class value(s) outside the declared path set "
|
||||
f"{sorted(PATHS)}: {sorted(stray)} — a value no routing rule selects and no admission "
|
||||
"policy allows is a workload on no path at all (the only exemption is a Kyverno "
|
||||
"policy-test fixture that a kyverno-test.yaml still expects to FAIL)")
|
||||
|
||||
# The renderer must project the DECLARED class onto the pod label, unmodified: that label is
|
||||
# what the routing rule selects on, so a renderer that rewrote it would break the coupling
|
||||
# between the declaration and the path.
|
||||
rtext = RENDER.read_text()
|
||||
check(re.search(rf'f?"\{{LBL\}}/egress-class":\s*s\["egress"\]', rtext) is not None
|
||||
or re.search(r'egress-class\"\]?\s*:\s*s\[\"egress\"\]', rtext) is not None,
|
||||
"catalog/render.py does not project the ServiceDefinition's own `egress` value onto the "
|
||||
f"{CLASS_LABEL} pod label — the declaration and the routing rule's selector would be "
|
||||
"two independent facts that could disagree")
|
||||
|
||||
|
||||
# ============================== (2) SELECTION IS BY RULE; THE RULE SET IS TOTAL & DISJOINT
|
||||
policies: list[tuple[Path, dict]] = []
|
||||
for p in TRACKED_YAML:
|
||||
for d in docs(p):
|
||||
if d.get("kind") == "CiliumEgressGatewayPolicy":
|
||||
policies.append((p, d))
|
||||
check(len(policies) == 1,
|
||||
f"{len(policies)} CiliumEgressGatewayPolicy object(s) are authored "
|
||||
f"({[str(p.relative_to(ROOT)) for p, _ in policies]}) — R-NET-04's steering must be ONE "
|
||||
"rule. Two rules can overlap, and which one wins for an overlapping pod is not "
|
||||
"something the repository states")
|
||||
|
||||
for path, pol in policies:
|
||||
rel = str(path.relative_to(ROOT))
|
||||
spec = pol.get("spec") or {}
|
||||
selectors = spec.get("selectors") or []
|
||||
check(len(selectors) == 1,
|
||||
f"{rel}: the steering rule has {len(selectors)} selector(s) — every additional "
|
||||
"selector widens the set of pods steered off the default path, and R-NET-04's "
|
||||
"selection must be exactly 'the vpn class'")
|
||||
for sel in selectors:
|
||||
pod_sel = sel.get("podSelector")
|
||||
check(isinstance(pod_sel, dict) and pod_sel,
|
||||
f"{rel}: a selector has no podSelector — an empty selector in Cilium matches "
|
||||
"EVERY pod, which would put the whole cluster on the VPN path and leave no "
|
||||
"default path at all")
|
||||
ml = (pod_sel or {}).get("matchLabels") or {}
|
||||
me = (pod_sel or {}).get("matchExpressions") or []
|
||||
check(not me,
|
||||
f"{rel}: the podSelector uses matchExpressions {me!r}. The class contract is a "
|
||||
"single label equality; expressions (In/NotIn/Exists) can select pods that "
|
||||
"never declared a class, so the rule set would no longer be disjoint from the "
|
||||
"default path")
|
||||
check(set(ml) == {CLASS_LABEL},
|
||||
f"{rel}: the podSelector matches on {sorted(ml)} — it must select on exactly "
|
||||
f"{CLASS_LABEL}. Any other key makes path selection depend on something other "
|
||||
"than the declared class (and a namespace/app key would silently exempt pods)")
|
||||
check(ml.get(CLASS_LABEL) == "vpn",
|
||||
f"{rel}: the steering rule selects {CLASS_LABEL}={ml.get(CLASS_LABEL)!r}. Only "
|
||||
"the `vpn` class may be steered: docs/03 — 'Only services labeled "
|
||||
"egress-class=vpn are steered to the tunnel'. A rule selecting `internet` would "
|
||||
"route the default path through the tunnel, i.e. leave one path, not two")
|
||||
|
||||
# The gateway is chosen by a NODE LABEL, not a hostname: the path's endpoint is itself a
|
||||
# rule, so the gateway can move (or gain a standby) with no rule rewrite.
|
||||
gw = spec.get("egressGateway") or {}
|
||||
node_sel = (gw.get("nodeSelector") or {}).get("matchLabels") or {}
|
||||
check(bool(node_sel),
|
||||
f"{rel}: the egressGateway selects no node by label — pinning the path's exit to a "
|
||||
"named host means the vpn path cannot fail over (docs/03 requires two gateway "
|
||||
"nodes + Egress Gateway HA)")
|
||||
check("nodeName" not in gw,
|
||||
f"{rel}: the egressGateway pins a nodeName — the exit node must be selected by "
|
||||
"label so the path survives losing that node")
|
||||
check(isinstance(gw.get("interface"), str) and gw.get("interface"),
|
||||
f"{rel}: the egressGateway names no egress interface — without it the steered class "
|
||||
"is not actually sent down a different path from the default")
|
||||
|
||||
# ------------------- (3) THE PATHS COEXIST: the choice is per DESTINATION -------------
|
||||
dests = spec.get("destinationCIDRs") or []
|
||||
excl = spec.get("excludedCIDRs") or []
|
||||
check([str(c) for c in dests] == ["0.0.0.0/0"],
|
||||
f"{rel}: destinationCIDRs is {dests!r}. The vpn path is for PUBLIC destinations "
|
||||
"('Routed through the in-cluster VPN tunnel'), expressed as the whole internet "
|
||||
"minus the exclusions below — narrowing it here would silently leave public "
|
||||
"destinations on the WAN for a vpn-class service")
|
||||
try:
|
||||
excluded = [ipaddress.ip_network(str(c)) for c in excl]
|
||||
except ValueError as e:
|
||||
excluded = []
|
||||
check(False, f"{rel}: excludedCIDRs contains an unparsable network ({e})")
|
||||
check(len(excluded) >= 3,
|
||||
f"{rel}: only {len(excluded)} excluded CIDR(s) — docs/03 requires intra-cluster, "
|
||||
"pod->service and LAN traffic to stay on the LOCAL path, which is all of RFC1918")
|
||||
|
||||
def covered(net: ipaddress.IPv4Network | ipaddress.IPv6Network) -> bool:
|
||||
return any(net.subnet_of(e) for e in excluded if e.version == net.version)
|
||||
|
||||
# RFC1918 in full — computed, not string-matched, so a typo'd mask is caught.
|
||||
for rfc in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"):
|
||||
net = ipaddress.ip_network(rfc)
|
||||
check(covered(net),
|
||||
f"{rel}: the private range {rfc} is NOT excluded from the tunnel — a vpn-class "
|
||||
"pod's cluster/LAN traffic would be SNAT'd out tun0, so the two paths would not "
|
||||
"coexist for the same pod and east-west traffic would break")
|
||||
|
||||
# Every address each ENVIRONMENT declares for itself must fall in the excluded set: that
|
||||
# is what makes 'local stays local' true per environment rather than by coincidence.
|
||||
def overlay_addresses() -> list[tuple[str, str]]:
|
||||
found: list[tuple[str, str]] = []
|
||||
base = CLUSTERS / "base" / "cluster-vars.yaml"
|
||||
data = {}
|
||||
if base.is_file():
|
||||
for d in docs(base):
|
||||
if d.get("kind") == "ConfigMap":
|
||||
data = {k: str(v) for k, v in (d.get("data") or {}).items()}
|
||||
for f in sorted(CLUSTERS.glob("*/kustomization.yaml")):
|
||||
env = {**data}
|
||||
for m in re.finditer(
|
||||
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n"
|
||||
r"\s*value:\s*(?P<v>\S.*?)\s*$", f.read_text(), re.M):
|
||||
env[m.group("k")] = m.group("v").strip().strip('"').strip("'")
|
||||
label = f.parent.name
|
||||
for key in ("CLUSTER_POD_CIDR", "GATEWAY_VIP", "CILIUM_LB_POOL_START",
|
||||
"CILIUM_LB_POOL_STOP", "K3S_API_HOST"):
|
||||
v = env.get(key, "")
|
||||
if v and not v.startswith("TBD"):
|
||||
found.append((f"{label}/{key}", v))
|
||||
return found
|
||||
|
||||
addrs = overlay_addresses()
|
||||
check(len(addrs) >= 5,
|
||||
f"only {len(addrs)} environment address(es) could be read from the Flux overlays — "
|
||||
"the 'local traffic stays local' assertion would be checking almost nothing")
|
||||
for label, value in addrs:
|
||||
try:
|
||||
net = ipaddress.ip_network(value, strict=False)
|
||||
except ValueError:
|
||||
continue
|
||||
check(covered(net),
|
||||
f"{rel}: {label}={value}, an address THIS repository assigns to a cluster, is "
|
||||
"NOT inside the tunnel's excludedCIDRs — a vpn-class pod in that environment "
|
||||
"would try to reach its own cluster/LAN through the VPN")
|
||||
|
||||
|
||||
# ============ (4) THE VPN PATH IS A SECOND TABLE, NEVER THE DEFAULT (split tunnel) ========
|
||||
TPL = VPN_ROLE / "templates"
|
||||
up = (TPL / "vpn-egress-up.sh.j2")
|
||||
down = (TPL / "vpn-egress-down.sh.j2")
|
||||
conf = (TPL / "vpn-egress.conf.j2")
|
||||
nft = (TPL / "vpn-egress-killswitch.nft.j2")
|
||||
for f in (up, down, conf, nft):
|
||||
check(f.is_file(), f"{f.relative_to(ROOT)} is missing — the second egress path's routing "
|
||||
"rules are not authored")
|
||||
if FAILURES:
|
||||
die()
|
||||
|
||||
defaults = yaml.safe_load((VPN_ROLE / "defaults" / "main.yml").read_text()) or {}
|
||||
|
||||
|
||||
def render(text: str) -> str:
|
||||
"""Substitute `{{ var }}` with the role's own default value, for every default.
|
||||
|
||||
The templates reference their knobs by NAME, so an analysis that looked for the VALUE
|
||||
(e.g. the nft set name) would silently match nothing and pass vacuously — which is
|
||||
exactly what an earlier draft of this check did. Rendering first means the assertions
|
||||
below examine the text the node actually gets. Only simple `{{ name }}` expressions are
|
||||
substituted; anything else is left as-is (and the assertions are written so an
|
||||
unsubstituted expression fails rather than passes).
|
||||
"""
|
||||
out = text
|
||||
for k, v in defaults.items():
|
||||
if isinstance(v, (str, int)):
|
||||
out = re.sub(r"\{\{\s*" + re.escape(str(k)) + r"\s*\}\}", str(v), out)
|
||||
return out
|
||||
|
||||
|
||||
up_t, conf_t, nft_t = render(up.read_text()), render(conf.read_text()), render(nft.read_text())
|
||||
table_id = str(defaults.get("vpn_egress_gateway_route_table_id", ""))
|
||||
check(table_id.isdigit() and table_id not in ("0", "253", "254", "255"),
|
||||
f"the tunnel's routing table id is {table_id!r} — the second path must live in a "
|
||||
"DEDICATED table. 254 (main), 255 (local) and 253 (default) are the kernel's own, and "
|
||||
"putting the tunnel's default route in any of them makes the VPN the node's default "
|
||||
"route instead of a selectable second path")
|
||||
check(str(defaults.get("vpn_egress_gateway_rule_priority", "")).isdigit(),
|
||||
"no ip-rule priority is declared for the tunnel table — 'selected by routing rules' "
|
||||
"needs a rule, at a defined precedence")
|
||||
|
||||
# Every `ip route ... default` the up-script installs must name the dedicated table.
|
||||
route_lines = [l.strip() for l in up_t.splitlines()
|
||||
if re.search(r"^\s*ip\s+route\s+(replace|add)\b", l)]
|
||||
check(len(route_lines) >= 2,
|
||||
f"the up-script installs {len(route_lines)} route(s) — the vpn path needs its own "
|
||||
"default route via the tunnel AND the fail-closed blackhole behind it")
|
||||
for l in route_lines:
|
||||
check(re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
|
||||
f"the up-script installs a route WITHOUT naming the dedicated table: {l!r}. A route "
|
||||
"landing in the main table changes the path every other pod and the node itself "
|
||||
"uses — the VPN would stop being opt-in")
|
||||
check("blackhole" not in l or re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
|
||||
f"a blackhole route is installed outside the tunnel table: {l!r}. docs/03: the "
|
||||
"kill-switch 'blackholes **only** that class'; `internet` fails OPEN and must never "
|
||||
"be blackholed")
|
||||
|
||||
# Every `ip rule` must be SCOPED. An unscoped rule (`ip rule add lookup <table>`) sends
|
||||
# everything into the tunnel table, which would leave one path instead of two.
|
||||
rule_lines = [l.strip() for l in up_t.splitlines() if re.search(r"^\s*ip\s+rule\s+add\b", l)]
|
||||
check(len(rule_lines) >= 1,
|
||||
"the up-script adds no `ip rule` — nothing SELECTS the tunnel path, so the second "
|
||||
"routing table would never be consulted (R-NET-04's 'selected by routing rules')")
|
||||
for l in rule_lines:
|
||||
check(re.search(r"\bfrom\s+\S|\bfwmark\s+\S|\biif\s+\S", l) is not None,
|
||||
f"an unscoped ip rule is installed: {l!r}. Without a `from`/`fwmark`/`iif` selector "
|
||||
"EVERY packet is looked up in the tunnel table, so the WAN path disappears and the "
|
||||
"VPN is no longer opt-in")
|
||||
check(re.search(r'lookup\s+"?\$\{?TABLE_ID', l) is not None,
|
||||
f"an ip rule looks up a table other than the tunnel's: {l!r}")
|
||||
check(re.search(r'ip rule add from\s+"\$\{SRC\}"', up_t) is not None,
|
||||
"the steering rule is not scoped to ${SRC}, the tunnel's own local address — that "
|
||||
"address is what Cilium SNATs vpn-class pods to, and it is the ONLY thing that "
|
||||
"distinguishes the steered class from every other flow on the gateway node")
|
||||
check(re.search(r"^\s*if \[ -z \"\$\{SRC\}\" \]", up_t, re.M) is not None,
|
||||
"the up-script does not refuse to run when the tunnel address is unknown — installing "
|
||||
"a partial rule set is how a path ends up half-selected")
|
||||
|
||||
# The provider must not be able to make the tunnel the default route either.
|
||||
check(re.search(r"^\s*route-noexec\s*$", conf_t, re.M) is not None,
|
||||
"the OpenVPN wrapper config lacks `route-noexec` — OpenVPN would install the pushed "
|
||||
"routes itself, in the MAIN table, and the VPN would become the node's default route "
|
||||
"(docs/03: 'does **not** set the node's default route (`route-noexec` + policy "
|
||||
"routing)')")
|
||||
check(re.search(r'pull-filter ignore "redirect-gateway"', conf_t) is not None,
|
||||
"the OpenVPN wrapper config does not ignore a pushed `redirect-gateway` — the VPN "
|
||||
"provider could redirect the node's whole default route into the tunnel, collapsing "
|
||||
"the two paths into one")
|
||||
check(re.search(r"^\s*redirect-gateway", conf_t, re.M) is None,
|
||||
"the OpenVPN wrapper config itself sets `redirect-gateway` — that IS making the VPN the "
|
||||
"default route")
|
||||
|
||||
|
||||
# ================= (5) THE KILL-SWITCH BLACKHOLES ONLY THE VPN CLASS (internet fails OPEN)
|
||||
src_set = str(defaults.get("vpn_egress_gateway_nft_src_set", ""))
|
||||
fwmark = str(defaults.get("vpn_egress_gateway_fwmark", ""))
|
||||
check(bool(src_set),
|
||||
"the role declares no nftables source set — the kill-switch would have no way to name "
|
||||
"'the steered class' and could only match broadly")
|
||||
policies_lines = [l.strip() for l in nft_t.splitlines() if re.search(r"^\s*type filter hook", l)]
|
||||
check(len(policies_lines) >= 1,
|
||||
"the kill-switch nft template declares no base chain — nothing would be enforced")
|
||||
for l in policies_lines:
|
||||
check(re.search(r"policy\s+accept\s*;", l) is not None,
|
||||
f"an nft base chain is not `policy accept`: {l!r}. A default-drop chain here "
|
||||
"blackholes the `internet` path too, and docs/03 requires the two classes to fail "
|
||||
"in OPPOSITE directions — 'internet fails **open** (WAN, never blackholed)'")
|
||||
drop_lines = [l.strip() for l in nft_t.splitlines()
|
||||
if re.search(r"\bdrop\b", l) and not l.lstrip().startswith("#")]
|
||||
check(len(drop_lines) >= 1,
|
||||
"the kill-switch nft template contains no drop rule at all — there is no kill-switch, "
|
||||
"and R-NET-04's 'the kill-switch blackholes only that class' has nothing to scope")
|
||||
for l in drop_lines:
|
||||
scoped = (src_set and src_set in l) or (fwmark and "mark" in l)
|
||||
check(scoped,
|
||||
f"an nft DROP rule is not scoped to the vpn class: {l!r}. It must match the tunnel "
|
||||
f"source set (@{src_set}) or the class's fwmark; an unqualified drop would blackhole "
|
||||
"the default `internet` path, which must never be blackholed")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
||||
check(CHECKS >= 35,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
die()
|
||||
print(f"check OK: egress-paths-selected-by-rules — {CHECKS} assertions; "
|
||||
f"paths={sorted(PATHS)} (default={egress_prop.get('default')}), "
|
||||
f"{len(policies)} steering rule selecting {CLASS_LABEL}=vpn only, tunnel default route "
|
||||
f"in dedicated table {table_id} behind a scoped ip rule, blackhole and every nft DROP "
|
||||
f"scoped to the vpn class. R-NET-04's RULE-SET half only: the paths' actual BEHAVIOUR "
|
||||
"is L2 (egress-class for the two labelled classes, egress-default-path for the "
|
||||
"unlabelled default) and the kill-switch firing is L3 (vpn-killswitch).")
|
||||
PY
|
||||
@@ -8,6 +8,18 @@
|
||||
# the empty string while the Kustomization still reports Ready — which emptied Cilium's
|
||||
# k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01).
|
||||
#
|
||||
# It is also LOSSY IN THE OTHER DIRECTION, and that is the second incident on this record.
|
||||
# envsubst consumes dollars in PAIRS and emits one per pair, unconditionally and everywhere —
|
||||
# so no substituted path can carry a literal `$$` at all. On 2026-09-13 a ConfigMap under
|
||||
# apps-wave-0 shipped a dollar-quoted PL/pgSQL `DO $$ ... $$;` self-check; Postgres received
|
||||
# a half-quoted `DO $` and every Flux-reconciled run of the nextcloud-db grant reconciler died
|
||||
# with a syntax error, nine Error pods deep before anyone looked (commit 16b7fcd). It passed
|
||||
# review because it was hand-applied with `kubectl apply`, which runs no envsubst: the defect
|
||||
# existed ONLY on the rendered path, and it landed on the assertion whose whole job was to
|
||||
# make a silent regression visible. Nothing here failed — the manifest rendered
|
||||
# "successfully" into something broken, because the check treated any doubled dollar as a
|
||||
# deliberate escape and stopped looking.
|
||||
#
|
||||
# GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom,
|
||||
# WHEN each `spec.path` is resolved to the exact file set it renders (following
|
||||
# kustomization.yaml `resources:` transitively, not globbing the repo),
|
||||
@@ -15,7 +27,12 @@
|
||||
# or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default
|
||||
# operator on an undefined name (the silent-empty class strict mode does NOT catch);
|
||||
# no unescaped expansion is unparseable by envsubst (which fails the WHOLE
|
||||
# Kustomization); every path rendering a knob actually substitutes cluster-vars;
|
||||
# Kustomization); NO RUN OF TWO OR MORE DOLLARS IS SILENTLY REWRITTEN — the only shape
|
||||
# that passes unremarked is exactly `$${`, the documented total escape, and every other
|
||||
# run (a bare `$$` that cannot survive verbatim, an ODD `$$${...}` that only LOOKS
|
||||
# escaped and in fact expands, an over-counted `$$$${...}`) is named and justified in
|
||||
# `dollar-collapse.allow`, which is itself held to being exact and non-stale;
|
||||
# every path rendering a knob actually substitutes cluster-vars;
|
||||
# every overlay-patched knob has a base default; and the check is wired into the
|
||||
# merge gate as a blocking, unconditionally-run stage.
|
||||
#
|
||||
@@ -26,6 +43,19 @@
|
||||
# only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
|
||||
# Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven, and deliberately NOT proxied: the wider lesson of 2026-09-13 is that the bug
|
||||
# survived review because verification used a path (`kubectl apply`) that runs no
|
||||
# substitution. That property is NOT statically decidable from this repo — which command
|
||||
# someone ran is a fact about a shell session, and git stores the manifest, never the
|
||||
# invocation. Grepping commit prose or runbooks for `kubectl apply` would be trivially
|
||||
# satisfiable by rewording and would fire on the break-glass runbooks whose whole content is
|
||||
# that command, so no such proxy is attempted. The dollar-run assertion is the honest answer
|
||||
# instead: it moves the proof off the verification PATH and onto the SOURCE, where it IS
|
||||
# decidable, so the mangling is caught by reading the manifest however a human chose to try
|
||||
# it. An end-to-end `kustomize build | flux envsubst --strict` per substituted path would be
|
||||
# the real thing, and it is how the arithmetic in check.sh's header was measured — but the L0
|
||||
# CI toolchain installs neither tool, so as a gate stage it would SKIP permanently.
|
||||
#
|
||||
# NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests
|
||||
# a path includes — a deliberate superset of the post-`kustomize build` stream. It does
|
||||
# not render, does not prove the feature gate is enabled on the live kustomize-controller,
|
||||
@@ -43,18 +73,36 @@ metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3c
|
||||
# (UNCONDITIONAL)", which this annotation replaces:
|
||||
#
|
||||
# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
|
||||
# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
|
||||
# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
|
||||
# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN").
|
||||
#
|
||||
# WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map
|
||||
# keys only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or
|
||||
# platform/ attributes no requirement, so the requirement-scoped selection would filter
|
||||
# this check out of exactly the change that introduces the bug. The invariant spans every
|
||||
# path any Flux Kustomization substitutes (apps/, platform/, platform-config/), so it runs
|
||||
# on EVERY gate invocation.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every variable a substituted Flux path renders is defined in cluster-vars or
|
||||
escaped, so post-build substitution cannot silently empty a value and enabling
|
||||
StrictPostBuildSubstitutions cannot fail a legitimate manifest.
|
||||
StrictPostBuildSubstitutions cannot fail a legitimate manifest; and no dollar run
|
||||
envsubst rewrites reaches the cluster unreviewed, so content that must arrive
|
||||
verbatim cannot be silently mangled on the Flux-rendered path (16b7fcd).
|
||||
spec:
|
||||
steps:
|
||||
- name: audit-every-substituted-flux-path
|
||||
description: >-
|
||||
Resolve each postBuild.substituteFrom Kustomization to the files it renders and
|
||||
assert every ${...} is defined or escaped, then assert the gate wires it blocking.
|
||||
assert every ${...} is defined or escaped and no dollar run is silently collapsed,
|
||||
then assert the gate wires it blocking.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -18,6 +18,32 @@
|
||||
# substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
|
||||
# the repo asserted that. This check does.
|
||||
#
|
||||
# SECOND INCIDENT, 2026-09-13 — THE INVERSE HAZARD (commit 16b7fcd)
|
||||
# ------------------------------------------------------------------
|
||||
# The first incident was a variable the author wanted EXPANDED that rendered EMPTY. This one
|
||||
# is its mirror: content the author wanted to reach the workload UNTOUCHED that substitution
|
||||
# silently REWROTE. A CronJob reconciler in apps/wave-0/nextcloud-db.yaml asserted its own
|
||||
# convergence in a dollar-quoted PL/pgSQL block —
|
||||
#
|
||||
# DO <dq> BEGIN ... END <dq>; where <dq> is two literal dollars: standard Postgres
|
||||
#
|
||||
# — inside a ConfigMap under apps-wave-0, which declares postBuild.substituteFrom. envsubst
|
||||
# collapses a doubled dollar into ONE dollar, ALWAYS, so Postgres received a half-quoted
|
||||
# `DO <one dollar>` and died with `syntax error at or near "$"`. Three scheduled runs left
|
||||
# nine Error pods in nextcloud-db before anyone looked.
|
||||
#
|
||||
# It PASSED verification because it was hand-applied with `kubectl apply`, which runs NO
|
||||
# envsubst. The defect existed ONLY on the Flux-rendered path. And the damage landed
|
||||
# precisely on the self-check: the GRANT on line 1 still ran, so the privilege stayed
|
||||
# correct and the only thing broken was the assertion meant to make a silent regression
|
||||
# visible — the part whose whole job is to stop a blind spot being quiet.
|
||||
#
|
||||
# THE ASSERTION GAP THIS CLOSES: assertions (1)-(3) below ask whether every `${...}` is
|
||||
# defined, escaped, or parseable. They said NOTHING about a doubled dollar that is not an
|
||||
# escape, because the old classifier treated a `$` run of length >= 2 as "escaped, therefore
|
||||
# fine" and stopped looking. Nothing failed the gate; the manifest rendered "successfully"
|
||||
# into something broken. Assertion (7) is that hole, closed.
|
||||
#
|
||||
# ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
|
||||
# `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
|
||||
# ${DEFINED} -> substituted with the cluster-vars value.
|
||||
@@ -33,6 +59,66 @@
|
||||
# shell/cloud-init consumer. Escaping is total: it neutralises
|
||||
# even the indexed form.
|
||||
# $VAR, $(cmd) -> left alone by envsubst; they need no escaping.
|
||||
#
|
||||
# DOLLAR-RUN ARITHMETIC — the exact rule, MEASURED, not inferred (see assertion (7))
|
||||
# ---------------------------------------------------------------------------------
|
||||
# envsubst here is `github.com/fluxcd/pkg/envsubst` v1.1.0 — Flux's own maintained FORK of
|
||||
# `github.com/drone/envsubst` ("This package is a fork of drone/envsubst", its README) — and it
|
||||
# is exposed by the `flux envsubst` CLI ("replicate the behavior of the Flux Kustomization
|
||||
# post-build substitutions", its own --help). Named from the binary, not from memory: the
|
||||
# pinned flux 2.4.0 in the dev image embeds the Go module path
|
||||
# `/home/runner/go/pkg/mod/github.com/fluxcd/pkg/[email protected]/parse`
|
||||
# (`grep -ao 'github.com/fluxcd/pkg/envsubst[^ ]*' "$(command -v flux)"`). Its scanner walks
|
||||
# the input left to right; on a `$` it looks at the NEXT byte:
|
||||
# * another `$` -> BOTH are consumed and ONE `$` is emitted. This is UNCONDITIONAL. What
|
||||
# follows the pair is irrelevant — `{`, a letter, a digit, a semicolon,
|
||||
# a newline, end of input. `$$` is not "the escape for `${`"; it is the
|
||||
# escape for `$` itself, everywhere.
|
||||
# * `{` -> the braced expression is parsed and EXPANDED.
|
||||
# * anything else -> the `$` is emitted literally (hence `$VAR` and `$(cmd)` survive).
|
||||
# So a run of N consecutive dollars emits floor(N/2) literal dollars, and when N is ODD the
|
||||
# final dollar is still LIVE. Measured with flux 2.4.0 `flux envsubst`, DEFINED=VALUE — the
|
||||
# left column is authored bytes, the right is what reaches the cluster:
|
||||
#
|
||||
# $ -> $ ${DEFINED} -> VALUE
|
||||
# $$ -> $ $${DEFINED} -> ${DEFINED} <- the escape
|
||||
# $$$ -> $$ $$${DEFINED} -> $VALUE <- ODD: brace LIVE
|
||||
# $$$$ -> $$ $$$${DEFINED} -> $${DEFINED}
|
||||
# $$$$$ -> $$$ $$$$${DEFINED} -> $$VALUE
|
||||
# DO $$ .. $$; -> DO $ .. $; $$${NOPE:-d} -> $d
|
||||
# awk '{print $$2}' -> awk '{print $2}' $${ARR[0]} -> ${ARR[0]} (safe)
|
||||
# $$@ $$< $$^ -> $@ $< $^ $$${ARR[0]} -> HARD PARSE ERROR, whole
|
||||
# Kustomization dies
|
||||
#
|
||||
# TWO CONSEQUENCES THIS CHECK IS BUILT ON:
|
||||
# (a) THERE IS NO WAY TO SHIP A LITERAL `$$` THROUGH A SUBSTITUTED PATH. To make two
|
||||
# dollars arrive you must author FOUR. So any authored `$$` that is meant to reach the
|
||||
# workload verbatim is a BUG, full stop — that is the 2026-09-13 incident, and Postgres
|
||||
# dollar-quoting is its commonest shape (Makefile `$$@`, awk `$$2` and docker-compose
|
||||
# `$${}` are the same hazard arriving from the other direction: there the collapse is
|
||||
# what the author wanted).
|
||||
# (b) `$$$` + brace is NOT "more escaped" than `$$` + brace — it is NOT escaped at all. An
|
||||
# odd run leaves one live dollar, so the brace expands, or, for an inner envsubst
|
||||
# cannot parse, fails the ENTIRE Kustomization (proven above with $$${ARR[0]}, which
|
||||
# aborted the whole stream while $${ARR[0]} on the line before it was fine).
|
||||
# `--strict` reports NONE of this: every row of that table except the parse error was
|
||||
# accepted SILENTLY by `flux envsubst --strict`, exit 0, and its output was BYTE-IDENTICAL to
|
||||
# non-strict (`cmp`). Strict mode cannot see this class, so a source-level assertion is the
|
||||
# only available guard.
|
||||
#
|
||||
# TO REPRODUCE THE TABLE (30 seconds, no cluster) — it is a measurement, so re-measure it
|
||||
# after any FLUX_VERSION bump in infra/devcontainer/versions.env. Open the toolbox with
|
||||
# `./infra/devcontainer/dev.sh` and run these INSIDE it (do NOT pipe into `dev.sh run` from
|
||||
# the host: it starts the container with `-it`, so a piped stdin hangs):
|
||||
# printf '[$$] [$$$] [$${D}] [$$${D}] [DO $$ x $$;]\n' | D=VALUE flux envsubst
|
||||
# -> [$] [$$] [${D}] [$VALUE] [DO $ x $;]
|
||||
# And end-to-end, on the actual delivery path, with the pre-16b7fcd manifest checked out
|
||||
# (`git show 16b7fcd^:apps/wave-0/nextcloud-db.yaml > apps/wave-0/nextcloud-db.yaml`):
|
||||
# kubectl kustomize apps/wave-0 | grep 'DO \$' -> "DO $$" (the kubectl path)
|
||||
# kubectl kustomize apps/wave-0 | flux envsubst --strict | grep 'DO \$'
|
||||
# -> "DO $" (the FLUX path), exit 0
|
||||
# (`kubectl kustomize` IS the kustomize build here; the image ships no standalone binary.)
|
||||
# That two-command contrast IS the 2026-09-13 incident, and is why review missed it.
|
||||
# ${...}, ${a b}, ${ARR[0]} (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
|
||||
# variable name" / "missing closing brace") which fails the
|
||||
# WHOLE Kustomization — every resource in the path, not just
|
||||
@@ -55,7 +141,14 @@
|
||||
# (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
|
||||
# the "add the knob with a base default BEFORE referencing it" rule that the
|
||||
# incident's interim discipline depends on;
|
||||
# AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and
|
||||
# (7) NO DOLLAR RUN IS SILENTLY REWRITTEN: the only run of length >= 2 that passes
|
||||
# unremarked is exactly `$$` immediately followed by `{` — the documented, total,
|
||||
# already-inventoried escape. EVERY other run of >= 2 dollars (a bare `$$`, an odd
|
||||
# `$$$`, an over-counted `$$$$`) is rewritten by envsubst between git and the
|
||||
# container, so it must be named and justified in `dollar-collapse.allow`, a file
|
||||
# this check also holds to being exact and non-stale;
|
||||
# AND (6) this check is LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`,
|
||||
# so ci/l0.sh's `--include-scope whole-repo` runs it on every gate run, and
|
||||
# ci/gate.sh sets the merge block when the l0 stage fails.
|
||||
# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
|
||||
# preserves body comments into the stream envsubst reads (verified: a `$`-carrying
|
||||
@@ -78,6 +171,21 @@
|
||||
# kustomize-controller. Both are cluster facts (L2), untested here. Tagging
|
||||
# R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
|
||||
# is fully determined, with no variable silently emptied on the way in.
|
||||
# - It CANNOT prove the thing that actually let 16b7fcd's bug reach a cluster: that a
|
||||
# change was VERIFIED through a substituting path. "It was hand-applied with kubectl" is
|
||||
# a fact about a shell session, not about any tracked file — git records the manifest,
|
||||
# never the command someone ran to try it, so the property is not statically decidable
|
||||
# here and no honest formulation of it exists. Nor is a proxy worth having: grepping
|
||||
# commit prose or runbooks for `kubectl apply` would be trivially satisfiable by
|
||||
# rewording, and would fire on the many legitimate break-glass runbooks whose whole
|
||||
# content is that command. So this check does not attempt it, and does not pretend to.
|
||||
# Assertion (7) answers that incident a different way: it moves the proof off the
|
||||
# verification PATH, which is unknowable, and onto the SOURCE, which is not — the
|
||||
# mangling is now caught by reading the manifest, whichever way a human chose to try it.
|
||||
# (The one decidable half-measure, `kustomize build <path> | flux envsubst --strict` per
|
||||
# substituted path in CI, is already ruled out above: the L0 toolchain installs neither
|
||||
# tool, so the stage would SKIP permanently. The dev image has both, and it is how the
|
||||
# arithmetic table above was measured.)
|
||||
# - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
|
||||
# that an overlay's LAN block is self-consistent). Presence, not correctness.
|
||||
# - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
|
||||
@@ -107,6 +215,7 @@ definition of "what a Flux path actually renders" in the harness.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import bisect
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
@@ -230,9 +339,18 @@ LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")
|
||||
|
||||
|
||||
def classify(dollars: int, inner: str) -> tuple[str, str] | None:
|
||||
"""Return (severity-tag, remediation) for a hazardous expansion, else None."""
|
||||
if dollars >= 2:
|
||||
return None # escaped: emitted as a literal
|
||||
"""Return (severity-tag, remediation) for a hazardous expansion, else None.
|
||||
|
||||
PARITY, not ">= 2" (the 2026-09-13 correction). envsubst consumes dollars in PAIRS and
|
||||
emits one per pair, so a run of N dollars in front of a brace leaves the brace ESCAPED
|
||||
only when N is EVEN. An ODD run emits floor(N/2) literal dollars and its last dollar is
|
||||
still LIVE, so `$$${X}` expands exactly as `${X}` does — and `$$${ARR[0]}` fails the
|
||||
whole Kustomization where `$${ARR[0]}` is harmless. The old `dollars >= 2 -> return None`
|
||||
called every such run "escaped" and stopped looking. Assertion (7) flags the run itself;
|
||||
this function additionally classifies the still-live expansion an odd run carries.
|
||||
"""
|
||||
if dollars % 2 == 0:
|
||||
return None # fully escaped: emitted as a literal
|
||||
if NAME_RE.match(inner):
|
||||
if inner in KEYS:
|
||||
return None # defined knob — the happy path
|
||||
@@ -267,6 +385,9 @@ def rendered_files(path_spec: str) -> tuple[list[str], bool]:
|
||||
findings: list[str] = []
|
||||
referenced: dict[str, set[str]] = {}
|
||||
inventory: list[str] = []
|
||||
# rel path -> the Kustomization name(s) that render it, for assertion (7)'s second pass.
|
||||
# A file can be rendered by more than one Kustomization; all of them substitute.
|
||||
subst_files: dict[str, set[str]] = {}
|
||||
|
||||
for k in kustomizations:
|
||||
files, exists = rendered_files(k["path"])
|
||||
@@ -280,13 +401,14 @@ for k in kustomizations:
|
||||
text = f.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
continue
|
||||
subst_files.setdefault(rel, set()).add(k["name"])
|
||||
for lineno, line in enumerate(text.splitlines(), 1):
|
||||
for m in TOKEN_RE.finditer(line):
|
||||
dollars, inner = len(m.group(1)), m.group(2)
|
||||
verdict = classify(dollars, inner)
|
||||
if verdict is None:
|
||||
if dollars >= 2:
|
||||
n_esc += 1
|
||||
if dollars % 2 == 0: # PARITY, see classify(): an odd run is
|
||||
n_esc += 1 # live, not escaped
|
||||
else:
|
||||
n_def += 1
|
||||
referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
|
||||
@@ -312,6 +434,156 @@ for k in kustomizations:
|
||||
check(not findings,
|
||||
f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")
|
||||
|
||||
# ----------------------------------- (7) no dollar run is silently rewritten on the way out
|
||||
# THE 2026-09-13 CLASS. envsubst consumes dollars in PAIRS and emits one per pair,
|
||||
# unconditionally and everywhere (header: DOLLAR-RUN ARITHMETIC, measured against flux 2.4.0).
|
||||
# So every run of >= 2 dollars in a substituted manifest is REWRITTEN between git and the
|
||||
# container, and from the text alone the two intents are indistinguishable:
|
||||
#
|
||||
# (a) a deliberate escape the author WANTS collapsed — `$${VAR}` -> literal `${VAR}`;
|
||||
# (b) content that must arrive VERBATIM — `DO $$ ... $$;` -> `DO $ ... $;`
|
||||
# i.e. the incident. A BUG.
|
||||
#
|
||||
# THE RULE CHOSEN, AND WHY IT IS NEITHER NOISY NOR VACUOUS.
|
||||
# Exactly ONE shape passes unremarked: a run of EXACTLY TWO dollars immediately followed by
|
||||
# `{`. That is not a guess about intent, it is the escape contract this repo already
|
||||
# documents in clusters/base/apps-vms.yaml, that assertions (1)-(3) already validate the
|
||||
# inside of, and that 18 reviewed occurrences already rely on (unifi cloud-init, the gitea
|
||||
# and nextcloud OIDC bootstrap scripts, two longhorn comments). Failing those would be a
|
||||
# permanently-firing alert on documented, correct intent, and this repo has already been
|
||||
# bitten by one of those.
|
||||
#
|
||||
# EVERY OTHER run of >= 2 dollars FAILS unless it is named in `dollar-collapse.allow`:
|
||||
# * DOLLAR-COLLAPSE — a run not followed by `{`. Cannot survive verbatim: `$$` -> `$`.
|
||||
# This is the incident's exact shape and a blanket bug by default.
|
||||
# * DOLLAR-HALF-ESCAPED — an ODD run >= 3 before `{`. Looks escaped, is not: the last
|
||||
# dollar is live. Occasionally legitimate (`$$${PRICE}` to render a
|
||||
# literal `$` before a value), so it is allowlistable, not banned.
|
||||
# * DOLLAR-OVER-ESCAPED — an EVEN run >= 4 before `{`. Emits N/2 dollars then a literal
|
||||
# brace; nearly always a miscount, sometimes deliberate.
|
||||
#
|
||||
# WHY AN ALLOWLIST AND NOT A BLANKET BAN: a blanket ban was the first choice and was
|
||||
# rejected on evidence, not taste — main carries two real non-braced `$$` occurrences today
|
||||
# (prose inside embedded shell comments), so a blanket ban would have shipped RED. Nor is
|
||||
# the allowlist a hole: an entry is a hand-written edit to a file inside the conformance
|
||||
# test, which is loud in review; it must quote a >= 24-character VERBATIM context, so it can
|
||||
# only ever cover the occurrence it was written for and never a future one that drifts into
|
||||
# the same file; it must carry a >= 40-character reason; and every entry must still MATCH
|
||||
# something, so the allowlist cannot rot into a blanket exemption after the code it excused
|
||||
# is deleted. A wildcard, a bare `$$` context and a stale entry each FAIL.
|
||||
ALLOW_FILE = (ROOT / "conformance" / "tests" / "l0-static"
|
||||
/ "flux-postbuild-substitution" / "dollar-collapse.allow")
|
||||
MIN_CONTEXT = 24
|
||||
MIN_REASON = 40
|
||||
DOLLAR_RUN_RE = re.compile(r"\$+")
|
||||
|
||||
allow_entries: list[dict] = []
|
||||
check(ALLOW_FILE.is_file(),
|
||||
f"{ALLOW_FILE.relative_to(ROOT)} is missing — assertion (7) needs its allowlist to "
|
||||
f"exist even when empty, because an absent file and an empty file are the same green "
|
||||
f"and the next author would not know the mechanism is there")
|
||||
if ALLOW_FILE.is_file():
|
||||
for lineno, raw in enumerate(ALLOW_FILE.read_text().splitlines(), 1):
|
||||
if not raw.strip() or raw.lstrip().startswith("#"):
|
||||
continue
|
||||
parts = [x.strip() for x in raw.split("|")]
|
||||
if len(parts) != 3:
|
||||
check(False, f"dollar-collapse.allow:{lineno}: expected exactly 3 '|'-separated "
|
||||
f"fields (path | verbatim-context | reason), got {len(parts)}")
|
||||
continue
|
||||
allow_entries.append({"line": lineno, "path": parts[0], "ctx": parts[1],
|
||||
"reason": parts[2], "used": 0})
|
||||
|
||||
for e in allow_entries:
|
||||
tag = f"dollar-collapse.allow:{e['line']}"
|
||||
check((ROOT / e["path"]).is_file(),
|
||||
f"{tag}: names '{e['path']}', which is not a file in this repo — a stale entry is "
|
||||
f"a standing exemption for a path someone could recreate")
|
||||
check(len(e["ctx"]) >= MIN_CONTEXT,
|
||||
f"{tag}: context {e['ctx']!r} is {len(e['ctx'])} chars; at least {MIN_CONTEXT} are "
|
||||
f"required so an entry cannot be a near-wildcard that swallows future occurrences")
|
||||
check(DOLLAR_RUN_RE.search(e["ctx"]) is not None
|
||||
and max((len(m.group(0)) for m in DOLLAR_RUN_RE.finditer(e["ctx"])), default=0) >= 2,
|
||||
f"{tag}: context {e['ctx']!r} contains no run of two or more dollars, so it excuses "
|
||||
f"nothing and is either a typo or a decoy")
|
||||
check(len(e["reason"]) >= MIN_REASON,
|
||||
f"{tag}: reason is {len(e['reason'])} chars; at least {MIN_REASON} are required. "
|
||||
f"State what envsubst will actually emit and why that is acceptable — an entry "
|
||||
f"nobody had to explain is an accident wearing an approval")
|
||||
|
||||
RUN_TAGS = {
|
||||
"collapse": ("DOLLAR-COLLAPSE",
|
||||
"envsubst collapses each PAIR of dollars into one, everywhere and "
|
||||
"unconditionally, so this CANNOT reach the workload verbatim (a literal $$ "
|
||||
"needs $$$$ in source). This is the 2026-09-13 nextcloud-db shape: a "
|
||||
"dollar-quoted PL/pgSQL DO block arrived half-quoted and every Flux-"
|
||||
"reconciled run died. Rewrite without dollars, double them again if the "
|
||||
"collapse is what you want, or justify it in dollar-collapse.allow"),
|
||||
"half": ("DOLLAR-HALF-ESCAPED",
|
||||
"an ODD run of dollars before a brace is NOT escaped: floor(N/2) literal "
|
||||
"dollars are emitted and the LAST dollar is still live, so this expansion "
|
||||
"EVALUATES at reconcile time (and if its inner is unparseable it fails the "
|
||||
"entire Kustomization). Use an EVEN run to escape, or justify it in "
|
||||
"dollar-collapse.allow if a literal dollar before a substituted value is "
|
||||
"really what you meant"),
|
||||
"over": ("DOLLAR-OVER-ESCAPED",
|
||||
"an EVEN run of four or more dollars emits N/2 literal dollars followed by a "
|
||||
"LITERAL brace — the value is never substituted and extra dollars appear in "
|
||||
"the rendered manifest. Almost always a miscount of the escape; justify it in "
|
||||
"dollar-collapse.allow if it is not"),
|
||||
}
|
||||
|
||||
run_findings: list[str] = []
|
||||
n_runs_ok = 0
|
||||
for rel in sorted(subst_files):
|
||||
text = (ROOT / rel).read_text(errors="replace")
|
||||
# Absolute offsets, not per-line: the gitea/nextcloud bootstrap scripts are single
|
||||
# 4000-character folded scalars, so "the line" is not a reviewable unit of context here.
|
||||
nl = [i for i, ch in enumerate(text) if ch == "\n"]
|
||||
spans = [] # allowlisted [start, end) regions
|
||||
for e in allow_entries:
|
||||
if e["path"] != rel:
|
||||
continue
|
||||
start = text.find(e["ctx"])
|
||||
while start >= 0:
|
||||
spans.append((start, start + len(e["ctx"]), e))
|
||||
start = text.find(e["ctx"], start + 1)
|
||||
for m in DOLLAR_RUN_RE.finditer(text):
|
||||
n = len(m.group(0))
|
||||
if n < 2:
|
||||
continue
|
||||
braced = text[m.end():m.end() + 1] == "{"
|
||||
if n == 2 and braced:
|
||||
n_runs_ok += 1 # the documented escape; (1)-(3) own it
|
||||
continue
|
||||
covering = [e for (a, b, e) in spans if a <= m.start() and m.end() <= b]
|
||||
if covering:
|
||||
for e in covering:
|
||||
e["used"] += 1
|
||||
continue
|
||||
kind = "collapse" if not braced else ("half" if n % 2 else "over")
|
||||
tag, fix = RUN_TAGS[kind]
|
||||
lineno = bisect.bisect_right(nl, m.start()) + 1
|
||||
ctx = text[max(0, m.start() - 60):m.end() + 60].replace("\n", "\\n")
|
||||
run_findings.append(
|
||||
f" [{tag}] {rel}:{lineno} a run of {n} dollar(s)"
|
||||
f"{' before a brace' if braced else ''}\n"
|
||||
f" rendered by Flux Kustomization(s) {', '.join(sorted(subst_files[rel]))}"
|
||||
f" — postBuild substitution REWRITES it\n"
|
||||
f" {fix}\n"
|
||||
f" | ...{ctx}...")
|
||||
|
||||
check(not run_findings,
|
||||
f"{len(run_findings)} dollar run(s) in substituted paths that envsubst rewrites and "
|
||||
f"nothing justifies (the 2026-09-13 nextcloud-db class)")
|
||||
|
||||
for e in allow_entries:
|
||||
check(e["used"] > 0,
|
||||
f"dollar-collapse.allow:{e['line']}: STALE — its context is not present in "
|
||||
f"{e['path']} any more (or no longer contains a rewritten dollar run), so the entry "
|
||||
f"now excuses nothing. Delete it. A stale allowlist is how an opt-in decays into a "
|
||||
f"blanket exemption for whatever lands in that file next")
|
||||
|
||||
# --------------------------------------------- (5) overlay knobs exist in the base
|
||||
# Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
|
||||
# `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
|
||||
@@ -328,10 +600,27 @@ for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
||||
# ------------------------------------------------------- (6) the check is load-bearing
|
||||
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
check("flux-postbuild-substitution/check.sh" in L0,
|
||||
"ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped "
|
||||
"conformance selection can filter this check out of a gate run, so l0.sh must invoke "
|
||||
"it directly and unconditionally or the gate is not actually enforcing it")
|
||||
# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
|
||||
# It used to assert `"flux-postbuild-substitution/check.sh" in L0`: ci/l0.sh carried a hand-written
|
||||
# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
|
||||
# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
|
||||
# to write it). The same property is now asserted two ways: this test DECLARES
|
||||
# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
|
||||
# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
|
||||
# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
|
||||
# passing) is proven end-to-end and falsifiably by
|
||||
# conformance/tests/l0-static/whole-repo-invariants-wired.
|
||||
SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "flux-postbuild-substitution" / "chainsaw-test.yaml"
|
||||
SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
|
||||
check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
|
||||
"flux-postbuild-substitution/chainsaw-test.yaml does not declare "
|
||||
'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
|
||||
"when a change happens to touch one of the requirements it is labelled with. "
|
||||
"The path map keys only clusters/* to R-ENV-02, so an edit to a SUBSTITUTED manifest under apps/ or platform/ attributes nothing and this check would be filtered out of exactly the change that empties a variable (the 2026-09-01 Cilium k8sServiceHost CNI outage)")
|
||||
check("--include-scope" in L0,
|
||||
"ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
|
||||
"a purely requirement-scoped run again and the scope annotation is inert decoration — "
|
||||
"this check would then run only by coincidence")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
@@ -345,6 +634,10 @@ print("Flux postBuild substitution inventory (per Kustomization):")
|
||||
for line in inventory:
|
||||
print(line)
|
||||
print(f"\ncluster-vars knobs defined: {len(KEYS)} referenced: {len(referenced)}")
|
||||
print(f"dollar runs >=2 in substituted files: {n_runs_ok} documented $${{...}} escapes, "
|
||||
f"{sum(e['used'] for e in allow_entries)} justified by dollar-collapse.allow "
|
||||
f"({len(allow_entries)} entr{'y' if len(allow_entries) == 1 else 'ies'}), "
|
||||
f"{len(run_findings)} unjustified")
|
||||
|
||||
if ADVISORIES:
|
||||
print("\nADVISORIES (not failures):")
|
||||
@@ -356,6 +649,12 @@ if findings:
|
||||
for f in findings:
|
||||
print(f, file=sys.stderr)
|
||||
|
||||
if run_findings:
|
||||
print("\nHAZARDS — dollar runs envsubst rewrites between git and the container:",
|
||||
file=sys.stderr)
|
||||
for f in run_findings:
|
||||
print(f, file=sys.stderr)
|
||||
|
||||
print()
|
||||
if not FAILURES:
|
||||
print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "
|
||||
|
||||
Loaded 100 of 319 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user