Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 4.8
|
7a34590b19 |
No files matched your search
+31
-48
@@ -1,27 +1,19 @@
|
||||
# .gitea/workflows/gate.yml — the ENFORCED PR merge gate (Phase 6, WS-GITOSS).
|
||||
#
|
||||
# This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
|
||||
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `make -C ci gate` — so CI
|
||||
# runs BYTE-FOR-BYTE the same command a human runs in the devcontainer (R-TEST-02): the
|
||||
# runner just installs the pinned tools from ci/versions.env first. `make -C ci gate`
|
||||
# (ci/gate.sh) runs every stage in order — spec-immutability, lint, Kyverno L0,
|
||||
# render==golden, L1 offline render — and applies the R-TEST-03 decision, printing
|
||||
# "MERGE GATE: PASS" / "MERGE GATE: BLOCKED". It is the single required status check on
|
||||
# `main` (see branch protection). A non-zero exit fails the job, so a BLOCKED gate blocks
|
||||
# the merge.
|
||||
# Docker executor + DinD). It runs the STATIC layers of the conformance gate — the ones
|
||||
# docs/14 says run "without a cluster": spec-immutability (best-effort), lint, Kyverno L0,
|
||||
# the render==golden reproducibility check, and L1 offline render checks. It is the single
|
||||
# required status check on `main`
|
||||
# (see branch protection). `make -C ci lint l0 render-golden l1` here is byte-identical to a laptop run
|
||||
# (R-TEST-02) — the runner just installs the pinned tools from ci/versions.env first.
|
||||
#
|
||||
# The cluster-gated L2/L3 conformance stage SKIPS honestly here (exit 3, surfaced not faked
|
||||
# — HARD RULE 4) because this runner declares no cluster: `make -C ci gate` does not set
|
||||
# CLUSTER_AVAILABLE, so gate.sh reports the touched requirements UNVERIFIED without blocking
|
||||
# Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
|
||||
# no broad kubeconfig) driven by the separate `conformance` job below.
|
||||
#
|
||||
# NOTE (R-TEST-06 / gate-semantics): spec-immutability is ALSO surfaced as its own explicit
|
||||
# early step (below) so its result is a distinct, tolerant status in the Gitea UI; gate.sh
|
||||
# re-runs it as its first stage, which is the authoritative one. `make -C ci gate` here is
|
||||
# the byte-identical laptop gate; the extra pre-step is a read-only git-diff check and is
|
||||
# purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
|
||||
# is retained dormant in merge-gate.yaml.
|
||||
# The L2/L3 CONFORMANCE layer (behavioural + fault-injection, needs a live cluster) is
|
||||
# NOT wired yet — it is a deliberate design decision (a CI toolchain image + a scoped
|
||||
# in-cluster kubeconfig/RBAC for the runner). It is present below as an `if: false`
|
||||
# placeholder so the shape is visible and the wiring is a one-line flip once decided.
|
||||
# The full pipeline blueprint (spec-checkout, per-layer jobs, decision job) is retained
|
||||
# dormant in merge-gate.yaml.
|
||||
---
|
||||
name: gate
|
||||
|
||||
@@ -55,27 +47,19 @@ jobs:
|
||||
./.gitea/workflows/scripts/install-tools.sh l0
|
||||
./.gitea/workflows/scripts/install-tools.sh l1
|
||||
|
||||
# R-TEST-06: the spec is read-only to agents. An EXPLICIT early surfacing of the
|
||||
# spec-immutability check, so its result is a distinct, tolerant status line in the
|
||||
# Gitea UI before the (longer) aggregate gate log. The spec repo is NOT checked out
|
||||
# into CI here (separate repo; SPEC_REPO_SLUG not wired into this runner), so the guard
|
||||
# honestly SKIPS (exit 3) rather than faking green; a real drift/edit of a co-located
|
||||
# spec exits 1 and fails. Tolerated: exit 3 only. Call the script DIRECTLY, not via
|
||||
# `make spec-immutability`: GNU make masks a recipe's exit code as its own 2, so
|
||||
# `make spec-immutability || [ $? -eq 3 ]` would see 2 and fail. The AUTHORITATIVE
|
||||
# spec-immutability run is the FIRST stage of `make -C ci gate` below (gate.sh calls
|
||||
# the script directly, so it sees exit 3 → SKIP, not a mask); this pre-step is a cheap,
|
||||
# read-only git-diff duplicate that only adds early UI signal.
|
||||
# R-TEST-06: the spec is read-only to agents. The spec repo is NOT checked out into
|
||||
# CI here (it is a separate repo; SPEC_REPO_SLUG is not wired into this runner), so
|
||||
# the guard honestly SKIPS (exit 3) rather than faking green. A real drift/edit of a
|
||||
# co-located spec would exit 1 and fail. Tolerated: exit 3 only.
|
||||
# Call the script DIRECTLY, not via `make`: GNU make masks a recipe's exit code as
|
||||
# its own 2 on failure, so `make spec-immutability || [ $? -eq 3 ]` would see 2 and
|
||||
# fail. The script itself exits 3 on an honest skip (spec repo not checked out here),
|
||||
# which we tolerate; a real drift/edit exits 1 and fails the gate.
|
||||
- name: Spec immutability (R-TEST-06, skips if spec repo absent)
|
||||
run: ./ci/spec-immutability.sh || [ "$?" -eq 3 ]
|
||||
|
||||
# Aggregate merge gate: byte-for-byte the command a human runs in the devcontainer
|
||||
# (`make -C ci gate` -> ci/gate.sh). Runs spec-immutability, lint, Kyverno L0,
|
||||
# render==golden and L1 render, then the R-TEST-03 decision. Exits non-zero on a
|
||||
# BLOCKED gate, which fails this job. The cluster-gated L2/L3 stage SKIPS here (no
|
||||
# CLUSTER_AVAILABLE) — surfaced as UNVERIFIED, never faked green (see header).
|
||||
- name: Merge gate — make -C ci gate
|
||||
run: make -C ci gate
|
||||
- name: Static gate — lint + Kyverno L0 + render==golden + L1 render
|
||||
run: make -C ci lint l0 render-golden l1
|
||||
|
||||
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
||||
# ("not supported on GHES" — Gitea does not implement the artifact-v2 backend), which
|
||||
@@ -84,16 +68,15 @@ jobs:
|
||||
# Gitea-compatible @v3 action or Gitea's own artifact backend if retention is wanted.
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────
|
||||
# PLACEHOLDER — NOT WIRED (deferred; concrete plan written, awaiting operator decision).
|
||||
# L2/L3 conformance (Chainsaw behavioural + fault-injection) needs a LIVE cluster and is
|
||||
# gated behind vars.CLUSTER_AVAILABLE + a narrow secrets.CONFORMANCE_TRIGGER_KUBECONFIG.
|
||||
# The intended model: an in-cluster Chainsaw Job (conformance/incluster/) under its OWN
|
||||
# scoped ServiceAccount — CI holds NO broad kubeconfig, only a trigger credential. The
|
||||
# blocker is a real privilege-escalation design question (a `create jobs` credential can
|
||||
# create a Job running as the privileged runner SA), plus a runner image + repo-clone +
|
||||
# results-collection that don't exist yet. Full analysis + step sequence:
|
||||
# docs/plans/l2-l3-conformance-ci-wiring.md
|
||||
# The blueprint job shape is retained dormant in merge-gate.yaml.
|
||||
# PLACEHOLDER — NOT WIRED (STOP-and-report item, WS-GITOSS Phase 6).
|
||||
# L2/L3 conformance (Chainsaw behavioural + fault-injection) needs a LIVE cluster.
|
||||
# Turning this on is a design decision with two open sub-decisions:
|
||||
# 1. Toolchain: build+push a pinned CI image to the Gitea container registry and
|
||||
# reference it as the job image, vs. install-in-job from ci/versions.env (slower).
|
||||
# 2. Cluster creds: how the runner reaches the cluster — the intended model is an
|
||||
# in-cluster Chainsaw Job driven by its OWN scoped ServiceAccount (CI holds only a
|
||||
# narrow trigger credential; the runner mounts NO SA token today). That SA + RBAC
|
||||
# + the trigger secret do not exist yet.
|
||||
# Kept `if: false` so it never runs (shows skipped, never fakes green — HARD RULE 4).
|
||||
# ───────────────────────────────────────────────────────────────────────────
|
||||
conformance:
|
||||
|
||||
@@ -41,9 +41,3 @@ infra/tofu/.r2.env
|
||||
# ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
|
||||
# stage (which scans untracked files) from tripping on .claude/worktrees/.
|
||||
.claude/
|
||||
|
||||
# Vendored ansible collections (installed by ansible-galaxy for node re-provision; not tracked or linted)
|
||||
infra/ansible/collections/
|
||||
|
||||
# standup.sh DNS snapshots — run artifacts, not authored state
|
||||
.standup-dns/
|
||||
@@ -1150,887 +1150,3 @@ gitignored (agent worktrees were tripping the lint stage's untracked-file scan).
|
||||
Plex uses `/config` → Plex writes the ephemeral layer, is unclaimed, loses config on restart
|
||||
(seerr↔plex connectivity is actually fine — 200 on `/identity`). FIX PENDING operator go (resets
|
||||
Plex, needs re-claim): mount PVC at `/config` + wire `PLEX_CLAIM`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-08-30 — Upgrade campaign landed: cluster now k3s v1.36.4 / Debian 13 / Cilium 1.20.1 / Longhorn 1.12.1
|
||||
|
||||
The platform-wide upgrade campaign completed and is live. Verified current state of the dev cluster:
|
||||
|
||||
- **k3s v1.36.4+k3s1** (was v1.31.5+k3s1), 3-server embedded-etcd HA.
|
||||
- **Debian 13 (trixie)** node OS (was Debian 12 bookworm).
|
||||
- **Cilium 1.20.1** (was 1.16.5); **Gateway API v1.6.1** (was v1.2.x).
|
||||
- **Longhorn 1.12.1** (was 1.7.3).
|
||||
- cert-manager >=1.21; Velero chart 12.x / Velero v1.18; kube-prometheus-stack 88 (operator
|
||||
v0.93.1, Grafana 13.2.0); kube-vip v1.2.3.
|
||||
- Node RAM **24 GiB / 24576 MB** (tofu default; earlier docs said 16 GiB).
|
||||
- Each node now carries a **dedicated virtio2 SSD etcd disk** (`/dev/vdc`, `local-zfs`) isolating
|
||||
etcd WAL fsync from HDD Longhorn I/O — see `docs/plans/ssd-etcd-isolation.md`.
|
||||
|
||||
**The "LIVE STATE VERIFIED 2026-08-26" line above (k3s v1.31.5) is SUPERSEDED by this entry.**
|
||||
That older line is retained as historical record — this append is the current source of truth.
|
||||
Documentation and comments across the repo were reconciled to these facts in the same change;
|
||||
toolchain kubectl bumped 1.31.4 → 1.36.4 (the 1.31 pin exceeded the ±1 version skew against a 1.36
|
||||
cluster). Revalidation items deliberately left open: external-snapshotter v8.2.0 pin vs 1.36, the
|
||||
Cilium ext_authz/forward-auth workaround relevance under 1.20.1, and the conformance ImageVolume
|
||||
feature-gate assertions under 1.36.
|
||||
|
||||
## 2026-08-31 — Post-upgrade stabilization: SSO fixed, all UIs gated, 8 vCPU, CI live. **PM HANDOFF.**
|
||||
|
||||
Everything the 08-30 campaign deferred is resolved; the platform is fully green. **This entry is the
|
||||
current source of truth and the handoff for the next PM.** (Session ran on a different workstation; a
|
||||
fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` and start at Next Steps.)
|
||||
|
||||
### LIVE STATE (verified 2026-08-31, all green)
|
||||
- 3× k3s **v1.36.4+k3s1**, Debian 13, **8 vCPU / 24 GiB** each (raised from 6 — see #5). Proxmox host 24 cores, ~idle.
|
||||
- Cilium 1.20.1 (GW-API v1.6.1), Longhorn 1.12.1, cert-manager 1.21, Velero chart 12/v1.18, kube-prometheus-stack 88
|
||||
(Grafana 13.2), kube-vip v1.2.3, **Loki chart 7.x / v3.6.11**, **Kyverno chart 3.9 / v1.19** (the last two were the
|
||||
08-30 "deferred" items — now done). etcd on the dedicated SSD virtio2 disk per node.
|
||||
- All Flux kustomizations + HelmReleases Ready; 0 not-ready pods.
|
||||
- **SSO end-to-end**: native OIDC for grafana/gitea/nextcloud/semaphore/headlamp; the 5 forward-auth UIs
|
||||
(pihole/jackett/qbittorrent/radarr/sonarr) gated by per-app oauth2-proxy sidecars (fail-closed, 302→Keycloak).
|
||||
Platform-admins require TOTP MFA.
|
||||
- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
|
||||
required status check; runner `incluster-dind` online (ns gitea-actions).
|
||||
|
||||
### What changed this session (durable)
|
||||
1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
|
||||
Gateway L7 proxy (`enforce_policy_on_l7lb`) re-check a pod's egress policy against Keycloak's REAL backend
|
||||
identity on the split-horizon hairpin; `toEntities: world` no longer covers it → all app SSO 403'd
|
||||
("Access denied", `server: envoy`). Fix: explicit `toEndpoints` egress allow to the keycloak backend —
|
||||
`catalog/render.py` emits it for `auth.mode ∈ {oidc,forward-auth}`; headlamp/crafty hand-written. (68690e5, 8a424bc)
|
||||
2. **Keycloak MFA config bug**: browser-mfa conditional-OTP used a non-existent config key → "Invalid role name: null"
|
||||
→ 403 for every platform-admin login. Fixed to `condUserRole: platform-admin` in git + live via kcadm. (45deac9)
|
||||
3. **VPN egress restored**: the Debian 12→13 re-provision wiped k3s-dev-3's node-host OpenVPN (PIA); re-ran
|
||||
`vpn.egress.gateway.yml`. Re-provision runbook (infra/ansible/README.md) updated — node-host roles (k3s, kube-vip,
|
||||
vpn.egress.gateway) must be re-run after any reprovision.
|
||||
4. **Grafana OOM** (kps 65→88 ships Grafana 13) — the hidden blocker that stalled the whole kps upgrade + platform
|
||||
tier. limits 256→512→768Mi; Prometheus 2→3Gi. (b2b1aa5 and follow-ups)
|
||||
5. **Node vCPU 6→8** — steady CPU PSI was 15-23% / load at-or-over core count (no spike headroom); now PSI ~4.5-7.7%.
|
||||
bpg reboots the VM on a cores change → rolled ONE node at a time, waiting Longhorn healthy between. (676c37a)
|
||||
+ node CPU PSI/load alerts + a "Nodes — CPU/Memory Pressure" Grafana dashboard. (a720c73)
|
||||
6. **Security: 5 forward-auth UIs gated** with per-app oauth2-proxy sidecars (Option B, crafty's proven pattern) +
|
||||
per-app Keycloak clients + SOPS creds. Closes the un-gated-admin-UI exposure. (72d4720, b2752ae)
|
||||
7. **Gitea Actions CI finished**: `ci/touched-requirements.sh` now excludes comment-only R-ID prose, so the workflow
|
||||
runs the exact aggregate `make -C ci gate`. (27bc024)
|
||||
8. **external-snapshotter vendored** — was remote github kustomize bases; the kustomize-controller's per-reconcile
|
||||
`git fetch github.com` flaked past its 27s timeout → `BuildFailed` stalled the platform tier. Now local
|
||||
`crds.yaml` + `snapshot-controller.yaml` (offline, deterministic build). (7a79cea)
|
||||
9. **08-30 revalidations closed**: ImageVolume gate is now ON in 1.36 → conformance rationale comments corrected
|
||||
(d60e553); external-snapshotter v8.2.0 pin kept (CRDs served at v1, works on 1.36); Cilium native ExternalAuth
|
||||
evaluated → **DEFER** (see plan).
|
||||
|
||||
### Operational gotchas that STILL matter (for the next PM)
|
||||
- **Access**: `./infra/devcontainer/dev.sh run <cmd>` (kubectl/flux/helm/tofu/ansible/kcadm). Do NOT run `dev.sh`/`make
|
||||
gate` concurrently — execs into the one container interfere → transient failures, just retry. Deploy = `git push
|
||||
gitea main` → Flux. Gate before push: `dev.sh run make -C ci gate` → "MERGE GATE: PASS".
|
||||
- **NAT topology** (see PHASE1-STATUS): API via `192.168.2.50:6443`→VIP .1.49; node SSH `192.168.2.102:225X`;
|
||||
Proxmox API `192.168.2.102:8006`. A single node reboot can briefly drop the kubeconfig NAT — wait ~3 min for boot.
|
||||
- **Keycloak realm import is ONE-SHOT**: the operator logs "Job already executed - not recreating" — a git change to
|
||||
`keycloak-realm-import.yaml` does NOT re-import. Edit the live realm via `kcadm.sh` in keycloak-0 (admin creds:
|
||||
secret `keycloak-admin-credentials`; kcadm present, curl/python3 are NOT), AND fix git for reproducibility.
|
||||
- **Cilium world-hairpin (CVE-2025-30162)**: any in-cluster→`*.home` call hairpins through the Gateway and gets its
|
||||
egress re-checked against the DESTINATION's real cluster identity — `toEntities: world` won't cover it. render.py
|
||||
handles the keycloak case for SSO apps; a NEW app→app-by-hostname dependency needs an explicit `toEndpoints` allow.
|
||||
- **Remote github kustomize bases flake** (27s git-fetch timeout). external-snapshotter is now vendored;
|
||||
**platform/kubevirt still uses remote bases** — vendor it the same way if it `BuildFail`s.
|
||||
- **Longhorn on the HDD**: every node reboot degrades all 18 volumes → full rebuild (slow; 30%+ Proxmox IO-delay).
|
||||
Roll reboots one node at a time; wait `robustness=healthy` before the next. Fix = `docs/plans/longhorn-ssd-tiering.md`.
|
||||
- **Gitea** can go read-only (ext4 remount-ro) during node churn → `kubectl -n gitea rollout restart deploy/gitea`.
|
||||
|
||||
### NEXT STEPS (prioritized) — where to start
|
||||
1. **Longhorn → SSD hot-tier move** — `docs/plans/longhorn-ssd-tiering.md`. The "won't fit" blocker was a measurement
|
||||
artifact (working set ~79 GB; SSD ~323 GB free). Moving the hot tier (Prometheus/CNPG DBs/Loki/Gitea ≈ 51 GB) off
|
||||
the HDD removes the reboot-rebuild tax on every future maintenance op. **Highest-leverage infra item.** Pre-flight:
|
||||
re-verify live SSD free + read HDD SMART.
|
||||
2. **L2/L3 conformance in CI** — `docs/plans/l2-l3-conformance-ci-wiring.md`. Needs the trigger-credential decision
|
||||
(scoped in-cluster SA + Kyverno-pinned Job shape; never a broad kubeconfig in CI). The gate runs no-cluster stages today.
|
||||
3. **Forward-auth long-term** — `docs/plans/cilium-native-externalauth-evaluation.md` (DEFER now: experimental CRD
|
||||
channel + fail-open history). The 5 UIs are safely sidecar-gated in the meantime.
|
||||
4. **Remaining Phase-6 onboarding** — see DESIGN-waves.md / `catalog/` (services rendered-but-not-onboarded).
|
||||
5. **Vendor platform/kubevirt** remote bases proactively (same flake class as #8).
|
||||
6. **Human action**: finish Headlamp login = complete TOTP enrollment in the browser.
|
||||
|
||||
---
|
||||
|
||||
## 2026-08-31 — **PHASE 7 OPENED (prep only)**: first prod node on the bench
|
||||
|
||||
The operator added the first **production** node and opened roadmap Phase 7. This entry records the
|
||||
opening, the state of the hardware, the operator's three decisions, and the blockers that stand
|
||||
between "a node exists" and "prod reconciles from `clusters/prod`".
|
||||
|
||||
### Governance — Phase 7 entry gate, recorded not waived
|
||||
|
||||
`roadmap/phases.md` gates Phase 7 on **two** conditions: *"Phases 1–6 proven in dev/test; operator
|
||||
declares prod hardware ready."* The operator has declared hardware ready. **The first condition is
|
||||
not met** — conformance is 37/51, and the uncovered set includes **R-ENV-01/02/03**, the exact
|
||||
requirements prod provisioning serves. We proceed with *prep only* on that basis, with no prod
|
||||
install until the §7 repo blockers close and the gate is green. This is an accepted, explicit risk,
|
||||
not an oversight.
|
||||
|
||||
### Node 1 as-found (`192.168.2.10`, verified by SSH 2026-08-31)
|
||||
|
||||
Not blank — imaged and reachable. **HP EliteDesk 800 G5 Desktop Mini** (matches the spec's `3× mini-PC
|
||||
bare metal`), i7-9700T 8c/8t, **31 GiB RAM**, Debian 13 trixie / kernel 6.12, hostname already
|
||||
`k3s-prod-1`, `k3s` user reachable with the `home_network` key, VT-x + `/dev/kvm` present
|
||||
(KubeVirt-capable), k3s not installed. Firmware HP R21 Ver. 02.26.00.
|
||||
|
||||
**One 477 GB SATA SSD** (`C400-MTFDDAC512M` — a ~2011-era Micron/Crucial consumer drive) carrying
|
||||
everything: `/` 30G, `/var` 12G, swap, and `/srv` 433G. No second device. Dev, by contrast, runs two
|
||||
dedicated data disks per node (Longhorn + a dedicated SSD for etcd, added after HDD contention
|
||||
starved etcd WAL fsync).
|
||||
|
||||
### Operator decisions (2026-08-31)
|
||||
|
||||
1. **Phase 7 opens, prep work only** — repo work and node baselining now; no k3s install until the
|
||||
parameterization blockers land and the gate is green.
|
||||
2. **Wait for all three nodes before `cluster-init`** — prod comes up as spec-conformant 3-node HA
|
||||
embedded etcd on day one; no single-server-etcd window, no later member-add.
|
||||
3. **Check drive health before choosing the disk layout** — a 15-year-old consumer SSD is not an
|
||||
obvious home for prod state.
|
||||
|
||||
### Findings — prod would build WRONG today
|
||||
|
||||
1. **R-ENV-02 is violated in the base (the headline blocker).** The overlay model promises dev and
|
||||
prod differ *only* by `cluster-vars` values. In fact dev LAN addresses are hardcoded into base
|
||||
manifests: `k8sServiceHost: 192.168.1.50` (`platform/cilium/helmrelease.yaml:50` + the
|
||||
`values.yaml` mirror), the Cilium LB pool `192.168.1.53-.60`
|
||||
(`platform-config/cilium/l2-ip-pool.yaml`), crafty `.55`, the UniFi VM `.54`, and three S3 backup
|
||||
endpoints at `192.168.1.9:9000`. **Prod is on `192.168.2.x` — none of these follow.** Fix is to
|
||||
tokenize each into a `cluster-vars` knob and set it in both overlays; the `postBuild.substituteFrom`
|
||||
seam already exists on every relevant Kustomization. Note `platform/cilium/values.yaml` is only a
|
||||
*reference mirror* — the authoritative pair is the HelmRelease (Flux) and the Ansible j2 template
|
||||
(already parameterized), so the Ansible side needs no change.
|
||||
2. **`clusters/prod` never sets `GATEWAY_VIP`** → it would inherit the base default `10.73.0.240`,
|
||||
an address on the **retired libvirt network**. `cluster-vars`' own comment flags this as owed
|
||||
"when hardware is built".
|
||||
3. **Sudo on node 1 requires a password** — every Ansible playbook fails at its first `become`.
|
||||
**Operator ruling: prod does NOT get NOPASSWD sudo** (my initial recommendation to mirror dev was
|
||||
rejected, correctly — dev's passwordless sudo is a convenience that should not propagate to
|
||||
prod). Instead prod uses Ansible `become` with an **ansible-vault-encrypted**
|
||||
`ansible_become_password` in `inventory/group_vars/k3s_prod/vault.yml`, mirroring the existing
|
||||
`hypervisor` and `proxmox_hosts` groups; the decrypting passphrase stays outside the repo at
|
||||
`~/.config/home-k3s/ansible-vault-pass`. Plays still run unattended, no plaintext secret in git,
|
||||
and prod keeps a real sudo boundary.
|
||||
4. **`node_baseline` device defaults are virtio** (`/dev/vdb`, `/dev/vdc`) and do not exist on bare
|
||||
metal; the role makes a filesystem on a *block device*, so a single-disk node needs either a
|
||||
second drive, `/dev/sda5`, or a repartition. Prod `group_vars` must override.
|
||||
5. **Single upstream resolver on node 1** (`192.168.2.1`) — the same SPOF class that broke
|
||||
containerd pulls and CoreDNS forwarding on dev. The fix is **stranded unmerged** on
|
||||
`fix/node-dns-resilient-upstream`.
|
||||
|
||||
### RESOLVED BY OPERATOR — status comes from the repository, not the task board
|
||||
|
||||
I raised an escalation here: `AGENTS.md` §11 designates `agents/task-board.md` as the one surface
|
||||
agents may edit, yet `ci/spec-immutability.sh` fails the merge gate on **any** spec-repo dirt with no
|
||||
path exception, and committing the board there moves spec `HEAD` off `ci/SPEC_PIN` — which also
|
||||
fails. Editing the board as instructed blocks every merge. That is why the board still reads
|
||||
all-`todo` after six delivered phases.
|
||||
|
||||
**Operator ruling (2026-08-31): the task board is not the status surface. The repository is.**
|
||||
Status of each requirement/spec is to be *derived from the state of the repo* — what is authored,
|
||||
what is reconciled, what has a green requirement-tagged test — not from a hand-maintained board.
|
||||
|
||||
Consequences, adopted:
|
||||
- The spec board is left alone. It is a historical seed, not a live surface. No agent edits it, and
|
||||
no `SPEC_PIN` churn is spent keeping it current.
|
||||
- `ci/spec-immutability.sh` keeps its full strictness — **no path exception was added**. Weakening
|
||||
an R-TEST-06 gate to make a bookkeeping file writable would have been a defect, not a fix.
|
||||
- The **repo-derived spec status report** becomes the artifact of record, alongside the existing
|
||||
requirement-coverage report and this log. Coverage already answers "does a tagged test exist";
|
||||
the status view must also answer "is this requirement actually *implemented and reconciled* in
|
||||
the repo as it stands".
|
||||
|
||||
### Next steps
|
||||
|
||||
1. **Operator:** configure NOPASSWD sudo on `k3s-prod-1`, then capture `smartctl -a /dev/sda`
|
||||
(commands in `infra/PROD-PROVISIONING.md` §5). The disk decision waits on that output.
|
||||
2. **Land the R-ENV-02 tokenization** — the critical path for prod reconciling at all. Dev overlay
|
||||
must set every new knob to today's exact value so live dev sees zero drift (`make -C ci render-golden`).
|
||||
3. **Complete `clusters/prod`** — `GATEWAY_VIP`, the L2 pool, and the prod backup target.
|
||||
4. **Land the two stranded fix branches** (`fix/node-dns-resilient-upstream`,
|
||||
`fix/act-runner-dind-readiness`) — the DNS one now matters for prod too.
|
||||
5. Resolve the task-board escalation above.
|
||||
6. Nodes 2 and 3 through `infra/PROD-PROVISIONING.md` §1–§5 as they arrive.
|
||||
|
||||
---
|
||||
|
||||
## 2026-08-31 — CI is the enforcement point; the LOCAL gate produced a FALSE GREEN
|
||||
|
||||
**Operator directive: "Conformance and gates are enforced at the gitea runner and CI/CD level.
|
||||
Use it."** Adopted. The authority on whether a change is mergeable is the Gitea Actions run, not a
|
||||
local `make -C ci gate`. Local runs are a fast pre-check only.
|
||||
|
||||
This was proven immediately and expensively. Branch `docs/phase-7-open-prod-node-1`:
|
||||
|
||||
- **Local** `make -C ci gate` → `no changed files vs base; empty touched-requirement set` →
|
||||
**MERGE GATE: PASS**.
|
||||
- **CI**, same commit (`a1fb31a`, task 200) → 17 touched requirements correctly computed →
|
||||
`REQUIRE: FAILED — required requirement(s) uncovered: ['R-SEC-01','R-SEC-02','R-TEST-06']` →
|
||||
**MERGE GATE: BLOCKED**.
|
||||
|
||||
The local run resolved an empty diff base and therefore checked nothing, reporting green. **A local
|
||||
false green is the most dangerous failure mode this project has** — it hid real coverage debt from
|
||||
the PM and would have hidden it from any future agent. `ci/touched-requirements.sh` base-ref
|
||||
resolution is being fixed so local agrees with CI, or announces loudly that it cannot determine a
|
||||
base. An honest noisy skip beats a quiet false pass (`ci/README.md` exit-code contract).
|
||||
|
||||
The gate itself was RIGHT: we authored a vaulted-secret flow (R-SEC-01) and status tooling that
|
||||
reasons about spec immutability (R-TEST-06) while those requirements had **no covering test at
|
||||
all**. Tests are being authored rather than the rule being narrowed.
|
||||
|
||||
**How to read CI from a workstation** (the API paths are non-obvious, so recording them):
|
||||
```bash
|
||||
CRED=$(printf 'protocol=https\nhost=gitea.home.keithjackson.dev\n\n' | git credential fill)
|
||||
# list recent runs (id | workflow | branch | status | sha):
|
||||
curl -sk -u "$U:$P" "$B/api/v1/repos/Home-Network/home-network-k3s-impl/actions/tasks?limit=10"
|
||||
# fetch a task's log — NOTE the job-id/task-id OFFSET: task N is at jobs/(N+5)/logs on this instance
|
||||
curl -sk -u "$U:$P" "$B/api/v1/repos/Home-Network/home-network-k3s-impl/actions/jobs/205/logs"
|
||||
```
|
||||
`.../actions/runs/<run>/jobs` is 404 on this Gitea version; use `actions/tasks` + `actions/jobs/<id>/logs`.
|
||||
|
||||
### OPERATOR ATTESTATIONS (2026-08-31) — capability vs. automated proof
|
||||
|
||||
The operator states, and this log records as fact:
|
||||
|
||||
1. **DR is PROVEN** — Velero backups to the Proxmox server, restore demonstrated. R-STATE-03's
|
||||
capability is delivered and in operational use.
|
||||
2. **Cluster resilience is PROVEN at the dev-cluster level** — node loss / rebuild demonstrated.
|
||||
It is slow because Longhorn storage is on HDD (etcd is the exception, on its dedicated SSD),
|
||||
which is a performance property, not a correctness gap.
|
||||
|
||||
**Distinguish capability from automated proof.** The new repo-derived status report flags
|
||||
`R-STATE-03` as a safety-critical requirement whose L3 test body asserts nothing. That finding
|
||||
stands and is NOT contradicted by the attestation above — it is **test debt, not capability debt**.
|
||||
The system does the thing; the suite does not yet assert that it does. Per the project's own
|
||||
definition of done, only a green requirement-tagged test is acceptance — so the remaining work is
|
||||
to make `dr-restore/chainsaw-test.yaml` and the node-loss test actually assert the behaviour the
|
||||
operator has already demonstrated, and let the runner enforce it. Recording the attestation here
|
||||
does not close the requirement, and the gate must not be softened to make it appear closed.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-01 — Phase-7 prep session complete. **PM HANDOFF.**
|
||||
|
||||
Everything below is on branch `docs/phase-7-open-prod-node-1` / **PR #4**, 13 commits, the
|
||||
enforced Gitea gate **green**. Nothing has been installed on prod hardware. `main` is untouched
|
||||
until the operator merges.
|
||||
|
||||
### Delivered
|
||||
1. **Phase 7 opened (prep only)** with the entry-gate shortfall recorded, not waived.
|
||||
2. **T-7.1 checklist** (`infra/PROD-PROVISIONING.md`) written against the real hardware.
|
||||
3. **R-ENV-02 tokenization** — 6 `cluster-vars` knobs; prod would otherwise have come up
|
||||
announcing DEV addresses. Dev proven byte-identical.
|
||||
4. **Prod Ansible**: inventory, vaulted `become` (prod gets NO passwordless sudo), and
|
||||
**target-group parameterisation** so prod reuses the roles unchanged.
|
||||
5. **`node_health` role** — read-only, portable dev+prod, run against both.
|
||||
6. **Repo-derived spec status report** (`run.sh --status`).
|
||||
7. **CI false-green killed** + L0 static tests for R-SEC-01/02 and R-TEST-06.
|
||||
8. **dind readiness fix**; **DNS resilience re-authored** for the current stack.
|
||||
9. **`dr-restore` now asserts** — safety-critical scaffolds 0.
|
||||
|
||||
### Numbers
|
||||
Coverage 37/51 → **40/51**. Executable tests 32 → **36**. Scaffold-only 5 → **4**.
|
||||
Safety-critical without an executable L3 1 → **0**. **PROVEN: 0 — still not derivable**, by
|
||||
design; proof-of-pass is not recoverable from repo state, so a green-run ledger is the only
|
||||
thing that ever closes a requirement.
|
||||
|
||||
### Corrections made during the session (recorded so they are not re-litigated)
|
||||
- **NOPASSWD sudo on prod: REJECTED** by the operator. Prod uses vaulted `become`. My original
|
||||
checklist copied dev's convenience onto prod; that was wrong.
|
||||
- **The "`$i` envsubst bug" was FALSE.** An assessment claimed Flux envsubst ate bare `$i`,
|
||||
making the dind cert-wait bound a no-op. Running the real `flux envsubst` disproved it: bare
|
||||
`$VAR` passes through; only `${VAR}` expands — matching this repo's own contract in
|
||||
`clusters/base/apps-vms.yaml`, itself written after empirical probing. **No escaping was
|
||||
applied**; `$${i}` would have been an active regression (correct only on the envsubst path —
|
||||
on a plain `kustomize build`, `sh` reads `$$` as its PID). Agent reports are not evidence;
|
||||
the tool is.
|
||||
- **The stranded DNS branch was NOT landed.** Its netplan mechanism no longer exists on Debian
|
||||
13; landing it would have broken `node-baseline.yml` on every node.
|
||||
|
||||
### Near-misses worth remembering
|
||||
- A YAML **comment** nearly broke every merge in the repo: Flux envsubst runs over comments too,
|
||||
so a braced name with punctuation is a fatal parse error that fails the whole `apps-wave-1`
|
||||
Kustomization.
|
||||
- `roles/cilium`, `cluster_verify`, `flux_bootstrap` all hardcoded `.cache/dev.kubeconfig` — the
|
||||
file `dev.sh` exports as `$KUBECONFIG`. The moment prod became targetable, a prod run would
|
||||
have silently overwritten it and left the operator's `kubectl` answering from **prod**.
|
||||
- Prod's `GATEWAY_VIP` outside the LB pool would have brought the Gateway up on `.21` while
|
||||
split-horizon DNS answered `.20` — blackholing in-cluster SSO while reporting healthy.
|
||||
|
||||
### BLOCKED ON THE OPERATOR — nothing else moves without these
|
||||
1. **Create `inventory/group_vars/k3s_prod/vault.yml`** (`ansible_become_password`). The
|
||||
passphrase file already exists. This gates SMART, the packages-only baseline, and every
|
||||
future prod play.
|
||||
2. **Drive health decision** — run `smartctl -a /dev/sda` once `smartmontools` lands via
|
||||
`node-baseline.yml -e k3s_hosts=k3s_prod --tags packages`. Prod state on an unmeasured
|
||||
~2011 consumer SSD is the open risk.
|
||||
3. **Prod's second DNS upstream** — `192.168.2.1` is both gateway and only IPv4 resolver.
|
||||
4. **Prod backup target** — `S3_BACKUP_ENDPOINT` is still `TBD-PROD-*`; dev's versitygw is on
|
||||
the far side of the pfSense boundary.
|
||||
5. **`UNIFI_L4_VIP` / `CRAFTY_L4_VIP`** — allocatable from `.20-.30`, not invented here.
|
||||
6. **Merge PR #4.**
|
||||
|
||||
### Known, documented, NOT fixed
|
||||
- `roles/flux_bootstrap` shells to `bootstrap/flux/bootstrap.sh` with a hardcoded
|
||||
`--path=clusters/dev` — retargeting prod needs a `bootstrap/` change.
|
||||
- Prod has no VPN egress group (needs its own vaulted PIA credentials).
|
||||
- **`dr-restore` is AUTHORED, NEVER EXECUTED** (`conformance.home.k3s/never-executed`). It is
|
||||
destructive; converting it to proven needs an operator maintenance-window run. The log must
|
||||
show `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`, **not** `DR_RESTORE_SKIPPED_OK` — green but
|
||||
proving nothing.
|
||||
- `dind` → native sidecar `initContainer` with a `docker info` startupProbe (NOT a bare
|
||||
tcpSocket probe) is the right shape on k3s 1.36; deferred so a merge-gate failure is
|
||||
unambiguous.
|
||||
- **Non-ECC RAM — RISK ACCEPTED by the operator (2026-09-01), not an open item.** `node_health`
|
||||
correctly reports ECC status as unknowable (no EDAC controller bound). Operator rationale:
|
||||
this is a 3-node k3s homelab, not enterprise k8s; corruption on one node is outweighed by the
|
||||
other two and results in node ejection. That holds for the dominant failure mode — etcd raft
|
||||
(per-entry CRC + boltdb page checksums, quorum of 2 survives) and Longhorn (3 replicas, hard
|
||||
anti-affinity, rebuild from healthy peers) both detect a *divergent* node and eject/rebuild it.
|
||||
The residual it does not cover is corruption introduced *upstream* of replication: bytes
|
||||
corrupted before the checksum is computed replicate faithfully to all three replicas, so
|
||||
quorum sees no disagreement. Low probability, and the mitigation (ECC hardware) is out of
|
||||
scope for this platform. **Do not re-litigate.** The one thing that operationalises the
|
||||
operator's own argument is Longhorn's `snapshotDataIntegrity`, which is currently left at the
|
||||
chart default rather than set explicitly — see the follow-up in `platform/longhorn/`.
|
||||
- L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs
|
||||
the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium on 2/3 dev nodes. **Root cause: a substitution race that fails OPEN.**
|
||||
|
||||
Merging the R-ENV-02 tokenization (PR #4, `dce0e8f`) triggered a Cilium Helm upgrade that put
|
||||
`cilium` agents into `Init:Error` on `k3s-dev-1` and `k3s-dev-2`. It **self-healed in ~6 minutes**
|
||||
and the cluster is fully green (12/12 Kustomizations Ready, 0 HelmReleases unready, 0 pods down).
|
||||
Recording it because the mechanism is systemic, will recur, and is far more dangerous on a cold
|
||||
prod bootstrap than it was here.
|
||||
|
||||
### What happened
|
||||
|
||||
The HelmRelease config digest tells the whole story:
|
||||
|
||||
```
|
||||
v8 05:49:39 digest e83bf695… <- before
|
||||
v9 05:50:11 digest eebd6c88… <- the bad render
|
||||
v10 05:55:23 digest e83bf695… <- identical to v8 again
|
||||
```
|
||||
|
||||
The config changed and then changed **back to byte-identical**. Nothing in git differed
|
||||
semantically — `git diff` over `platform/cilium/helmrelease.yaml` shows only comments plus
|
||||
`k8sServiceHost: "192.168.1.50"` → `k8sServiceHost: "${K3S_API_HOST}"`, and the dev overlay sets
|
||||
`K3S_API_HOST=192.168.1.50`. The pre-merge byte-identity verification was **correct**; the steady
|
||||
state is identical. The damage happened in a transient intermediate state.
|
||||
|
||||
### Root cause — the var and its consumer are reconciled by DIFFERENT Kustomizations
|
||||
|
||||
- `cluster-vars` (which *defines* `K3S_API_HOST`) is owned by the **`flux-system`** Kustomization.
|
||||
- `platform/cilium/helmrelease.yaml` (which *references* it) is owned by the **`platform`**
|
||||
Kustomization, via `postBuild.substituteFrom: cluster-vars`.
|
||||
- There is **no dependency ordering between defining a variable and using it.**
|
||||
|
||||
So there is a window where `platform` renders the NEW manifest against the OLD `cluster-vars`,
|
||||
which has no `K3S_API_HOST` key yet. **Flux's post-build substitution is non-strict by default: an
|
||||
undefined variable becomes the EMPTY STRING, silently.** That produced `k8sServiceHost: ""`, Helm
|
||||
upgraded Cilium with it, and the agents fell back to the in-cluster ClusterIP `10.43.0.1:443` —
|
||||
which only the Cilium agent itself can translate. The agent was the thing restarting, so its own
|
||||
`config` init container could not reach the API server:
|
||||
|
||||
```
|
||||
level=error msg="Unable to contact k8s api-server" ipAddr=https://10.43.0.1:443
|
||||
error="dial tcp 10.43.0.1:443: i/o timeout"
|
||||
level=fatal msg="Build config failed"
|
||||
```
|
||||
|
||||
Once `flux-system` applied the updated `cluster-vars`, the next `platform` reconcile rendered the
|
||||
correct value, digest returned to `e83bf695…`, and all three agents came up.
|
||||
|
||||
`optional: false` on the `substituteFrom` entry does **not** protect against this — it only requires
|
||||
the ConfigMap to *exist*, not that every referenced key is defined.
|
||||
|
||||
### Why this matters much more for prod
|
||||
|
||||
On dev, one Cilium agent (`k3s-dev-3`) stayed up throughout, the API server was reachable, and the
|
||||
race closed on its own. **On a cold prod bootstrap every one of these knobs is new simultaneously**,
|
||||
there is no already-running CNI to fall back on, and an empty `k8sServiceHost` means Cilium may
|
||||
never come up at all — with no healthy peer to recover from. This is a plausible
|
||||
bring-up-blocker for `clusters/prod`.
|
||||
|
||||
### The fix — make substitution FAIL CLOSED
|
||||
|
||||
`kustomize-controller` currently runs with **no** `--feature-gates` argument, i.e. non-strict
|
||||
substitution. Flux supports `StrictPostBuildSubstitutions`, which makes an **undefined variable fail
|
||||
the Kustomization** instead of silently emptying it. That is exactly this project's stated doctrine
|
||||
("if unsure, deny/close"; "never fake green") applied to the substitution layer, and it converts
|
||||
this entire bug class from *silently corrupt a critical value* into *a loud, blocking reconcile
|
||||
failure*.
|
||||
|
||||
**Not enabled unilaterally** — turning it on cluster-wide would immediately fail any Kustomization
|
||||
that references a variable not defined in `cluster-vars`, including legitimate shell `${…}` in
|
||||
embedded scripts. The safe sequence, before flipping it:
|
||||
1. Enumerate every `${VAR}` in every substituted path and prove each is either defined in
|
||||
`cluster-vars` or correctly escaped as `$${VAR}`.
|
||||
2. Then enable the gate, and let CI + a dev reconcile prove it.
|
||||
|
||||
Interim discipline, independent of the gate: **add a `cluster-vars` knob in one commit and
|
||||
reference it in a later one**, so the definition is always reconciled before its first use.
|
||||
|
||||
### Operational note worth keeping
|
||||
|
||||
A Cilium rolling upgrade on this cluster is not risk-free even when the values are right: the
|
||||
`config` init container resolves the API server through the ClusterIP unless `k8sServiceHost` is
|
||||
set, and the agent that translates that ClusterIP is the one being replaced. Keep
|
||||
`k8sServiceHost`/`k8sServicePort` correct at all times, and roll Cilium one node at a time.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-01 — RESEARCH: Flux substitution + HelmRelease best practice. **Measured semantics + ranked plan.**
|
||||
|
||||
Three parallel research passes (substitution/ordering, HelmRelease/CNI, and a full repo audit),
|
||||
prompted by the incident above. Two of my own earlier conclusions were WRONG and are corrected
|
||||
here. Everything below marked *measured* was run against flux 2.4.0 / the live cluster; everything
|
||||
else is cited inference.
|
||||
|
||||
### THE SUBSTITUTION SEMANTICS TABLE (measured — trust this over prose elsewhere)
|
||||
|
||||
| form | behaviour |
|
||||
|---|---|
|
||||
| `${DEFINED}` | substituted |
|
||||
| `${UNDEFINED}` | **silently empty** (non-strict) · fails loudly (strict) |
|
||||
| `${UNDEFINED:-d}` | **default applied at reconcile time — `--strict` does NOT catch it** |
|
||||
| `${VAR:?msg}` | renders **the message as the value** — a non-empty WRONG value. ANTI-PATTERN. |
|
||||
| `${VAR:=192.168.1.50}` | silently ships the dev value to prod. ANTI-PATTERN. |
|
||||
| `$${VAR}` / `$${VAR:-d}` / `$${ARR[0]}` | literal — **escaping is TOTAL** |
|
||||
| `$VAR` / `$(cmd)` | untouched, no escaping needed |
|
||||
| `${...}` / `${a b}` / `${ARR[0]}` unescaped | **hard parse error — fails the whole Kustomization** |
|
||||
|
||||
Flux exempts default-providing operators (`-`, `:-`, `+`, `:+`, `=`, `:=`, `?`, `:?`) from strict
|
||||
mode by design ("they intentionally handle the unset case"). **That is why strict mode would never
|
||||
have caught the gitea/nextcloud dead-guard bug**, and why a targeted check was required.
|
||||
|
||||
**Doctrine corrected:** `clusters/base/apps-vms.yaml` claimed an indexed expansion is fatal
|
||||
"escaped or not". Measured: escaping is total; only the UNESCAPED form is fatal. Fixed in place.
|
||||
|
||||
**Comment nuance:** `kustomize build` strips *structural* YAML comments before envsubst sees them,
|
||||
so those are usually harmless. But `#` lines INSIDE a block scalar are data and DO reach envsubst,
|
||||
and a `$`-carrying body comment was observed surviving into built `apps/wave-1`. Do not rely on
|
||||
stripping — escape in comments too. (An earlier claim in this log that a YAML comment "nearly broke
|
||||
every merge" was overstated for the structural case; the hazard is real for block-scalar bodies.)
|
||||
|
||||
### CORRECTION 1 (mine) — `StrictPostBuildSubstitutions` alone is INSUFFICIENT
|
||||
|
||||
It fixes nothing about ordering; it converts "render a corrupt value and ship it" into "fail loudly
|
||||
and keep last-good". **The actual ordering fix is one line**, verified against
|
||||
`kustomization_controller.go :: checkDependencies` at our exact version (v1.4.0): `dependsOn` is
|
||||
**revision-gated**, not merely Ready-gated — when the dependency shares a GitRepository, the
|
||||
dependent cannot reconcile revision N until the dependency has APPLIED revision N.
|
||||
|
||||
`platform` is the only Kustomization in the graph with no `dependsOn`; every other consumer chains
|
||||
transitively through it. So:
|
||||
|
||||
```yaml
|
||||
# clusters/base/platform.yaml
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: flux-system # cluster-vars must be applied at THIS revision first
|
||||
```
|
||||
|
||||
**Cost, recorded honestly:** the root becomes a gate — a malformed manifest anywhere under
|
||||
`clusters/dev` now stops the whole graph where today it would not. Fail-closed, consistent with
|
||||
doctrine, but a real blast-radius change. No deadlock risk: the root runs `wait: false`.
|
||||
|
||||
Rejected alternative: moving `cluster-vars` into the consuming Kustomization is **strictly worse** —
|
||||
`SubstituteVariables` reads the ConfigMap from the API at build time, *before* the Kustomization
|
||||
applies its own resources, so it would always read the PREVIOUS value. That converts an occasional
|
||||
race into a permanent one-reconcile lag.
|
||||
|
||||
### CORRECTION 2 (an agent's) — the safety machinery DID fire
|
||||
|
||||
A research pass concluded that Helm considered the bad release healthy (because
|
||||
`updateStrategy.rollingUpdate.maxUnavailable: 2` on 3 nodes means `expectedReady = 1`), and
|
||||
therefore remediation never engaged. **Measured on the live cluster: that is wrong.** The Helm
|
||||
release secrets show **v9 = `failed`**; Helm's wait DID fail and the release was remediated, which
|
||||
is why it self-healed. `maxUnavailable: 2 of desired 3` IS confirmed live — so the real harm was
|
||||
blast radius (2 of 3 agents down at once), not a missed detection.
|
||||
|
||||
### Verified, and worth knowing
|
||||
|
||||
- **Flux 2.4.0 is OUT OF SUPPORT** — the project supports the last three minors; 2.9.x is current.
|
||||
- **Strict substitution became DEFAULT-ON** in kustomize-controller v1.9.0. So upgrading Flux flips
|
||||
this behaviour whether or not we opt in — the variable audit is a prerequisite for the upgrade
|
||||
regardless of today's decision.
|
||||
- **Cilium's `values.schema.json` permits `k8sServiceHost: ""`** — no `minLength`, no `pattern`.
|
||||
Helm was never going to reject the bad render.
|
||||
- **Cilium docs make `k8sServiceHost` REQUIRED under kube-proxy replacement**, precisely because the
|
||||
environment exports the API ClusterIP and there is no kube-proxy to provision it.
|
||||
- **The Cilium chart ships zero test templates**, so `spec.test` is useless for it. A post-upgrade
|
||||
check must live outside Helm.
|
||||
- `valuesFrom` fails HARD on a missing key (verified in `chartutil/values.go`) unlike postBuild's
|
||||
silent empty — but only on a MISSING key, not an empty value, and it cannot cross namespaces
|
||||
(`cluster-vars` is in `flux-system`; the Cilium HelmRelease is in `kube-system`).
|
||||
- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** It compares the cluster against the
|
||||
Helm storage manifest — during the incident that manifest WAS the bad render, so drift correction
|
||||
would have actively re-asserted the broken value.
|
||||
|
||||
### RANKED PLAN
|
||||
|
||||
| P | Action | State |
|
||||
|---|---|---|
|
||||
| **P0** | `platform.dependsOn: [flux-system]` — closes the race completely | **PENDING — operator call** (blast-radius change) |
|
||||
| **P0** | Escape the dead OIDC guards; renderer now refuses the class | ✅ DONE |
|
||||
| **P1** | Substitution-safety L0 gate, unconditional in `ci/l0.sh` | ✅ DONE — 102 assertions |
|
||||
| **P1** | Enable `StrictPostBuildSubstitutions=true` | **READY — proven safe**, all 11 paths exit 0 under `flux envsubst --strict`. Sequence AFTER P0. |
|
||||
| **P2** | `updateStrategy.rollingUpdate.maxUnavailable: 1` on Cilium, as a new **[parity]** field in BOTH the HelmRelease and the Ansible j2 | PENDING — bounds blast radius to one node |
|
||||
| **P3** | Split Cilium into its own Kustomization (`wait: true`, `interval: 1h`, `timeout: 10m`); `platform` dependsOn it | PENDING — gives `flux suspend kustomization cilium` as an emergency valve |
|
||||
| **P4** | Upgrade Flux 2.4.0 → 2.9.x | PENDING — out of support; brings strict-by-default |
|
||||
| **P5** | Cilium MINOR bumps become manual (Renovate PR → human runs the documented pre-flight → merge). Never automerge a CNI minor. | PENDING — process |
|
||||
| — | Record `${VAR:?}` / `${VAR:=}` as anti-patterns | ✅ DONE (table above) |
|
||||
| — | *Not recommended*: drift detection on Cilium; `spec.test` on Cilium; `k8sServiceHost: auto` | — |
|
||||
|
||||
### Follow-ups surfaced, not yet actioned
|
||||
- 6 dead `cluster-vars` knobs; `BASE_DOMAIN` and `CLUSTER_ENV` look like genuine rot.
|
||||
- `platform/{cilium,longhorn,velero,oauth2-proxy}/values.yaml` reference mirrors carry decorative
|
||||
`${...}` that can silently drift from the real HelmRelease.
|
||||
- `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time
|
||||
out while a legitimate Longhorn upgrade is still running.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-01 — P0/P1/P2 LANDED + a second incident: two bugs stacked, the outer one hid the inner
|
||||
|
||||
All three approved changes are merged (PR #5) and reconciled on dev, plus a hotfix (PR #6).
|
||||
|
||||
### Landed and verified on the live cluster
|
||||
|
||||
| Change | Verified |
|
||||
|---|---|
|
||||
| **P0** `platform.dependsOn: [flux-system]` | Flux on `63002d1`; all 12 Kustomizations Ready; steady-state graph unchanged |
|
||||
| **P1** strict substitution | Committed to `bootstrap/flux/components/`. **NOT YET ROLLED OUT** — see below |
|
||||
| **P2** Cilium `maxUnavailable: 1` | Applied. **Pod ages unchanged (15h/39h), restart counts unchanged** — the clean proof nothing rolled, exactly as predicted |
|
||||
| OIDC guard fix | Deployed guards now read `${..._OIDC_CLIENT_SECRET:-}` instead of `""`; both providers registered |
|
||||
| Prod overlay | SOPS decryption (same age key as dev) + source on `main` |
|
||||
|
||||
### THE SECOND INCIDENT — nextcloud CrashLoopBackOff
|
||||
|
||||
Immediately after the PR #5 reconcile, `nextcloud` went `CrashLoopBackOff` (7 restarts,
|
||||
`FailedPostStartHook`) and `apps-wave-1`/`apps-wave-2` stalled. Cause:
|
||||
|
||||
```sh
|
||||
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries" # LAST line of the hook
|
||||
```
|
||||
|
||||
On the **success** path (`p < 15`) the test returns 1, and as the script's last command that became
|
||||
the hook's exit status. **A non-zero postStart exit makes the kubelet kill the container.**
|
||||
|
||||
**The previous commit did not introduce this — it UNCOVERED it.** The bug was unreachable for as
|
||||
long as the OIDC guard above it was dead code that `exit 0`'d first. Fixing the guard made the block
|
||||
execute for the first time in these apps' deployed lifetime, and the latent defect fired instantly.
|
||||
**Two bugs stacked, and the outer one hid the inner one.** Worth remembering as a pattern: fixing
|
||||
dead code is not a safe no-op — it is the first execution of never-tested code, and should be
|
||||
treated with the caution of new code, not of a one-character escape.
|
||||
|
||||
Fixed by converting to a plain `if` and adding an explicit `exit 0`. Gitea's hook already returned 0
|
||||
on both paths, but only by accident of which statement happened to be last — made explicit there too
|
||||
so the class cannot return. Recovery confirmed: both pods `1/1 Running`, 0 restarts, all
|
||||
Kustomizations Ready, both OIDC providers still registered.
|
||||
|
||||
### STILL TO DO — strict substitution is committed but NOT ACTIVE
|
||||
|
||||
`StrictPostBuildSubstitutions` does **not** arrive on a git push. This repo does not use
|
||||
`flux bootstrap`; the controllers were installed imperatively by `flux install` and carry no Flux
|
||||
ownership labels, so they are outside Flux's inventory and nothing under `clusters/**` can reach a
|
||||
controller flag. An operator must run `bootstrap/flux/bootstrap.sh` step 1 (idempotent; the
|
||||
standalone command is in `bootstrap/flux/README.md`), which restarts kustomize-controller for
|
||||
~10-20s. Flux is level-triggered, so an interrupted reconcile is simply retried.
|
||||
|
||||
**Ordering is satisfied:** P0 is merged and reconciled, so the gate can now be rolled out safely.
|
||||
With the gate on but the ordering edge absent, the same race would turn a transient empty render
|
||||
into a transient hard Kustomization failure.
|
||||
|
||||
### Prod consequences recorded this session
|
||||
|
||||
- **Every merge to `main` will reconcile BOTH clusters** once prod is live — no promotion step, no
|
||||
soak window. A deliberate consequence of dev being temporary, but it removes the "prove it on dev
|
||||
first" safety net that the last two days have leaned on heavily, twice.
|
||||
- **Gitea migration is the FIRST task of the dev decommission, not the last.** Prod's GitOps source
|
||||
currently points at Gitea running on the dev cluster.
|
||||
- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
|
||||
with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
|
||||
Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
|
||||
|
||||
## 2026-09-05 — **PROD IS UP.** Overnight standup: 12/13 tiers, LE wildcard, dev untouched
|
||||
|
||||
Operator asleep; standing instruction was "finish out what is necessary and I want it working by
|
||||
the morning." **Prod is serving.** Dev was verified unharmed at every step.
|
||||
|
||||
### Final state
|
||||
|
||||
| | prod | dev |
|
||||
|---|---|---|
|
||||
| Kustomizations | **12/13 Ready** (`velero` suspended, deliberately) | **13/13 Ready** |
|
||||
| Source | bootstrap Gitea `main@c8c1d29` | bootstrap Gitea `dev-live@a4a3c2e` |
|
||||
| `BASE_DOMAIN` | `prod.home.keithjackson.dev` | `home.keithjackson.dev` (apex) |
|
||||
| Pods | **113, 0 not Running/Completed** | healthy, all HelmReleases Ready |
|
||||
| Wildcard cert | **Ready** — LE prod, `*.prod.home.keithjackson.dev` | unchanged |
|
||||
| DNS | **36 records**, every TXT `owner=prod` | **57 records, byte-identical to baseline** |
|
||||
|
||||
End-to-end HTTPS with certificate validation, pinned to prod's gateway: apex `200`,
|
||||
keycloak `302` (login), gitea `200`, grafana `302` (SSO).
|
||||
|
||||
### What was actually blocking, and what it taught
|
||||
|
||||
**Both blockers were the same bug in different clothes: Flux server-side dry-runs every object in a
|
||||
Kustomization before applying any of them, so ONE unknown `kind` aborts the entire tier.**
|
||||
|
||||
1. **Kyverno** — chart sets `crds.install: false` (5.6 MiB bundle > kubectl's 256 KiB annotation
|
||||
limit) and nothing else installed them. Its pods had **55 restarts each over 4.5h**, controllers
|
||||
with no API types to watch. The failed install then tripped `install.remediation.retries: 3`,
|
||||
whose uninstall ALSO timed out, wedging the release in `uninstalling` — a state helm-controller
|
||||
cannot advance from. Cleared with `helm uninstall` + a cold reinstall.
|
||||
2. **CDI — a true deadlock, and the one worth remembering.** `platform-config` applies the CDI CR
|
||||
*and* `cdi-storageprofiles.yaml` in the SAME pass, but `StorageProfile`'s CRD does not exist until
|
||||
`cdi-operator` reconciles that CR. Dry-run fails → CR never applied → CRD never created → the tier
|
||||
can NEVER converge. `cdi-operator` had been Available for **12h** with no CR. **Invisible on dev**
|
||||
(its CDI CRDs have existed for months) **and fatal on any cold cluster.** Broken by pre-applying
|
||||
only the CR; the CRD appeared in 10s.
|
||||
|
||||
> **DURABLE FIX STILL OWED:** move `cdi-storageprofiles.yaml` out of `platform-config` into its own
|
||||
> Kustomization with `dependsOn: platform-config`. Not done overnight — it edits the reconciliation
|
||||
> graph both clusters share, while prod reads a pinned revision and dev sits mid-migration on a
|
||||
> separate branch. Until it lands, `standup.sh crds` is a prerequisite on any cold cluster.
|
||||
|
||||
### The armed footgun, disarmed
|
||||
|
||||
Prod's external-dns Deployment was running `--policy=sync --registry=txt
|
||||
--domain-filter=keithjackson.dev` **with no `--txt-owner-id`** — it would have defaulted to `default`,
|
||||
the owner of **all 36 of dev's TXT registry entries**, and `policy: sync` would have deleted the
|
||||
records behind them. The 2026-09-04 incident, re-armed. **The only thing preventing it was
|
||||
`replicas=0` — manual drift that any Helm upgrade resets to the chart default of 1. That is not a
|
||||
guard, it is luck.**
|
||||
|
||||
The repo was never at fault: the live HelmRelease `spec.values` carried `txtOwnerId: "prod"`. The
|
||||
fault was Helm/Flux STATE — the installed release predated `c8c1d29`, yet helm-controller reported
|
||||
`Ready=True` with `configDigest == lastAttemptedConfigDigest`, believing it had already deployed the
|
||||
current config, so it would never have upgraded on its own.
|
||||
|
||||
**The fix pattern is the reusable part: PROVE THE OUTCOME BEFORE CAUSING IT.** A Helm upgrade sets
|
||||
replicas to 1 and applies new args in one operation — there is no window to check first. So
|
||||
`standup.sh externaldns` renders the chart with the LIVE values and asserts `--txt-owner-id=<owner>`
|
||||
appears, aborting if not. Rendering is deterministic and touches nothing, which converts a race into
|
||||
a precondition.
|
||||
|
||||
### Three bugs in my own verification — recorded because a check you cannot trust is worse than none
|
||||
|
||||
1. **`SNAP_DIR` defaulted to `$TMPDIR`.** Every `dev.sh run` is a fresh container, so the DNS baseline
|
||||
was discarded between steps and the before/after comparison — the check that detects damage to the
|
||||
other cluster — could never run across invocations. Now repo-relative and gitignored.
|
||||
2. **The first real comparison cried wolf**, reporting "RECORDS OUTSIDE prod.home CHANGED" and listing
|
||||
pairs of IDENTICAL lines. Locale-dependent sort collation: the host ordered `*.keithjackson.dev`
|
||||
differently from the container. Proved it was noise under `LC_ALL=C` (all 57 identical); snapshots
|
||||
now strip CRs and sort under a fixed collation.
|
||||
3. **preflight `die`d on `EXTERNALDNS_TXT_OWNER_ID=default`** — dev's legitimate id — so a script
|
||||
claiming "no dev/prod fork" (R-ENV-02) could never run against dev. A fork in disguise. preflight
|
||||
is now a pure diagnostic; the refusal moved to the mutating path where acting on a bad value is
|
||||
what actually causes harm.
|
||||
|
||||
### I also fell into the pfSense trap — ACTION REQUIRED
|
||||
|
||||
The local override is a wildcard on `*.home.keithjackson.dev` and now swallows the whole new
|
||||
`prod.home.*` subtree, resolving **every prod hostname to `192.168.2.50` (dev)**. Cloudflare
|
||||
correctly returns `192.168.2.20`.
|
||||
|
||||
**Prod is reachable from anywhere using public DNS, but NOT from behind pfSense.** Narrow the
|
||||
override, or add one pointing `prod.home.*` at `192.168.2.20`.
|
||||
|
||||
It is subtle because it **partially works**: dev's `*.home.keithjackson.dev` wildcard matches
|
||||
`prod.home.keithjackson.dev` (one label) so the apex returns 200 over valid TLS, while
|
||||
`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. I read that
|
||||
as a broken prod cert before checking the resolver. **This is the same trap that produced the false
|
||||
healthy reading during the incident. Never verify DNS or TLS here through a local resolver.**
|
||||
|
||||
### Still open for the operator
|
||||
|
||||
- **`S3_BACKUP_ENDPOINT` is `TBD-PROD-*`** — `velero` stays suspended, and note `apps-wave-0` consumes
|
||||
it too and applies GREEN while anything writing there fails at runtime. **Prod has no backup
|
||||
coverage.** Needs a real prod endpoint (prod cannot reach dev's `192.168.1.9:9000`).
|
||||
- **Dev's migration off the apex** (§10a sequence D) — Keycloak issuer, ~12 OIDC redirect URIs, a
|
||||
rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
|
||||
- **Dev's GitOps source is this workstation** — `192.168.2.132` is a **DHCP lease on wifi**. A lease
|
||||
change stalls dev's fetching (cluster keeps serving; fix is a `kubectl patch`). Static reservation.
|
||||
- **Six L2 tests hardcode the apex** and must be parameterized per environment.
|
||||
- Nothing merged to `main`; nothing pushed. `infra/PROD-STANDUP.md` is the command sequence.
|
||||
|
||||
## 2026-09-05 (afternoon) — **PM autonomous session.** Backups to R2, CDI deadlock repaired, coverage 82%→88% — and a guard I broke
|
||||
|
||||
Operator away ~2h, PM directed to work autonomously through agents. Four agents ran;
|
||||
every result was re-verified by the PM before commit. **Nothing merged, nothing pushed.**
|
||||
Both clusters healthy throughout — dev 13/13, prod 12/13 (velero deliberately suspended).
|
||||
|
||||
Commits: `32d6766` R2 · `765ad03` CDI · `59ffbaf` L2 params · `8f9b2e4` s3check · `fe45d0f` coverage.
|
||||
|
||||
### THE MISTAKE, AND WHY IT MATTERS MORE THAN THE FEATURES
|
||||
|
||||
In `32d6766` I changed the backup placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to
|
||||
`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`. The guards in `standup.sh` matched
|
||||
`TBD-*` — a **prefix** glob. `fnmatch('https://TBD-R2-…','TBD-*')` is **False**, so all three
|
||||
guards silently died, and **`resume` would have enabled velero on prod against a nonexistent
|
||||
hostname**: installs clean, reports healthy, no backup exists. Precisely the failure the guard
|
||||
existed to prevent, disarmed by a value change in a *different file*.
|
||||
|
||||
**The class, now seen three times today** (this; the CRD count that read a blank line as `1`;
|
||||
two coverage checks that passed when perturbed): *a check that degrades to success*. The
|
||||
structural lesson is narrower and worth keeping — **a guard keyed on a value's SHAPE, where the
|
||||
value lives in another file, has no test tying the two together.** Fixed by substring match;
|
||||
a test for the coupling is **not** yet written and should be.
|
||||
|
||||
### Operator decisions waiting
|
||||
|
||||
1. **R-TLS-02 "revoked" is NOT satisfied — a real platform gap.** The spec requires certs be
|
||||
"issued, renewed, **and revoked** without manual steps"; docs/08 reads revocation as a
|
||||
declarative removal. That needs `enableCertificateOwnerRef: true`, which
|
||||
`platform/cert-manager/helmrelease.yaml` does not set (chart default false). **Verified on
|
||||
both live clusters: `gateway/wildcard-home-tls` has ZERO ownerReferences** — deleting the
|
||||
Certificate leaves the private key serving indefinitely.
|
||||
**Deliberately not flipped.** The wildcard Certificate is under `./platform`, which runs
|
||||
`prune: true`. With owner refs ON, a transient prune of that Certificate — *exactly what
|
||||
happened to the CDI StorageProfiles today* — deletes the SERVING TLS Secret for every host,
|
||||
and reissuance is bounded by LE's duplicate-certificate limit (5/week per identical name set),
|
||||
risking **no trusted cert for a week**. Security gap vs plausible week-long outage: operator's
|
||||
call. The L2 test asserts the requirement as written and **will fail until decided** — it was
|
||||
not softened.
|
||||
2. **R2 needs the two token values.** `infra/BACKUP-R2.md` is the procedure. The **account id is
|
||||
already committed** (`infra/tofu/backend.r2.hcl` → `7ce452a1b3d6c86daf2c020edcb82d37`), so if
|
||||
the buckets go in that account only the token is missing. Placeholder kept on purpose.
|
||||
3. **The apex flip is also the legacy-prod 443 cutover.** The SOHO router NATs inbound 443 to the
|
||||
pfSense WAN for the non-k3s prod. §10a sequence P does not cover repointing it.
|
||||
|
||||
### Merge preconditions — read before merging anything here
|
||||
|
||||
- **CDI split:** `flux get kustomization platform-config` must be **Ready on the target first**.
|
||||
Otherwise the two StorageProfiles are pruned and sit at CDI defaults until it recovers (prod was
|
||||
in that state hours earlier). Measured exposure: CDI unaided infers **ReadWriteOnce**
|
||||
(`longhorn-static`) where we author **ReadWriteMany** (`longhorn-vm`) — so a DataVolume created
|
||||
in the ~30–60s window is **permanently non-live-migratable**. `cloneStrategy` is *not* at risk.
|
||||
- **R2 switch RESETS dev's PITR window** — fresh kopia repo, new CNPG base backups, the 30-day
|
||||
Barman window rebuilds from zero. The MinIO copy is retained and restorable.
|
||||
- **R-STATE-03 is PROVEN MinIO-ONLY** and its ledger row predates the R2 move. `BACKUP-R2.md` §5
|
||||
is the re-proof and it ends in a **restore**, not an upload.
|
||||
|
||||
### Findings worth keeping
|
||||
|
||||
- **Second-order CRDs need a third pass.** A CR whose CRD is created by a *controller at runtime*
|
||||
cannot share a Kustomization with resources of that CRD. `cdi-operator` registers exactly one
|
||||
CRD; `storageprofiles`/`datavolumes`/`datasources` appear only after it reconciles the CDI CR.
|
||||
Audit found `StorageProfile` was the only second-order kind in `platform-config`.
|
||||
- **`dns-exposure`'s teardown is ASYNCHRONOUS.** It reports PASS when chainsaw deletes the
|
||||
namespace; the Cloudflare record is reaped later by external-dns's 1-minute sync. A green run
|
||||
does **not** prove cleanup — if external-dns were suspended the record would persist silently.
|
||||
(Observed, then observed clearing.)
|
||||
- **The apex was not the only per-environment literal in the L2 suite** — two DNS tests hardcoded
|
||||
dev's LB range `192.168.1.53-.60`. They now assert the *observed* `GATEWAY_VIP` / assigned LB IP,
|
||||
which is stricter as well as portable.
|
||||
- **Velero's kopia backend has no checksum knob**, so BSL metadata and PVC-data R2 compatibility
|
||||
are **separate questions**. CNPG's boto3 1.35.60 is below the 1.36 default-checksum line — clear
|
||||
today, but **a CNPG image bump is an R2 re-verification event** (barman-cloud has no escape hatch).
|
||||
- **`ansible-inventory --list` prints DECRYPTED vault values** (the passphrase file is mounted into
|
||||
the dev container). It emitted the prod become-password to an agent's stdout. Never put it in CI.
|
||||
Confirmed no plaintext reached disk.
|
||||
|
||||
## 2026-09-06 — Prod solidified. Node 2 at full 32 GB, storage recovered, four latent cold-start bugs closed
|
||||
|
||||
Operator away; PM directed to oversee, delegate, and pay down debt while skipping anything needing
|
||||
their input. Five agents ran; every result re-verified by the PM before commit. **Nothing merged,
|
||||
nothing pushed. Both clusters healthy.**
|
||||
|
||||
Commits: `04569ab` node-maintenance runbook · `da3e9cc` Kyverno/CDI · `a984442` timezone + guard test.
|
||||
|
||||
### PROD IS FUNCTIONING
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Nodes | 3/3 Ready, **all three at ~`32638256Ki`** — node 2's DIMM upgrade landed, matching 1 and 3 to within 4 KiB |
|
||||
| Flux | **12/13 tiers Ready** (`velero` suspended: no backup endpoint) |
|
||||
| Pods | **112 Running**, 0 Running-but-NotReady |
|
||||
| Storage | **17/17 volumes healthy**, replicas symmetric 17-running+1-stopped per node |
|
||||
| TLS | LE wildcard `*.prod.home.keithjackson.dev` Ready |
|
||||
| etcd | 3 members, survived an ungraceful power-down without losing a member |
|
||||
|
||||
The memattr saga (§0b) is **closed on the hardware side**: with the 7.1.8 backports kernel, all three
|
||||
nodes address their full 32 GB.
|
||||
|
||||
### THE FINDING THAT MATTERS MOST — a failed drain silently stops storage self-healing
|
||||
|
||||
`kubectl drain` **cordons before it fails**, and Longhorn's `disable-scheduling-on-cordoned-node` is
|
||||
`true`. After prod-2's memory-upgrade reboot, **all 17 volumes sat `attached/degraded` with prod-2
|
||||
holding 2 of 18 replicas and ZERO rebuilds in flight** — nothing driving recovery, every pod
|
||||
`Running`, no alert. `kubectl uncordon` started rebuilds immediately; converged 17/17 in ~6 minutes.
|
||||
|
||||
**Uncordon is not tidying up — it is what re-enables redundancy.** Full runbook + post-maintenance
|
||||
checklist in `infra/PROD-STANDUP.md` §Node maintenance, including why `drain` can never complete here
|
||||
(instance-manager PDBs at 0 allowed disruptions; at 3 nodes/3 replicas replica eviction has nowhere to
|
||||
go, which is structural, not a setting).
|
||||
|
||||
### FOUR BUGS OF ONE SHAPE — *a thing that looks applied and isn't*
|
||||
|
||||
Worth naming as a class, because it has now appeared four times in two days:
|
||||
|
||||
1. **Kyverno's own CRDs** — `crds.install: false`, nothing else installed them; pods crash-looped 55×.
|
||||
2. **CDI StorageProfiles** — CR and its dependent resource in the same Flux pass; the dry-run aborts
|
||||
the tier, so the CR never applies and its CRD never appears. *Repaired in the graph* (`765ad03`).
|
||||
3. **The placeholder guard** — a `TBD-*` prefix glob stopped matching when the value it guards changed
|
||||
*in a different file*; `resume` would have enabled velero against a nonexistent host. Now tested
|
||||
(`a984442`).
|
||||
4. **PolicyException** — the CRD is installed but every controller runs
|
||||
`--enablePolicyException=false`. An exception would apply cleanly through Flux and be **silently
|
||||
ignored**. Verified live on prod.
|
||||
|
||||
Two more of the same family: a coverage check that passed when perturbed (asked a question always
|
||||
satisfiable), and `ci/touched-requirements.sh` having **no `bootstrap/*` entry**, which scoped the L0
|
||||
run away from the very check guarding the placeholder class. Both fixed.
|
||||
|
||||
**The rule:** dev's age hides cold-start defects. Anything that only runs on a fresh cluster has never
|
||||
been exercised on dev, and prod is the first honest test. And: *a guard keyed on the shape of a value
|
||||
that lives in another file needs a test tying the two together.*
|
||||
|
||||
### Kyverno blocking CDI — fixed by SATISFYING the policy, not exempting it
|
||||
|
||||
Prod's UniFi VM had no disk: Kyverno denied CDI's generated `importer-*` pod for missing the
|
||||
`auth-mode` / `egress-class` labels, which CDI cannot set. Fixed with a **mutate** policy that adds
|
||||
them when the requester is ServiceAccount `cdi/cdi-sa` — identity from AdmissionReview userInfo, **not
|
||||
a name prefix** (a prefix is attacker-choosable; the test proves it with a forged
|
||||
`importer-prime-forged-by-an-app` fixture that is still denied). Two-way proof: same pod, different
|
||||
requester, opposite outcomes. Kyverno L0 17 → 29 tests.
|
||||
|
||||
Rejected `exclude` scoped to the same SA because any userInfo match forces `background: false`, which
|
||||
would drop background scanning of R-NET-05/R-IDP-02 for **all existing pods** to fix one pod class.
|
||||
|
||||
### Conformance 82.4% → 88.2%
|
||||
|
||||
45/51 scaffolded, 0 safety-critical gaps. Six L2 tests made environment-agnostic (they derive
|
||||
`BASE_DOMAIN`/`GATEWAY_VIP` from the cluster they run against, so they cannot drift); R-TLS-02,
|
||||
R-ENV-01 and R-ENV-03 covered — the last two matter because the Phase-7 entry-gate note recorded
|
||||
R-ENV-01/02/03 as uncovered while prod provisioning was proceeding.
|
||||
|
||||
### STAGED FOR ONE PUSH — `prod/deliver-fixes`
|
||||
|
||||
Prod reconciles bootstrap-Gitea `main@c8c1d29` and **cannot take the working branch** — that carries
|
||||
the DNS inversion, which would move prod to the apex and collide with dev. `prod/deliver-fixes` is
|
||||
`c8c1d29` + the CDI graph fix + the Kyverno mutate policy: a clean fast-forward, verified free of any
|
||||
cluster-vars/externaldns/overlay change, prod's `BASE_DOMAIN` untouched, render checked. **Not pushed
|
||||
— that changes what a live cluster reconciles.**
|
||||
|
||||
### Waiting on the operator (deliberately skipped)
|
||||
|
||||
- **R2 credentials.** Two token values; the account id is already committed in
|
||||
`infra/tofu/backend.r2.hcl`. `infra/BACKUP-R2.md` is the procedure. **Prod has no backup coverage.**
|
||||
- **`enableCertificateOwnerRef`** — R-TLS-02's "revoked" clause is unmet; both clusters' TLS Secrets
|
||||
have zero ownerReferences. Not flipped because the wildcard Certificate lives under a `prune: true`
|
||||
tier, so a transient prune would delete the serving Secret and reissuance is capped at 5/week.
|
||||
- **Node timezone** — authored (`Etc/UTC`, matching dev), `--check` confirms the drift, **not applied**:
|
||||
a live clock change during storage resync is not an unattended action.
|
||||
- **Database HA** — all three CNPG clusters are `instances: 1` and all three primaries are on prod-1.
|
||||
Losing that node takes Keycloak, Nextcloud and Semaphore together. `instances: 2` spends resources.
|
||||
- **The apex flip** is also the legacy-prod 443 cutover; §10a sequence P does not cover the SOHO NAT.
|
||||
- **R-STATE-04** stays a gap: `bulk` is a non-provisioning stub pending **OQ-2**.
|
||||
- Orphaned `prime-be3f9d1c-…` 40 GiB scratch PVC from the blocked imports — operator cleanup.
|
||||
@@ -69,17 +69,6 @@ infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
|
||||
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
|
||||
```
|
||||
|
||||
`kubectl`/`flux`/`helm` in there talk to the **dev** cluster by default. Prefix `CLUSTER=prod` to
|
||||
target prod instead:
|
||||
|
||||
```bash
|
||||
infra/devcontainer/dev.sh run kubectl get nodes # dev (default)
|
||||
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod
|
||||
```
|
||||
|
||||
Both clusters are always present in the container's `KUBECONFIG` as the contexts `k3s-dev` and
|
||||
`k3s-prod`; `CLUSTER` picks which one is current, and each run prints the cluster it is targeting.
|
||||
|
||||
The container mounts the repo at `/work`, your age keyring, and the sibling spec repo read-only
|
||||
so the spec-immutability check can actually run. See
|
||||
[`infra/devcontainer/README.md`](infra/devcontainer/README.md).
|
||||
|
||||
@@ -89,20 +89,11 @@ spec:
|
||||
# userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
|
||||
# signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
|
||||
# Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
|
||||
# NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
|
||||
# ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
|
||||
# Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
|
||||
# as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
|
||||
# by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
|
||||
# an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
|
||||
# and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
|
||||
# flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
|
||||
# text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
|
||||
# wording in clusters/base/apps-vms.yaml overstated it.
|
||||
# NOTE: apps-vms has NO postBuild.substituteFrom, so shell ${...} below is NOT envsubst'd.
|
||||
userData: |-
|
||||
#cloud-config
|
||||
hostname: unifi
|
||||
fqdn: unifi.${BASE_DOMAIN}
|
||||
fqdn: unifi.home.keithjackson.dev
|
||||
package_update: true
|
||||
package_upgrade: false
|
||||
packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
|
||||
@@ -112,11 +103,11 @@ spec:
|
||||
content: |
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
if [ -z "$${UOS_URL:-}" ]; then
|
||||
if [ -z "${UOS_URL:-}" ]; then
|
||||
UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
|
||||
fi
|
||||
[ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
|
||||
t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
|
||||
[ -n "${UOS_URL}" ] && [ "${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
|
||||
t="$(mktemp)"; echo "downloading ${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
|
||||
runcmd:
|
||||
- [ systemctl, enable, --now, qemu-guest-agent ]
|
||||
- [ systemctl, enable, --now, podman.socket ]
|
||||
|
||||
@@ -8,9 +8,8 @@
|
||||
# CANNOT express: raw TCP on non-HTTP ports (8080 inform, 6789), UDP at all (3478 STUN, 10001
|
||||
# discovery), or the device-facing protocols that never speak HTTP. UniFi device ADOPTION rides
|
||||
# exactly those L4 ports. So the controller is exposed as a type=LoadBalancer Service whose VIP
|
||||
# is pulled from the CiliumLoadBalancerIPPool `default-pool` (dev 192.168.1.53-.60 — the
|
||||
# CILIUM_LB_POOL_START/STOP cluster-vars knobs) and ARP-announced on the flat LAN by the existing
|
||||
# CiliumL2AnnouncementPolicy `default-l2` — the SAME L2/LB-IPAM
|
||||
# is pulled from the CiliumLoadBalancerIPPool `default-pool` (192.168.1.53-.60) and ARP-announced
|
||||
# on the flat LAN by the existing CiliumL2AnnouncementPolicy `default-l2` — the SAME L2/LB-IPAM
|
||||
# machinery the Gateway ingress VIP already uses (platform-config/cilium/*). No new Cilium config
|
||||
# is required; these Services just consume the pool.
|
||||
#
|
||||
@@ -24,11 +23,8 @@
|
||||
# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
|
||||
# VIP pinned to the node running the VM (also what L2 announcement does).
|
||||
#
|
||||
# VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
|
||||
# default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
|
||||
# Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for
|
||||
# exactly this token; every shell $${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
|
||||
# accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool.
|
||||
# FLAG: confirm 192.168.1.54 is free, inside default-pool, and reserved in tofu (the Gateway
|
||||
# already holds one pool IP, typically .53). Adjust if it collides.
|
||||
#
|
||||
# REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an
|
||||
# L4-only workload as a Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing
|
||||
@@ -43,10 +39,10 @@ metadata:
|
||||
app.kubernetes.io/name: unifi
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "unifi"
|
||||
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
||||
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
|
||||
lbipam.cilium.io/ips: "192.168.1.54"
|
||||
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> 192.168.1.54.
|
||||
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
|
||||
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
|
||||
external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
@@ -69,7 +65,7 @@ metadata:
|
||||
app.kubernetes.io/name: unifi
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "unifi" # same key + same IP => shares the VIP with -tcp
|
||||
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
||||
lbipam.cilium.io/ips: "192.168.1.54"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
@@ -90,9 +90,9 @@ spec:
|
||||
name: gitea-bootstrap
|
||||
key: secret-key
|
||||
- name: GITEA__server__DOMAIN
|
||||
value: gitea.${BASE_DOMAIN}
|
||||
value: gitea.home.keithjackson.dev
|
||||
- name: GITEA__server__ROOT_URL
|
||||
value: https://gitea.${BASE_DOMAIN}/
|
||||
value: https://gitea.home.keithjackson.dev/
|
||||
- name: GITEA__actions__ENABLED
|
||||
value: 'true'
|
||||
- name: GITEA__actions__DEFAULT_ACTIONS_URL
|
||||
@@ -127,8 +127,8 @@ spec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1;\
|
||||
\ break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
|
||||
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name\
|
||||
\ groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n"
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
@@ -185,10 +185,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute gitea-3000 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -207,7 +203,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- gitea.${BASE_DOMAIN}
|
||||
- gitea.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: jackett-oauth2-proxy
|
||||
namespace: jackett
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:n5h6NG0emA==,iv:d6lvZfgRq6+IKNpsSoRa5a38R2f2+tWBO+j9QKDThMY=,tag:P+cvQXlWR4aZs7a9/lVuMQ==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:WL5G7PuVTdHQoe7Zp2eiE9ppxSGE0+W2RMfISuwqD6JA9auG97ZYnPUjG8HNx7sosOvsmqeE9kONK23IqIeaQw==,iv:kSP10UKeDv/8BsgX36+uKwEY+LBovcicce1RkD14M4g=,tag:fEUsVuajWPxAmAo8Nnrrrg==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:WChW9YLhzG9Nwp8dCWTFtirufXxGk3Ep9HXpfqcNz69PXQW3RaMC4EJpw2U=,iv:f6obIerkog8Q2gfj1tVL259LOl5jj59VE29KnJwmCXQ=,tag:s0FwNtU6vffKgDcGUNUhCg==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLWVptVHc4aFl0MXRWTHVP
|
||||
cC9qcVoyZUV2TGZnTEpHY3RLOU8vQnNNNm00ClZRejE2bzFaWXNSUmQ5MEZtV3VK
|
||||
djB2TXJqQjVUOE93QjN6U0lMVDBEbm8KLS0tIFFTZnB0c2pCZWZMZ084WEIxT2tW
|
||||
WUxxemJHdWRZbnBpL280dzlpMmhwSW8Kra0bkx3iJYQi+3rnpU/89fJbJHzQjKi2
|
||||
ZWNwYhLpULw1xNo3xbCm+PUp5f30kpyMKQld2qaLOZAqWI5vNl9Byw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:41:37Z"
|
||||
mac: ENC[AES256_GCM,data:zU9nWCrSomAXAZ/Ve8U3yLHnoxhu8nkQ7523neJTGv810ONzAAZ7Gr8nFHR7bTqz2QwuGYD3P1ypdqYnKLM8ri2Zf1LDo/Su9ptnvQhUbfXKlRorcW6uAWsViCwvTM05wZNkdzZfza7tGwJQVTD6hc0cP+/7Y7JnpT5FEstK254=,iv:oggTRjcWqAJizA/a22+mtslkEWka8hUG0wudM8obO4g=,tag:i0TtNmqGe+b5t4iq8LpsqA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:9117
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: jackett-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: jackett-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: jackett-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-9117
|
||||
port: 9117
|
||||
targetPort: 9117
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): jackett-allow-radarr-9117
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -233,7 +166,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '9117'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
|
||||
@@ -251,10 +184,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute jackett-9117 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -273,7 +202,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- jackett.${BASE_DOMAIN}
|
||||
- jackett.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -281,7 +210,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: jackett
|
||||
port: 4180
|
||||
port: 9117
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
|
||||
@@ -8,15 +8,12 @@ kind: Kustomization
|
||||
resources:
|
||||
- gitea-bootstrap-secret.sops.yaml
|
||||
- gitea.yaml
|
||||
- jackett-oauth2-proxy-credentials.sops.yaml
|
||||
- jackett.yaml
|
||||
- nextcloud-db-app.sops.yaml
|
||||
- nextcloud-db-backup-credentials.sops.yaml
|
||||
- nextcloud-db.yaml
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
- pihole.yaml
|
||||
- plex.yaml
|
||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||
- qbittorrent.yaml
|
||||
- semaphore-db-app.sops.yaml
|
||||
- semaphore-db-backup-credentials.sops.yaml
|
||||
|
||||
@@ -74,17 +74,11 @@ spec:
|
||||
# (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP.
|
||||
# Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials
|
||||
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
|
||||
# endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
|
||||
# substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
|
||||
backup:
|
||||
retentionPolicy: "30d"
|
||||
barmanObjectStore:
|
||||
# PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER
|
||||
# NAME — and both environments have a `nextcloud-db`. Sharing one MinIO with a shared path
|
||||
# would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and
|
||||
# destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests.
|
||||
destinationPath: ${CNPG_BACKUP_PATH}
|
||||
endpointURL: ${S3_BACKUP_ENDPOINT}
|
||||
destinationPath: s3://cnpg
|
||||
endpointURL: http://192.168.1.9:9000
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: nextcloud-db-backup-credentials
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: pihole-oauth2-proxy
|
||||
namespace: pihole
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:C6IXgQxi,iv:A9bWygwabOKIRi55jV9+TRU7eiZels3vqqLDTZ469aA=,tag:hC7sn/Xr4vQFkRjDLMgEmQ==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:50HEfMG+CoWXokS+IxQgCPQEt+EBCUontOc0GrTyauHuV/U7w9IPo83iJECO0QhTPiQIgD3j+F+YXfubCbTnuw==,iv:ejQxin0NMs0Ye0fhqfPuh8JmRrEimDrSAcZj28uS+GM=,tag:FVassLZPZMY9ry48ZjBgmw==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:nfTEg5DlIXCLI/hhHngyqLwsQh2SgXU21LaSkVnr91sTi4wODrEH4C7wvPY=,iv:WWXnHnFP2IqLrw1yXAlD0Sw+Vc1m4rxXIHT1u2UWH3o=,tag:pYZEuwPYH+b8S4kGtaWqEA==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuTnJiWTVPR01aY2h3dkJI
|
||||
OTV0WTFQeHoyUGluVzV4NFpSZjd6bjlHZkE0ClZmTGgzQWo4VTROaUg5Wkd1TTVV
|
||||
cGRjZjJuMitPZC9OaVZUZzRGeEJmemcKLS0tIGRjUFd3SzFzeFBBQVRacjZWQmNH
|
||||
ZjllZTQxbzRWQXNiVllkc1hESGN4alEK/hFOtchsYMx09/WE5uaCe6JetUurzhhQ
|
||||
7imb8RIjh0cfP4PDT3dZhhPF+ehcV7ttslFkcyaqlG1F8Gk2jtN0pw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:33:15Z"
|
||||
mac: ENC[AES256_GCM,data:Tse6fLhc08DxCAuZW5KSZ3M5OQUTlbeHnWiO3jIROUVLrBfbXCuDNWnQyOPjwVPqyzeAzC3APEKFzQ2S8rYjRyO08SJ4ZFPPUeRblcXgXHZ4Y/E4Renq7axk61VvFGtZteHWuQz9ss//Qt+KwPMDBGTivLCoVTOh2K01AXPFXdA=,iv:fyhkDLVN28umXDR4/f4ZzQuu3OG6QBkpUgf3E9rknKk=,tag:iNqor6cD9tJ6MsqAmmRP0Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+3
-74
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:80
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: pihole-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: pihole-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: pihole-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-80
|
||||
port: 80
|
||||
targetPort: 80
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): pihole-allow-gateway-ingress
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -189,7 +122,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '80'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
|
||||
@@ -207,10 +140,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute pihole-80 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -229,7 +158,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- pihole.${BASE_DOMAIN}
|
||||
- pihole.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -237,7 +166,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: pihole
|
||||
port: 4180
|
||||
port: 80
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
|
||||
@@ -183,7 +183,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- plex.${BASE_DOMAIN}
|
||||
- plex.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
namespace: qbittorrent
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:ytfVRDabEfRM8NI=,iv:ERtFy5iss10id/cGLMZDmIAazkYtFG/spkxRYkoT2pw=,tag:FzOepKeYT05BBrz6RlnACQ==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:klZzTdyAqQgG+1s074/0Bh5ybuHehiEo/RYceRGAGwuUrgCMb0AmYHCrE9gvtnPkSOBqWvFnPnorWkERc+V73g==,iv:nOUO/p+WnI5657I3Lm6zQAsumkscjF7J0fK0EVFHGo0=,tag:dF3AHYP4sxDChroRer9kMQ==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:jVpsZjYHRrPvblfsb2f9OuneqoZ0cwcTLKrPb8XJ5oves1PmHtN5njrkk4o=,iv:bNjUK/ar4nniR7AjJ9fsGVkKd8nxwphkptmhFaYh9Ek=,tag:2axf+AshSdIvvqwyIfhq6g==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAreWdPdUxiYXBxYWxZNlhy
|
||||
WHFTZmo1RW1McXJJTjlWeTBCd2Y4WVU2TVRNCnExbEQrQXZCS2RRaGxJQlQ5eHpY
|
||||
Z1UrZWlubUJpRC9YeEhJZW8zNVdLUVUKLS0tIGtmblFDRzk3ZC8wWVo2ZHJHWmEx
|
||||
OW9DRUpaSlFrbHNDTGYvQUdyamh2bWsK5VvV4cl1iNXYECnSk3fi1ZuAPKGN+n8B
|
||||
g51lT4gi03KA67ARzwVGVBfP5E1Cdmx4rJhi305tG+Ug99VyLBCUfQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:42:02Z"
|
||||
mac: ENC[AES256_GCM,data:7GdlUKnZ/ST7EOhfPB6I2e2wyVHWmZ2G554yBcZQzqoZZWZ1hn+lkosNJxWD7E0hRId6P320EKj+Sf0g2ur3Mna5zii1GKFJHGtDEoccKvJQRx2JMkZlEC/x6xwc4BeJNtvidECVRvKC2WdrByRMr9rrI8NfXpmWH4e7iz1Xac0=,iv:UqsNOPRwzT+Dt043ldxIubkAAkPr0/nvjxlgVPiXIbw=,tag:7Wex5Rng3Nwtr8wxkFJD9Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:8080
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-8080
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): qbittorrent-allow-radarr-8080
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -233,7 +166,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '8080'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
|
||||
@@ -251,10 +184,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute qbittorrent-8080 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -273,7 +202,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- qbittorrent.${BASE_DOMAIN}
|
||||
- qbittorrent.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -281,7 +210,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: qbittorrent
|
||||
port: 4180
|
||||
port: 8080
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
|
||||
@@ -75,17 +75,11 @@ spec:
|
||||
# (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
|
||||
# Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
|
||||
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
|
||||
# endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
|
||||
# substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
|
||||
backup:
|
||||
retentionPolicy: "30d"
|
||||
barmanObjectStore:
|
||||
# PER-ENVIRONMENT PATH (cluster-vars). CNPG appends serverName, which defaults to the CLUSTER
|
||||
# NAME — and both environments have a `semaphore-db`. Sharing one MinIO with a shared path
|
||||
# would interleave two Postgres servers' WAL into ONE Barman stream, corrupting the chain and
|
||||
# destroying PITR for both. Dev keeps `s3://cnpg` (its existing, PROVEN backups); prod nests.
|
||||
destinationPath: ${CNPG_BACKUP_PATH}
|
||||
endpointURL: ${S3_BACKUP_ENDPOINT}
|
||||
destinationPath: s3://cnpg
|
||||
endpointURL: http://192.168.1.9:9000
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: semaphore-db-backup-credentials
|
||||
|
||||
@@ -103,69 +103,25 @@ spec:
|
||||
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
|
||||
- name: runner
|
||||
image: gitea/act_runner:0.2.11
|
||||
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
|
||||
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
|
||||
# boot and runs `act_runner daemon`.
|
||||
#
|
||||
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
|
||||
# and hard-exits if it is missing (a startup race against the sidecar).
|
||||
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
|
||||
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
|
||||
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
|
||||
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
|
||||
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
|
||||
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
|
||||
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
|
||||
#
|
||||
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
|
||||
# braced form or the double-dollar form; both are wrong here, for different reasons.
|
||||
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
|
||||
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
|
||||
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
|
||||
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
|
||||
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
|
||||
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
|
||||
# cluster verbatim and both 60s bounds below are real, enforced bounds.
|
||||
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
|
||||
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
|
||||
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
|
||||
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
|
||||
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
|
||||
# form — still wrong for us, because on any path where envsubst does NOT run
|
||||
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
|
||||
# reads that leading double-dollar as its own PID, so a bound would compare a
|
||||
# string like 1234{i} against 60. The bare form is the only spelling that is
|
||||
# correct on BOTH paths.
|
||||
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
|
||||
# name") that fails this whole Kustomization — and this Kustomization gates every
|
||||
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
|
||||
# comment. That is why this note names the three forms in words rather than
|
||||
# showing them: an earlier draft of it spelled them out literally and broke
|
||||
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
|
||||
# clusters/base/apps-vms.yaml describes.)
|
||||
# Wait for the DinD sidecar to publish its TLS client certs into the shared
|
||||
# emptyDir before starting the daemon — act_runner reads /certs/client/ca.pem at
|
||||
# startup and hard-exits if it is missing (a startup race against the sidecar).
|
||||
# Then exec the image's normal entrypoint (/sbin/tini -- /opt/act/run.sh), which
|
||||
# registers-from-env on first boot and runs `act_runner daemon`.
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
|
||||
i=0
|
||||
until [ -f /certs/client/ca.pem ]; do
|
||||
i=$((i + 1))
|
||||
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
|
||||
echo "waiting for dind TLS certs ($i)"; sleep 1
|
||||
done
|
||||
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
|
||||
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
|
||||
j=0
|
||||
until nc -z -w1 127.0.0.1 2376; do
|
||||
j=$((j + 1))
|
||||
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
|
||||
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
|
||||
done
|
||||
exec /sbin/tini -- /opt/act/run.sh
|
||||
env:
|
||||
- name: GITEA_INSTANCE_URL
|
||||
value: https://gitea.${BASE_DOMAIN}
|
||||
value: https://gitea.home.keithjackson.dev
|
||||
- name: GITEA_RUNNER_REGISTRATION_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
@@ -199,31 +155,6 @@ spec:
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
|
||||
#
|
||||
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
|
||||
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
|
||||
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
|
||||
# ordering that the loops above hand-roll — which also removes the envsubst hazard
|
||||
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
|
||||
# Shape to build:
|
||||
# 1. Move this whole container into `initContainers:` as the FIRST entry.
|
||||
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
|
||||
# initContainer a sidecar: it starts before the regular containers, keeps
|
||||
# running alongside them, and is terminated after them.
|
||||
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
|
||||
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
|
||||
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
|
||||
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
|
||||
# it would re-open the same "port accepting but TLS/certs not ready" gap that
|
||||
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
|
||||
# preserves today's 60s cap.
|
||||
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
|
||||
# `runner` can drop its `command:` override completely and run the image's
|
||||
# default entrypoint.
|
||||
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
|
||||
# entire repository. If the conversion is wrong, nothing can merge — so it must land
|
||||
# alone, where the failure is unambiguous and not entangled with anything else, and
|
||||
# only after a real workflow run has been observed green on the converted pod.
|
||||
- name: dind
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
|
||||
@@ -10,9 +10,7 @@ resources:
|
||||
- gitea-actions.yaml
|
||||
- nextcloud-bootstrap-secret.sops.yaml
|
||||
- nextcloud.yaml
|
||||
- radarr-oauth2-proxy-credentials.sops.yaml
|
||||
- radarr.yaml
|
||||
- semaphore-bootstrap-secret.sops.yaml
|
||||
- semaphore.yaml
|
||||
- sonarr-oauth2-proxy-credentials.sops.yaml
|
||||
- sonarr.yaml
|
||||
@@ -104,13 +104,13 @@ spec:
|
||||
name: nextcloud-bootstrap
|
||||
key: admin-password
|
||||
- name: NEXTCLOUD_TRUSTED_DOMAINS
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
- name: OVERWRITEHOST
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
- name: OVERWRITEPROTOCOL
|
||||
value: https
|
||||
- name: OVERWRITECLIURL
|
||||
value: https://nextcloud.${BASE_DOMAIN}
|
||||
value: https://nextcloud.home.keithjackson.dev
|
||||
- name: NEXTCLOUD_OIDC_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
@@ -125,9 +125,8 @@ spec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc]\
|
||||
\ client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
|
||||
); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
|
||||
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra\
|
||||
\ positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n"
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
@@ -206,10 +205,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute nextcloud-80 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -228,7 +223,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- nextcloud.${BASE_DOMAIN}
|
||||
- nextcloud.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: radarr-oauth2-proxy
|
||||
namespace: radarr
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:vFLxqzxt,iv:6k0ZjKP8rvg56ZFPWEGdyj+FLgK+LBOE0pLBWfC0bjA=,tag:LiZb5BasJAEWM1c3I8vFTA==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:JzfHF8JF4YUxBxiC738N4UsMlcCOvxn4/AE2B4meZNGAZb2uhG4n51uEY8OQsbsM0Ni4qM2Wi17KuLDwAicfUA==,iv:A3AnUhYxw/IWfvfcZf6MgC5u37dMoNToIWxRG9AxEJQ=,tag:1OY2pWGTbCxPYYdaiBa/oA==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:LPt5nghLfoNvr7yBes8rLeEK3M6UVIO1h9POHMOAj8iZX3l2RJJa4w8GcSg=,iv:Q5+7zH8Tb1SWe92B0pxekfMgOXlBG8nixWn/ho4oJw4=,tag:D+41dBFPVTGj86xv5/GEng==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBHVG5MWVRyOVl1RWFpOExC
|
||||
SjZKcEd1ZlFnZVZ5NmtyUlQzRGI1b1o3dG04CkwrTUNjd3crM1VzaUdPczliSDY1
|
||||
a3dONnBFWk8yOW5kL2xES1RZbUN6L3cKLS0tIG9hWG9Cbm01RlMrcC9zMWkxeFgr
|
||||
ZnRGRUVTTS81SFRyenNsZUZEcG1TUm8K5XUGP/1NVsoaFcBZUER3/zMPTtCuVisT
|
||||
LrbSmTkIyVfzjN/vRpRH92b1VzENJvhGhRghhQaKY0rbG34i6yh3Vw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:42:25Z"
|
||||
mac: ENC[AES256_GCM,data:TxdPz2zE6+ypgQG2b0Rp3VFiPL7LLls4NNkC7YQ/bVCLvyvaXdGsEWkiHJE+Po7hcESmZRt/NMS25mEXPMfvlSCltwFUTcrxoZDS0NI27kIzbtceTbxmznt/dMS/iZm21lCgA6Aag9ByLjWusPLkRKWhep+G+0G54qtfQzv3s80=,iv:44f0KikrpKpnSaLLyzTvnCfrwEXylRgVIlKMtSpMyNc=,tag:jDx43n0usDtbwzo8KzDBqw==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+3
-74
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:7878
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: radarr-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: radarr-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: radarr-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-7878
|
||||
port: 7878
|
||||
targetPort: 7878
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy egress allow (R-NET-02): radarr-allow-qbittorrent-8080
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -255,7 +188,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '7878'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
|
||||
@@ -273,10 +206,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute radarr-7878 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -295,7 +224,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- radarr.${BASE_DOMAIN}
|
||||
- radarr.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -303,7 +232,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: radarr
|
||||
port: 4180
|
||||
port: 7878
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
|
||||
@@ -110,7 +110,7 @@ spec:
|
||||
- name: SEMAPHORE_ADMIN_NAME
|
||||
value: Semaphore Admin
|
||||
- name: SEMAPHORE_ADMIN_EMAIL
|
||||
value: admin@${BASE_DOMAIN}
|
||||
value: admin@home.keithjackson.dev
|
||||
- name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
@@ -209,10 +209,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute semaphore-3000 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -231,7 +227,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- semaphore.${BASE_DOMAIN}
|
||||
- semaphore.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: sonarr-oauth2-proxy
|
||||
namespace: sonarr
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:JMYoJymH,iv:CM9HF71QZCeETzrOtkeTUs+BLTp/gGSExulXUP8yS/M=,tag:l73RaFQolO7+wZBo8nYCvw==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:Fq271KdbtKWKWOe1Sun6+SLv/kSBsvAeFUhZIq6J2DNPGM4E8dgc2YANITcpezcJR07PAbSlUcS0L/qF6K+YhQ==,iv:/mVD1uOkweyO26RMV3qRQmLdyxXJR4I7p+OjYiiuSMU=,tag:9ZOKye5Sp2NHU2RXH1Y+wQ==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:jMZZEi8fdh20BbPzCoVfwdS1g0/ma5n/QISU5A/gQCu/2Rkylt7mp798CUg=,iv:6IPqDNQyJHze2DUOQ93IwruWSge5nXjkVcQIOMfJSwY=,tag:cOKRcDKywV3PBOBrhoddRQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoNU5NbnJnV0dKRnpIN2dw
|
||||
QmhBRWI4aEpWajRHVkxZcGxCRk1hMDBjSlRVCm91b0U2N08vNi8rSEFaVlRJazFL
|
||||
bllPcm9EMUgrQ3ZlUDlrdEZpZE8zNmMKLS0tIDU0ajdycXZrazlQb3RVRGQyY25h
|
||||
R3RDajltcmFvQWdLYWQ2Mjh0VkplaWsKJprnLNJYzMd0yHbxjsD8P66Wu+sefmaV
|
||||
nduEgVQY0t4EsYN6iKVDLMUAp+o3gXU8QuMuBnFZznrmBghfG+ZruQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:42:49Z"
|
||||
mac: ENC[AES256_GCM,data:ckVR9OyFvle0fLhBjHtJ6hH2St2CRJeu6abTCnqtxAfMDVXdbIkgHEesmN2kAgA7s3LkKJC2m7EFIpt3pO94VQNnZvC1KKD7qUSIfJN1DLVv78m8JbEQfVTJk/WgWy1ObxLriw5hDuMiEuOvKND+s6eqS5j6er4Q72lBsBpoEeY=,iv:uguThOQbR9Ks1qs1U0HO7EhUj1BT60fvM+3xVEC9hUU=,tag:m4P5nhlspCZyJRuOqffMog==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+3
-74
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:8989
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-8989
|
||||
port: 8989
|
||||
targetPort: 8989
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy egress allow (R-NET-02): sonarr-allow-qbittorrent-8080
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -255,7 +188,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '8989'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
|
||||
@@ -273,10 +206,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute sonarr-8989 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -295,7 +224,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- sonarr.${BASE_DOMAIN}
|
||||
- sonarr.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -303,7 +232,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: sonarr
|
||||
port: 4180
|
||||
port: 8989
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
|
||||
@@ -223,7 +223,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- seerr.${BASE_DOMAIN}
|
||||
- seerr.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
|
||||
+10
-118
@@ -11,120 +11,16 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux
|
||||
## Files
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → **pre-apply chart CRDs (`preapply-crds.sh`)** → verify `sops-age` Secret → `kubectl apply -k ${FLUX_ENV_PATH}`. |
|
||||
| `preapply-crds.sh` | **Server-side pre-apply of the Kyverno CRD bundle** — the CRDs `platform/kyverno/admission/helmrelease.yaml` deliberately tells Helm *not* to install (`crds: {install: false}`, because the ~5.6 MiB bundle exceeds kubectl's 256 KiB last-applied annotation limit and stalls the release). Nothing else installs them, so without this step `platform` never goes Ready on a cold cluster. Chart name/version/repo are **derived from that HelmRelease** — no second pin. Idempotent; runnable standalone. |
|
||||
| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
|
||||
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
|
||||
| `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
|
||||
| `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
|
||||
| `patches/apps-sops-decryption.yaml` | Same, for the `apps` Kustomization. |
|
||||
| `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |
|
||||
|
||||
## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster
|
||||
|
||||
Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
|
||||
and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):
|
||||
|
||||
```
|
||||
$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
|
||||
{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
|
||||
"app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
|
||||
# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1
|
||||
```
|
||||
|
||||
No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under
|
||||
`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux
|
||||
documents for hand-installed clusters ([Customize Flux
|
||||
manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)):
|
||||
|
||||
```
|
||||
flux install --export … -> <tmp>/gotk-components.yaml # regenerated from the pinned flux CLI
|
||||
components/kustomization.yaml -> <tmp>/kustomization.yaml # the only hand-authored part (in git)
|
||||
kubectl kustomize <tmp> | kubectl apply -f -
|
||||
```
|
||||
|
||||
Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION`
|
||||
(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations,
|
||||
more feature gates) is another entry in `components/kustomization.yaml`.
|
||||
|
||||
### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)
|
||||
|
||||
> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the
|
||||
> controller Deployment is outside Flux's inventory. Someone must run step 1.
|
||||
|
||||
Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1:
|
||||
|
||||
```bash
|
||||
# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
|
||||
d=$(mktemp -d)
|
||||
flux install --export --namespace=flux-system \
|
||||
--components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
|
||||
cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
|
||||
kubectl kustomize "$d" | kubectl apply -f - # add --server-side --force-conflicts on a field-manager conflict
|
||||
kubectl -n flux-system rollout status deploy/kustomize-controller
|
||||
```
|
||||
|
||||
**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any
|
||||
reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing
|
||||
is lost. Confirm afterwards:
|
||||
|
||||
```bash
|
||||
kubectl -n flux-system get deploy kustomize-controller \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].args}' # must contain --feature-gates=StrictPostBuildSubstitutions=true
|
||||
flux get kustomizations -A # all 11 must return to Ready
|
||||
```
|
||||
|
||||
**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
|
||||
simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
|
||||
|
||||
## The CRD pre-apply step (`preapply-crds.sh`) — why bootstrap has to do this
|
||||
|
||||
`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`. That is correct and must
|
||||
stay: the Kyverno CRD bundle is **5,636,155 bytes across 22 CRDs** at chart 3.9.0 (measured with
|
||||
`helm template … --include-crds`, 2026-09-04), so Helm's client-side apply hits kubectl's 256 KiB
|
||||
`last-applied-configuration` annotation limit — `metadata.annotations: Too long` — which stalls the
|
||||
HelmRelease and blocks `platform` plus every tier that `dependsOn` it.
|
||||
|
||||
The part that was never automated: **with CRD install off, nothing installs the CRDs.** Kyverno's
|
||||
Deployments start, its API types do not exist, `platform` never reaches Ready. Dev has them only
|
||||
because somebody once ran the command in a shell. `preapply-crds.sh` is that command, in git:
|
||||
|
||||
```
|
||||
helm template <chart> --repo <url> --version <constraint> --include-crds --set crds.install=true
|
||||
| yq 'select(.kind == "CustomResourceDefinition")'
|
||||
| kubectl apply --server-side --force-conflicts
|
||||
```
|
||||
|
||||
- **Version derivation, not a second pin.** `chart`, `version`, `namespace` and the repo `url` are
|
||||
read out of the HelmRelease/HelmRepository pair with `yq`. `spec.chart.spec.version` is a semver
|
||||
**constraint** (`3.9.x`), and `helm template --version` accepts the same constraint syntax
|
||||
helm-controller resolves — so the rendered CRDs are the ones Flux will install, and a Renovate bump
|
||||
of that one line carries this step with it. There is nothing here that can drift.
|
||||
- **CRDs only.** Piping the *whole* `helm template` output into kubectl (the by-hand command in the
|
||||
HelmRelease comment) would create Deployments/webhooks outside Helm's ownership and then fight the
|
||||
HelmRelease over field managers. The `yq` filter is load-bearing.
|
||||
- **Idempotent.** Server-side apply is an upsert; `--force-conflicts` adopts the fields dev's
|
||||
hand-applied CRDs left owned by `kubectl-client-side-apply`. Re-running prints
|
||||
`serverside-applied` and exits 0.
|
||||
- **Cannot degrade to a no-op.** Three guards: every derived coordinate must be non-empty and not
|
||||
`null`; the rendered CRD count must be `> 0` (an empty stream would otherwise `apply` cleanly and
|
||||
exit 0 — the exact silent-success bug this repo keeps hitting); and after the apply every CRD is
|
||||
waited on for `condition=Established`. There is **no skip flag** on purpose.
|
||||
- **Environment-agnostic (R-ENV-02).** It names no environment and takes no overlay input — it reads
|
||||
repo files and acts on `$KUBECONFIG`. Dev and prod run the identical step.
|
||||
|
||||
Standalone use (an already-bootstrapped cluster whose `platform` is blocked on the missing CRDs):
|
||||
|
||||
```bash
|
||||
# inside infra/devcontainer, KUBECONFIG pointed at the target cluster
|
||||
bash bootstrap/flux/preapply-crds.sh
|
||||
```
|
||||
|
||||
## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
|
||||
|
||||
The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):
|
||||
`flux-system.yaml` is the `flux-system` **GitRepository**; `platform.yaml`, `platform-config.yaml`
|
||||
and the app-wave Kustomizations (`apps-wave-0/1/2.yaml` + `apps-vms.yaml` — the single `apps.yaml`
|
||||
was retired) are the Flux **Kustomizations**; `clusters/dev` is the overlay that patches
|
||||
`cluster-vars` (R-ENV-02) and layers SOPS decryption onto each Kustomization by name.
|
||||
`flux-system.yaml` is the `flux-system` **GitRepository**; `platform.yaml`/`apps.yaml` are the Flux
|
||||
**Kustomizations**; `clusters/dev` is the overlay that patches `cluster-vars` (R-ENV-02).
|
||||
|
||||
`flux bootstrap git` **generates and owns its own** `gotk-sync` GitRepository + Kustomization named
|
||||
`flux-system` and commits them into the repo — which would **duplicate/conflict** with the authored
|
||||
@@ -142,12 +38,11 @@ tofu apply → cloud-init → Ansible (k3s + Cilium) [Infra worker, T-1
|
||||
▼ (final Ansible play calls the entrypoints below)
|
||||
1. bootstrap/sops/age-keygen.sh # operator generates age key OUTSIDE repo
|
||||
2. bootstrap/sops/age-key-secret.md # operator installs private key as sops-age Secret
|
||||
3. bootstrap/flux/bootstrap.sh # flux install → preapply-crds.sh → verify sops-age → apply -k ${FLUX_ENV_PATH}
|
||||
3. bootstrap/flux/bootstrap.sh # flux install → verify sops-age → apply -k clusters/dev
|
||||
│
|
||||
▼
|
||||
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
|
||||
→ Kustomization(apps-wave-0 → apps-wave-1 → apps-wave-2) + Kustomization(apps-vms),
|
||||
with per-Kustomization SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).
|
||||
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(apps),
|
||||
with SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).
|
||||
```
|
||||
|
||||
## Interface note — for the Infra worker (T-1.3 Ansible)
|
||||
@@ -162,10 +57,8 @@ live in `infra/devcontainer/versions.env`).
|
||||
template). `bootstrap.sh` **fails fast** if this Secret is absent.
|
||||
3. `bootstrap/flux/bootstrap.sh` (this entrypoint).
|
||||
|
||||
`bootstrap.sh` is idempotent-friendly (`flux install`, `preapply-crds.sh`'s server-side apply, and
|
||||
`kubectl apply -k` all re-run safely) and never handles key material — it only *verifies* the Secret
|
||||
exists. Beyond `flux`/`kubectl` it now also needs **`helm` + `yq`** on PATH (step 2's CRD render;
|
||||
both are in `infra/devcontainer`) and outbound network to the Kyverno chart repo.
|
||||
`bootstrap.sh` is idempotent-friendly (`flux install` and `kubectl apply -k` re-run safely) and
|
||||
never handles key material — it only *verifies* the Secret exists.
|
||||
|
||||
## Handoff — clusters owner / PM (files this worker may not edit)
|
||||
|
||||
@@ -179,8 +72,7 @@ patches:
|
||||
- path: ../../bootstrap/flux/patches/apps-sops-decryption.yaml # SOPS on apps
|
||||
```
|
||||
|
||||
Alternatively the base TODOs in `clusters/base/platform.yaml` and each app-wave Kustomization
|
||||
(`apps-wave-0/1/2.yaml`, `apps-vms.yaml`) and the `url` in
|
||||
Alternatively the base TODOs in `clusters/base/{platform,apps}.yaml` and the `url` in
|
||||
`clusters/base/flux-system.yaml` can be filled in directly (same effect) — the clusters owner's call.
|
||||
Either way the `sops-age` `secretRef` name and `flux-system` source name are kept in lockstep with
|
||||
the authored base.
|
||||
|
||||
+19
-94
@@ -1,9 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
|
||||
# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
|
||||
# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
|
||||
# same script bootstraps dev and prod with no fork (R-ENV-02).
|
||||
# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
|
||||
#
|
||||
# Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
|
||||
# =============================================================================
|
||||
@@ -28,28 +25,16 @@
|
||||
# We do NOT use `flux bootstrap git` in Phase 1 because that command commits
|
||||
# and OWNS its own gotk-sync GitRepository+Kustomization named `flux-system`.
|
||||
# Our entrypoints are ALREADY authored in clusters/base (flux-system.yaml =
|
||||
# the GitRepository; platform.yaml + platform-config.yaml + the app-wave
|
||||
# Kustomizations apps-wave-0/1/2.yaml + apps-vms.yaml = the Kustomizations;
|
||||
# the single apps.yaml was retired). Letting
|
||||
# the GitRepository; platform.yaml/apps.yaml = the Kustomizations). Letting
|
||||
# bootstrap regenerate them would DUPLICATE/conflict. `flux install` +
|
||||
# applying the authored clusters/dev overlay is the reconciling path.
|
||||
# 2. CRD pre-apply — ./preapply-crds.sh applies the Kyverno CRD bundle SERVER-SIDE, because
|
||||
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`
|
||||
# (the ~5.6 MiB bundle blows kubectl's 256 KiB last-applied annotation limit,
|
||||
# which STALLS the HelmRelease and blocks the `platform` Kustomization and
|
||||
# every tier below it). With CRD install off, NOTHING else installs them, so
|
||||
# a cold cluster can never reconcile `platform` without this step. The chart
|
||||
# version is DERIVED from that same HelmRelease — no second pin. Dev only
|
||||
# ever had these CRDs because they were applied BY HAND; this is that step,
|
||||
# in git, idempotent and version-derived. See ./preapply-crds.sh.
|
||||
# 3. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
|
||||
# 2. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
|
||||
# decryption works on first pass (../sops/age-key-secret.md). This script
|
||||
# only VERIFIES it exists; it never handles key material.
|
||||
# 4. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
|
||||
# 3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
|
||||
# GitRepository `flux-system` (source) -> Kustomization `platform` ->
|
||||
# `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
|
||||
# (chained dependsOn) + `apps-vms`, with cluster-vars overlay
|
||||
# (R-ENV-02) and per-Kustomization SOPS decryption enabled via ./patches/.
|
||||
# Kustomization `apps` (dependsOn platform), with cluster-vars overlay
|
||||
# (R-ENV-02) and SOPS decryption enabled via the patches in ./patches/.
|
||||
#
|
||||
# PREREQUISITES the operator/PM resolves before this runs (see ../README.md "Handoff"):
|
||||
# - clusters/base/flux-system.yaml url placeholder is overridden — wire ./patches/flux-system-
|
||||
@@ -64,10 +49,7 @@
|
||||
set -euo pipefail
|
||||
|
||||
# --- config (env-overridable) ------------------------------------------------
|
||||
# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
|
||||
# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
|
||||
# every existing caller — override it, never edit it.
|
||||
FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
|
||||
FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}" # dev is the ONLY Phase-1 build target
|
||||
FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
|
||||
SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
|
||||
# The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
|
||||
@@ -77,94 +59,37 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
|
||||
|
||||
echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
|
||||
echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
|
||||
|
||||
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
|
||||
# helm + yq are needed by step 2 (./preapply-crds.sh renders the CRD bundle from the authored
|
||||
# HelmRelease). Checked HERE as well as in that script so a missing binary stops the run BEFORE the
|
||||
# Flux controllers are installed, instead of half-way through.
|
||||
for bin in flux kubectl helm yq; do
|
||||
for bin in flux kubectl; do
|
||||
command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
|
||||
done
|
||||
|
||||
# --- 1. install Flux controllers (rendered through ./components) --------------
|
||||
# --- 1. install Flux controllers ---------------------------------------------
|
||||
# Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates,
|
||||
# ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the
|
||||
# image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly.
|
||||
#
|
||||
# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently:
|
||||
# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined
|
||||
# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see
|
||||
# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md).
|
||||
#
|
||||
# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the
|
||||
# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux
|
||||
# documents for hand-installed clusters ("Customize Flux manifests"):
|
||||
#
|
||||
# flux install --export > gotk-components.yaml # exact manifests `flux install` would apply
|
||||
# kustomize overlay = ./components/kustomization.yaml (checked in — the ONLY hand-authored part)
|
||||
# kubectl apply -k
|
||||
#
|
||||
# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION,
|
||||
# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is
|
||||
# a temp dir — the repo tree is never written to.
|
||||
#
|
||||
# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
|
||||
# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
|
||||
echo "==> [1/4] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
|
||||
flux_install_args=(--namespace="${FLUX_NAMESPACE}")
|
||||
[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
|
||||
|
||||
if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then
|
||||
echo " WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED."
|
||||
echo " StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN."
|
||||
flux install "${flux_install_args[@]}"
|
||||
if [ -n "${FLUX_COMPONENTS_EXTRA}" ]; then
|
||||
echo "==> [1/3] flux install (default controllers + extra: ${FLUX_COMPONENTS_EXTRA})"
|
||||
flux install --namespace="${FLUX_NAMESPACE}" --components-extra="${FLUX_COMPONENTS_EXTRA}"
|
||||
else
|
||||
components_dir="${script_dir}/components"
|
||||
[ -f "${components_dir}/kustomization.yaml" ] || {
|
||||
echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2
|
||||
echo " (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2
|
||||
exit 1
|
||||
}
|
||||
render_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "${render_dir}"' EXIT
|
||||
echo " rendering flux install --export through ${components_dir}"
|
||||
flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml"
|
||||
cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml"
|
||||
# Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux
|
||||
# version bump) — better a hard stop than silently installing without the feature gate.
|
||||
kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml"
|
||||
grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || {
|
||||
echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2
|
||||
echo " Check ./components/kustomization.yaml against this flux CLI version." >&2
|
||||
exit 1
|
||||
}
|
||||
kubectl apply -f "${render_dir}/rendered.yaml"
|
||||
echo "==> [1/3] flux install (default controllers only)"
|
||||
flux install --namespace="${FLUX_NAMESPACE}"
|
||||
fi
|
||||
|
||||
# --- 2. pre-apply the CRDs Helm is not allowed to install ---------------------
|
||||
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`, so the 22 Kyverno CRDs
|
||||
# must exist BEFORE the platform Kustomization reconciles that HelmRelease — otherwise Kyverno's
|
||||
# controllers come up with no API types, `platform` never goes Ready, and all of its dependents stall.
|
||||
# ./preapply-crds.sh derives the chart version from the HelmRelease itself (no second pin), applies
|
||||
# only the CRD documents SERVER-SIDE (no 256 KiB annotation limit), and is idempotent. It FAILS HARD
|
||||
# rather than skipping — there is deliberately NO opt-out flag: a skipped CRD apply produces a
|
||||
# cluster that looks bootstrapped and reconciles nothing.
|
||||
echo "==> [2/4] pre-applying chart CRDs (server-side)"
|
||||
bash "${script_dir}/preapply-crds.sh"
|
||||
|
||||
# --- 3. verify the out-of-band age key Secret exists -------------------------
|
||||
# --- 2. verify the out-of-band age key Secret exists -------------------------
|
||||
# We DO NOT create it here (it holds the private key). The operator installs it per
|
||||
# ../sops/age-key-secret.md. Fail fast if missing so decryption can't silently no-op.
|
||||
echo "==> [3/4] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
|
||||
echo "==> [2/3] verifying ${SOPS_SECRET_NAME} Secret exists in ${FLUX_NAMESPACE}"
|
||||
if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null 2>&1; then
|
||||
echo "ERROR: Secret ${FLUX_NAMESPACE}/${SOPS_SECRET_NAME} not found." >&2
|
||||
echo " Install the age private key first: bootstrap/sops/age-key-secret.md" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- 4. apply the authored environment entrypoint ----------------------------
|
||||
echo "==> [4/4] kubectl apply -k ${FLUX_ENV_PATH}"
|
||||
# --- 3. apply the authored dev entrypoint ------------------------------------
|
||||
echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
|
||||
kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
|
||||
|
||||
echo
|
||||
|
||||
@@ -1,64 +0,0 @@
|
||||
---
|
||||
# Controller-level customization of the Flux CONTROLLERS themselves (the `flux-system` Deployments).
|
||||
#
|
||||
# WHY THIS DIRECTORY EXISTS
|
||||
# =========================
|
||||
# This repo does NOT use `flux bootstrap` (see ../README.md "Why `flux install` + apply"): the
|
||||
# reconciliation graph is authored by hand in clusters/base, and the controllers are installed
|
||||
# imperatively by `flux install`. Consequence, verified on the live dev cluster 2026-09-01:
|
||||
#
|
||||
# kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
|
||||
# -> NO kustomize.toolkit.fluxcd.io/name label, deployment.kubernetes.io/revision: 1
|
||||
#
|
||||
# i.e. **the controllers are not managed by Flux and are not in git.** There was therefore no
|
||||
# existing mechanism to customize them, and no amount of editing under clusters/** can reach them.
|
||||
#
|
||||
# This is the mechanism Flux documents for exactly that situation ("Customize Flux manifests" —
|
||||
# https://fluxcd.io/flux/installation/#customize-flux-manifests): export the component manifests
|
||||
# with `flux install --export`, overlay them with kustomize `patches:`, and apply the result.
|
||||
# `../bootstrap.sh` step 1 does that render — it copies THIS file next to a freshly exported
|
||||
# `gotk-components.yaml` in a temp dir and runs `kubectl apply -k`. Nothing is generated into the
|
||||
# repo, so the manifests can never drift from the pinned FLUX_VERSION
|
||||
# (infra/devcontainer/versions.env) the way a vendored copy would.
|
||||
#
|
||||
# Every future controller customization (vertical scaling, tolerations, extra feature gates,
|
||||
# --concurrent) belongs HERE, as another entry under `patches:`.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
# Written by `bootstrap.sh` from `flux install --export ...` at render time. NOT checked in.
|
||||
- gotk-components.yaml
|
||||
patches:
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# StrictPostBuildSubstitutions — make postBuild substitution FAIL CLOSED.
|
||||
#
|
||||
# Default behaviour is fail-OPEN: a `${VAR}` that is not defined in `cluster-vars` renders as the
|
||||
# EMPTY STRING, silently. On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium
|
||||
# HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry). With this
|
||||
# gate the undefined variable FAILS the Kustomization instead, and Flux keeps the last-good state.
|
||||
#
|
||||
# PROVEN SAFE before enabling: every one of the 11 substituted paths was rendered with
|
||||
# `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst --strict` and
|
||||
# exited 0, with a negative control reproducing the incident. The permanent guard that keeps it
|
||||
# true is the L0 gate conformance/tests/l0-static/flux-postbuild-substitution/.
|
||||
#
|
||||
# NOTE the gate does NOT catch default-providing forms (`${VAR:-d}`, `${VAR:?msg}`, `${VAR:=v}`) —
|
||||
# Flux exempts them by design. That class is what the L0 gate covers; the two are complementary.
|
||||
#
|
||||
# This also pre-pays for the Flux 2.4.0 -> 2.9.x upgrade: strict substitution became DEFAULT-ON in
|
||||
# kustomize-controller v1.9.0, so the upgrade flips this behaviour whether or not we opt in.
|
||||
#
|
||||
# JSON-6902 `add ... /args/-` (append), NOT a strategic merge: a container's `args` is an atomic
|
||||
# list, so a strategic-merge patch would REPLACE every arg `flux install` generates
|
||||
# (--events-addr, --watch-all-namespaces, --log-level, --log-encoding, --enable-leader-election)
|
||||
# and silently drop any the CLI adds in a future version.
|
||||
- target:
|
||||
group: apps
|
||||
version: v1
|
||||
kind: Deployment
|
||||
name: kustomize-controller
|
||||
namespace: flux-system
|
||||
patch: |-
|
||||
- op: add
|
||||
path: /spec/template/spec/containers/0/args/-
|
||||
value: --feature-gates=StrictPostBuildSubstitutions=true
|
||||
@@ -1,169 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# preapply-crds.sh — server-side pre-apply of the CRDs that Helm is NOT allowed to install.
|
||||
# ENVIRONMENT-AGNOSTIC: it reads the chart coordinates out of the authored HelmRelease and talks to
|
||||
# whatever cluster $KUBECONFIG points at, so dev and prod run the SAME step with no fork (R-ENV-02).
|
||||
# Called by ./bootstrap.sh (step 2/4); also safe to run STANDALONE against an already-bootstrapped
|
||||
# cluster (that is how prod gets unblocked without re-running the whole bootstrap).
|
||||
#
|
||||
# R-GITOPS-01 (the graph must reconcile from git, not from someone's shell history) · R-ENV-02.
|
||||
# =============================================================================
|
||||
# *** AUTHORED, NOT RUN. Agents must not execute it — it MUTATES a cluster. ***
|
||||
# =============================================================================
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
#
|
||||
# platform/kyverno/admission/helmrelease.yaml sets `crds: {install: false}`. That is deliberate:
|
||||
# the Kyverno CRD bundle is ~5.6 MiB, far past kubectl's 256 KiB last-applied-configuration
|
||||
# annotation limit, so Helm's client-side apply fails with "metadata.annotations: Too long" and
|
||||
# STALLS the HelmRelease — which blocks the `platform` Kustomization and every tier that dependsOn
|
||||
# it (the kps-cascade failure mode; see the HelmRelease's own comment).
|
||||
#
|
||||
# The consequence nobody automated: with CRD install off, NOTHING installs those CRDs. Kyverno's
|
||||
# Deployments come up but its API types do not exist, so `platform` never goes Ready on a cold
|
||||
# cluster. The dev cluster only has them because they were once applied BY HAND — exactly the
|
||||
# undocumented manual step this repo exists to eliminate. This script is that step, in git,
|
||||
# version-derived and idempotent.
|
||||
#
|
||||
# HOW THE VERSION IS DERIVED (there is deliberately NO second copy of it)
|
||||
#
|
||||
# Everything comes out of the HelmRelease/HelmRepository pair that Flux itself reconciles:
|
||||
#
|
||||
# chart name spec.chart.spec.chart (HelmRelease)
|
||||
# chart version spec.chart.spec.version (HelmRelease) e.g. "3.9.x"
|
||||
# chart repo url spec.url (HelmRepository)
|
||||
# namespace metadata.namespace (HelmRelease)
|
||||
#
|
||||
# The version is a semver CONSTRAINT, not a fixed tag. `helm template --version` accepts the same
|
||||
# constraint syntax Flux's helm-controller resolves (both use Masterminds/semver), so passing it
|
||||
# through UNCHANGED renders the very chart version Flux will install. Renovate bumps one line in
|
||||
# one file and this step follows automatically — it cannot drift, because there is nothing here to
|
||||
# drift from.
|
||||
#
|
||||
# WHY ONLY THE CRD DOCUMENTS ARE APPLIED
|
||||
#
|
||||
# The by-hand command recorded in the HelmRelease comment piped the WHOLE `helm template` output
|
||||
# into kubectl. Do not do that: it would create Deployments/Services/webhooks outside Helm's
|
||||
# ownership, which then fight the HelmRelease over field managers and leave orphans behind. We
|
||||
# filter to `kind: CustomResourceDefinition` and apply nothing else. Values are the chart DEFAULTS
|
||||
# (plus an explicit crds.install=true) — NOT the repo's values, which switch CRD rendering off.
|
||||
#
|
||||
# IDEMPOTENCY
|
||||
#
|
||||
# `kubectl apply --server-side --force-conflicts` is a declarative upsert: the first run creates,
|
||||
# every later run reports "serverside-applied" and exits 0, and field ownership is taken from
|
||||
# whatever previously wrote those objects (that is what --force-conflicts is for — it resolves the
|
||||
# hand-applied dev CRDs' `kubectl-client-side-apply` manager without a human deciding anything).
|
||||
# Server-side apply does not write the last-applied annotation, which is the whole reason the
|
||||
# 5.6 MiB bundle can be applied at all.
|
||||
#
|
||||
# LOUD FAILURE (this script must never degrade to a successful no-op)
|
||||
#
|
||||
# `set -euo pipefail`, plus three explicit guards, because a silent skip here reproduces the exact
|
||||
# class of bug this repo has been bitten by more than once:
|
||||
# 1. every derived coordinate must be non-empty and not "null" — a moved YAML path fails here;
|
||||
# 2. the rendered CRD count must be > 0 — if a future chart moves/renames its CRD templates the
|
||||
# filter would match nothing and `kubectl apply` on an empty stream EXITS 0. Guarded;
|
||||
# 3. after applying, every CRD is waited on for condition=Established — so "applied" is proven
|
||||
# to mean "the API type exists", not merely "kubectl returned 0".
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
# --- config (env-overridable) ------------------------------------------------
|
||||
# The single source of truth for the chart coordinates. Repo-relative on purpose: this is the file
|
||||
# Flux reconciles, so it is the file this step must agree with.
|
||||
CRD_HELMRELEASE="${CRD_HELMRELEASE:-platform/kyverno/admission/helmrelease.yaml}"
|
||||
# How long to wait for the applied CRDs to become Established (apiextensions publishes the type).
|
||||
CRD_ESTABLISH_TIMEOUT="${CRD_ESTABLISH_TIMEOUT:-120s}"
|
||||
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
|
||||
|
||||
echo "==> CRD pre-apply — source of truth: ${CRD_HELMRELEASE}"
|
||||
|
||||
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
|
||||
# helm renders the bundle, yq selects the CRD documents, kubectl applies them server-side. All three
|
||||
# are baked into infra/devcontainer (HELM_VERSION / YQ_VERSION / KUBECTL_VERSION in versions.env).
|
||||
for bin in helm yq kubectl; do
|
||||
command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
|
||||
done
|
||||
|
||||
helmrelease="${repo_root}/${CRD_HELMRELEASE}"
|
||||
[ -f "${helmrelease}" ] || {
|
||||
echo "ERROR: ${helmrelease} not found — CRD_HELMRELEASE points nowhere." >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# --- 1. derive the chart coordinates from the authored manifests --------------
|
||||
chart_name="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.chart' "${helmrelease}")"
|
||||
chart_version="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .spec.chart.spec.version' "${helmrelease}")"
|
||||
chart_ns="$(yq eval-all --no-doc 'select(.kind == "HelmRelease") | .metadata.namespace' "${helmrelease}")"
|
||||
chart_repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' "${helmrelease}")"
|
||||
|
||||
# Guard 1: a moved/renamed YAML path must STOP the bootstrap, not render a bogus chart. yq prints
|
||||
# "null" for a missing path and an empty string for a missing document, so both are rejected.
|
||||
for pair in "chart:${chart_name}" "version:${chart_version}" "namespace:${chart_ns}" "repo:${chart_repo}"; do
|
||||
key="${pair%%:*}"; val="${pair#*:}"
|
||||
case "${val}" in
|
||||
""|null|*$'\n'*)
|
||||
echo "ERROR: could not derive a single '${key}' from ${CRD_HELMRELEASE} (got: '${val}')." >&2
|
||||
echo " The HelmRelease/HelmRepository shape changed — fix the yq paths in $(basename "${BASH_SOURCE[0]}")." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo " chart=${chart_name} version=${chart_version} repo=${chart_repo} namespace=${chart_ns}"
|
||||
echo " (version is the constraint Flux resolves — NOT a second copy pinned here)"
|
||||
|
||||
# --- 2. render the chart and keep ONLY the CRD documents ----------------------
|
||||
render_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "${render_dir}"' EXIT
|
||||
crd_file="${render_dir}/crds.yaml"
|
||||
|
||||
# --include-crds covers charts that ship a native crds/ dir; crds.install=true covers Kyverno, whose
|
||||
# CRDs are subchart TEMPLATES that the repo's own values switch off. Chart defaults otherwise.
|
||||
echo "==> rendering ${chart_name} ${chart_version} (CRD documents only)"
|
||||
helm template "${chart_name}" "${chart_name}" \
|
||||
--repo "${chart_repo}" \
|
||||
--version "${chart_version}" \
|
||||
--namespace "${chart_ns}" \
|
||||
--include-crds \
|
||||
--set crds.install=true \
|
||||
| yq eval-all 'select(.kind == "CustomResourceDefinition")' - > "${crd_file}"
|
||||
|
||||
# Blank/"null" lines are dropped on purpose: an EMPTY render makes `yq .metadata.name` emit one
|
||||
# empty line, which would otherwise count as a CRD and sail straight past guard 2 below.
|
||||
mapfile -t crd_names < <(yq eval-all --no-doc '.metadata.name' "${crd_file}" | grep -vx -e '' -e 'null' || true)
|
||||
crd_count="${#crd_names[@]}"
|
||||
|
||||
# Guard 2: an empty stream applies cleanly and exits 0 — the silent-no-op trap. Refuse it.
|
||||
[ "${crd_count}" -gt 0 ] || {
|
||||
echo "ERROR: rendered 0 CustomResourceDefinitions from ${chart_name} ${chart_version}." >&2
|
||||
echo " The chart's CRD layout or the crds.install value key changed. Refusing to continue:" >&2
|
||||
echo " applying nothing would look like success and leave 'platform' blocked forever." >&2
|
||||
exit 1
|
||||
}
|
||||
echo " ${crd_count} CRDs, $(wc -c < "${crd_file}") bytes (>256 KiB — this is why it must be server-side)"
|
||||
|
||||
# --- 3. server-side apply (idempotent upsert) ---------------------------------
|
||||
# --server-side: no last-applied annotation, so the size limit does not apply.
|
||||
# --force-conflicts: adopt fields owned by an earlier manager (e.g. dev's hand-applied CRDs, or
|
||||
# Helm's own manager if a past chart version did install them). Makes a re-run a plain no-op.
|
||||
echo "==> kubectl apply --server-side --force-conflicts (${crd_count} CRDs)"
|
||||
kubectl apply --server-side --force-conflicts -f "${crd_file}"
|
||||
|
||||
# --- 4. prove the API types actually exist -----------------------------------
|
||||
# Without this, "kubectl returned 0" is the only evidence. Establishment is what Kyverno's
|
||||
# controllers and the platform Kustomization actually need.
|
||||
echo "==> waiting for condition=Established (timeout ${CRD_ESTABLISH_TIMEOUT})"
|
||||
crd_refs=()
|
||||
for n in "${crd_names[@]}"; do crd_refs+=("crd/${n}"); done
|
||||
kubectl wait --for=condition=Established --timeout="${CRD_ESTABLISH_TIMEOUT}" \
|
||||
"${crd_refs[@]}" >/dev/null || {
|
||||
echo "ERROR: not all ${crd_count} CRDs reached Established within ${CRD_ESTABLISH_TIMEOUT}." >&2
|
||||
echo " Inspect: kubectl get crd | grep ${chart_name}" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
echo "==> ${crd_count} ${chart_name} CRDs applied and Established."
|
||||
@@ -1,30 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
# bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
|
||||
# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
|
||||
# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
|
||||
# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
|
||||
# using it now.
|
||||
# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
|
||||
# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
|
||||
# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
|
||||
# CONTRIBUTING.md).
|
||||
#
|
||||
# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
|
||||
# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
|
||||
#
|
||||
# PROD (bare metal, current use): HOST_NAT_IP=192.168.2.132
|
||||
# The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
|
||||
# NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
|
||||
# three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
|
||||
# apply to prod, so do not "fix" prod by opening zones.
|
||||
#
|
||||
# DEV (libvirt VMs, HISTORICAL): HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
|
||||
# *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
|
||||
# bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
|
||||
# recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
|
||||
#
|
||||
# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
|
||||
# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
|
||||
# author-only work and must never be fired off casually or "to see if it works": it starts a
|
||||
# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
|
||||
# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
|
||||
# it will then reconcile. Know which HEAD you are publishing before you run it.
|
||||
# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
|
||||
set -euo pipefail
|
||||
ENGINE="${ENGINE:-podman}"
|
||||
GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}" # pinned; bump if needed
|
||||
@@ -35,24 +16,11 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
|
||||
# subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
|
||||
DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
|
||||
DATA_DIR="podman volume ${DATA_VOLUME}" # display label only (data lives in the named volume)
|
||||
# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
|
||||
# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
|
||||
# clone/redirect URLs are correct from inside the cluster.
|
||||
#
|
||||
# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
|
||||
# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
|
||||
# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
|
||||
# Pass it explicitly, e.g. for prod: HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
|
||||
#
|
||||
# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
|
||||
# VM->host:3000 by default and had to be opened once with
|
||||
# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
|
||||
# sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
|
||||
# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
|
||||
if [ -z "${HOST_NAT_IP:-}" ]; then
|
||||
echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
|
||||
echo " prod: HOST_NAT_IP=192.168.2.132 (same LAN as the nodes, no NAT, no firewall change)" >&2
|
||||
echo " There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
|
||||
exit 1
|
||||
fi
|
||||
HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
|
||||
ORG="${GITEA_ORG:-Home-Network}"
|
||||
REPO="${GITEA_REPO:-home-network-k3s-impl}"
|
||||
|
||||
@@ -107,17 +75,8 @@ cat <<EOF
|
||||
|
||||
Gitea bootstrap source ready (${DATA_DIR}).
|
||||
UI (host): ${BASE}/${ORG}/${REPO}
|
||||
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
|
||||
(patch this url into the overlay being bootstrapped:
|
||||
clusters/prod today; clusters/dev has already cut over)
|
||||
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git (clusters/dev)
|
||||
Admin creds (host, 0600): ${ADMIN_CRED_FILE}
|
||||
Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
|
||||
firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
|
||||
must allow VM->host:${HTTP_PORT} —
|
||||
Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
|
||||
ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
|
||||
REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
|
||||
at this source will reconcile THAT revision.
|
||||
TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
|
||||
any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
|
||||
procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
|
||||
EOF
|
||||
@@ -1,988 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# standup.sh — bring a bootstrapped cluster's reconciliation graph all the way up.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): it reads everything it needs from the cluster it is pointed at and
|
||||
# from the authored manifests. There is no dev/prod fork. What makes it safe to run against prod is
|
||||
# the MANDATORY --expect-server guard, not a name in the filename.
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
# `bootstrap/flux/bootstrap.sh` installs Flux and applies the graph. It does NOT finish the job when
|
||||
# tiers have been deliberately suspended (a DNS/TLS migration window, an incident recovery) or when
|
||||
# a HelmRelease has wedged. Those last steps were living in shell history and in a human's head,
|
||||
# which is the same failure the Kyverno CRD pre-apply had. This is that sequence, in git, ordered,
|
||||
# idempotent, and gated.
|
||||
#
|
||||
# THE ORDERING IS LOAD-BEARING. Read infra/PROD-STANDUP.md before deviating from it.
|
||||
#
|
||||
# USAGE
|
||||
# bootstrap/standup.sh --kubeconfig <path> --expect-server <url> [--confirm-destructive] <step...>
|
||||
#
|
||||
# steps: preflight crds unwedge externaldns s3check resume verify (or: all)
|
||||
#
|
||||
# --expect-server is REQUIRED and is compared against the live API server before ANY command runs.
|
||||
# Pointing this at the wrong cluster is the one mistake with unbounded blast radius: dev is live and
|
||||
# shares a Cloudflare zone with prod. The guard is not optional and there is no override flag.
|
||||
#
|
||||
# --confirm-destructive is required by `unwedge` only. Everything else is additive or reversible.
|
||||
#
|
||||
# --s3-endpoint <url> makes `s3check` probe a CANDIDATE backup endpoint instead of the live
|
||||
# cluster-vars value, so a proposed target can be PROVEN before it is authored into cluster-vars.
|
||||
# `s3check` writes and then deletes ONE throwaway object under `<prefix>/.s3check/`; it changes
|
||||
# nothing else and applies nothing. See infra/BACKUP-R2.md.
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
# ---------------------------------------------------------------------------- args
|
||||
KUBECONFIG_ARG="" EXPECT_SERVER="" CONFIRM_DESTRUCTIVE=0 S3_ENDPOINT_OVERRIDE=""
|
||||
STEPS=()
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--kubeconfig) KUBECONFIG_ARG="${2:?--kubeconfig needs a value}"; shift 2 ;;
|
||||
--expect-server) EXPECT_SERVER="${2:?--expect-server needs a value}"; shift 2 ;;
|
||||
--confirm-destructive) CONFIRM_DESTRUCTIVE=1; shift ;;
|
||||
--s3-endpoint) S3_ENDPOINT_OVERRIDE="${2:?--s3-endpoint needs a value}"; shift 2 ;;
|
||||
-h|--help) sed -n '2,30p' "$0"; exit 0 ;;
|
||||
-*) echo "ERROR: unknown flag '$1'" >&2; exit 2 ;;
|
||||
*) STEPS+=("$1"); shift ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ -n "${KUBECONFIG_ARG}" ] || { echo "ERROR: --kubeconfig is required." >&2; exit 2; }
|
||||
[ -n "${EXPECT_SERVER}" ] || { echo "ERROR: --expect-server is required (safety guard)." >&2; exit 2; }
|
||||
[ "${#STEPS[@]}" -gt 0 ] || { echo "ERROR: no step given. One of: preflight crds unwedge externaldns s3check resume verify all" >&2; exit 2; }
|
||||
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/.." && pwd))"
|
||||
cd "${repo_root}"
|
||||
|
||||
export KUBECONFIG="${KUBECONFIG_ARG}"
|
||||
[ -f "${KUBECONFIG}" ] || { echo "ERROR: kubeconfig not found: ${KUBECONFIG}" >&2; exit 1; }
|
||||
|
||||
# Where DNS snapshots go. This MUST be a path that survives between invocations: each `dev.sh run` is
|
||||
# a FRESH container, so a $TMPDIR default silently threw the baseline away and made `verify`'s
|
||||
# before/after comparison — the check that detects damage to the OTHER cluster's records — impossible
|
||||
# to run across steps. The repo is bind-mounted, so a repo-relative dir persists on the host.
|
||||
# Gitignored: these are run artifacts, not authored state.
|
||||
SNAP_DIR="${STANDUP_SNAP_DIR:-${repo_root}/.standup-dns}"
|
||||
mkdir -p "${SNAP_DIR}"
|
||||
|
||||
say() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
info() { printf ' %s\n' "$*"; }
|
||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
||||
warn() { printf '\033[33mWARN: %s\033[0m\n' "$*" >&2; }
|
||||
|
||||
for bin in kubectl flux helm yq sops; do
|
||||
command -v "$bin" >/dev/null 2>&1 || die "'$bin' not found — run this inside infra/devcontainer."
|
||||
done
|
||||
|
||||
# ------------------------------------------------------------------ THE SAFETY GUARD
|
||||
# Checked once, before any step, and re-checked by every mutating helper. The failure mode this
|
||||
# prevents is running a prod bring-up against the LIVE dev cluster, whose public DNS records prod
|
||||
# has already destroyed once (2026-09-04).
|
||||
live_server="$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')"
|
||||
[ "${live_server}" = "${EXPECT_SERVER}" ] || die \
|
||||
"TARGET MISMATCH — refusing to run.
|
||||
--expect-server : ${EXPECT_SERVER}
|
||||
live API server : ${live_server}
|
||||
The kubeconfig does not point where you think it does. Nothing has been changed."
|
||||
say "target confirmed: ${live_server}"
|
||||
|
||||
# Dependency order. Flux enforces the edges itself via dependsOn; resuming in this order simply means
|
||||
# each wait is meaningful instead of blocking on a dependency that is still suspended.
|
||||
# flux-system -> cilium -> platform -> platform-config -> platform-config-cdi
|
||||
# -> {leaf tiers, apps-wave-0} -> wave-1 -> wave-2 -> apps-vms
|
||||
#
|
||||
# `platform-config-cdi` MUST stay in this list and MUST stay before apps-vms. It carries the CDI
|
||||
# StorageProfiles, which were split out of platform-config because StorageProfile's CRD does not exist
|
||||
# until cdi-operator has reconciled the CDI CR (clusters/base/platform-config-cdi.yaml). apps-vms
|
||||
# dependsOn it: apps/vms/unifi provisions on `longhorn-vm`, and without the profile applied CDI infers
|
||||
# ReadWriteOnce instead of the authored ReadWriteMany — permanently non-live-migratable for that disk.
|
||||
# Omitting it here would make `resume` return success without ever waiting on the tier that guarantees it.
|
||||
TIER_ORDER=(platform platform-config platform-config-cdi oauth2-proxy headlamp homepage crafty apps-wave-0 apps-wave-1 apps-wave-2 apps-vms)
|
||||
# velero is NOT in that list on purpose — see step `resume`: it consumes S3_BACKUP_ENDPOINT, which is
|
||||
# an intentionally-invalid placeholder until the operator supplies the real R2 endpoint.
|
||||
|
||||
cvar() { kubectl -n flux-system get cm cluster-vars -o jsonpath="{.data.$1}" 2>/dev/null || true; }
|
||||
|
||||
k_ready() { kubectl -n flux-system get kustomization "$1" \
|
||||
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true; }
|
||||
k_msg() { kubectl -n flux-system get kustomization "$1" \
|
||||
-o jsonpath='{.status.conditions[?(@.type=="Ready")].message}' 2>/dev/null || true; }
|
||||
k_susp() { kubectl -n flux-system get kustomization "$1" -o jsonpath='{.spec.suspend}' 2>/dev/null || true; }
|
||||
|
||||
wait_ready() { # <kustomization> <timeout-seconds>
|
||||
local n="$1" t="${2:-600}" i=0
|
||||
while [ "$i" -lt "$t" ]; do
|
||||
[ "$(k_ready "$n")" = "True" ] && { info "$n Ready (${i}s)"; return 0; }
|
||||
sleep 10; i=$((i+10))
|
||||
done
|
||||
warn "$n did NOT reach Ready in ${t}s: $(k_msg "$n")"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Snapshot the Cloudflare zone. This is how damage to the OTHER cluster's records becomes detectable.
|
||||
# Queried against the Cloudflare API, never a local resolver: the workstation's own dnsmasq wildcard
|
||||
# (address=/.home.keithjackson.dev/) answers the whole tree locally and produced a false
|
||||
# healthy reading during the 2026-09-04 incident and hid the damage for a while.
|
||||
dns_snapshot() { # <output-file>
|
||||
local out="$1" tok zid page tp
|
||||
tok="$(sops -d platform/externaldns/cloudflare-api-token.sops.yaml \
|
||||
| yq '.stringData."api-token"' | tr -d '"')"
|
||||
[ -n "${tok}" ] && [ "${tok}" != "null" ] || die "could not read the Cloudflare token (SOPS/age key available?)"
|
||||
zid="$(curl -sS -H "Authorization: Bearer ${tok}" \
|
||||
'https://api.cloudflare.com/client/v4/zones?name=keithjackson.dev' | yq '.result[0].id' | tr -d '"')"
|
||||
[ -n "${zid}" ] && [ "${zid}" != "null" ] || die "Cloudflare zone lookup failed"
|
||||
page=1; : > "${out}"
|
||||
while :; do
|
||||
local r; r="$(curl -sS -H "Authorization: Bearer ${tok}" \
|
||||
"https://api.cloudflare.com/client/v4/zones/${zid}/dns_records?per_page=100&page=${page}")"
|
||||
printf '%s' "${r}" | yq -o=json '.result' \
|
||||
| yq -r '.[] | (.type + "|" + .name + "|" + .content)' | tr -d '"' >> "${out}"
|
||||
tp="$(printf '%s' "${r}" | yq '.result_info.total_pages' | tr -d '"')"
|
||||
[ "${page}" -ge "${tp:-1}" ] && break
|
||||
page=$((page+1))
|
||||
done
|
||||
# Strip CRs, and sort under a FIXED collation. Both matter, and both were learned the hard way:
|
||||
# - a stray \r makes every line differ from a baseline captured without one;
|
||||
# - locale-dependent sort ordering puts `*.keithjackson.dev` in a different position on the host
|
||||
# than in the container, so a diff reports pairs of IDENTICAL lines as changed. That produced a
|
||||
# "RECORDS OUTSIDE ... CHANGED" alarm on a zone where nothing had changed at all.
|
||||
# A regression check that cries wolf is worse than none — it trains you to skip the one alarm that
|
||||
# matters. LC_ALL=C makes the ordering reproducible everywhere.
|
||||
tr -d '\r' < "${out}" | grep -vE '^\s*$' | LC_ALL=C sort -o "${out}"
|
||||
info "$(wc -l < "${out}") records -> ${out}"
|
||||
}
|
||||
|
||||
dns_diff() { # <before> <after> — returns non-zero if anything changed
|
||||
if diff -u "$1" "$2" > "${SNAP_DIR}/dns.diff"; then
|
||||
info "Cloudflare zone IDENTICAL — nothing created, modified or deleted."
|
||||
return 0
|
||||
fi
|
||||
warn "Cloudflare zone CHANGED:"; cat "${SNAP_DIR}/dns.diff"
|
||||
return 1
|
||||
}
|
||||
|
||||
# ======================================================================= s3check plumbing
|
||||
# WHY THIS IS THIS BIG. The previous version of `s3check` curl'd /minio/health/live and reported
|
||||
# REACHABLE. That probe could not pass against Cloudflare R2 (no such path), and reachability is not
|
||||
# the property anyone cares about: a reachable endpoint with a wrong key, a missing bucket, or R2's
|
||||
# checksum rejection still yields a velero tier that reports Ready while NO backup exists. What must
|
||||
# be proven is the write path Velero and CNPG-barman actually use, with the credentials they actually
|
||||
# hold, from the network position they actually occupy.
|
||||
|
||||
# Pinned. The AWS CLI is a deliberate choice over `minio/mc`:
|
||||
# - it prints the S3 ERROR CODE (NoSuchBucket / AccessDenied / InvalidAccessKeyId / InvalidArgument),
|
||||
# so a failure can be NAMED and turned into a fix instead of guessed at from an HTTP status;
|
||||
# - it exposes the FLEXIBLE-CHECKSUM behaviour that is the R2 trap
|
||||
# (AWS_REQUEST_CHECKSUM_CALCULATION), so the trap can be reproduced on purpose here instead of
|
||||
# discovered by a silently-failing backup at 03:00;
|
||||
# - its image carries bash + coreutils. `minio/mc` ships no sed/awk/diff — that minimalism has
|
||||
# already broken one probe in this repo — and it cannot be made to send the checksum headers, so
|
||||
# it cannot test for the trap at all.
|
||||
S3CHECK_IMAGE="amazon/aws-cli:2.36.40"
|
||||
S3CHECK_RUN_ID="$(date +%s)"
|
||||
|
||||
# Throwaway probe pods, tracked so they are removed on EVERY exit path (including `die`). A probe
|
||||
# must never leave a pod behind on a live cluster.
|
||||
S3CHECK_PODS=()
|
||||
s3_cleanup_pods() {
|
||||
local p
|
||||
for p in ${S3CHECK_PODS[@]+"${S3CHECK_PODS[@]}"}; do
|
||||
kubectl -n "${p%%/*}" delete pod "${p#*/}" --ignore-not-found --wait=false >/dev/null 2>&1 || true
|
||||
done
|
||||
S3CHECK_PODS=()
|
||||
}
|
||||
trap s3_cleanup_pods EXIT
|
||||
|
||||
# TRUE when the endpoint is not a usable target. `TBD-*` alone was WRONG once the placeholder became
|
||||
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`: that does not START with TBD-, so the old
|
||||
# prefix pattern matched nothing and `resume` would have resumed velero straight onto a placeholder
|
||||
# hostname — the exact silent-no-backups failure the guard exists to prevent. Match anywhere.
|
||||
s3_is_placeholder() { # <endpoint>
|
||||
case "$1" in
|
||||
""|*TBD*|*tbd-*|*REPLACE-ME*|*replace-me*|*CHANGE-ME*|*change-me*) return 0 ;;
|
||||
esac
|
||||
return 1
|
||||
}
|
||||
|
||||
# Refuse any value that is not a plain URL/name fragment. This is BOTH a safety guard — these values
|
||||
# are interpolated into the probe pod's JSON overrides AND into a shell script that runs inside it, so
|
||||
# a quote, a backslash or a newline could rewrite either — and a correctness guard: a cluster-vars
|
||||
# value with a trailing space or a stray quote is a defect, not something to paper over.
|
||||
s3_safe_token() { # <label> <value>
|
||||
case "$2" in
|
||||
*[!A-Za-z0-9:/._-]*) die "refusing to probe: $1 contains a character outside [A-Za-z0-9:/._-].
|
||||
That is either a corrupt cluster-vars value or an injection attempt. Value (quoted): $(printf '%q' "$2")" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- the in-pod probe
|
||||
# Emitted to the pod base64-encoded, so nothing in it has to survive JSON or shell quoting.
|
||||
# CONTRACT WITH THE CALLER (this is what makes a silent no-op impossible):
|
||||
# - every proven property prints exactly one `S3CHECK-OK[n] ...` line and increments n;
|
||||
# - every failure prints `S3CHECK-FAIL ...` and is counted;
|
||||
# - the LAST line is `S3CHECK-RESULT PASS|FAIL assertions=<n> failures=<n>`.
|
||||
# The caller independently knows how many assertions this environment must produce and re-counts the
|
||||
# OK lines itself, so neither a truncated log, a skipped bucket, an empty log, nor a lying result line
|
||||
# can be read as success. (A CRD guard in this repo once counted a blank line as "1 CRD" and sailed
|
||||
# past; that class of bug is designed out here rather than commented about.)
|
||||
s3_probe_script() {
|
||||
cat <<'PROBE'
|
||||
set -u
|
||||
umask 077
|
||||
A=0; F=0
|
||||
ok() { A=$((A+1)); printf 'S3CHECK-OK[%d] %s\n' "$A" "$*"; }
|
||||
bad() { F=$((F+1)); printf 'S3CHECK-FAIL %s\n' "$*"; }
|
||||
note() { printf 'S3CHECK-NOTE %s\n' "$*"; }
|
||||
verdict() {
|
||||
if [ "${F}" -eq 0 ] && [ "${A}" -gt 0 ]; then
|
||||
printf 'S3CHECK-RESULT PASS assertions=%d failures=0\n' "${A}"; exit 0
|
||||
fi
|
||||
printf 'S3CHECK-RESULT FAIL assertions=%d failures=%d\n' "${A}" "${F}"; exit 1
|
||||
}
|
||||
|
||||
for v in S3EP S3REGION S3TARGETS S3KEY CRED_MODE; do
|
||||
eval "x=\${${v}-}"
|
||||
[ -n "${x}" ] || { bad "required env ${v} is empty — the probe was launched wrong"; verdict; }
|
||||
done
|
||||
|
||||
export AWS_EC2_METADATA_DISABLED=true
|
||||
export AWS_PAGER=""
|
||||
export AWS_DEFAULT_REGION="${S3REGION}"
|
||||
export AWS_REGION="${S3REGION}"
|
||||
export AWS_MAX_ATTEMPTS=2
|
||||
export AWS_RETRY_MODE=standard
|
||||
export AWS_CONFIG_FILE=/tmp/s3check-aws-config
|
||||
# Path-style addressing, because both targets need it: MinIO on a bare IP has no virtual-host names,
|
||||
# and R2's S3 API is documented path-style. Checksums OFF, mirroring `checksumAlgorithm: ""` on the
|
||||
# Velero BSL — this run must exercise the same wire shape the real backup will use. Written to the
|
||||
# CONFIG FILE as well as the environment so the setting holds even if one channel is ignored by a
|
||||
# future CLI version.
|
||||
{
|
||||
echo '[default]'
|
||||
echo 'request_checksum_calculation = when_required'
|
||||
echo 'response_checksum_validation = when_required'
|
||||
echo 's3 ='
|
||||
echo ' addressing_style = path'
|
||||
} > "${AWS_CONFIG_FILE}"
|
||||
|
||||
# ---- (1) the credential loads at all -------------------------------------------------
|
||||
# NEVER print a credential value, in either `key: value` or `key=value` shape. Only booleans,
|
||||
# byte counts and lengths leave this pod.
|
||||
case "${CRED_MODE}" in
|
||||
file)
|
||||
cf="${AWS_SHARED_CREDENTIALS_FILE-}"
|
||||
if [ -z "${cf}" ] || [ ! -s "${cf}" ]; then
|
||||
bad "credential file '${cf:-<unset>}' is missing or empty — the Secret key did not mount"
|
||||
elif ! grep -Eq '^[[:space:]]*\[default\][[:space:]]*$' "${cf}"; then
|
||||
bad "credential file has no [default] profile. velero-plugin-for-aws reads an AWS INI file from
|
||||
the 'cloud' key of velero-cloud-credentials; without [default] every backup fails at RUNTIME"
|
||||
elif ! grep -Eq '^[[:space:]]*aws_access_key_id[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}" \
|
||||
|| ! grep -Eq '^[[:space:]]*aws_secret_access_key[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}"; then
|
||||
bad "credential file [default] lacks a non-empty aws_access_key_id / aws_secret_access_key"
|
||||
else
|
||||
ok "credential LOADS: AWS INI with [default] + both keys ($(wc -c < "${cf}") bytes; values never printed)"
|
||||
fi
|
||||
;;
|
||||
env)
|
||||
if [ -z "${AWS_ACCESS_KEY_ID-}" ] || [ -z "${AWS_SECRET_ACCESS_KEY-}" ]; then
|
||||
bad "AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY did not arrive from the Secret (missing ACCESS_KEY_ID / ACCESS_SECRET_KEY key?)"
|
||||
else
|
||||
ok "credential LOADS: access key id present (${#AWS_ACCESS_KEY_ID} chars; values never printed)"
|
||||
fi
|
||||
;;
|
||||
*) bad "unknown CRED_MODE '${CRED_MODE}'" ;;
|
||||
esac
|
||||
[ "${F}" -eq 0 ] || verdict # no point talking to S3 with no credential
|
||||
|
||||
OUT=""; RC=0
|
||||
awsx() { OUT="$(aws --endpoint-url "${S3EP}" "$@" 2>&1)"; RC=$?; return 0; }
|
||||
|
||||
detail() { printf '%s\n' "${OUT}" | head -n 12 | while IFS= read -r l; do printf 'S3CHECK-DETAIL %s\n' "${l}"; done; }
|
||||
|
||||
# Every failure is NAMED and given the fix. An unclassified failure is still a failure.
|
||||
diagnose() { # <operation label>
|
||||
local op="$1"
|
||||
case "${OUT}" in
|
||||
*InvalidArgument*|*checksum*|*Checksum*)
|
||||
bad "${op}: 400 InvalidArgument — THE R2 CHECKSUM TRAP. velero-plugin-for-aws v1.14+ bundles an
|
||||
AWS SDK that sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT, and
|
||||
Cloudflare R2 REJECTS those headers with HTTP 400 InvalidArgument, so every Velero backup and
|
||||
every kopia blob upload fails. FIX: checksumAlgorithm: \"\" must be present in the
|
||||
backupStorageLocation config in platform/velero/helmrelease.yaml. This probe already disables
|
||||
checksums, so seeing this here means the client sent one anyway — treat it as a hard blocker." ;;
|
||||
*"Could not connect"*|*EndpointConnectionError*|*ConnectTimeoutError*|*"Failed to establish"*|\
|
||||
*"Name or service not known"*|*"nodename nor servname"*|*"Temporary failure in name resolution"*|\
|
||||
*"Connection reset"*|*"Read timeout"*)
|
||||
bad "${op}: this POD cannot reach ${S3EP} (connect / DNS / timeout). Velero and CNPG would
|
||||
install perfectly clean and then fail every backup at RUNTIME — a tier that looks healthy
|
||||
while no backup exists. This is the reading that matters: the workstation's own reachability
|
||||
is irrelevant (prod sits on the SOHO net, pfSense's WAN; the dev LAN's MinIO sits INSIDE
|
||||
pfSense) and a host-side probe has already misled this project once. Fix pod egress/routing,
|
||||
or give this environment its own endpoint." ;;
|
||||
*NoSuchBucket*|*"(404)"*|*"Not Found"*)
|
||||
bad "${op}: the bucket does not exist at this endpoint. Create it before enabling velero/CNPG
|
||||
(R2: dashboard -> R2 -> Create bucket; see infra/BACKUP-R2.md)." ;;
|
||||
*InvalidAccessKeyId*|*SignatureDoesNotMatch*|*"(401)"*|*Unauthorized*)
|
||||
bad "${op}: the endpoint REJECTED these credentials. Wrong key, or a key belonging to a
|
||||
different R2 account / MinIO. Re-encrypt the credential Secrets — infra/BACKUP-R2.md." ;;
|
||||
*AccessDenied*|*"(403)"*|*Forbidden*)
|
||||
bad "${op}: authenticated but FORBIDDEN. The token is not scoped to this bucket. An R2 API token
|
||||
needs Object Read & Write on BOTH the velero and cnpg buckets." ;;
|
||||
*) bad "${op}: unclassified failure (rc=${RC}). Treated as a FAILURE, never as unknown-so-fine." ;;
|
||||
esac
|
||||
detail
|
||||
}
|
||||
|
||||
PAY=/tmp/s3check.payload
|
||||
GOT=/tmp/s3check.got
|
||||
head -c 4096 /dev/urandom | base64 > "${PAY}"
|
||||
[ -s "${PAY}" ] || { bad "could not generate a probe payload inside the pod"; verdict; }
|
||||
|
||||
TRAP_DONE=0
|
||||
for t in ${S3TARGETS}; do
|
||||
B="${t%%:*}"; P="${t#*:}"
|
||||
[ "${P}" = "${B}" ] && P=""
|
||||
if [ -n "${P}" ]; then K="${P}/.s3check/${S3KEY}"; else K=".s3check/${S3KEY}"; fi
|
||||
|
||||
# ---- (2) the bucket exists and is LISTABLE. This is the operation Velero's BSL sync and Barman's
|
||||
# WAL scan both perform; ListBuckets deliberately is NOT used, because neither consumer
|
||||
# needs it and a correctly bucket-scoped R2 token may not grant it.
|
||||
args=(s3api list-objects-v2 --bucket "${B}" --max-keys 1)
|
||||
[ -n "${P}" ] && args+=(--prefix "${P}/")
|
||||
awsx "${args[@]}"
|
||||
if [ "${RC}" -eq 0 ]; then
|
||||
ok "${B}: bucket EXISTS, AUTHENTICATES and is LISTABLE (prefix '${P:-<bucket root>}')"
|
||||
else
|
||||
diagnose "LIST ${B}"
|
||||
continue # a short assertion count is how the caller learns this target was abandoned
|
||||
fi
|
||||
|
||||
# ---- (3) PUT -> GET -> byte-compare -> DELETE. A real write, not a HEAD: HEAD on a bucket proves
|
||||
# nothing about whether an object can be STORED (wrong scope, checksum rejection and
|
||||
# read-only tokens all pass a HEAD and fail a backup).
|
||||
awsx s3api put-object --bucket "${B}" --key "${K}" --body "${PAY}"
|
||||
if [ "${RC}" -eq 0 ]; then ok "${B}: PUT ${K} accepted (genuine write path)"
|
||||
else diagnose "PUT ${B}/${K}"; continue; fi
|
||||
|
||||
rm -f "${GOT}"
|
||||
awsx s3api get-object --bucket "${B}" --key "${K}" "${GOT}"
|
||||
if [ "${RC}" -eq 0 ]; then ok "${B}: GET ${K} returned an object"
|
||||
else diagnose "GET ${B}/${K}"
|
||||
aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${K}" >/dev/null 2>&1 || true
|
||||
continue; fi
|
||||
|
||||
sp="$(wc -c < "${PAY}")"; sg="$(wc -c < "${GOT}" 2>/dev/null || echo 0)"
|
||||
hp="$(sha256sum < "${PAY}")"; hp="${hp%% *}"
|
||||
hg="$(sha256sum < "${GOT}")"; hg="${hg%% *}"
|
||||
if [ "${sp}" = "${sg}" ] && [ -n "${hp}" ] && [ "${hp}" = "${hg}" ]; then
|
||||
ok "${B}: round trip is BYTE-IDENTICAL (${sp} bytes, sha256 matches)"
|
||||
else
|
||||
bad "${B}: round trip CORRUPTED — wrote ${sp} bytes (sha256 ${hp}), read back ${sg} bytes (sha256 ${hg})"
|
||||
fi
|
||||
|
||||
awsx s3api delete-object --bucket "${B}" --key "${K}"
|
||||
if [ "${RC}" -eq 0 ]; then ok "${B}: DELETE ${K} accepted"
|
||||
else diagnose "DELETE ${B}/${K}"; fi
|
||||
|
||||
awsx s3api head-object --bucket "${B}" --key "${K}"
|
||||
if [ "${RC}" -ne 0 ]; then ok "${B}: scratch object is GONE — the probe left nothing behind"
|
||||
else bad "${B}: scratch object ${K} STILL EXISTS after DELETE. Remove it by hand; the probe leaked."; fi
|
||||
|
||||
# ---- (4) the R2 checksum trap, probed ON PURPOSE. Informational by design: the authored BSL sets
|
||||
# checksumAlgorithm: "", so a rejection here does NOT mean backups are broken — it means
|
||||
# that line is LOAD-BEARING. Reported either way so the operator learns which world this
|
||||
# endpoint lives in before, not after, enabling velero.
|
||||
if [ "${TRAP_DONE}" -eq 0 ]; then
|
||||
TRAP_DONE=1
|
||||
TK="${K}.checksum-trap-probe"
|
||||
OUT="$(AWS_REQUEST_CHECKSUM_CALCULATION=when_supported \
|
||||
aws --endpoint-url "${S3EP}" s3api put-object --bucket "${B}" --key "${TK}" --body "${PAY}" 2>&1)"; RC=$?
|
||||
if [ "${RC}" -eq 0 ]; then
|
||||
note "checksum probe: this endpoint ACCEPTS a checksum-bearing PUT — the R2 checksum trap is NOT
|
||||
present here (expected for MinIO). checksumAlgorithm: \"\" stays harmless."
|
||||
aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${TK}" >/dev/null 2>&1 \
|
||||
|| note "checksum probe: could NOT delete ${TK} — remove it by hand"
|
||||
else
|
||||
case "${OUT}" in
|
||||
*InvalidArgument*)
|
||||
note "checksum probe: R2 CHECKSUM TRAP CONFIRMED at this endpoint — a PUT carrying
|
||||
x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 is rejected with 400 InvalidArgument.
|
||||
checksumAlgorithm: \"\" in platform/velero/helmrelease.yaml is LOAD-BEARING: delete it, or bump
|
||||
velero-plugin-for-aws without re-checking it, and EVERY backup breaks." ;;
|
||||
*) note "checksum probe: inconclusive (rc=${RC}) — neither confirms nor rules out the trap." ;;
|
||||
esac
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
verdict
|
||||
PROBE
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- run one probe pod
|
||||
# <ns> <file|env> <secret> <space-separated bucket:prefix targets> <endpoint> <region>
|
||||
s3_probe() {
|
||||
local ns="$1" mode="$2" sec="$3" targets="$4" ep="$5" region="$6"
|
||||
# $$ alone is NOT unique here: inside the dev container this script is frequently PID 1, so two runs
|
||||
# would reuse the same pod name AND the same scratch object key. Mix in the epoch second.
|
||||
local pod="s3check-$$-${S3CHECK_RUN_ID}-${mode}"
|
||||
|
||||
# What this environment MUST produce: 1 credential assertion + 6 per bucket (list, put, get,
|
||||
# byte-compare, delete, delete-verified). Computed HERE, by the caller, from the target list — the
|
||||
# pod's own claim is never the only source of the expected count.
|
||||
local t n=0
|
||||
for t in ${targets}; do n=$((n+1)); done
|
||||
local min=$((1 + 6 * n))
|
||||
|
||||
local volsjson="" mountsjson="" creds=""
|
||||
case "${mode}" in
|
||||
file) # exactly how velero consumes it: the `cloud` key as an AWS INI credentials FILE
|
||||
volsjson='"volumes":[{"name":"s3creds","secret":{"secretName":"'"${sec}"'","items":[{"key":"cloud","path":"credentials"}],"defaultMode":256}}],'
|
||||
mountsjson='"volumeMounts":[{"name":"s3creds","mountPath":"/s3creds","readOnly":true}],'
|
||||
creds='{"name":"AWS_SHARED_CREDENTIALS_FILE","value":"/s3creds/credentials"},{"name":"CRED_MODE","value":"file"}' ;;
|
||||
env) # exactly how CNPG/barman consumes it: discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY keys
|
||||
creds='{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_KEY_ID"}}},{"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_SECRET_KEY"}}},{"name":"CRED_MODE","value":"env"}' ;;
|
||||
*) die "internal: unknown probe mode '${mode}'" ;;
|
||||
esac
|
||||
|
||||
local b64; b64="$(s3_probe_script | base64 | tr -d '\n')"
|
||||
[ -n "${b64}" ] || die "internal: probe script encoded to nothing"
|
||||
|
||||
# egress-class label: harmless here (the velero namespace is not an app namespace) but required by
|
||||
# the require-egress-class-label ClusterPolicy in any namespace labelled app-namespace=true, which
|
||||
# the CNPG credential namespaces are. Without it the probe pod would be REJECTED by admission and
|
||||
# the check would report "unknown" instead of a result.
|
||||
local overrides
|
||||
overrides='{"apiVersion":"v1","metadata":{"labels":{"app.kubernetes.io/name":"s3check","homelab.keithjackson.dev/egress-class":"internet"}},"spec":{"restartPolicy":"Never","automountServiceAccountToken":false,'"${volsjson}"'"containers":[{"name":"s3check","image":"'"${S3CHECK_IMAGE}"'","imagePullPolicy":"IfNotPresent","command":["/bin/bash","-c","printf %s \"$PROBE_B64\" | base64 -d > /tmp/probe.sh && exec bash /tmp/probe.sh"],'"${mountsjson}"'"env":['"${creds}"',{"name":"PROBE_B64","value":"'"${b64}"'"},{"name":"S3EP","value":"'"${ep}"'"},{"name":"S3REGION","value":"'"${region}"'"},{"name":"S3TARGETS","value":"'"${targets}"'"},{"name":"S3KEY","value":"'"${pod}"'.probe"},{"name":"AWS_REQUEST_CHECKSUM_CALCULATION","value":"when_required"},{"name":"AWS_RESPONSE_CHECKSUM_VALIDATION","value":"when_required"}],"resources":{"requests":{"cpu":"50m","memory":"64Mi"},"limits":{"memory":"256Mi"}}}]}}'
|
||||
|
||||
say "probe: ns=${ns} secret=${sec} (${mode}) targets='${targets}' → ${min} assertions required"
|
||||
kubectl -n "${ns}" delete pod "${pod}" --ignore-not-found >/dev/null 2>&1 || true
|
||||
S3CHECK_PODS+=("${ns}/${pod}")
|
||||
kubectl -n "${ns}" run "${pod}" --restart=Never --image="${S3CHECK_IMAGE}" \
|
||||
--overrides="${overrides}" >/dev/null 2>&1 \
|
||||
|| die "could NOT create the probe pod ${ns}/${pod} (admission policy? RBAC? quota?).
|
||||
That is a FAILURE, not an unknown: nothing about ${ep} has been proven."
|
||||
|
||||
local i=0 phase=""
|
||||
while [ "${i}" -lt 300 ]; do
|
||||
phase="$(kubectl -n "${ns}" get pod "${pod}" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
case "${phase}" in Succeeded|Failed) break ;; esac
|
||||
sleep 5; i=$((i+5))
|
||||
done
|
||||
case "${phase}" in
|
||||
Succeeded|Failed) ;;
|
||||
*) local reason
|
||||
reason="$(kubectl -n "${ns}" get pod "${pod}" \
|
||||
-o jsonpath='{.status.containerStatuses[0].state.waiting.reason}' 2>/dev/null || true)"
|
||||
die "probe pod ${ns}/${pod} did not finish in ${i}s (phase=${phase:-<none>}, waiting=${reason:-none}).
|
||||
NOT a pass. If waiting=ErrImagePull/ImagePullBackOff this cluster cannot pull ${S3CHECK_IMAGE}." ;;
|
||||
esac
|
||||
|
||||
local log; log="$(kubectl -n "${ns}" logs "${pod}" 2>/dev/null || true)"
|
||||
[ -n "${log}" ] || die "probe pod ${ns}/${pod} produced NO output (phase=${phase}). Failing closed —
|
||||
an empty log is the shape of a probe that never ran, not of a healthy backup target."
|
||||
printf '%s\n' "${log}" | while IFS= read -r l; do printf ' %s\n' "${l}"; done
|
||||
|
||||
# ---- the four independent accounting checks that make a silent no-op impossible ----
|
||||
local okcount failcount resline nres verd a f
|
||||
okcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-OK\[' || true)"
|
||||
failcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-FAIL' || true)"
|
||||
nres="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-RESULT ' || true)"
|
||||
[ "${nres}" = "1" ] || die "expected exactly ONE S3CHECK-RESULT line from ${ns}/${pod}, found ${nres}.
|
||||
A missing verdict line means the probe died mid-way; it is never a pass."
|
||||
resline="$(printf '%s\n' "${log}" | grep '^S3CHECK-RESULT ')"
|
||||
read -r _ verd a f <<<"${resline}"
|
||||
a="${a#assertions=}"; f="${f#failures=}"
|
||||
case "${a}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
|
||||
case "${f}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
|
||||
[ "${verd}" = "PASS" ] || die "S3 ROUND TRIP FAILED against ${ep} (${ns}/${pod}: ${f} failure(s)).
|
||||
Read the S3CHECK-FAIL lines above — each one names the cause and the fix. Do NOT enable velero
|
||||
or CNPG backups against this endpoint until they are all gone."
|
||||
[ "${f}" = "0" ] || die "${ns}/${pod} claimed PASS while reporting ${f} failures — refusing to believe it."
|
||||
[ "${failcount}" = "0" ] || die "${ns}/${pod} claimed PASS but the log carries ${failcount} S3CHECK-FAIL line(s)."
|
||||
[ "${a}" -ge "${min}" ] || die "${ns}/${pod} claimed PASS with only ${a} assertions; ${min} are required
|
||||
(1 credential + 6 per bucket). A short count means steps were SKIPPED — exactly the
|
||||
degrade-to-a-successful-no-op this check exists to prevent."
|
||||
[ "${okcount}" = "${a}" ] || die "assertion accounting mismatch for ${ns}/${pod}: ${okcount} S3CHECK-OK
|
||||
lines in the log vs a claimed ${a}. Refusing to pass on a self-report that does not add up."
|
||||
[ "${phase}" = "Succeeded" ] || die "${ns}/${pod} printed PASS but the pod ended ${phase}."
|
||||
info "PROVEN: ${a} assertions, 0 failures, pod ${phase}, no object left behind."
|
||||
}
|
||||
|
||||
# ============================================================================ steps
|
||||
step_preflight() {
|
||||
say "PREFLIGHT (read-only)"
|
||||
info "nodes Ready: $(kubectl get nodes --no-headers | awk '$2=="Ready"' | wc -l)/$(kubectl get nodes --no-headers | wc -l)"
|
||||
info "flux controllers Running: $(kubectl -n flux-system get pods --no-headers | awk '$3=="Running"' | wc -l)"
|
||||
# Count BOTH groups the Kyverno chart ships. Grepping only 'kyverno.io' undercounts by 2 — the
|
||||
# policy-report types are in the wgpolicyk8s.io group — which reads as "CRDs are missing" when the
|
||||
# pre-apply in fact succeeded. Expect 22 for kyverno 3.9.x.
|
||||
info "kyverno CRDs present: $(kubectl get crd --no-headers 2>/dev/null | grep -cE 'kyverno\.io|wgpolicyk8s\.io' || true) (expect 22 for kyverno 3.9.x)"
|
||||
|
||||
local base owner filter s3 acme
|
||||
base="$(cvar BASE_DOMAIN)"; owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
|
||||
filter="$(cvar EXTERNALDNS_DOMAIN_FILTER)"; s3="$(cvar S3_BACKUP_ENDPOINT)"; acme="$(cvar ACME_CLUSTER_ISSUER)"
|
||||
say "cluster-vars guards"
|
||||
info "BASE_DOMAIN = ${base}"
|
||||
info "EXTERNALDNS_TXT_OWNER_ID = ${owner}"
|
||||
info "EXTERNALDNS_DOMAIN_FILTER = ${filter}"
|
||||
info "ACME_CLUSTER_ISSUER = ${acme}"
|
||||
info "S3_BACKUP_ENDPOINT = ${s3}"
|
||||
|
||||
# THE GUARD THAT MATTERS. Two clusters sharing one Cloudflare zone are isolated by txtOwnerId and
|
||||
# by serving disjoint hostnames — NOT by domainFilters (finding F-DNS1: a subdomain filter drops the
|
||||
# parent zone and publishes nothing). Both clusters inheriting the chart default `default` is the
|
||||
# single root cause of the 2026-09-04 incident, in which prod deleted every one of dev's records.
|
||||
# preflight is a DIAGNOSTIC: it must always finish and report. The only thing it refuses over is the
|
||||
# target guard, because that one means you are pointed at the wrong cluster. Everything else is
|
||||
# reported and warned about here, and REFUSED on the mutating path (step_externaldns) where acting
|
||||
# on a bad value is what actually causes harm. Dying here instead hid the rest of dev's health
|
||||
# report behind a condition that was, for dev, simply the long-standing status quo.
|
||||
if [ -z "${owner}" ]; then
|
||||
warn "EXTERNALDNS_TXT_OWNER_ID is EMPTY — external-dns falls back to the chart default 'default'.
|
||||
Safe only if this is the sole cluster publishing into the zone. 'externaldns' will refuse."
|
||||
elif [ "${owner}" = "unset-see-cluster-vars" ]; then
|
||||
warn "EXTERNALDNS_TXT_OWNER_ID is still the base sentinel — the overlay did not apply.
|
||||
'externaldns' will refuse until a real per-environment value is set."
|
||||
elif [ "${owner}" = "default" ]; then
|
||||
# NOT fatal here, because `default` is DEV's legitimate, established owner id — its live records
|
||||
# are registered under it and cannot be renamed without orphaning them. Dying on it would make
|
||||
# this script refuse to run against dev at all, which would be an environment fork (R-ENV-02) in
|
||||
# a script whose whole claim is that it has none. preflight only READS, so it warns; the
|
||||
# MUTATING path (step_externaldns) still refuses outright.
|
||||
warn "EXTERNALDNS_TXT_OWNER_ID is 'default' — the Helm chart's default value.
|
||||
That is correct for DEV (its records are registered under it) and WRONG for any second cluster
|
||||
sharing this zone: two clusters both defaulting to 'default' is the exact root cause of the
|
||||
2026-09-04 incident, in which prod treated dev's TXT registry entries as its own and
|
||||
policy: sync deleted the records behind them."
|
||||
else
|
||||
info "owner id is distinct and non-default: OK"
|
||||
fi
|
||||
|
||||
if s3_is_placeholder "${s3}"; then
|
||||
warn "S3_BACKUP_ENDPOINT is unset/placeholder ('${s3}'). velero AND apps-wave-0 consume it.
|
||||
velero will be SKIPPED by the resume step. apps-wave-0 will still apply (the token IS defined,
|
||||
so strict substitution succeeds) but anything writing to that endpoint fails at RUNTIME, not at
|
||||
apply time — so a green apps-wave-0 does NOT mean backups work. Operator decision required."
|
||||
else
|
||||
info "S3 backup endpoint set: OK"
|
||||
fi
|
||||
|
||||
# COLD-START CRD ORDERING. Flux server-side dry-runs EVERY object in a Kustomization before applying
|
||||
# any of them, so a single unknown `kind` aborts the whole tier. platform-config applies the CDI CR
|
||||
# *and* CDI StorageProfiles in the same pass — but the StorageProfile CRD does not exist until the
|
||||
# cdi-operator has reconciled that CR. On an established cluster the CRDs are long since there and
|
||||
# this is invisible; on a COLD one it is a deadlock, and it is the same class of bug as Kyverno's
|
||||
# 5.7 MiB CRD bundle. Report it here so `resume` does not burn 15 minutes discovering it.
|
||||
say "CRDs that later tiers apply resources against"
|
||||
local crd missing=0
|
||||
for crd in storageprofiles.cdi.kubevirt.io cdis.cdi.kubevirt.io kubevirts.kubevirt.io \
|
||||
volumesnapshotclasses.snapshot.storage.k8s.io; do
|
||||
if kubectl get crd "${crd}" >/dev/null 2>&1; then
|
||||
printf ' %-52s present\n' "${crd}"
|
||||
else
|
||||
printf ' \033[33m%-52s MISSING\033[0m\n' "${crd}"; missing=1
|
||||
fi
|
||||
done
|
||||
if [ "${missing}" = "1" ]; then
|
||||
warn "a MISSING crd above will fail platform-config's dry-run and block every tier below it.
|
||||
CDI's own CRDs (StorageProfile, DataVolume, ...) are created by cdi-operator only AFTER it
|
||||
reconciles the CDI CR. Check that the CDI CR is applied and cdi-operator is Available; if the
|
||||
CRD appeared only recently this can also be a stale discovery cache in kustomize-controller,
|
||||
which a forced reconcile clears."
|
||||
kubectl -n cdi get cdi,deploy 2>/dev/null | head -12 || info "(no cdi namespace yet)"
|
||||
fi
|
||||
|
||||
say "tiers"
|
||||
local n
|
||||
for n in flux-system cilium "${TIER_ORDER[@]}" velero; do
|
||||
printf ' %-16s suspend=%-6s ready=%-6s %s\n' "$n" "$(k_susp "$n")" "$(k_ready "$n")" "$(k_msg "$n" | head -c 90)"
|
||||
done
|
||||
|
||||
say "HelmReleases not Ready"
|
||||
kubectl get helmrelease -A \
|
||||
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPEND:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message \
|
||||
--no-headers 2>/dev/null | grep -v ' True ' || info "(all Ready)"
|
||||
|
||||
# SUSPENDED releases are invisible above: a suspended HelmRelease keeps whatever Ready condition it
|
||||
# last had, so a STALE `Ready=True` hides the fact that Flux is not managing it at all. external-dns
|
||||
# is deliberately in this state until `standup.sh externaldns` runs.
|
||||
say "HelmReleases SUSPENDED (Flux is not reconciling these — a stale Ready=True means nothing)"
|
||||
kubectl get helmrelease -A \
|
||||
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPEND:.spec.suspend --no-headers 2>/dev/null \
|
||||
| awk '$3=="true"' || true
|
||||
}
|
||||
|
||||
step_crds() {
|
||||
say "CRD PRE-APPLY (idempotent)"
|
||||
# Charts whose CRD bundle exceeds kubectl's 256 KiB annotation limit set crds.install=false, so
|
||||
# nothing installs them and `platform` can never go Ready. See bootstrap/flux/preapply-crds.sh.
|
||||
bash "${script_dir}/flux/preapply-crds.sh"
|
||||
|
||||
# ---------------------------------------------------------------- the CDI cold-start deadlock
|
||||
# SAME CLASS OF BUG, DIFFERENT OPERATOR. `platform-config` applies the CDI CR *and*
|
||||
# platform/kubevirt/cdi-storageprofiles.yaml in ONE pass. But StorageProfile's CRD does not exist
|
||||
# until cdi-operator has reconciled that CR — and Flux server-side dry-runs EVERY object before
|
||||
# applying ANY of them, so the unknown kind aborts the whole tier, the CR is never applied, the CRD
|
||||
# is never created, and platform-config can never converge. A true deadlock, invisible on an
|
||||
# established cluster (the CRDs are long since there) and fatal on a cold one.
|
||||
#
|
||||
# Breaking it needs exactly one thing: get the CDI CR in. Then cdi-operator creates the remaining
|
||||
# CDI CRDs and platform-config's next reconcile dry-runs clean. We apply ONLY the CR — the minimum
|
||||
# — and leave every other object to Flux. The CR carries no ${} tokens, so no substitution is
|
||||
# needed and applying it here cannot diverge from what Flux would render.
|
||||
local cdi_cr="${repo_root}/platform/kubevirt/cdi-cr.yaml"
|
||||
if kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1; then
|
||||
info "storageprofiles CRD already present — CDI is past its cold start"
|
||||
elif [ ! -f "${cdi_cr}" ]; then
|
||||
warn "expected ${cdi_cr} and it is missing — skipping the CDI unblock"
|
||||
else
|
||||
say "CDI cold-start unblock: applying the CDI CR so cdi-operator can create its CRDs"
|
||||
kubectl get ns cdi >/dev/null 2>&1 || die "namespace 'cdi' absent — the platform tier has not applied the CDI operator yet. Run 'resume' far enough to make platform Ready first."
|
||||
kubectl apply --server-side --force-conflicts -f "${cdi_cr}"
|
||||
local i=0
|
||||
while [ "$i" -lt 300 ]; do
|
||||
kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1 && break
|
||||
sleep 10; i=$((i+10))
|
||||
done
|
||||
kubectl get crd storageprofiles.cdi.kubevirt.io >/dev/null 2>&1 \
|
||||
|| die "cdi-operator did not create storageprofiles.cdi.kubevirt.io within ${i}s.
|
||||
Inspect: kubectl -n cdi get cdi,pods,deploy — platform-config stays blocked until it exists."
|
||||
kubectl wait --for=condition=Established --timeout=60s crd/storageprofiles.cdi.kubevirt.io >/dev/null
|
||||
info "storageprofiles.cdi.kubevirt.io created and Established (${i}s)"
|
||||
fi
|
||||
}
|
||||
|
||||
step_unwedge() {
|
||||
say "UNWEDGE STUCK HELM RELEASES"
|
||||
# A Helm release left in uninstalling/pending-* cannot be advanced by helm-controller: it reports
|
||||
# StateError ("unable to determine state for release with status 'uninstalling'") forever, which
|
||||
# keeps `platform` (wait: true) not-Ready and every tier below it blocked.
|
||||
#
|
||||
# How prod got here: Kyverno's install timed out because its pods were crash-looping with no CRDs;
|
||||
# install.remediation.retries:3 then fired an uninstall, and that uninstall timed out too.
|
||||
#
|
||||
# Pre-applied CRDs live OUTSIDE the Helm release and SURVIVE this — that is precisely why the
|
||||
# pre-apply is a separate step and must run before this one.
|
||||
local found=0 ns rel st
|
||||
while read -r ns rel st; do
|
||||
[ -n "${ns:-}" ] || continue
|
||||
found=1
|
||||
warn "${ns}/${rel} is '${st}'"
|
||||
[ "${CONFIRM_DESTRUCTIVE}" = "1" ] || {
|
||||
info "DRY RUN. This DELETES the release's Kubernetes objects and lets Flux install cold."
|
||||
info "Re-run with --confirm-destructive to act."
|
||||
continue
|
||||
}
|
||||
say "clearing ${ns}/${rel}"
|
||||
# --no-hooks and --wait=false: the WAIT is what hung the original remediation, so do not repeat it.
|
||||
helm uninstall "${rel}" -n "${ns}" --no-hooks --wait=false || warn "helm uninstall returned non-zero; continuing to check state"
|
||||
if helm list -n "${ns}" --all -q 2>/dev/null | grep -qx "${rel}"; then
|
||||
warn "${ns}/${rel} still present after uninstall — inspect manually, NOT deleting release secrets by hand."
|
||||
else
|
||||
info "${ns}/${rel} cleared"
|
||||
flux reconcile helmrelease "${rel}" -n "${ns}" --force --timeout=10m || warn "reconcile of ${ns}/${rel} did not complete in 10m"
|
||||
fi
|
||||
done < <(
|
||||
for ns in $(kubectl get helmrelease -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' | sort -u); do
|
||||
helm list -n "${ns}" --all -o json 2>/dev/null \
|
||||
| yq -r '.[] | select(.status | test("uninstalling|pending-install|pending-upgrade|pending-rollback")) | "'"${ns}"' " + .name + " " + .status' 2>/dev/null || true
|
||||
done
|
||||
)
|
||||
[ "${found}" = "1" ] || info "no wedged releases found."
|
||||
}
|
||||
|
||||
step_externaldns() {
|
||||
say "EXTERNAL-DNS — SAFE UPGRADE (prove the owner id BEFORE the pod can run)"
|
||||
# THE HAZARD: a Helm upgrade sets replicas back to the chart default (1) and applies the new args in
|
||||
# ONE operation. There is no window in which to check the result first. If --txt-owner-id is missing
|
||||
# the pod starts as owner `default`, claims the other cluster's TXT registry entries, and under
|
||||
# `policy: sync` DELETES the records behind them. That is the 2026-09-04 incident.
|
||||
#
|
||||
# THE FIX IS TO VERIFY IN ADVANCE, NOT AFTER: render the chart with the LIVE HelmRelease values and
|
||||
# assert the flag appears. Rendering is deterministic and touches nothing, so this converts a race
|
||||
# into a precondition.
|
||||
local ns=external-dns rel=external-dns
|
||||
local owner chart ver repo vals
|
||||
owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
|
||||
[ -n "${owner}" ] && [ "${owner}" != "default" ] && [ "${owner}" != "unset-see-cluster-vars" ] \
|
||||
|| die "refusing: EXTERNALDNS_TXT_OWNER_ID is '${owner}'"
|
||||
|
||||
chart="$(kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.chart.spec.chart}')"
|
||||
ver="$(kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.chart.spec.version}')"
|
||||
repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' platform/externaldns/helmrepository.yaml | tr -d '"')"
|
||||
[ -n "${repo}" ] || die "could not derive the external-dns chart repo url"
|
||||
|
||||
vals="${SNAP_DIR}/ed-values.yaml"
|
||||
kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.values}' | yq -P > "${vals}"
|
||||
info "chart=${chart} version=${ver} repo=${repo}"
|
||||
info "live spec.values.txtOwnerId = $(yq '.txtOwnerId' "${vals}")"
|
||||
|
||||
say "rendering the chart with the LIVE values to prove the flag"
|
||||
local rendered="${SNAP_DIR}/ed-rendered.yaml"
|
||||
helm template "${rel}" "${chart}" --repo "${repo}" --version "${ver}" -n "${ns}" -f "${vals}" > "${rendered}"
|
||||
local args; args="$(yq -r 'select(.kind == "Deployment") | .spec.template.spec.containers[0].args[]' "${rendered}")"
|
||||
printf '%s\n' "${args}" | sed 's/^/ /'
|
||||
printf '%s\n' "${args}" | grep -qx -- "--txt-owner-id=${owner}" \
|
||||
|| die "the render does NOT contain --txt-owner-id=${owner}.
|
||||
An upgrade would start external-dns as owner 'default' and delete the other cluster's records.
|
||||
REFUSING. Fix the chart values key before retrying."
|
||||
info "PROVEN: the upgrade will carry --txt-owner-id=${owner}"
|
||||
|
||||
say "snapshotting the Cloudflare zone (baseline for the post-check)"
|
||||
dns_snapshot "${SNAP_DIR}/dns-before.txt"
|
||||
|
||||
say "resuming + forcing the upgrade"
|
||||
kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.suspend}' | grep -q true \
|
||||
&& flux resume helmrelease "${rel}" -n "${ns}" || info "(HelmRelease was not suspended)"
|
||||
flux reconcile helmrelease "${rel}" -n "${ns}" --force --timeout=10m || warn "reconcile did not complete in 10m"
|
||||
|
||||
say "verifying the LIVE deployment args"
|
||||
local live; live="$(kubectl -n "${ns}" get deploy "${rel}" -o jsonpath='{.spec.template.spec.containers[0].args}' 2>/dev/null || true)"
|
||||
printf ' %s\n' "${live}"
|
||||
printf '%s' "${live}" | grep -q -- "--txt-owner-id=${owner}" \
|
||||
|| { kubectl -n "${ns}" scale deploy "${rel}" --replicas=0 || true
|
||||
die "live args LACK --txt-owner-id=${owner}. Scaled to 0. Investigate before scaling up."; }
|
||||
info "live deployment carries --txt-owner-id=${owner}"
|
||||
|
||||
say "post-check: did anything in the shared zone change?"
|
||||
sleep 45 # external-dns --interval=1m; give it a full cycle to act before judging it safe
|
||||
dns_snapshot "${SNAP_DIR}/dns-after.txt"
|
||||
dns_diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-after.txt" \
|
||||
|| warn "records changed — expected ONLY additions under this cluster's own BASE_DOMAIN ($(cvar BASE_DOMAIN)).
|
||||
Any change to a name outside it, or to a TXT entry owned by another cluster, is a REGRESSION."
|
||||
}
|
||||
|
||||
step_s3check() {
|
||||
say "S3 BACKUP TARGET — PROVE A REAL ROUND TRIP FROM INSIDE THE CLUSTER"
|
||||
# Velero and CNPG run in-cluster, so pod egress is the only reachability that counts: prod sits on
|
||||
# the SOHO network (pfSense's WAN) while the dev LAN's MinIO sits INSIDE pfSense, so host and pod
|
||||
# egress differ — and a healthy host-side reading has already misled this project once.
|
||||
#
|
||||
# WHAT THIS PROVES (and it fails closed on every one of them):
|
||||
# 1. the committed credential loads and AUTHENTICATES, in the exact shape each consumer reads it
|
||||
# (velero: an AWS INI file from the `cloud` key; CNPG: discrete ACCESS_KEY_ID/ACCESS_SECRET_KEY);
|
||||
# 2. each required bucket EXISTS and is LISTABLE under this environment's own prefix;
|
||||
# 3. PUT -> GET -> byte-compare -> DELETE of a throwaway object under that prefix — the genuine
|
||||
# write path, not a HEAD (a HEAD passes with a read-only token and with R2 rejecting writes);
|
||||
# 4. whether the R2 CHECKSUM TRAP is present at this endpoint, named explicitly either way.
|
||||
#
|
||||
# WHAT IT STILL DOES NOT PROVE: that Velero's kopia uploader and CNPG's barman-cloud themselves work
|
||||
# against this target. Both are UNPROVEN against R2 in this repo. See infra/BACKUP-R2.md.
|
||||
local ep live region
|
||||
live="$(cvar S3_BACKUP_ENDPOINT)"
|
||||
info "live S3_BACKUP_ENDPOINT = ${live:-<unset>}"
|
||||
if [ -n "${S3_ENDPOINT_OVERRIDE}" ]; then
|
||||
# Probe a CANDIDATE endpoint before adopting it. The whole point is to find out whether a proposed
|
||||
# backup target works from this cluster's pods BEFORE authoring it into cluster-vars — otherwise
|
||||
# the first evidence is silently-missing backups.
|
||||
ep="${S3_ENDPOINT_OVERRIDE}"
|
||||
info "probing CANDIDATE endpoint from --s3-endpoint: ${ep}"
|
||||
else
|
||||
ep="${live}"
|
||||
fi
|
||||
if s3_is_placeholder "${ep}"; then
|
||||
die "no usable endpoint to probe (endpoint='${ep:-<unset>}').
|
||||
cluster-vars still carries the loud placeholder. Pass --s3-endpoint https://<account-id>.r2.cloudflarestorage.com
|
||||
to test a CANDIDATE before authoring it. See infra/BACKUP-R2.md."
|
||||
fi
|
||||
s3_safe_token S3_BACKUP_ENDPOINT "${ep}"
|
||||
case "${ep}" in
|
||||
http://*|https://*) ;;
|
||||
*) die "endpoint '${ep}' carries no http:// or https:// scheme — the AWS SDK needs one." ;;
|
||||
esac
|
||||
|
||||
# Region is a knob, not a constant: MinIO ignores it, R2 requires exactly `auto`, and an EMPTY value
|
||||
# in cluster-vars silently builds a BSL with no region.
|
||||
region="$(cvar S3_BACKUP_REGION)"
|
||||
case "${ep}" in
|
||||
*.r2.cloudflarestorage.com*)
|
||||
if [ "${region}" != "auto" ]; then
|
||||
warn "endpoint is Cloudflare R2 but cluster-vars S3_BACKUP_REGION='${region:-<unset>}'. R2 requires
|
||||
exactly 'auto'. Probing with 'auto' anyway; FIX clusters/base/cluster-vars.yaml before enabling
|
||||
velero, or the BackupStorageLocation is built with the wrong region."
|
||||
fi
|
||||
region=auto ;;
|
||||
*)
|
||||
if [ -z "${region}" ]; then
|
||||
region="us-east-1"
|
||||
warn "cluster-vars S3_BACKUP_REGION is EMPTY on this cluster. Probing with 'us-east-1' (MinIO
|
||||
ignores it) — but the BSL would be rendered with region: \"\". Set it explicitly."
|
||||
fi ;;
|
||||
esac
|
||||
s3_safe_token S3_BACKUP_REGION "${region}"
|
||||
|
||||
# Buckets come from the AUTHORED manifests + cluster-vars, never from a hardcoded name here: the
|
||||
# whole point is to probe what this environment will really write to (R-ENV-02, no dev/prod fork).
|
||||
local vbucket vprefix cpath cbucket cprefix rest
|
||||
vbucket="$(yq -r '.spec.values.configuration.backupStorageLocation[0].bucket' \
|
||||
platform/velero/helmrelease.yaml 2>/dev/null | tr -d '"')"
|
||||
[ -n "${vbucket}" ] && [ "${vbucket}" != "null" ] \
|
||||
|| die "could not read the Velero bucket name from platform/velero/helmrelease.yaml"
|
||||
vprefix="$(cvar VELERO_BACKUP_PREFIX)"
|
||||
|
||||
cpath="$(cvar CNPG_BACKUP_PATH)"
|
||||
if [ -z "${cpath}" ]; then
|
||||
warn "cluster-vars CNPG_BACKUP_PATH is EMPTY on this cluster. barmanObjectStore.destinationPath would
|
||||
substitute to the empty string and EVERY WAL archive would fail at runtime. Probing the historical
|
||||
default s3://cnpg so the round trip is still proven — but SET THIS VALUE."
|
||||
cpath="s3://cnpg"
|
||||
fi
|
||||
case "${cpath}" in s3://*) ;; *) die "CNPG_BACKUP_PATH='${cpath}' is not an s3:// URL." ;; esac
|
||||
rest="${cpath#s3://}"; cbucket="${rest%%/*}"
|
||||
if [ "${rest}" = "${cbucket}" ]; then cprefix=""; else cprefix="${rest#*/}"; fi
|
||||
cprefix="${cprefix%/}"
|
||||
[ -n "${cbucket}" ] || die "CNPG_BACKUP_PATH='${cpath}' names no bucket."
|
||||
|
||||
s3_safe_token velero-bucket "${vbucket}"; s3_safe_token VELERO_BACKUP_PREFIX "${vprefix}"
|
||||
s3_safe_token cnpg-bucket "${cbucket}"; s3_safe_token CNPG_BACKUP_PATH-prefix "${cprefix}"
|
||||
|
||||
say "what is about to be proven"
|
||||
info "endpoint ${ep}"
|
||||
info "region ${region}"
|
||||
info "velero bucket '${vbucket}' prefix '${vprefix:-<bucket root>}' (authored bucket + VELERO_BACKUP_PREFIX)"
|
||||
info "cnpg bucket '${cbucket}' prefix '${cprefix:-<bucket root>}' (from CNPG_BACKUP_PATH)"
|
||||
info "scratch key <prefix>/.s3check/<pod>.probe — written, read back, byte-compared, DELETED"
|
||||
|
||||
local ran=0 keys
|
||||
# (a) Velero's own credential, from its own namespace, mounted the way the aws plugin reads it.
|
||||
if kubectl get ns velero >/dev/null 2>&1 \
|
||||
&& kubectl -n velero get secret velero-cloud-credentials >/dev/null 2>&1; then
|
||||
# List KEY NAMES only. Never pipe or print a value (a careless redaction has leaked a password
|
||||
# into a transcript on this project already). go-template, not jsonpath: kubectl's jsonpath
|
||||
# cannot range over a map's keys (`range $k,$v :=` is rejected, U+002C) and returns EMPTY, which
|
||||
# silently looked like "the key is missing".
|
||||
keys="$(kubectl -n velero get secret velero-cloud-credentials \
|
||||
-o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}' 2>/dev/null || true)"
|
||||
printf '%s\n' "${keys}" | grep -qx 'cloud' \
|
||||
|| die "Secret velero/velero-cloud-credentials carries no 'cloud' key (keys: $(printf '%s' "${keys}" | tr '\n' ' ')).
|
||||
velero-plugin-for-aws reads exactly that key as an AWS INI file; without it every backup fails
|
||||
at RUNTIME while the HelmRelease reports Ready."
|
||||
s3_probe velero file velero-cloud-credentials "${vbucket}:${vprefix}" "${ep}" "${region}"
|
||||
ran=$((ran+1))
|
||||
else
|
||||
warn "no Secret velero/velero-cloud-credentials on this cluster — Velero's own credential could NOT
|
||||
be proven. That is expected before the velero tier has ever reconciled here; it is not a pass."
|
||||
fi
|
||||
|
||||
# (b) A CNPG/barman credential, discovered rather than hardcoded (R-ENV-02: no per-environment list
|
||||
# of app names in this script). Any one of them proves the key + the cnpg bucket + the discrete
|
||||
# ACCESS_KEY_ID/ACCESS_SECRET_KEY shape; they are all kept in sync by design.
|
||||
local pick cns csec
|
||||
pick="$(kubectl get secret -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}' 2>/dev/null \
|
||||
| grep -- '-db-backup-credentials$' | head -n 1 || true)"
|
||||
if [ -n "${pick}" ]; then
|
||||
cns="${pick%%/*}"; csec="${pick#*/}"
|
||||
s3_probe "${cns}" env "${csec}" "${cbucket}:${cprefix}" "${ep}" "${region}"
|
||||
ran=$((ran+1))
|
||||
else
|
||||
warn "no *-db-backup-credentials Secret found in any namespace — the CNPG/barman credential could
|
||||
NOT be proven."
|
||||
fi
|
||||
|
||||
[ "${ran}" -gt 0 ] || die "NO backup credential Secret exists on this cluster, so NOTHING could be
|
||||
proven about ${ep}. That is a FAILURE, not a skip: a green s3check must never mean 'I found
|
||||
nothing to check'."
|
||||
|
||||
say "S3 ROUND TRIP PROVEN from inside the cluster: ${ep} (${ran} credential path(s))"
|
||||
info "STILL UNPROVEN by this step: Velero's kopia uploader and CNPG barman-cloud against this target,"
|
||||
info "and a RESTORE. A backup that has never been restored is not a backup — see infra/BACKUP-R2.md."
|
||||
}
|
||||
|
||||
step_resume() {
|
||||
say "RESUME TIERS (dependency order)"
|
||||
local n s3; s3="$(cvar S3_BACKUP_ENDPOINT)"
|
||||
for n in "${TIER_ORDER[@]}"; do
|
||||
if [ "$(k_susp "$n")" = "true" ]; then
|
||||
info "resuming $n"; flux resume kustomization "$n" --timeout=1s >/dev/null 2>&1 || true
|
||||
else
|
||||
info "$n already unsuspended"
|
||||
fi
|
||||
wait_ready "$n" 900 || { warn "stopping at $n — a partial, understood bring-up beats a forced green."; return 1; }
|
||||
done
|
||||
|
||||
# The placeholder test MUST be substring-based. It was `TBD-*` (a PREFIX match) while the placeholder
|
||||
# was `TBD-PROD-…`; the R2 move changed it to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`,
|
||||
# which does not START with TBD- — so this case arm silently stopped matching and `resume` would have
|
||||
# RESUMED velero onto a placeholder hostname on prod. That is the precise failure this guard exists
|
||||
# to prevent, reintroduced by a value change in another file. s3_is_placeholder matches anywhere.
|
||||
if s3_is_placeholder "${s3}"; then
|
||||
warn "velero left SUSPENDED: S3_BACKUP_ENDPOINT='${s3}' is not a usable endpoint.
|
||||
Resuming it would create a BackupStorageLocation that can never reach a bucket, i.e. a tier
|
||||
that reports Ready-ish while backups silently do not exist. Set the real endpoint first, prove
|
||||
it with \`s3check\`, and only then resume (infra/BACKUP-R2.md)."
|
||||
else
|
||||
info "resuming velero"; flux resume kustomization velero --timeout=1s >/dev/null 2>&1 || true
|
||||
wait_ready velero 900 || warn "velero did not converge"
|
||||
fi
|
||||
}
|
||||
|
||||
step_verify() {
|
||||
say "VERIFY"
|
||||
local base; base="$(cvar BASE_DOMAIN)"
|
||||
info "BASE_DOMAIN = ${base}"
|
||||
say "kustomizations"
|
||||
kubectl get kustomization -A \
|
||||
-o custom-columns=NAME:.metadata.name,SUSPEND:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status --no-headers
|
||||
say "certificates (must be Ready before any hostname is trusted)"
|
||||
kubectl get certificate -A --no-headers 2>/dev/null || info "(none yet)"
|
||||
say "pods not Running/Completed"
|
||||
kubectl get pods -A --no-headers | awk '$4!="Running" && $4!="Completed"' || true
|
||||
info "total pods: $(kubectl get pods -A --no-headers | wc -l)"
|
||||
say "gateway / LoadBalancer VIPs"
|
||||
kubectl get svc -A --field-selector spec.type=LoadBalancer --no-headers 2>/dev/null || true
|
||||
say "published DNS for this cluster's own domain"
|
||||
local owner; owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)"
|
||||
dns_snapshot "${SNAP_DIR}/dns-verify.txt"
|
||||
info "records under ${base}: $(grep -c "${base}" "${SNAP_DIR}/dns-verify.txt" || true)"
|
||||
|
||||
# Every TXT registry entry this cluster publishes must carry OUR owner id. One reading another
|
||||
# cluster's id means we have claimed its records and `policy: sync` will delete them next cycle.
|
||||
local foreign
|
||||
foreign="$(grep "${base}" "${SNAP_DIR}/dns-verify.txt" | grep 'heritage=external-dns' \
|
||||
| grep -v "external-dns/owner=${owner}" || true)"
|
||||
if [ -n "${foreign}" ]; then
|
||||
warn "TXT entries under ${base} NOT owned by '${owner}' — this cluster has claimed records it does not own:"
|
||||
printf '%s\n' "${foreign}" | sed 's/^/ /'
|
||||
else
|
||||
info "every TXT registry entry under ${base} reads owner=${owner}: OK"
|
||||
fi
|
||||
|
||||
# THE REGRESSION CHECK. Additions under our own BASE_DOMAIN are expected. ANY change to a name
|
||||
# outside it means we have touched the other cluster's zone — the 2026-09-04 incident.
|
||||
if [ -f "${SNAP_DIR}/dns-before.txt" ]; then
|
||||
say "regression check against the pre-standup baseline"
|
||||
local outside
|
||||
outside="$(diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-verify.txt" \
|
||||
| grep -E '^[<>]' | grep -v "${base}" || true)"
|
||||
if [ -n "${outside}" ]; then
|
||||
warn "RECORDS OUTSIDE ${base} CHANGED — investigate before doing anything else:"
|
||||
printf '%s\n' "${outside}" | sed 's/^/ /'
|
||||
else
|
||||
info "no record outside ${base} was added, changed or removed: OK"
|
||||
info "added under ${base}: $(diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-verify.txt" | grep -c '^>' || true)"
|
||||
fi
|
||||
else
|
||||
warn "no ${SNAP_DIR}/dns-before.txt baseline — cannot prove the other cluster's records are untouched.
|
||||
Take one BEFORE the next standup so this check can run."
|
||||
fi
|
||||
}
|
||||
|
||||
# ============================================================================ dispatch
|
||||
for s in "${STEPS[@]}"; do
|
||||
case "$s" in
|
||||
preflight) step_preflight ;;
|
||||
s3check) step_s3check ;;
|
||||
crds) step_crds ;;
|
||||
unwedge) step_unwedge ;;
|
||||
externaldns) step_externaldns ;;
|
||||
resume) step_resume ;;
|
||||
verify) step_verify ;;
|
||||
all) step_preflight; step_crds; step_unwedge; step_externaldns; step_resume; step_verify ;;
|
||||
*) die "unknown step '$s' (preflight crds unwedge externaldns s3check resume verify all)" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
say "done: ${STEPS[*]}"
|
||||
+1
-11
@@ -10,7 +10,7 @@ policy, egress, TLS, DNS, monitoring, and update policy from it (see
|
||||
[`docs/13-service-catalog.md`](../../home-network-k3s/docs/13-service-catalog.md) (14-service table +
|
||||
SSO/egress/dependency intent).
|
||||
|
||||
## Current examples
|
||||
## Current examples (Phase 0)
|
||||
| File | docs/13 row | exposure | egress | auth | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| `pihole.yaml` | 6 | internal | internet | forward-auth | admin UI behind oauth2-proxy; config on Longhorn (DR-critical); also platform DNS |
|
||||
@@ -28,16 +28,6 @@ inherit every guarantee automatically.
|
||||
`auth.groups`; `native`/`none` are documented exceptions only.
|
||||
- Irreplaceable state (`class: longhorn` configs/DBs) must set `backup.policy` to an off-cluster
|
||||
schedule — never `reacquirable` (R-STATE-03). Only bulk media may be `reacquirable`.
|
||||
- **Shell braces must be escaped `$${...}`.** Anything rendered verbatim onto the workload — a
|
||||
`lifecycle` postStart script, an `env` value — lands under `apps/wave-<N>/`, and every
|
||||
`apps-wave-<N>` Flux Kustomization declares `postBuild.substituteFrom: cluster-vars`. Flux runs
|
||||
envsubst over the WHOLE manifest at reconcile time, so an unescaped `${VAR:-}` meant for the
|
||||
*shell* is replaced with the empty string before the container ever starts (this silently killed
|
||||
gitea's and nextcloud's OIDC registration for days). Bare `$VAR` and `$(cmd)` are left alone and
|
||||
need no escaping; a genuine environment knob is declared in `clusters/base/cluster-vars.yaml` and
|
||||
written bare `${KNOB}`. Full contract: [`../clusters/base/apps-vms.yaml`](../clusters/base/apps-vms.yaml).
|
||||
`render.py` REFUSES to emit a manifest that breaks this rule (Flux's *strict* substitution mode
|
||||
cannot: it deliberately exempts the `${V:-}` / `${V:=x}` / `${V:?}` default-operator forms).
|
||||
|
||||
## Validate before commit — single command, no cluster (R-TEST-02)
|
||||
```bash
|
||||
|
||||
+6
-24
@@ -24,9 +24,9 @@ env:
|
||||
name: gitea-bootstrap
|
||||
key: secret-key
|
||||
- name: GITEA__server__DOMAIN # correct public clone/redirect URLs for the TLS-terminated host
|
||||
value: gitea.${BASE_DOMAIN}
|
||||
value: gitea.home.keithjackson.dev
|
||||
- name: GITEA__server__ROOT_URL
|
||||
value: https://gitea.${BASE_DOMAIN}/
|
||||
value: https://gitea.home.keithjackson.dev/
|
||||
- name: GITEA__actions__ENABLED # Phase 6 CI/CD: declared (not implicit upstream default) so the
|
||||
value: "true" # in-cluster act_runner (ns gitea-actions) registers against it
|
||||
- name: GITEA__actions__DEFAULT_ACTIONS_URL # resolve `uses: <org>/<action>@ref` from github.com (cluster
|
||||
@@ -104,24 +104,12 @@ lifecycle:
|
||||
# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
|
||||
# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
|
||||
# usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
|
||||
GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
|
||||
GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
||||
gitea_oidc_fail() {
|
||||
echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
|
||||
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
|
||||
}
|
||||
# NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild.
|
||||
# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
|
||||
# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not
|
||||
# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE
|
||||
# — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on
|
||||
# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was
|
||||
# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.
|
||||
# Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design.
|
||||
# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.
|
||||
# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the
|
||||
# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.
|
||||
# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
|
||||
if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
|
||||
else
|
||||
# (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
|
||||
@@ -162,13 +150,7 @@ lifecycle:
|
||||
gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
|
||||
fi
|
||||
fi
|
||||
# A postStart hook MUST exit 0 — a non-zero exit makes the kubelet KILL the container
|
||||
# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but
|
||||
# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to
|
||||
# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
|
||||
# the moment its OIDC guard was fixed and the block actually ran for the first time.
|
||||
exit 0
|
||||
exposure: public # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
|
||||
exposure: public # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
||||
storage:
|
||||
- name: data
|
||||
@@ -178,7 +160,7 @@ storage:
|
||||
# the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
|
||||
dependencies: [] # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: gitea.${BASE_DOMAIN}
|
||||
- host: gitea.home.keithjackson.dev
|
||||
port: 3000 # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
||||
auth:
|
||||
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
|
||||
|
||||
@@ -17,7 +17,7 @@ name: jackett
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/jackett:latest
|
||||
exposure: internal # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
|
||||
exposure: internal # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
|
||||
egress: vpn # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
- name: config
|
||||
@@ -25,7 +25,7 @@ storage:
|
||||
class: longhorn # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
|
||||
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: jackett.${BASE_DOMAIN}
|
||||
- host: jackett.home.keithjackson.dev
|
||||
port: 9117 # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
|
||||
auth:
|
||||
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
|
||||
|
||||
@@ -18,7 +18,7 @@ dependencies:
|
||||
port: 5432
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: keycloak.${BASE_DOMAIN}
|
||||
- host: keycloak.home.keithjackson.dev
|
||||
port: 8080 # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
|
||||
auth:
|
||||
mode: native # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.
|
||||
|
||||
+7
-29
@@ -50,13 +50,13 @@ env:
|
||||
# the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
|
||||
# though it speaks plain HTTP behind the proxy. ---
|
||||
- name: NEXTCLOUD_TRUSTED_DOMAINS
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
- name: OVERWRITEHOST
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
- name: OVERWRITEPROTOCOL
|
||||
value: https
|
||||
- name: OVERWRITECLIURL
|
||||
value: https://nextcloud.${BASE_DOMAIN}
|
||||
value: https://nextcloud.home.keithjackson.dev
|
||||
# --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
|
||||
# `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
|
||||
# (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
|
||||
@@ -88,7 +88,7 @@ lifecycle:
|
||||
# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
|
||||
# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
|
||||
OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
|
||||
NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
|
||||
NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
||||
nextcloud_oidc_fail() {
|
||||
echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
|
||||
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
|
||||
@@ -111,19 +111,7 @@ lifecycle:
|
||||
# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.
|
||||
OCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \
|
||||
|| echo "[nextcloud-oidc] WARN: failed to set allow_local_remote_servers" >&2
|
||||
# NOTE the `$$` escape below — it is load-bearing. apps-wave-1 declares postBuild.
|
||||
# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
|
||||
# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is
|
||||
# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS
|
||||
# TRUE — so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never
|
||||
# installing/enabling user_oidc nor registering the provider. SSO kept working only because
|
||||
# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with
|
||||
# NO OIDC at all. Flux STRICT substitution cannot catch this — it EXEMPTS default-operator
|
||||
# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion
|
||||
# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why
|
||||
# `--clientsecret-env` and the bare uses below always worked and only the guard broke.
|
||||
# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
|
||||
if [ -z "$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
echo "[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)"
|
||||
exit 0
|
||||
fi
|
||||
@@ -203,17 +191,7 @@ lifecycle:
|
||||
[ "$missing" = 0 ] && break
|
||||
p=$((p + 1)); sleep 2
|
||||
done
|
||||
if [ "$p" -ge 15 ]; then
|
||||
nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
|
||||
fi
|
||||
# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container
|
||||
# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:
|
||||
# the final statement used to be `[ "$p" -ge 15 ] && nextcloud_oidc_fail ...`, so on the
|
||||
# SUCCESS path the test returned 1 and became the script's exit status. That was latent for
|
||||
# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard
|
||||
# exposed it immediately. Every failure path here logs loudly and writes a marker file and
|
||||
# deliberately does NOT crash the pod — the break-glass local admin must stay usable.
|
||||
exit 0
|
||||
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
|
||||
exposure: public # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks
|
||||
storage:
|
||||
@@ -228,7 +206,7 @@ dependencies:
|
||||
port: 5432 # Renders the egress half; the DB namespace mirrors keycloak-db (no
|
||||
protocol: tcp # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
|
||||
ingressRoutes:
|
||||
- host: nextcloud.${BASE_DOMAIN}
|
||||
- host: nextcloud.home.keithjackson.dev
|
||||
port: 80 # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
|
||||
auth:
|
||||
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
|
||||
|
||||
+1
-1
@@ -14,7 +14,7 @@ storage:
|
||||
class: longhorn # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
|
||||
dependencies: [] # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: pihole.${BASE_DOMAIN}
|
||||
- host: pihole.home.keithjackson.dev
|
||||
port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09)
|
||||
auth:
|
||||
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
|
||||
|
||||
+1
-1
@@ -72,7 +72,7 @@ storage:
|
||||
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
|
||||
dependencies: [] # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: plex.${BASE_DOMAIN}
|
||||
- host: plex.home.keithjackson.dev
|
||||
port: 32400 # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
|
||||
auth:
|
||||
mode: native # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
|
||||
|
||||
@@ -32,7 +32,7 @@ name: qbittorrent
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/qbittorrent:5.1.2 # pinned stable tag (R-UPD-01); update.range governs advancement
|
||||
exposure: internal # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
|
||||
exposure: internal # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
|
||||
egress: vpn # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
- name: config
|
||||
@@ -40,7 +40,7 @@ storage:
|
||||
class: longhorn # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
|
||||
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: qbittorrent.${BASE_DOMAIN}
|
||||
- host: qbittorrent.home.keithjackson.dev
|
||||
port: 8080 # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
|
||||
auth:
|
||||
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
|
||||
|
||||
+2
-2
@@ -9,7 +9,7 @@ name: radarr
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/radarr:5.21.1 # pinned stable v5 tag (R-UPD-01); update.range governs advancement
|
||||
exposure: internal # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
|
||||
exposure: internal # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
|
||||
egress: vpn # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
|
||||
# kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
@@ -29,7 +29,7 @@ dependencies:
|
||||
port: 9117
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: radarr.${BASE_DOMAIN}
|
||||
- host: radarr.home.keithjackson.dev
|
||||
port: 7878 # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
||||
auth:
|
||||
mode: forward-auth # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
|
||||
|
||||
+20
-478
@@ -29,7 +29,6 @@ conformance harness stays stdlib-only and only shells out to this script + diffs
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
@@ -44,15 +43,7 @@ except ImportError: # pragma: no cover - explicit, actionable failure
|
||||
|
||||
# --- Label / naming conventions (match platform/kyverno/policies + platform/*/) --------
|
||||
LBL = "homelab.keithjackson.dev"
|
||||
# The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
|
||||
# Every rendered manifest lands under a Flux path whose Kustomization declares
|
||||
# postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
|
||||
# reconcile time: dev renders home.keithjackson.dev, prod renders prod.home.keithjackson.dev.
|
||||
# It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
|
||||
# the escape contract below reserves `$${...}` for expansions meant for a container's shell.
|
||||
# Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
|
||||
# and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
|
||||
ZONE = "${BASE_DOMAIN}"
|
||||
ZONE = "home.keithjackson.dev"
|
||||
GATEWAY_NAME = "shared"
|
||||
GATEWAY_NS = "gateway"
|
||||
KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery label
|
||||
@@ -61,20 +52,6 @@ KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery
|
||||
EGRESS_CLASSES = ("internet", "vpn")
|
||||
AUTH_MODES = ("oidc", "forward-auth", "native", "none")
|
||||
|
||||
# --- forward-auth (Option B) per-app oauth2-proxy sidecar ------------------------------
|
||||
# A `forward-auth` app has a bare web UI with no usable native OIDC (PiHole, *arr, Jackett,
|
||||
# qBittorrent). Cilium 1.16.x cannot inject ext_authz into the shared Gateway's Envoy config
|
||||
# (docs/plans/cilium-native-externalauth-evaluation.md; platform/oauth2-proxy/README.md), so
|
||||
# the platform's forward-auth CEC is an UNWIRED template. Instead each forward-auth app is
|
||||
# gated by a PER-POD oauth2-proxy REVERSE-PROXY sidecar (Option B) — EXACTLY the pattern
|
||||
# platform/crafty proves working: the shared Gateway routes <app>.home -> the sidecar (:4180),
|
||||
# the sidecar authenticates the browser against Keycloak (homelab realm, the app's auth.groups)
|
||||
# and only THEN proxies to the co-located app UI at http://127.0.0.1:<uiport>. The gate is IN
|
||||
# the data path (fail-closed: no valid session in an authorized group => no upstream; R-IDP-02).
|
||||
OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
|
||||
OAUTH2_PROXY_PORT = 4180
|
||||
KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
|
||||
@@ -275,13 +252,6 @@ def r_deployment(s: dict) -> dict:
|
||||
|
||||
pod_spec = {"containers": [container]}
|
||||
|
||||
# forward-auth (Option B): append the per-pod oauth2-proxy reverse-proxy sidecar AFTER the app
|
||||
# container (app stays containers[0]). The shared Gateway routes to the sidecar (:4180); it gates
|
||||
# the co-located UI against Keycloak and only then proxies to http://127.0.0.1:<uiport>. `oidc`
|
||||
# apps (native client) and `native`/`none` render byte-identical (no sidecar).
|
||||
if _needs_oauth2_sidecar(s):
|
||||
pod_spec["containers"].append(_oauth2_sidecar_container(s))
|
||||
|
||||
# podSecurityContext (schema-optional): the k8s PodSecurityContext is emitted VERBATIM onto the
|
||||
# pod spec. Its primary use is `fsGroup`, which makes a mounted Longhorn PVC group-writable by a
|
||||
# non-root image (e.g. semaphore runs as UID 1001 and cannot otherwise write its config PVC).
|
||||
@@ -324,14 +294,6 @@ def r_service(s: dict):
|
||||
ports = _service_ports(s)
|
||||
if not ports:
|
||||
return None
|
||||
# forward-auth (Option B): the shared Gateway must reach the oauth2-proxy sidecar, so front it
|
||||
# with a `authproxy` :4180 Service port (the HTTPRoute backendRef targets this). The app UI/API
|
||||
# port(s) stay in the Service for EAST-WEST callers (e.g. arr -> qbittorrent:8080 / jackett:9117);
|
||||
# north-south to those app ports is denied at the Gateway (see r_gateway_ingress_policy).
|
||||
if _needs_oauth2_sidecar(s):
|
||||
ports = ports + [
|
||||
{"name": "authproxy", "port": OAUTH2_PROXY_PORT, "targetPort": OAUTH2_PROXY_PORT}
|
||||
]
|
||||
return {
|
||||
"apiVersion": "v1",
|
||||
"kind": "Service",
|
||||
@@ -502,7 +464,7 @@ def r_gateway_ingress_policy(s: dict):
|
||||
north-south). This CiliumNetworkPolicy is the ADDITIVE, least-privilege allow that opens
|
||||
ONLY the north-south path (docs/09 "Ingress opens north-south only — it does not relax
|
||||
east-west default-deny"): ingress from Cilium's reserved `ingress` identity (the Gateway/
|
||||
Ingress Envoy proxy, `reserved:ingress`, numeric identity 8 in Cilium 1.20.1) to ONLY the
|
||||
Ingress Envoy proxy, `reserved:ingress`, numeric identity 8 in Cilium 1.16.5) to ONLY the
|
||||
exposed service port(s). Everything else stays denied; undeclared ports stay denied.
|
||||
|
||||
Verified empirically against the cluster: with default-deny in place, Hubble shows the
|
||||
@@ -512,15 +474,7 @@ def r_gateway_ingress_policy(s: dict):
|
||||
"""
|
||||
if not s["ingressRoutes"]:
|
||||
return None
|
||||
# forward-auth (Option B): the ONLY north-south path is the oauth2-proxy sidecar. Open the
|
||||
# Gateway to :4180 ONLY — never the app UI port — so the un-gated UI is unreachable from the
|
||||
# Gateway (fail-closed; R-IDP-02). Every other app opens its exposed service port(s) as before.
|
||||
if _needs_oauth2_sidecar(s):
|
||||
allow_ports = [{"port": str(OAUTH2_PROXY_PORT), "protocol": "TCP"}]
|
||||
else:
|
||||
allow_ports = [
|
||||
{"port": str(p["port"]), "protocol": "TCP"} for p in _service_ports(s)
|
||||
]
|
||||
ports = _service_ports(s)
|
||||
return {
|
||||
"apiVersion": "cilium.io/v2",
|
||||
"kind": "CiliumNetworkPolicy",
|
||||
@@ -536,7 +490,14 @@ def r_gateway_ingress_policy(s: dict):
|
||||
"ingress": [
|
||||
{
|
||||
"fromEntities": ["ingress"],
|
||||
"toPorts": [{"ports": allow_ports}],
|
||||
"toPorts": [
|
||||
{
|
||||
"ports": [
|
||||
{"port": str(p["port"]), "protocol": "TCP"}
|
||||
for p in ports
|
||||
]
|
||||
}
|
||||
],
|
||||
}
|
||||
],
|
||||
},
|
||||
@@ -566,28 +527,6 @@ def r_internet_egress(s: dict) -> dict:
|
||||
switch (a cluster-wide singleton keyed on the egress-class pod label) decide the PATH; this
|
||||
policy decides that north-south egress is ALLOWED at all.
|
||||
"""
|
||||
egress = [{"toEntities": ["world"]}]
|
||||
# SSO apps (oidc/forward-auth) reach Keycloak at keycloak.home, which coredns split-horizon
|
||||
# resolves to the shared Gateway VIP (an L2-announced LoadBalancer IP Cilium classifies as
|
||||
# `world`), so the pod hairpins THROUGH the Gateway. Under Cilium >=1.16.8/1.17.2 (the
|
||||
# CVE-2025-30162 fix, present in 1.20), the Gateway's L7 proxy re-evaluates THIS pod's egress
|
||||
# policy against Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) — `world`
|
||||
# matches the VIP at L3/L4 but NOT the resolved keycloak identity at L7, so OIDC discovery/
|
||||
# token 403s ("Access denied") without an explicit allow to the keycloak backend. Scope it to
|
||||
# the apps that actually talk to Keycloak (auth mode oidc or forward-auth).
|
||||
if s.get("auth", {}).get("mode") in ("oidc", "forward-auth"):
|
||||
egress.append(
|
||||
{
|
||||
"toEndpoints": [
|
||||
{
|
||||
"matchLabels": {
|
||||
"io.kubernetes.pod.namespace": "keycloak",
|
||||
"app": "keycloak",
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
return {
|
||||
"apiVersion": "cilium.io/v2",
|
||||
"kind": "CiliumNetworkPolicy",
|
||||
@@ -598,7 +537,7 @@ def r_internet_egress(s: dict) -> dict:
|
||||
},
|
||||
"spec": {
|
||||
"endpointSelector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
|
||||
"egress": egress,
|
||||
"egress": [{"toEntities": ["world"]}],
|
||||
},
|
||||
}
|
||||
|
||||
@@ -611,10 +550,6 @@ def r_httproutes(s: dict) -> list:
|
||||
"""
|
||||
out = []
|
||||
for route in s["ingressRoutes"]:
|
||||
# forward-auth (Option B): the Gateway backendRef targets the oauth2-proxy sidecar (:4180),
|
||||
# NOT the app UI port — so the gate is in the data path and the bare UI is never served
|
||||
# un-authenticated. Other apps route straight to their exposed app port.
|
||||
backend_port = OAUTH2_PROXY_PORT if _needs_oauth2_sidecar(s) else route["port"]
|
||||
out.append(
|
||||
{
|
||||
"apiVersion": "gateway.networking.k8s.io/v1",
|
||||
@@ -635,7 +570,7 @@ def r_httproutes(s: dict) -> list:
|
||||
{"path": {"type": "PathPrefix", "value": "/"}}
|
||||
],
|
||||
"backendRefs": [
|
||||
{"name": s["name"], "port": backend_port}
|
||||
{"name": s["name"], "port": route["port"]}
|
||||
],
|
||||
}
|
||||
],
|
||||
@@ -645,94 +580,6 @@ def r_httproutes(s: dict) -> list:
|
||||
return out
|
||||
|
||||
|
||||
def _needs_oauth2_sidecar(s: dict) -> bool:
|
||||
"""True iff this service gets a per-pod oauth2-proxy reverse-proxy sidecar (Option B).
|
||||
|
||||
A `forward-auth` app with an exposed UI has no usable native OIDC, so its Gateway route is
|
||||
gated IN the data path by a co-located oauth2-proxy sidecar (crafty's proven pattern). `oidc`
|
||||
apps authenticate themselves (native Keycloak client) and `native`/`none` are unchanged."""
|
||||
return s["auth"]["mode"] == "forward-auth" and bool(s["ingressRoutes"])
|
||||
|
||||
|
||||
def _oauth2_sidecar_container(s: dict) -> dict:
|
||||
"""The oauth2-proxy REVERSE-PROXY sidecar (Option B) that gates a forward-auth app's UI.
|
||||
|
||||
Mirrors platform/crafty/deployment.yaml's `oauth2-proxy` container EXACTLY, adapted per app:
|
||||
the upstream is the co-located app UI over plain HTTP (http://127.0.0.1:<uiport>) — the *arr /
|
||||
PiHole / Jackett / qBittorrent UIs are HTTP, so (unlike crafty's self-signed HTTPS :8443) there
|
||||
is NO --ssl-upstream-insecure-skip-verify. --allowed-group is emitted once per declared
|
||||
auth.group (R-IDP-06: a valid session is not enough — the user must be in an authorized group,
|
||||
else /oauth2 returns 403). client-id/secret + cookie-secret come by name from the SOPS-encrypted
|
||||
`<app>-oauth2-proxy` Secret (R-IDP-04). Single provider => --skip-provider-button 302s straight
|
||||
to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy."""
|
||||
route = s["ingressRoutes"][0]
|
||||
host = route["host"]
|
||||
upstream_port = route["port"]
|
||||
args = [
|
||||
f"--http-address=0.0.0.0:{OAUTH2_PROXY_PORT}",
|
||||
"--reverse-proxy=true",
|
||||
"--provider=oidc",
|
||||
f"--oidc-issuer-url={KEYCLOAK_ISSUER}",
|
||||
f"--upstream=http://127.0.0.1:{upstream_port}",
|
||||
"--email-domain=*",
|
||||
"--scope=openid email profile",
|
||||
"--oidc-groups-claim=groups",
|
||||
]
|
||||
# R-IDP-06: authorize by group. One flag per declared group (fail-closed to 403 otherwise).
|
||||
for group in s["auth"].get("groups", []):
|
||||
args.append(f"--allowed-group={group}")
|
||||
args += [
|
||||
"--skip-provider-button=true",
|
||||
f"--redirect-url=https://{host}/oauth2/callback",
|
||||
f"--cookie-domain=.{ZONE}",
|
||||
f"--whitelist-domain=.{ZONE}",
|
||||
"--cookie-secure=true",
|
||||
"--pass-access-token=true",
|
||||
"--set-xauthrequest=true",
|
||||
]
|
||||
secret = f"{s['name']}-oauth2-proxy"
|
||||
return {
|
||||
"name": "oauth2-proxy",
|
||||
"image": OAUTH2_PROXY_IMAGE,
|
||||
"imagePullPolicy": "IfNotPresent",
|
||||
"args": args,
|
||||
"env": [
|
||||
{
|
||||
"name": "OAUTH2_PROXY_CLIENT_ID",
|
||||
"valueFrom": {"secretKeyRef": {"name": secret, "key": "client-id"}},
|
||||
},
|
||||
{
|
||||
"name": "OAUTH2_PROXY_CLIENT_SECRET",
|
||||
"valueFrom": {"secretKeyRef": {"name": secret, "key": "client-secret"}},
|
||||
},
|
||||
{
|
||||
"name": "OAUTH2_PROXY_COOKIE_SECRET",
|
||||
"valueFrom": {"secretKeyRef": {"name": secret, "key": "cookie-secret"}},
|
||||
},
|
||||
],
|
||||
"ports": [{"name": "http", "containerPort": OAUTH2_PROXY_PORT, "protocol": "TCP"}],
|
||||
"resources": {
|
||||
"requests": {"cpu": "25m", "memory": "64Mi"},
|
||||
"limits": {"memory": "128Mi"},
|
||||
},
|
||||
"securityContext": {
|
||||
"allowPrivilegeEscalation": False,
|
||||
"readOnlyRootFilesystem": True,
|
||||
"capabilities": {"drop": ["ALL"]},
|
||||
},
|
||||
"livenessProbe": {
|
||||
"httpGet": {"path": "/ping", "port": "http"},
|
||||
"initialDelaySeconds": 5,
|
||||
"periodSeconds": 15,
|
||||
},
|
||||
"readinessProbe": {
|
||||
"httpGet": {"path": "/ping", "port": "http"},
|
||||
"initialDelaySeconds": 5,
|
||||
"periodSeconds": 15,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _auth_annotations(s: dict) -> dict:
|
||||
"""Auth wiring hints (renderer-contract §auth): the platform (oauth2-proxy/Keycloak)
|
||||
consumes these to protect ingressRoutes. oidc/forward-auth carry the authorized groups
|
||||
@@ -820,7 +667,7 @@ def r_image_automation(s: dict) -> list:
|
||||
upd = s["update"]
|
||||
interval = "1h0m0s"
|
||||
imagerepo = {
|
||||
"apiVersion": "image.toolkit.fluxcd.io/v1",
|
||||
"apiVersion": "image.toolkit.fluxcd.io/v1beta2",
|
||||
"kind": "ImageRepository",
|
||||
"metadata": {"name": s["name"], "namespace": s["name"]},
|
||||
"spec": {"image": image, "interval": interval},
|
||||
@@ -835,7 +682,7 @@ def r_image_automation(s: dict) -> list:
|
||||
else:
|
||||
policy_spec["policy"] = {"semver": {"range": ">=0.0.0"}}
|
||||
imagepolicy = {
|
||||
"apiVersion": "image.toolkit.fluxcd.io/v1",
|
||||
"apiVersion": "image.toolkit.fluxcd.io/v1beta2",
|
||||
"kind": "ImagePolicy",
|
||||
"metadata": {
|
||||
"name": s["name"],
|
||||
@@ -945,124 +792,6 @@ def r_virtualmachine(s: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Substitution safety (R-ENV-02 / R-GITOPS-02) — refuse to emit a manifest that
|
||||
# Flux's envsubst would silently eat before the workload ever runs.
|
||||
# =============================================================================
|
||||
# Every manifest this renderer writes lands under apps/wave-<N>/, and EVERY apps-wave-<N>
|
||||
# Flux Kustomization declares `postBuild.substituteFrom: cluster-vars` (see
|
||||
# flux_wave_kustomization_text below). Flux therefore runs envsubst over the WHOLE rendered
|
||||
# document — including any shell script carried verbatim in a `lifecycle` hook or an env
|
||||
# value — at reconcile time, long before a container sees it. The escape contract is stated
|
||||
# in clusters/base/apps-vms.yaml and proven with `flux envsubst`:
|
||||
#
|
||||
# `${VAR}` -> SUBSTITUTED (empty when unset) => a shell brace must be written `$${VAR}`,
|
||||
# which envsubst emits back as `${VAR}`.
|
||||
# `$VAR` -> left alone => bare shell vars need NO escaping.
|
||||
# `$(cmd)` -> left alone => command substitution needs NO escaping.
|
||||
# `${A[0]}` -> HARD ERROR ("missing closing brace"), failing the whole Kustomization.
|
||||
#
|
||||
# This bit us for real (fixed 2026-09-01): gitea/nextcloud guarded their whole Keycloak OIDC
|
||||
# registration with `if [ -z "${..._OIDC_CLIENT_SECRET:-}" ]`. Neither name is a cluster-var,
|
||||
# so envsubst rendered the guard as `if [ -z "" ]` — ALWAYS TRUE — and every boot skipped the
|
||||
# registration entirely. Flux's STRICT substitution mode cannot catch this class: it
|
||||
# deliberately EXEMPTS the default-operator forms (`${V:-}`, `${V:=x}`, `${V:?}`), which is
|
||||
# exactly the form a defensive shell script uses. So the renderer checks it itself.
|
||||
CLUSTER_VARS_PATH = (
|
||||
Path(__file__).resolve().parent.parent / "clusters" / "base" / "cluster-vars.yaml"
|
||||
)
|
||||
|
||||
# A run of `$` followed by `{...}`. envsubst consumes `$$` as a literal `$`, so a run of N
|
||||
# dollars leaves the brace SUBSTITUTED when N is odd (`${X}`, `$$${X}`) and LITERAL when N is
|
||||
# even (`$${X}` — the escape). Deliberately greedy about the body so malformed/indexed forms
|
||||
# (`${A[0]}`) are caught too rather than skipped.
|
||||
_BRACED_EXPANSION_RE = re.compile(r"(\$+)\{([^{}]*)\}")
|
||||
_PLAIN_IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
|
||||
|
||||
_CLUSTER_VARS_CACHE: dict = {}
|
||||
|
||||
|
||||
def cluster_vars_keys(path: Path | None = None) -> set:
|
||||
"""The declared cluster-vars keys — the ONLY names envsubst may legitimately expand.
|
||||
|
||||
Read from clusters/base/cluster-vars.yaml (the single knob surface, R-ENV-02) so the
|
||||
renderer never carries a stale hardcoded copy. Unreadable/absent => empty set, i.e.
|
||||
FAIL-CLOSED: with no known knobs, every braced expansion must be escaped for the shell.
|
||||
"""
|
||||
key = str(path or CLUSTER_VARS_PATH)
|
||||
if key not in _CLUSTER_VARS_CACHE:
|
||||
try:
|
||||
doc = yaml.safe_load(Path(key).read_text())
|
||||
data = doc.get("data") if isinstance(doc, dict) else None
|
||||
_CLUSTER_VARS_CACHE[key] = set(data) if isinstance(data, dict) else set()
|
||||
except Exception: # pragma: no cover - absent/malformed => fail closed
|
||||
_CLUSTER_VARS_CACHE[key] = set()
|
||||
return _CLUSTER_VARS_CACHE[key]
|
||||
|
||||
|
||||
def substitution_violations(text: str, known: set | None = None) -> list:
|
||||
"""Every unescaped `${...}` in `text` that is NOT a bare, declared cluster-vars key.
|
||||
|
||||
Returns `(line_number, token, reason)` triples in line order. An expansion is ALLOWED only
|
||||
when it is exactly `${KNOWN_KEY}` — a deliberate R-ENV-02 knob. Anything else is a bug:
|
||||
a shell expansion that envsubst will eat (`${V:-}`), a typo'd/undeclared knob that
|
||||
envsubst will blank, or an indexed form that hard-fails the whole Kustomization.
|
||||
"""
|
||||
keys = cluster_vars_keys() if known is None else known
|
||||
out = []
|
||||
for lineno, line in enumerate(text.splitlines(), start=1):
|
||||
for m in _BRACED_EXPANSION_RE.finditer(line):
|
||||
if len(m.group(1)) % 2 == 0:
|
||||
continue # even run of `$` => escaped; envsubst emits it back literally
|
||||
expr = m.group(2)
|
||||
if _PLAIN_IDENT_RE.match(expr):
|
||||
if expr in keys:
|
||||
continue # a declared cluster-vars knob, intentionally substituted
|
||||
reason = (
|
||||
f"'{expr}' is not a declared cluster-vars key — envsubst will replace "
|
||||
"it with the EMPTY STRING"
|
||||
)
|
||||
elif "[" in expr:
|
||||
reason = (
|
||||
"indexed expansion — envsubst hard-fails with 'missing closing brace' "
|
||||
"and the whole Kustomization stops reconciling"
|
||||
)
|
||||
else:
|
||||
reason = (
|
||||
"shell expansion (default-operator/parameter form) — envsubst eats it "
|
||||
"at reconcile time and STRICT mode exempts this form, so nothing else "
|
||||
"will catch it"
|
||||
)
|
||||
out.append((lineno, m.group(0), reason))
|
||||
return out
|
||||
|
||||
|
||||
def assert_substitution_safe(text: str, origin: str = "<render>") -> None:
|
||||
"""Refuse to emit a manifest carrying an unescaped braced expansion (see above).
|
||||
|
||||
Raises ValueError listing every offending token. The fix is almost always to escape the
|
||||
shell brace as `$${...}` (envsubst emits a literal `${...}` for the shell), or — if the
|
||||
value really is an environment knob — to declare it in clusters/base/cluster-vars.yaml
|
||||
and write it as a bare `${KNOB}`.
|
||||
"""
|
||||
bad = substitution_violations(text)
|
||||
if not bad:
|
||||
return
|
||||
lines = [
|
||||
f"{origin}: refusing to emit — unescaped `${{...}}` in a manifest that Flux's "
|
||||
"postBuild.substituteFrom (cluster-vars) will substitute at reconcile time:"
|
||||
]
|
||||
for lineno, token, reason in bad:
|
||||
lines.append(f" line {lineno}: {token} -> {reason}")
|
||||
lines.append(
|
||||
" FIX: a SHELL expansion must be escaped `$${...}` (envsubst emits a literal "
|
||||
"`${...}`); bare `$VAR` / `$(cmd)` need no escaping. A genuine environment knob "
|
||||
"must be declared in clusters/base/cluster-vars.yaml and written bare `${KNOB}`. "
|
||||
"Contract: clusters/base/apps-vms.yaml."
|
||||
)
|
||||
raise ValueError("\n".join(lines))
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Top-level render + emit
|
||||
# =============================================================================
|
||||
@@ -1153,14 +882,8 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
|
||||
return docs
|
||||
|
||||
|
||||
def emit(docs: list, origin: str = "<render>") -> str:
|
||||
"""Deterministic multi-document YAML text (golden-diffable).
|
||||
|
||||
Every emitted document is checked against the substitution contract before it is handed
|
||||
back (assert_substitution_safe): the rendered manifest lands on a Flux path that runs
|
||||
envsubst over it, so an unescaped `${...}` meant for a SHELL is silently blanked. Raises
|
||||
ValueError rather than emitting such a manifest.
|
||||
"""
|
||||
def emit(docs: list) -> str:
|
||||
"""Deterministic multi-document YAML text (golden-diffable)."""
|
||||
parts = []
|
||||
for comment, resource in docs:
|
||||
body = yaml.dump(
|
||||
@@ -1170,9 +893,7 @@ def emit(docs: list, origin: str = "<render>") -> str:
|
||||
width=4096,
|
||||
)
|
||||
parts.append(f"---\n# {comment}\n{body}".rstrip() + "\n")
|
||||
text = "".join(parts)
|
||||
assert_substitution_safe(text, origin)
|
||||
return text
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def load_sd(path: Path) -> dict:
|
||||
@@ -1211,9 +932,7 @@ def render_file(path: Path, image_automation: bool = True, catalog_dir: Path | N
|
||||
src_dir = catalog_dir if catalog_dir is not None else path.parent
|
||||
rev = build_reverse_map(_load_catalog(src_dir))
|
||||
inbound = rev.get(sd["name"], [])
|
||||
return emit(
|
||||
render(sd, image_automation=image_automation, inbound=inbound), origin=str(path)
|
||||
)
|
||||
return emit(render(sd, image_automation=image_automation, inbound=inbound))
|
||||
|
||||
|
||||
# =============================================================================
|
||||
@@ -1317,7 +1036,6 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
convention:
|
||||
- `<svc>.yaml` -> wave(svc)
|
||||
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
|
||||
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
|
||||
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
||||
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
||||
- `<svc>-db-backup-credentials.sops.yaml` -> wave(svc-db) (CNPG Barman S3 backup secret)
|
||||
@@ -1338,10 +1056,6 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
w = waves[name]
|
||||
add(w, f"{name}.yaml")
|
||||
add(w, f"{name}-bootstrap-secret.sops.yaml")
|
||||
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
|
||||
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
|
||||
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
|
||||
add(w, f"{name}-oauth2-proxy-credentials.sops.yaml")
|
||||
db = f"{name}{CO_DB_SUFFIX}"
|
||||
if any(dep["to"] == db for dep in by_name[name]["dependencies"]):
|
||||
wdb = waves[db]
|
||||
@@ -1589,15 +1303,12 @@ def _selftest() -> int:
|
||||
|
||||
# 4b) exposure (ingressRoutes) -> an ADDITIVE gateway-ingress CiliumNetworkPolicy that
|
||||
# allows ONLY the reserved `ingress` (Gateway proxy) identity to the exposed port,
|
||||
# while default-deny still denies all other ingress (R-NET-01/02). Uses an explicit
|
||||
# non-forward-auth mode so this exercises the direct app-port gateway path (a
|
||||
# forward-auth app instead routes the Gateway to the :4180 sidecar — see 6c).
|
||||
# while default-deny still denies all other ingress (R-NET-01/02).
|
||||
exp_docs = render(
|
||||
{
|
||||
"name": "z",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "z.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "none"},
|
||||
}
|
||||
)
|
||||
gw = [
|
||||
@@ -1711,91 +1422,6 @@ def _selftest() -> int:
|
||||
"an SD without podSecurityContext must not gain a pod securityContext",
|
||||
)
|
||||
|
||||
# 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
|
||||
# fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.
|
||||
fa_docs = render(
|
||||
{
|
||||
"name": "fa",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "fa.home.keithjackson.dev", "port": 9117}],
|
||||
"auth": {"mode": "forward-auth", "groups": ["media-users"]},
|
||||
}
|
||||
)
|
||||
fa_dep = next(d[1] for d in fa_docs if d[1]["kind"] == "Deployment")
|
||||
fa_containers = fa_dep["spec"]["template"]["spec"]["containers"]
|
||||
check(
|
||||
len(fa_containers) == 2
|
||||
and fa_containers[0]["name"] == "fa"
|
||||
and fa_containers[1]["name"] == "oauth2-proxy",
|
||||
"forward-auth app must render the app container FIRST + an oauth2-proxy sidecar SECOND",
|
||||
)
|
||||
side = fa_containers[1]
|
||||
check(
|
||||
"--upstream=http://127.0.0.1:9117" in side["args"],
|
||||
"sidecar upstream must be the co-located app UI over http on 127.0.0.1:<uiport>",
|
||||
)
|
||||
check(
|
||||
"--allowed-group=media-users" in side["args"],
|
||||
"sidecar must enforce the declared auth.group (R-IDP-06)",
|
||||
)
|
||||
check(
|
||||
"--redirect-url=https://fa.home.keithjackson.dev/oauth2/callback" in side["args"],
|
||||
"sidecar redirect-url must be the app host's /oauth2/callback",
|
||||
)
|
||||
check(
|
||||
not any(a.startswith("--ssl-upstream-insecure-skip-verify") for a in side["args"]),
|
||||
"http upstream must NOT skip TLS verify (that flag is only for crafty's self-signed https)",
|
||||
)
|
||||
check(
|
||||
side["env"][0]["valueFrom"]["secretKeyRef"]["name"] == "fa-oauth2-proxy",
|
||||
"sidecar creds must come by name from the per-app <app>-oauth2-proxy Secret (R-IDP-04)",
|
||||
)
|
||||
fa_svc = next(d[1] for d in fa_docs if d[1]["kind"] == "Service")
|
||||
fa_svc_ports = {p["port"] for p in fa_svc["spec"]["ports"]}
|
||||
check(
|
||||
fa_svc_ports == {9117, 4180},
|
||||
"forward-auth Service must expose the app port (east-west) AND :4180 (gateway->sidecar)",
|
||||
)
|
||||
fa_route = next(d[1] for d in fa_docs if d[1]["kind"] == "HTTPRoute")
|
||||
check(
|
||||
fa_route["spec"]["rules"][0]["backendRefs"][0]["port"] == 4180,
|
||||
"forward-auth HTTPRoute backendRef must target the sidecar :4180, not the app UI port",
|
||||
)
|
||||
fa_gw = next(
|
||||
d[1] for d in fa_docs
|
||||
if d[1]["kind"] == "CiliumNetworkPolicy"
|
||||
and d[1]["metadata"]["name"] == "fa-allow-gateway-ingress"
|
||||
)
|
||||
fa_gw_ports = {
|
||||
p["port"] for r in fa_gw["spec"]["ingress"] for tp in r["toPorts"] for p in tp["ports"]
|
||||
}
|
||||
check(
|
||||
fa_gw_ports == {"4180"},
|
||||
"fail-closed: the Gateway may reach ONLY the sidecar :4180, never the un-gated app port",
|
||||
)
|
||||
# an oidc app (native client) must NOT gain a sidecar or a :4180 path.
|
||||
oidc_docs = render(
|
||||
{
|
||||
"name": "oi",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "oi.home.keithjackson.dev", "port": 3000}],
|
||||
"auth": {"mode": "oidc", "groups": ["platform-admins"]},
|
||||
}
|
||||
)
|
||||
oi_containers = next(
|
||||
d[1] for d in oidc_docs if d[1]["kind"] == "Deployment"
|
||||
)["spec"]["template"]["spec"]["containers"]
|
||||
check(
|
||||
len(oi_containers) == 1
|
||||
and not any("oauth2-proxy" in c["name"] for c in oi_containers),
|
||||
"an oidc app must NOT gain an oauth2-proxy sidecar (native client authenticates itself)",
|
||||
)
|
||||
oi_route = next(d[1] for d in oidc_docs if d[1]["kind"] == "HTTPRoute")
|
||||
check(
|
||||
oi_route["spec"]["rules"][0]["backendRefs"][0]["port"] == 3000,
|
||||
"an oidc app's HTTPRoute must still target its own app port (unchanged)",
|
||||
)
|
||||
|
||||
# 7) wave partitioning: longest-path over the dependency DAG, CNPG-DB strictly below.
|
||||
wsds = [
|
||||
{"name": "a", "source": {"image": "x:1"}}, # leaf
|
||||
@@ -1866,90 +1492,6 @@ def _selftest() -> int:
|
||||
check(kt["kind"] == "Kustomization" and kt["resources"] == ["svc.yaml", "sonarr.yaml"],
|
||||
"wave kustomization must list exactly the wave's resources")
|
||||
|
||||
# 8) substitution safety (R-ENV-02): the renderer must REFUSE to emit a manifest whose
|
||||
# shell script carries an unescaped `${...}`. Every rendered manifest lands under a
|
||||
# Flux path with postBuild.substituteFrom cluster-vars, so envsubst rewrites it before
|
||||
# the container runs — this is the gitea/nextcloud OIDC-guard bug (`[ -z "${X:-}" ]`
|
||||
# -> `[ -z "" ]`, always true, whole registration block skipped on every boot).
|
||||
known = {"BASE_DOMAIN", "GATEWAY_VIP"}
|
||||
# allowed: a bare, DECLARED cluster-vars knob is a deliberate R-ENV-02 substitution.
|
||||
check(
|
||||
substitution_violations("host: app.${BASE_DOMAIN}", known) == [],
|
||||
"a bare declared cluster-vars key must be allowed (that is what substitution is FOR)",
|
||||
)
|
||||
# allowed: escaped shell braces, and the forms envsubst leaves alone.
|
||||
check(
|
||||
substitution_violations('if [ -z "$${X:-}" ]; then', known) == [],
|
||||
"an ESCAPED `$${...}` must be allowed (envsubst emits it back as a literal `${...}`)",
|
||||
)
|
||||
check(
|
||||
substitution_violations('t="$(mktemp)"; echo "$HOME $1"', known) == [],
|
||||
"bare `$VAR` / `$(cmd)` are left alone by envsubst and must NOT be flagged",
|
||||
)
|
||||
# `$$` is a literal `$` to envsubst, so an ODD run of dollars still leaves the brace live.
|
||||
check(
|
||||
len(substitution_violations("$$${X:-}", known)) == 1,
|
||||
"an ODD run of `$` still substitutes ($$ -> literal $, then `${X:-}` is eaten)",
|
||||
)
|
||||
# flagged: the exact regression — a default-operator form STRICT mode deliberately exempts.
|
||||
guard_bug = substitution_violations('if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then', known)
|
||||
check(
|
||||
len(guard_bug) == 1 and guard_bug[0][1] == "${GITEA_OIDC_CLIENT_SECRET:-}",
|
||||
"an unescaped `${VAR:-}` shell guard must be flagged (strict mode exempts this form)",
|
||||
)
|
||||
# flagged: an undeclared bare knob (envsubst blanks it), and an indexed form (hard error).
|
||||
check(
|
||||
len(substitution_violations("value: ${NOT_A_KNOB}", known)) == 1,
|
||||
"an undeclared bare `${VAR}` must be flagged (envsubst blanks it silently)",
|
||||
)
|
||||
idx = substitution_violations("echo ${PIPESTATUS[0]}", known)
|
||||
check(
|
||||
len(idx) == 1 and "missing closing brace" in idx[0][2],
|
||||
"an indexed `${A[0]}` must be flagged (envsubst hard-fails the whole Kustomization)",
|
||||
)
|
||||
# the checker must be wired into emit(): rendering an SD whose lifecycle hook carries the
|
||||
# bug RAISES rather than writing a manifest whose guard is dead on arrival.
|
||||
bad_sd = {
|
||||
"name": "subbad",
|
||||
"source": {"image": "a:1"},
|
||||
"lifecycle": {
|
||||
"postStart": {
|
||||
"exec": {
|
||||
"command": [
|
||||
"/bin/sh",
|
||||
"-c",
|
||||
'if [ -z "${SUBBAD_SECRET:-}" ]; then exit 0; fi',
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
}
|
||||
try:
|
||||
emit(render(bad_sd), origin="<selftest>")
|
||||
check(False, "emit must REFUSE a manifest with an unescaped `${...}` shell expansion")
|
||||
except ValueError as exc:
|
||||
check(
|
||||
"SUBBAD_SECRET" in str(exc) and "$${" in str(exc),
|
||||
"the refusal must name the offending token AND state the `$${...}` fix",
|
||||
)
|
||||
# the ESCAPED form of the same hook renders normally (the fix must be emittable).
|
||||
good_sd = dict(bad_sd, name="subgood")
|
||||
good_sd["lifecycle"] = {
|
||||
"postStart": {
|
||||
"exec": {
|
||||
"command": [
|
||||
"/bin/sh",
|
||||
"-c",
|
||||
'if [ -z "$${SUBBAD_SECRET:-}" ]; then exit 0; fi',
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
check(
|
||||
"$${SUBBAD_SECRET:-}" in emit(render(good_sd), origin="<selftest>"),
|
||||
"the escaped `$${...}` form must render (and reach the shell as `${...}`)",
|
||||
)
|
||||
|
||||
if failures:
|
||||
for f in failures:
|
||||
sys.stderr.write(f"SELFTEST FAIL: {f}\n")
|
||||
|
||||
+1
-1
@@ -57,7 +57,7 @@ dependencies:
|
||||
port: 32400
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: seerr.${BASE_DOMAIN}
|
||||
- host: seerr.home.keithjackson.dev
|
||||
port: 5055 # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
|
||||
auth:
|
||||
mode: native # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
|
||||
|
||||
@@ -58,7 +58,7 @@ env:
|
||||
- name: SEMAPHORE_ADMIN_NAME
|
||||
value: Semaphore Admin
|
||||
- name: SEMAPHORE_ADMIN_EMAIL
|
||||
value: admin@${BASE_DOMAIN}
|
||||
value: admin@home.keithjackson.dev
|
||||
# --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
|
||||
# SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
|
||||
- name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
|
||||
@@ -110,7 +110,7 @@ dependencies:
|
||||
port: 5432 # Renders the egress half; the DB namespace mirrors keycloak-db (no
|
||||
protocol: tcp # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
|
||||
ingressRoutes:
|
||||
- host: semaphore.${BASE_DOMAIN}
|
||||
- host: semaphore.home.keithjackson.dev
|
||||
port: 3000 # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
|
||||
auth:
|
||||
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
|
||||
|
||||
+2
-2
@@ -9,7 +9,7 @@ name: sonarr
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/sonarr:4.0.15 # pinned stable v4 tag (R-UPD-01); update.range governs advancement
|
||||
exposure: internal # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
|
||||
exposure: internal # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
|
||||
egress: vpn # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
|
||||
# kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
@@ -29,7 +29,7 @@ dependencies:
|
||||
port: 9117
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: sonarr.${BASE_DOMAIN}
|
||||
- host: sonarr.home.keithjackson.dev
|
||||
port: 8989 # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
||||
auth:
|
||||
mode: forward-auth # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
|
||||
|
||||
+3
-11
@@ -1,9 +1,8 @@
|
||||
# ci/Makefile — portable, single-command local equivalent of the Gitea Actions
|
||||
# merge gate (R-TEST-02: the gates are runnable TODAY, without Gitea).
|
||||
#
|
||||
# The LIVE Gitea Actions workflow (.gitea/workflows/gate.yml) invokes `make -C ci gate`
|
||||
# directly, so `make -C ci gate` locally IS the CI merge gate — byte-for-byte. Run from
|
||||
# anywhere:
|
||||
# The Gitea Actions workflow (.gitea/workflows/merge-gate.yaml) calls these exact
|
||||
# scripts, so `make -C ci gate` locally == the CI merge gate. Run from anywhere:
|
||||
# make -C ci gate
|
||||
#
|
||||
# Every stage is a thin wrapper over a ci/*.sh script; see ci/README.md.
|
||||
@@ -12,7 +11,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
|
||||
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
|
||||
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools
|
||||
|
||||
help: ## Show this help
|
||||
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf " \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
|
||||
@@ -43,13 +42,6 @@ all: gate ## Alias for `gate`
|
||||
touched: ## Print the touched-requirement subset for the current branch vs main
|
||||
@"$(CI_DIR)touched-requirements.sh"
|
||||
|
||||
test: ## Run ci/ script unit tests (no cluster; e.g. touched-requirements content scan)
|
||||
@rc=0; for t in "$(CI_DIR)"tests/*.test.sh; do \
|
||||
echo "== $$t"; bash "$$t" || rc=1; \
|
||||
done; \
|
||||
if [ $$rc -ne 0 ]; then echo "ci/tests: FAILED" >&2; else echo "ci/tests: all suites passed"; fi; \
|
||||
exit $$rc
|
||||
|
||||
versions: ## Print pinned tool versions (R-TEST-02)
|
||||
@cat "$(CI_DIR)versions.env"
|
||||
|
||||
|
||||
+5
-45
@@ -20,54 +20,17 @@ Individual stages:
|
||||
|
||||
```bash
|
||||
make -C ci lint # yaml/lint (no cluster; live now)
|
||||
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
|
||||
make -C ci l0 # ServiceDefinition schema + Kyverno (no cluster; live now)
|
||||
make -C ci l1 # L1 rendering checks (no cluster; live now)
|
||||
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
|
||||
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
|
||||
```
|
||||
|
||||
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
|
||||
(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
|
||||
per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
|
||||
spec-immutability enforcement). See `conformance/tests/README.md`.
|
||||
|
||||
## Exit-code contract
|
||||
|
||||
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
|
||||
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
||||
the gate stays honest in Phase 0 (no fake green).
|
||||
|
||||
### The touched-requirement subset can also be *undeterminable* (exit 3)
|
||||
|
||||
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
|
||||
order, **verifying each candidate exists** before using it:
|
||||
|
||||
```
|
||||
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
|
||||
```
|
||||
|
||||
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
|
||||
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
|
||||
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
|
||||
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
|
||||
|
||||
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
|
||||
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
|
||||
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
|
||||
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
|
||||
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
|
||||
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
|
||||
|
||||
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
|
||||
|
||||
| reason | headline | exit |
|
||||
|---|---|---|
|
||||
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
|
||||
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
|
||||
|
||||
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
|
||||
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
|
||||
|
||||
## Reproducibility (R-TEST-02)
|
||||
|
||||
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
|
||||
@@ -80,19 +43,16 @@ in that case — nothing goes untested — and says plainly that the *scoped* ch
|
||||
|
||||
```
|
||||
ci/
|
||||
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
|
||||
Makefile targets: lint · l0 · l1 · conformance · gate · touched · tools · versions
|
||||
versions.env PINNED tool versions (R-TEST-02) — single source of truth
|
||||
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
|
||||
yamllint.yaml lint config for the yaml/lint stage
|
||||
lint.sh Stage: yaml/lint
|
||||
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
|
||||
l0.sh Stage: ServiceDefinition schema validation + Kyverno L0 policy gate
|
||||
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
||||
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
|
||||
exits 3 LOUDLY rather than guessing when the diff base is unreliable
|
||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter)
|
||||
gate.sh run all stages + the merge decision (the single command)
|
||||
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
|
||||
throwaway git repos via the REPO_ROOT override — no network, no cluster
|
||||
README.md this file
|
||||
```
|
||||
|
||||
@@ -107,7 +67,7 @@ below are this workstream's **assumptions** — flagged for the PM to verify at
|
||||
|---|---|---|---|---|
|
||||
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
|
||||
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
|
||||
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` |
|
||||
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l1\|l2,l3> [--requirements R-A,R-B] [--trigger in-cluster-job] --junit <file> --coverage <dir>` |
|
||||
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
|
||||
|
||||
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
|
||||
|
||||
+8
-56
@@ -48,32 +48,6 @@ echo >&2
|
||||
log "==== gate decision ===="
|
||||
BLOCK=0
|
||||
|
||||
# --- R-TEST-03 input: the touched-requirement subset --------------------------
|
||||
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
|
||||
# reliable diff base; that used to degrade to an empty set, which silently disabled the
|
||||
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
|
||||
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
|
||||
# as an UNDECIDED merge decision, never as a pass.
|
||||
_reason_file="$(mktemp)"
|
||||
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
|
||||
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
|
||||
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
|
||||
TOUCHED_RC=$?
|
||||
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
|
||||
rm -f "$_reason_file"
|
||||
TOUCHED=()
|
||||
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
|
||||
UNDECIDED=0
|
||||
if [ "$TOUCHED_RC" -eq 3 ]; then
|
||||
UNDECIDED=1
|
||||
echo >&2
|
||||
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
|
||||
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
|
||||
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
|
||||
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED[*]}"
|
||||
fi
|
||||
|
||||
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
|
||||
# A skip means the spec repo wasn't available to verify — surfaced, not green.
|
||||
case "${OUTCOME[spec-immutability]}" in
|
||||
@@ -103,11 +77,9 @@ case "${OUTCOME[conformance]}" in
|
||||
skip)
|
||||
if cluster_available; then
|
||||
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
|
||||
else
|
||||
if [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
reqs=("${TOUCHED[@]}")
|
||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
||||
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
|
||||
warn " (no cluster — Phase 0). These are NOT certified green."
|
||||
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
|
||||
@@ -121,35 +93,15 @@ case "${OUTCOME[conformance]}" in
|
||||
fi ;;
|
||||
esac
|
||||
|
||||
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
|
||||
# on-base -> this checkout has no branch commits to gate (you are on main, or the
|
||||
# branch is already merged; a push-to-main CI run lands here). Legitimate,
|
||||
# so the job is not failed — but the headline says NOT APPLICABLE, never
|
||||
# PASS, so nobody reads it as "my branch is green".
|
||||
# anything -> the environment cannot support a merge decision (no base ref, broken
|
||||
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
|
||||
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
|
||||
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
||||
BLOCK=1
|
||||
fi
|
||||
|
||||
echo >&2
|
||||
if [ "$BLOCK" -ne 0 ]; then
|
||||
err "MERGE GATE: BLOCKED"
|
||||
summary ""
|
||||
summary "**Result: BLOCKED** — see failures above."
|
||||
exit 1
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
|
||||
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
|
||||
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
|
||||
warn " (or set BASE_REF) and run it again."
|
||||
summary ""
|
||||
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
|
||||
exit 0
|
||||
else
|
||||
if [ "$BLOCK" -eq 0 ]; then
|
||||
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
|
||||
summary ""
|
||||
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
|
||||
exit 0
|
||||
else
|
||||
err "MERGE GATE: BLOCKED"
|
||||
summary ""
|
||||
summary "**Result: BLOCKED** — see failures above."
|
||||
exit 1
|
||||
fi
|
||||
@@ -7,9 +7,6 @@
|
||||
# Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline
|
||||
# present, no plaintext secrets, required labels/egress-class,
|
||||
# ServiceDefinition schema conformance (owner: T-0.5, ADR-0011).
|
||||
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
|
||||
# a per-resource policy cannot express (secret hygiene across every
|
||||
# tracked file, least-privilege RBAC, spec-immutability enforcement).
|
||||
#
|
||||
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
|
||||
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
|
||||
@@ -76,67 +73,6 @@ else
|
||||
rm -rf "$l0_render"
|
||||
fi
|
||||
|
||||
# ---- Stage 3b: L0 STATIC conformance checks ---------------------------------
|
||||
# docs/14's L0 row is "manifests/ServiceDefinitions obey invariants", which is wider
|
||||
# than what Kyverno alone can express over single resources: whole-repo invariants
|
||||
# (no plaintext secrets anywhere in git, least-privilege RBAC across every tracked
|
||||
# binding, the spec-immutability gate's own behaviour) live in the conformance
|
||||
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
|
||||
# through the same single-command runner CI and a laptop both use (R-TEST-02).
|
||||
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
|
||||
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
|
||||
summary "- l0-static: **SKIPPED** (conformance runner missing)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
else
|
||||
load_touched_requirements
|
||||
if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
|
||||
ok "L0 static conformance checks passed"
|
||||
summary "- l0-static: **passed**"
|
||||
else
|
||||
err "L0 static conformance checks failed"
|
||||
summary "- l0-static: **FAILED**"
|
||||
rc=1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
|
||||
# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
|
||||
# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
|
||||
# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
|
||||
# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
|
||||
# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
|
||||
# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
|
||||
# a legitimate manifest, and that no expansion silently empties today.
|
||||
#
|
||||
# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
|
||||
# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
|
||||
# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
|
||||
# attributes no requirement, so the requirement-scoped selection would filter this check
|
||||
# out of exactly the change that introduces the bug. The invariant is whole-repo (same
|
||||
# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
|
||||
# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
|
||||
# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
|
||||
SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
|
||||
log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
|
||||
if [ ! -f "$SUBST_CHECK" ]; then
|
||||
skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
|
||||
summary "- subst-safety: **SKIPPED** (check missing)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
else
|
||||
subst_rc=0
|
||||
bash "$SUBST_CHECK" || subst_rc=$?
|
||||
case "$subst_rc" in
|
||||
0) ok "Flux postBuild substitution safety passed"
|
||||
summary "- subst-safety: **passed**" ;;
|
||||
3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
|
||||
summary "- subst-safety: **SKIPPED** (nothing to audit)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
|
||||
*) err "Flux postBuild substitution safety FAILED"
|
||||
summary "- subst-safety: **FAILED**"; rc=1 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
case "$rc" in
|
||||
0) ok "L0 stage complete" ;;
|
||||
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
|
||||
|
||||
@@ -20,16 +20,19 @@ if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# Touched-requirement subset (R-TEST-03). An empty OR undeterminable subset => run all
|
||||
# L1 as a superset sanity net, and SAY SO — see load_touched_requirements in ci/lib.sh.
|
||||
load_touched_requirements
|
||||
req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
||||
# Touched-requirement subset (R-TEST-03). Empty subset => run all L1 as a sanity net.
|
||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
||||
req_args=()
|
||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${reqs[*]}"
|
||||
req_args=(--requirements "$(IFS=,; echo "${reqs[*]}")")
|
||||
fi
|
||||
|
||||
# RECONCILED (PM 2026-08-16): conformance/run.sh's real interface is `--layer <l1|l2|l3>`
|
||||
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
|
||||
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
||||
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
||||
"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
|
||||
"$CONFORMANCE_CMD" --layer l1 "${req_args[@]}"
|
||||
|
||||
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
||||
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
||||
|
||||
@@ -10,12 +10,9 @@
|
||||
set -euo pipefail
|
||||
|
||||
# --- Paths -------------------------------------------------------------------
|
||||
# CI_DIR = this ci/ directory (always the real script location). REPO_ROOT = its
|
||||
# parent (impl repo root) by default, but OVERRIDABLE via the environment so the
|
||||
# scripts can be pointed at a throwaway git repo under test (see ci/tests/) without
|
||||
# copying them out of the tree. CI and normal runs leave it unset -> unchanged.
|
||||
# CI_DIR = this ci/ directory; REPO_ROOT = its parent (impl repo root).
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
: "${REPO_ROOT:=$(cd "${CI_DIR}/.." && pwd)}"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/versions.env"
|
||||
|
||||
@@ -85,38 +82,3 @@ cluster_available() {
|
||||
}
|
||||
|
||||
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
|
||||
|
||||
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
|
||||
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
|
||||
# (ready to splice into a conformance/run.sh invocation).
|
||||
#
|
||||
# ci/touched-requirements.sh exits 3 — LOUDLY — when it cannot establish a reliable
|
||||
# diff base, instead of the old silent degradation to an empty set. An empty set
|
||||
# means run.sh gets no `--requirements`, which means its coverage `--require` gate
|
||||
# never runs; treating "could not determine" as "nothing touched" is precisely the
|
||||
# false green this helper exists to prevent. A stage still runs its FULL layer in
|
||||
# that case (a superset, so nothing goes untested), but says plainly that the
|
||||
# SCOPED check did not happen; ci/gate.sh turns that into an UNDECIDED decision.
|
||||
load_touched_requirements() {
|
||||
local out rc=0
|
||||
out="$("${CI_DIR}/touched-requirements.sh")" || rc=$?
|
||||
TOUCHED_RC=$rc
|
||||
TOUCHED_REQS=()
|
||||
# `if`, not `[ ... ] && ...`: under `set -e` a false AND-list is the caller's last
|
||||
# command status and would abort the stage script on an EMPTY touched set.
|
||||
if [ -n "$out" ]; then mapfile -t TOUCHED_REQS <<< "$out"; fi
|
||||
REQ_ARGS=()
|
||||
if [ "$rc" -eq 3 ]; then
|
||||
warn "touched-requirement subset UNDETERMINED (reason logged above)."
|
||||
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
|
||||
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
|
||||
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED_REQS[*]}"
|
||||
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
|
||||
else
|
||||
log "empty touched-requirement set (base WAS established) — running the full layer."
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
@@ -39,7 +39,6 @@ PYTHON="${PYTHON:-python3}"
|
||||
# DESIGN-waves.md §"Catalog services not yet onboarded"). They render as a
|
||||
# by-product of `--all catalog` but must NOT have a golden; the gate ignores
|
||||
# their rendered output. Onboarding one = add its golden and drop it here.
|
||||
#
|
||||
: "${RENDER_GOLDEN_NOT_ONBOARDED:=keycloak vm-demo}"
|
||||
|
||||
log "Stage: render==golden reproducibility (canonical render vs committed apps/)"
|
||||
|
||||
@@ -1,169 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/touched-requirements-base.test.sh — regression tests for BASE-REF RESOLUTION
|
||||
# in ci/touched-requirements.sh.
|
||||
#
|
||||
# THE BUG THESE LOCK DOWN (2026-08-31, local/CI divergence):
|
||||
# `make -C ci gate` on a feature branch printed
|
||||
# "no changed files vs base; empty touched-requirement set"
|
||||
# and then "MERGE GATE: PASS" — while CI, with a correct base, computed 17 touched
|
||||
# requirements and BLOCKED on real coverage debt. Root cause: EVERY way of failing to
|
||||
# establish a diff base degraded silently to an EMPTY touched set and exit 0. An empty
|
||||
# set means conformance/run.sh is invoked without `--requirements`, which means its
|
||||
# coverage `--require` gate never runs — so the gate reported PASS having checked
|
||||
# nothing. A false green is worse than a red.
|
||||
#
|
||||
# The fix: an undeterminable base is a LOUD exit 3 (ci/README.md's SKIPPED code) with
|
||||
# EMPTY stdout and a machine-readable reason code, never exit 0. These tests assert
|
||||
# that for each way the base can be undeterminable, and assert the good paths still work.
|
||||
#
|
||||
# The tests point the real script at THROWAWAY git repos via the REPO_ROOT override
|
||||
# (ci/lib.sh honours a pre-set REPO_ROOT); the script runs from its true location so it
|
||||
# still sources the real lib.sh. No network, no cluster, and the working repo is never
|
||||
# touched.
|
||||
#
|
||||
# Run: ci/tests/touched-requirements-base.test.sh (or `make -C ci test`)
|
||||
set -uo pipefail # NOT -e: these tests deliberately run a script that exits non-zero.
|
||||
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
# newrepo <dir> — a throwaway repo with one baseline commit on `main`.
|
||||
newrepo() {
|
||||
local d="$1"
|
||||
mkdir -p "$d/notes"
|
||||
git -C "$d" init -q -b main
|
||||
git -C "$d" config user.email [email protected]
|
||||
git -C "$d" config user.name t
|
||||
git -C "$d" config commit.gpgsign false # throwaway: never sign (ignore host gpg config)
|
||||
echo baseline > "$d/notes/base.txt"
|
||||
git -C "$d" add -A && git -C "$d" commit -qm baseline
|
||||
}
|
||||
|
||||
# commit_touching <dir> <text> — a commit whose content declares an R-ID.
|
||||
commit_touching() {
|
||||
local d="$1" text="$2"
|
||||
printf 'name: %s\n' "$text" > "$d/notes/decl.yaml"
|
||||
git -C "$d" add -A && git -C "$d" commit -qm change
|
||||
}
|
||||
|
||||
# Runs the script in <dir>; sets RC, OUT (stdout), ERRTXT (stderr), REASON.
|
||||
run_in() {
|
||||
local d="$1"; shift
|
||||
local reason_file="${TMP}/reason.$$"
|
||||
: > "$reason_file"
|
||||
OUT="$(REPO_ROOT="$d" TOUCHED_REASON_FILE="$reason_file" "$@" "$SCRIPT" 2>"${TMP}/err.$$")"
|
||||
RC=$?
|
||||
ERRTXT="$(cat "${TMP}/err.$$")"
|
||||
REASON="$(cat "$reason_file")"
|
||||
}
|
||||
|
||||
# assert_refusal <label> <expected-reason>
|
||||
assert_refusal() {
|
||||
local label="$1" want="$2"
|
||||
[ "$RC" = 3 ] && ok "${label}: exits 3 (SKIPPED), not 0" || bad "${label}: rc=${RC}, expected 3"
|
||||
[ -z "$OUT" ] && ok "${label}: stdout is EMPTY (no silent requirement list)" \
|
||||
|| bad "${label}: stdout was '${OUT}'"
|
||||
[ "$REASON" = "$want" ] && ok "${label}: reason code '${want}'" \
|
||||
|| bad "${label}: reason '${REASON}', expected '${want}'"
|
||||
grep -q "NOT DETERMINABLE" <<<"$ERRTXT" && ok "${label}: announces the refusal LOUDLY on stderr" \
|
||||
|| bad "${label}: stderr lacked the loud banner"
|
||||
}
|
||||
|
||||
echo "== case 1: a branch ahead of local main (no remote) resolves and reports"
|
||||
R1="${TMP}/r1"; newrepo "$R1"
|
||||
git -C "$R1" checkout -q -b feature
|
||||
commit_touching "$R1" implements-R-ALPHA-01
|
||||
run_in "$R1"
|
||||
[ "$RC" = 0 ] && ok "ahead-of-main: exits 0" || bad "ahead-of-main: rc=${RC}"
|
||||
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "ahead-of-main: emits the touched R-ID" \
|
||||
|| bad "ahead-of-main: output was '${OUT}'"
|
||||
grep -q "diff base: main" <<<"$ERRTXT" && ok "ahead-of-main: names the base it used" \
|
||||
|| bad "ahead-of-main: base not reported: ${ERRTXT}"
|
||||
|
||||
echo "== case 2: THE FALSE GREEN — HEAD is the base branch itself"
|
||||
# This is the state that produced 'no changed files vs base' + 'MERGE GATE: PASS'
|
||||
# locally while CI blocked. It must now refuse, not report an empty set.
|
||||
R2="${TMP}/r2"; newrepo "$R2"
|
||||
git -C "$R2" checkout -q -b feature
|
||||
commit_touching "$R2" implements-R-ALPHA-01
|
||||
git -C "$R2" checkout -q main # <- sitting on the base branch
|
||||
run_in "$R2"
|
||||
assert_refusal "on-base" "on-base"
|
||||
|
||||
echo "== case 3: a branch already merged into the base (HEAD is an ancestor)"
|
||||
R3="${TMP}/r3"; newrepo "$R3"
|
||||
git -C "$R3" checkout -q -b feature
|
||||
commit_touching "$R3" implements-R-ALPHA-01
|
||||
git -C "$R3" checkout -q main
|
||||
git -C "$R3" merge -q --no-edit feature
|
||||
git -C "$R3" checkout -q feature # feature is now behind main
|
||||
run_in "$R3"
|
||||
assert_refusal "already-merged" "on-base"
|
||||
|
||||
echo "== case 4: no base ref resolves at all (no main, no origin/main)"
|
||||
R4="${TMP}/r4"; newrepo "$R4"
|
||||
git -C "$R4" checkout -q -b work
|
||||
commit_touching "$R4" implements-R-ALPHA-01
|
||||
git -C "$R4" branch -q -D main
|
||||
run_in "$R4"
|
||||
assert_refusal "no-base-ref" "no-base"
|
||||
|
||||
echo "== case 5: a single-commit repo with no main (the removed HEAD~1 last resort)"
|
||||
# HEAD~1 does not exist here. The old code fell back to it, the diff failed, and the
|
||||
# empty result was reported as 'no changed files' + exit 0 — a silent false green.
|
||||
R5="${TMP}/r5"; mkdir -p "$R5/notes"
|
||||
git -C "$R5" init -q -b work
|
||||
git -C "$R5" config user.email [email protected]; git -C "$R5" config user.name t
|
||||
git -C "$R5" config commit.gpgsign false
|
||||
echo only > "$R5/notes/only.txt"; git -C "$R5" add -A; git -C "$R5" commit -qm only
|
||||
run_in "$R5"
|
||||
assert_refusal "single-commit-no-main" "no-base"
|
||||
|
||||
echo "== case 6: BASE_REF names a ref that does not exist"
|
||||
R6="${TMP}/r6"; newrepo "$R6"
|
||||
git -C "$R6" checkout -q -b feature
|
||||
commit_touching "$R6" implements-R-ALPHA-01
|
||||
run_in "$R6" env BASE_REF=refs/heads/nope
|
||||
assert_refusal "bogus-BASE_REF" "no-base"
|
||||
|
||||
echo "== case 7: GITHUB_BASE_REF set but absent from the checkout (shallow clone)"
|
||||
R7="${TMP}/r7"; newrepo "$R7"
|
||||
git -C "$R7" checkout -q -b feature
|
||||
commit_touching "$R7" implements-R-ALPHA-01
|
||||
run_in "$R7" env GITHUB_BASE_REF=release-9
|
||||
assert_refusal "missing-GITHUB_BASE_REF" "no-base"
|
||||
|
||||
echo "== case 8: origin/main is preferred over a stale local main, and staleness warns"
|
||||
R8="${TMP}/r8"; newrepo "$R8"
|
||||
UP="${TMP}/r8-origin.git"; git init -q --bare -b main "$UP"
|
||||
git -C "$R8" remote add origin "$UP"
|
||||
git -C "$R8" push -q origin main
|
||||
# advance origin/main beyond the local main ref, then branch off it
|
||||
git -C "$R8" checkout -q -b ahead
|
||||
echo more > "$R8/notes/more.txt"; git -C "$R8" add -A; git -C "$R8" commit -qm more
|
||||
git -C "$R8" push -q origin ahead:main
|
||||
git -C "$R8" fetch -q origin
|
||||
git -C "$R8" checkout -q -b feature origin/main
|
||||
commit_touching "$R8" implements-R-ALPHA-01
|
||||
run_in "$R8"
|
||||
[ "$RC" = 0 ] && ok "origin-preferred: exits 0" || bad "origin-preferred: rc=${RC}"
|
||||
grep -q "diff base: origin/main" <<<"$ERRTXT" && ok "origin-preferred: uses origin/main, not local main" \
|
||||
|| bad "origin-preferred: base line was: ${ERRTXT}"
|
||||
grep -q "may be stale" <<<"$ERRTXT" && ok "origin-preferred: warns that local main differs" \
|
||||
|| bad "origin-preferred: no staleness warning"
|
||||
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "origin-preferred: emits the touched R-ID" \
|
||||
|| bad "origin-preferred: output was '${OUT}'"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then
|
||||
echo "touched-requirements base-resolution tests: PASS (${PASS} assertions)"; exit 0
|
||||
fi
|
||||
echo "touched-requirements base-resolution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
|
||||
exit 1
|
||||
@@ -1,98 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/touched-requirements.test.sh — regression tests for the content scan in
|
||||
# ci/touched-requirements.sh, focused on rule 3: R-IDs cited only in a whole-line
|
||||
# YAML/shell `#` comment are PROSE (a reference), not a declaration, and must NOT be
|
||||
# attributed — while R-IDs on real code/spec lines still ARE.
|
||||
#
|
||||
# The tests point the real script at a THROWAWAY git repo via the REPO_ROOT override
|
||||
# (ci/lib.sh honours a pre-set REPO_ROOT); the script itself is run from its true
|
||||
# location so it still sources the real lib.sh. No network, no cluster.
|
||||
#
|
||||
# Run: ci/tests/touched-requirements.test.sh (or `make -C ci test`)
|
||||
set -euo pipefail
|
||||
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
# assert_absent <needle> <haystack> <msg>
|
||||
assert_absent() {
|
||||
if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))";
|
||||
else ok "$3"; fi
|
||||
}
|
||||
# assert_present <needle> <haystack> <msg>
|
||||
assert_present() {
|
||||
if grep -qxF "$1" <<<"$2"; then ok "$3";
|
||||
else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
|
||||
}
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
git -C "$TMP" init -q -b main
|
||||
git -C "$TMP" config user.email [email protected]
|
||||
git -C "$TMP" config user.name t
|
||||
git -C "$TMP" config commit.gpgsign false # throwaway repo: never sign (ignore host gpg config)
|
||||
mkdir -p "$TMP/notes" # a path OUTSIDE path_requirements() so we isolate the content scan
|
||||
echo "baseline" > "$TMP/notes/base.txt"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
|
||||
|
||||
git -C "$TMP" checkout -q -b feature
|
||||
|
||||
# A YAML file: an R-ID in a whole-line comment (must be IGNORED) and an R-ID on a real
|
||||
# value line (must be DETECTED). Also an indented comment, to prove leading space is fine.
|
||||
cat > "$TMP/notes/example.yaml" <<'YAML'
|
||||
# This header explains R-DOC-01 and must NOT be attributed.
|
||||
kind: Thing
|
||||
metadata:
|
||||
name: implements-R-REAL-01
|
||||
spec:
|
||||
note: value # trailing inline comment mentioning R-INLINE-01 (conservatively still seen)
|
||||
# deeply indented comment about R-DOC-02 — also NOT attributed
|
||||
YAML
|
||||
|
||||
# A shell file: a `#` comment R-ID (IGNORED) plus a real command line R-ID (DETECTED).
|
||||
cat > "$TMP/notes/example.sh" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
# doc: this script relates to R-SHDOC-01 (comment only — ignored)
|
||||
echo "runs check for R-SHREAL-01"
|
||||
SH
|
||||
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm feature
|
||||
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
|
||||
# Comment-only citations are dropped (rule 3):
|
||||
assert_absent "R-DOC-01" "$OUT" "whole-line YAML comment R-ID is NOT attributed"
|
||||
assert_absent "R-DOC-02" "$OUT" "indented YAML comment R-ID is NOT attributed"
|
||||
assert_absent "R-SHDOC-01" "$OUT" "shell '#' comment R-ID is NOT attributed"
|
||||
# Real code/spec lines still count:
|
||||
assert_present "R-REAL-01" "$OUT" "R-ID on a real YAML value line IS attributed"
|
||||
assert_present "R-SHREAL-01" "$OUT" "R-ID on a real shell command line IS attributed"
|
||||
# Conservative: a trailing inline comment on a REAL line is still seen (only whole-line
|
||||
# comments are dropped) — documents the intentional boundary of the fix.
|
||||
assert_present "R-INLINE-01" "$OUT" "inline comment on a real line is (conservatively) still seen"
|
||||
|
||||
# Removed comment lines are also excluded, while removed real lines still count. The file
|
||||
# must EXIST at the base, so we diff the removal against the commit that added it.
|
||||
git -C "$TMP" checkout -q -b feature2 main
|
||||
cat > "$TMP/notes/todelete.yaml" <<'YAML'
|
||||
# will delete a comment citing R-DELDOC-01 and a real line citing R-DELREAL-01
|
||||
kind: Thing
|
||||
name: implements-R-DELREAL-01
|
||||
YAML
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm add
|
||||
ADD_SHA="$(git -C "$TMP" rev-parse HEAD)"
|
||||
git -C "$TMP" rm -q "$TMP/notes/todelete.yaml" && git -C "$TMP" commit -qm remove
|
||||
OUT2="$(REPO_ROOT="$TMP" BASE_REF="$ADD_SHA" "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (removal):\n%s\n---\n' "$OUT2"
|
||||
assert_absent "R-DELDOC-01" "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
|
||||
assert_present "R-DELREAL-01" "$OUT2" "REMOVED real-line R-ID IS attributed"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
|
||||
else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
+20
-154
@@ -28,99 +28,28 @@ CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
# ---------------------------------------------------------------------------------
|
||||
# BASE-REF RESOLUTION — and why it now REFUSES rather than guesses.
|
||||
#
|
||||
# BUG THIS FIXES (local/CI divergence, 2026-08-31): every failure to establish a
|
||||
# reliable diff base used to degrade SILENTLY to an EMPTY touched set, which main()
|
||||
# reported as "no changed files vs base" and exit 0. An empty set means the coverage
|
||||
# `--require` gate is never applied, so the merge gate printed "MERGE GATE: PASS"
|
||||
# having checked NOTHING — while CI, with a correct base, computed 17 touched
|
||||
# requirements and blocked. A false green is the worst outcome this project can
|
||||
# produce, so every one of those degradations is now a LOUD exit 3:
|
||||
#
|
||||
# * HEAD~1 last resort REMOVED. It silently narrowed the diff to the last commit
|
||||
# (and on a one-commit repo failed into an empty set).
|
||||
# * two-dot fallback REMOVED. It masked a failing three-dot diff as "no changes".
|
||||
# * unresolvable base exit 3 (was: HEAD~1 or empty).
|
||||
# * diff command fails exit 3 (was: empty set).
|
||||
# * HEAD not ahead of exit 3 (was: exit 0 with an empty set — this is the case
|
||||
# the base that produced the false green: running the gate while
|
||||
# sitting on `main`, or on a branch already merged).
|
||||
#
|
||||
# Exit 3 is the ci/README.md "SKIPPED / not-run" code: NOT a pass, NOT a fail. Callers
|
||||
# (ci/gate.sh) must surface it as an UNDECIDED merge decision. An honest, noisy skip
|
||||
# beats a quiet false pass.
|
||||
#
|
||||
# Order of preference: BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs, target
|
||||
# branch) > origin/main > main. Each candidate is VERIFIED to resolve to a commit
|
||||
# before it is used, so a shallow checkout that lacks origin/main is reported, not
|
||||
# papered over.
|
||||
# Base ref to diff against. In Gitea/GitHub Actions PR runs this is the target
|
||||
# branch (main). Locally, default to main; override with BASE_REF.
|
||||
: "${BASE_REF:=}"
|
||||
BASE="" # resolved base ref
|
||||
BASE_SOURCE="" # where it came from (for the log line)
|
||||
BASE_PROBLEM="" # why resolution failed, when it did
|
||||
|
||||
rev() { git -C "$REPO_ROOT" rev-parse --verify -q "$1^{commit}" 2>/dev/null; }
|
||||
|
||||
# Loud, structured refusal. Prints to stderr only; stdout stays empty so a caller that
|
||||
# captures stdout gets NO requirement list rather than a silently-empty one.
|
||||
#
|
||||
# $1 is a machine-readable REASON CODE, also written to $TOUCHED_REASON_FILE when that
|
||||
# env var names a path. ci/gate.sh uses it to tell the two cases apart:
|
||||
# on-base HEAD is the base branch (or already merged) — nothing to gate. Legitimate
|
||||
# (a push-to-main CI run lands here), so the gate reports NOT APPLICABLE
|
||||
# rather than PASS, and does not fail the job.
|
||||
# no-base no reliable base ref exists in this checkout — the ENVIRONMENT is broken,
|
||||
# diff-failed or git could not produce the diff. A merge decision is impossible, so the
|
||||
# gate BLOCKS rather than guessing.
|
||||
undecidable() {
|
||||
local code="$1"; shift
|
||||
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "$code" > "$TOUCHED_REASON_FILE"; fi
|
||||
err "TOUCHED-REQUIREMENT SET NOT DETERMINABLE (${code})"
|
||||
err " reason: $*"
|
||||
err " consequence: the R-TEST-03 coverage gate CANNOT be applied to this checkout."
|
||||
err " This is exit 3 (SKIPPED / not-run), never a pass — see ci/README.md."
|
||||
err " Fix one of:"
|
||||
err " * check out the feature branch you mean to gate (not the base branch), or"
|
||||
err " * git fetch origin main (so origin/main exists and is current), or"
|
||||
err " * BASE_REF=<ref> make -C ci gate to name the base explicitly."
|
||||
exit 3
|
||||
}
|
||||
|
||||
resolve_base() {
|
||||
local cand
|
||||
if [ -n "$BASE_REF" ]; then
|
||||
if rev "$BASE_REF" >/dev/null; then BASE="$BASE_REF"; BASE_SOURCE="BASE_REF"; return 0; fi
|
||||
BASE_PROBLEM="BASE_REF='${BASE_REF}' does not resolve to a commit in ${REPO_ROOT}"
|
||||
return 1
|
||||
fi
|
||||
if [ -n "${GITHUB_BASE_REF:-}" ]; then
|
||||
for cand in "origin/${GITHUB_BASE_REF}" "${GITHUB_BASE_REF}"; do
|
||||
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="GITHUB_BASE_REF"; return 0; fi
|
||||
done
|
||||
BASE_PROBLEM="GITHUB_BASE_REF='${GITHUB_BASE_REF}' is set, but neither 'origin/${GITHUB_BASE_REF}' nor '${GITHUB_BASE_REF}' exists here (shallow checkout? the workflow needs fetch-depth: 0)"
|
||||
return 1
|
||||
fi
|
||||
for cand in origin/main main; do
|
||||
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="default"; return 0; fi
|
||||
done
|
||||
BASE_PROBLEM="no base ref resolves (tried origin/main, main) in ${REPO_ROOT}"
|
||||
return 1
|
||||
if [ -n "$BASE_REF" ]; then echo "$BASE_REF"; return; fi
|
||||
if [ -n "${GITHUB_BASE_REF:-}" ]; then echo "origin/${GITHUB_BASE_REF}"; return; fi
|
||||
if git -C "$REPO_ROOT" rev-parse --verify -q origin/main >/dev/null; then echo "origin/main"; return; fi
|
||||
if git -C "$REPO_ROOT" rev-parse --verify -q main >/dev/null; then echo "main"; return; fi
|
||||
echo "HEAD~1" # last resort
|
||||
}
|
||||
|
||||
# Both diff helpers assume BASE is already resolved and validated by main(). They return
|
||||
# git's exit status so the CALLER can distinguish "no changes" from "the diff failed" —
|
||||
# conflating those two is exactly what produced the false green.
|
||||
changed_files() {
|
||||
local base; base="$(resolve_base)"
|
||||
# Three-dot: changes on this branch since it diverged from base.
|
||||
git -C "$REPO_ROOT" diff --name-only "${BASE}...HEAD"
|
||||
if git -C "$REPO_ROOT" diff --name-only "${base}...HEAD" 2>/dev/null; then return; fi
|
||||
git -C "$REPO_ROOT" diff --name-only "${base}" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# The ADDED/REMOVED lines of the change, for the R-ID content scan — excluding
|
||||
# documentation.
|
||||
#
|
||||
# THREE scoping rules, all learned from the gate blocking merges it should not have:
|
||||
# TWO scoping rules, both learned from the gate blocking merges it should not have:
|
||||
#
|
||||
# 1. Scan the DIFF, not whole file contents. Grepping whole files meant a change
|
||||
# inherited every requirement those files had EVER mentioned, on lines the author
|
||||
@@ -137,34 +66,14 @@ changed_files() {
|
||||
# that matter (conformance/, schemas/, catalog/, clusters/, ci/) are covered by
|
||||
# the path->requirement fallback regardless of file type.
|
||||
#
|
||||
# 3. Skip COMMENT-ONLY lines in non-Markdown files — rule 2's logic, applied to the
|
||||
# prose that lives INSIDE code files. A requirement ID cited in a YAML/shell `#`
|
||||
# comment is documentation, not a declaration, exactly like an ID in a `*.md` file.
|
||||
# Rule 2 drops whole Markdown files, but a `.gitea/workflows/*.yml` or `ci/*.sh`
|
||||
# HEADER that explains a requirement would otherwise re-attribute it. Concretely:
|
||||
# gate.yml's header cites R-TEST-06 (enforced by the spec-immutability STAGE, not by
|
||||
# any Chainsaw test, so a known-uncovered ID); before this rule, editing that comment
|
||||
# re-attributed R-TEST-06 and the coverage `--require` check false-blocked the merge —
|
||||
# the same self-referential trap rule 2 fixed for `*.md`. Applied in main()'s content
|
||||
# scan (see there): after the diff marker is stripped, a line whose first non-blank
|
||||
# character is `#` is dropped BEFORE the R-ID grep. Deliberately conservative — only
|
||||
# WHOLE-LINE comments are dropped; an R-ID on a real code/spec line is still detected
|
||||
# (authoritative declarations — test tags, policies, ServiceDefinitions — are never a
|
||||
# bare comment), so this only narrows PROSE, never a declaration.
|
||||
#
|
||||
# Removed lines count: deleting a test's requirement tag, or a policy's self-declared
|
||||
# ID, touches that requirement just as much as adding one.
|
||||
# `:(exclude)*.md` drops Markdown from the content scan (rule 2 above).
|
||||
# `:(exclude)ci/tests/` drops this dir's OWN unit-test fixtures: those files embed example
|
||||
# R-IDs as DATA (asserting the scan's behaviour), never as declarations — the same
|
||||
# "reference, not declaration" logic as rule 2. Their gate coverage is unaffected: any
|
||||
# ci/ change (ci/tests/ included) still attributes R-GITOPS-01/R-TEST-02/R-TEST-03 via the
|
||||
# path->requirement map below. NOTE: this excludes ci/tests/ only — conformance/tests/
|
||||
# carry AUTHORITATIVE `conformance.home.k3s/R-*` labels and are deliberately still scanned.
|
||||
# Three-dot to match changed_files(): changes on this branch since it diverged from base.
|
||||
# `:(exclude)*.md` drops Markdown from the content scan (rule 2 above). Three-dot to
|
||||
# match changed_files(): changes on this branch since it diverged from base.
|
||||
changed_diff() {
|
||||
local -a excl=(':(exclude)*.md' ':(exclude)ci/tests/')
|
||||
git -C "$REPO_ROOT" diff -U0 "${BASE}...HEAD" -- . "${excl[@]}"
|
||||
local base; base="$(resolve_base)"
|
||||
if git -C "$REPO_ROOT" diff -U0 "${base}...HEAD" -- . ':(exclude)*.md' 2>/dev/null; then return; fi
|
||||
git -C "$REPO_ROOT" diff -U0 "${base}" -- . ':(exclude)*.md' 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Coarse path -> requirement map. Deliberately conservative; extend as subtrees
|
||||
@@ -183,49 +92,15 @@ path_requirements() {
|
||||
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
|
||||
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
|
||||
clusters/*) echo "R-GITOPS-01 R-ENV-02" ;;
|
||||
# bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing:
|
||||
# bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be
|
||||
# enabled against a real backup endpoint. A change touching standup.sh ALONGSIDE any
|
||||
# already-mapped path scoped the L0 run to that other path's requirements and silently
|
||||
# excluded conformance/tests/l0-static/placeholder-guard-coupling — the check that exists
|
||||
# precisely to catch a guard/value mismatch. The 2026-09-05 guard bug (a prefix glob that
|
||||
# stopped matching when the value it guards changed, in a different file) is exactly the
|
||||
# class that omission would hide.
|
||||
bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;;
|
||||
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
|
||||
*) : ;;
|
||||
esac
|
||||
}
|
||||
|
||||
main() {
|
||||
local files reqs="" rc=0 base_sha head_sha local_main merge_base
|
||||
|
||||
resolve_base || undecidable no-base "$BASE_PROBLEM"
|
||||
base_sha="$(rev "$BASE")" || undecidable no-base "resolved base '${BASE}' vanished"
|
||||
head_sha="$(rev HEAD)" || undecidable no-base "HEAD does not resolve to a commit (empty repo?)"
|
||||
log "diff base: ${BASE} (${BASE_SOURCE}) @ ${base_sha:0:12} ... HEAD @ ${head_sha:0:12}"
|
||||
|
||||
# Advisory: a never-fetched origin/main silently WIDENS the set (over-broad, never a
|
||||
# false green) — surfaced so an unexpectedly large touched set has an explanation.
|
||||
if [ "$BASE" = "origin/main" ] && local_main="$(rev main)" && [ -n "$local_main" ] \
|
||||
&& [ "$local_main" != "$base_sha" ]; then
|
||||
warn "local 'main' (${local_main:0:12}) != origin/main (${base_sha:0:12}) — if origin/main"
|
||||
warn " was not fetched recently the touched set may be stale: git fetch origin main"
|
||||
fi
|
||||
|
||||
merge_base="$(git -C "$REPO_ROOT" merge-base "$BASE" HEAD 2>/dev/null)" \
|
||||
|| undecidable no-base "no merge base between '${BASE}' and HEAD (unrelated histories, or a shallow clone)"
|
||||
if [ "$merge_base" = "$head_sha" ]; then
|
||||
undecidable on-base "HEAD is not ahead of '${BASE}' (merge-base == HEAD): this checkout contains no branch commits to gate. The empty touched set would be empty BY DEFAULT, not by verification."
|
||||
fi
|
||||
|
||||
files="$(changed_files)" || rc=$?
|
||||
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff --name-only ${BASE}...HEAD failed (rc=${rc})"
|
||||
if [ -z "$files" ]; then
|
||||
warn "HEAD is ahead of ${BASE} but the three-dot diff is empty (self-cancelling commits?)"
|
||||
warn " -> genuinely empty touched-requirement set; base WAS established."
|
||||
exit 0
|
||||
fi
|
||||
local files reqs=""
|
||||
files="$(changed_files)"
|
||||
[ -n "$files" ] || { log "no changed files vs base; empty touched-requirement set"; exit 0; }
|
||||
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
@@ -234,18 +109,9 @@ main() {
|
||||
|
||||
# Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
|
||||
# `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
|
||||
# `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
|
||||
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
|
||||
# YAML/shell comments (rule 3 in changed_diff's header) BEFORE the R-ID grep — a
|
||||
# requirement cited only in a `#` comment is prose, not a declaration.
|
||||
local diff_txt
|
||||
diff_txt="$(changed_diff)" || rc=$?
|
||||
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff -U0 ${BASE}...HEAD failed (rc=${rc})"
|
||||
reqs+=" $(printf '%s\n' "$diff_txt" \
|
||||
reqs+=" $(changed_diff \
|
||||
| grep -E '^[+-]' \
|
||||
| grep -vE '^(\+\+\+|---)' \
|
||||
| sed -E 's/^[+-]//' \
|
||||
| grep -vE '^[[:space:]]*#' \
|
||||
| grep -hoE 'R-[A-Z]+-[0-9]+' \
|
||||
| tr '\n' ' ' || true)"
|
||||
|
||||
|
||||
+2
-2
@@ -9,7 +9,7 @@
|
||||
# whatever platform/kyverno and conformance/ pin — see integration notes in README.
|
||||
|
||||
# L0 policy gate (Kyverno CLI) + ServiceDefinition validation (kyverno-json)
|
||||
KYVERNO_VERSION="1.19.0"
|
||||
KYVERNO_VERSION="1.13.4"
|
||||
KYVERNO_JSON_VERSION="0.0.3"
|
||||
|
||||
# L1/L2/L3 declarative runner (Chainsaw) — invoked via conformance/ entrypoint,
|
||||
@@ -18,7 +18,7 @@ CHAINSAW_VERSION="0.2.12"
|
||||
|
||||
# Lint / manifest tooling
|
||||
YAMLLINT_VERSION="1.35.1"
|
||||
KUBECTL_VERSION="1.36.4"
|
||||
KUBECTL_VERSION="1.31.4"
|
||||
YQ_VERSION="4.44.6"
|
||||
|
||||
# ServiceDefinition JSON-Schema validation (schemas/validate.sh). Pip-installed;
|
||||
|
||||
+2
-49
@@ -29,9 +29,8 @@ clusters/
|
||||
|
||||
## How the overlay model satisfies R-ENV-02
|
||||
|
||||
The base authors the reconciliation graph **once**. `platform.yaml` and the app-wave Kustomizations
|
||||
(`apps-wave-0/1/2.yaml` — the single `apps.yaml` was retired; `apps-vms.yaml` deliberately omits it,
|
||||
see below) declare `postBuild.substituteFrom` the `cluster-vars` ConfigMap. Platform and
|
||||
The base authors the reconciliation graph **once**. `platform.yaml` and `apps.yaml` (Flux
|
||||
`Kustomization`s) both declare `postBuild.substituteFrom` the `cluster-vars` ConfigMap. Platform and
|
||||
app manifests use `${TOKEN}` placeholders for every value that legitimately differs between
|
||||
environments (issuer, replica counts, storage sizes, retention, node placement).
|
||||
|
||||
@@ -40,31 +39,6 @@ they add no resources of their own and never copy/fork a base manifest. Promotio
|
||||
([`docs/02`](../../home-network-k3s/docs/02-environments-and-provisioning.md)) is a change to the
|
||||
prod overlay's variables, nothing else.
|
||||
|
||||
### LAN addressing is a knob, not a literal (fixed 2026-08-31)
|
||||
|
||||
Until 2026-08-31 the base manifests hardcoded the **dev** LAN: the k3s API host Cilium dials, the
|
||||
Cilium LB IP pool range, the pinned crafty/unifi L4 VIPs and the off-cluster S3 backup endpoint. That
|
||||
broke R-ENV-02 outright — `clusters/prod` would have rendered dev addresses (192.168.1.x) onto the
|
||||
prod LAN (192.168.2.x). They are now the `cluster-vars` "LAN addressing" block
|
||||
(`K3S_API_HOST`, `CILIUM_LB_POOL_START`/`_STOP`, `UNIFI_L4_VIP`, `CRAFTY_L4_VIP`,
|
||||
`S3_BACKUP_ENDPOINT`, alongside the pre-existing `GATEWAY_VIP`), set explicitly by BOTH overlays.
|
||||
|
||||
Two rules when touching that block:
|
||||
|
||||
1. **Substitution coverage.** A `${TOKEN}` only resolves in a path whose Flux Kustomization declares
|
||||
`postBuild.substituteFrom: cluster-vars`; anywhere else the literal string `${TOKEN}` ships to the
|
||||
cluster. `apps-vms` was the one path without it and now has it — which means every shell `${...}`
|
||||
in a VM manifest's cloud-init must be escaped `$${...}` (see `clusters/base/apps-vms.yaml` for the
|
||||
full envsubst escape contract).
|
||||
2. **`GATEWAY_VIP` must be inside `CILIUM_LB_POOL_START..STOP`,** and in practice equal to START: the
|
||||
`shared` Gateway pins no address, so Cilium LB-IPAM gives it the pool's first free IP while
|
||||
split-horizon CoreDNS answers with `${GATEWAY_VIP}`. A VIP outside the pool blackholes in-cluster
|
||||
hairpin traffic while the Gateway still looks healthy.
|
||||
|
||||
Values the operator has not decided are `TBD-PROD-*` placeholders, not guesses: prod inherits the
|
||||
DEV base defaults if an overlay op is simply omitted, so an invalid value that fails loudly is
|
||||
strictly safer than a plausible one that quietly comes up on the wrong network.
|
||||
|
||||
## How Flux is pointed at an environment
|
||||
|
||||
`flux bootstrap` (run by WS-INFRA in Phase 1) targets one environment's path per cluster:
|
||||
@@ -97,27 +71,6 @@ Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0)
|
||||
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
|
||||
```
|
||||
|
||||
### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
|
||||
|
||||
The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
|
||||
controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
|
||||
and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
|
||||
reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
|
||||
before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
|
||||
**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
|
||||
`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
|
||||
have existed for months); it cost 12h on prod's cold start.
|
||||
|
||||
So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
|
||||
platform-config`), and a rule:
|
||||
|
||||
> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
|
||||
> **controller at runtime** rather than by a manifest in git.
|
||||
|
||||
`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
|
||||
`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
|
||||
profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
|
||||
|
||||
Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
|
||||
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
|
||||
(`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
|
||||
|
||||
@@ -4,11 +4,8 @@
|
||||
# NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
|
||||
# import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
|
||||
# health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
|
||||
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
|
||||
# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
|
||||
# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
|
||||
# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
|
||||
# adoption concern).
|
||||
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
|
||||
# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
@@ -22,47 +19,11 @@ spec:
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config
|
||||
# LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
|
||||
# storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
|
||||
# decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
|
||||
# (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
|
||||
# live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
|
||||
# with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
|
||||
# to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
|
||||
# conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
|
||||
#
|
||||
# HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
|
||||
# derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
|
||||
# status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
|
||||
# it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
|
||||
#
|
||||
# This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
|
||||
# deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
|
||||
# dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
|
||||
#
|
||||
# The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
|
||||
# edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
|
||||
# cannot silently drop this tier's KubeVirt/StorageClass gate.
|
||||
- name: platform-config-cdi
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./apps/vms
|
||||
# postBuild.substituteFrom ENABLED 2026-08-31 (R-ENV-02). It was previously omitted because Flux's
|
||||
# envsubst runs over the ENTIRE manifest — including VM cloud-init shell scripts — and would mangle
|
||||
# every `${...}`; the omission was safe only while no VM manifest referenced a cluster-var. It now
|
||||
# does: apps/vms/unifi/50-service-l4.yaml pins the UniFi L4 VIP from ${UNIFI_L4_VIP} instead of the
|
||||
# hardcoded dev-LAN literal that would have shipped 192.168.1.54 onto the PROD LAN.
|
||||
#
|
||||
# The escape contract the old comment prescribed is now in force and MUST be honoured by every VM
|
||||
# manifest under this path (verified with `flux envsubst`):
|
||||
# `${VAR}` -> substituted (empty when unset) => escape shell braces as `$${VAR}`, which
|
||||
# envsubst emits back as a literal `${VAR}`.
|
||||
# `$VAR` -> left alone by envsubst => bare shell vars need NO escaping.
|
||||
# `$(cmd)` -> left alone by envsubst => command substitution needs NO escaping.
|
||||
# `${ARR[0]}` (e.g. ${PIPESTATUS[0]}) -> HARD ERROR "missing closing brace", which fails this
|
||||
# whole Kustomization. NEVER use an indexed expansion here, escaped or not.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# NO postBuild.substituteFrom: Flux's envsubst would run over the ENTIRE manifest, including the
|
||||
# VM cloud-init shell scripts, and mangle every `${...}` (e.g. ${UOS_URL}, ${PIPESTATUS[0]}) —
|
||||
# corrupting the in-guest install. VM manifests reference no cluster-vars, so substitution is
|
||||
# unneeded here. A future VM that needs a cluster-var must escape its shell `${...}` as `$${...}`.
|
||||
@@ -1,60 +0,0 @@
|
||||
---
|
||||
# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
|
||||
# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
|
||||
#
|
||||
# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
|
||||
#
|
||||
# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
|
||||
# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
|
||||
# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
|
||||
# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
|
||||
#
|
||||
# * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
|
||||
# * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
|
||||
# tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
|
||||
# `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
|
||||
# the valve the 2026-09-01 substitution incident wanted and did not have.
|
||||
# * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
|
||||
# `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
|
||||
#
|
||||
# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
|
||||
# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
|
||||
# expressed as an explicit edge instead of an accident of co-tenancy.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: cilium
|
||||
namespace: flux-system
|
||||
spec:
|
||||
# A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
|
||||
# kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
|
||||
interval: 1h
|
||||
retryInterval: 2m
|
||||
# 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
|
||||
# rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
|
||||
timeout: 10m
|
||||
prune: true
|
||||
# TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
|
||||
wait: true
|
||||
# Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
|
||||
# by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
|
||||
# is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
|
||||
# revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
|
||||
# `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
|
||||
dependsOn:
|
||||
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/cilium
|
||||
# MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
|
||||
# kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
|
||||
# this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
|
||||
# renders empty). That is the intended fail-closed behaviour — but it means the stanza is
|
||||
# load-bearing for the CNI coming up at all. Do not remove it.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
|
||||
# oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).
|
||||
+10
-196
@@ -17,173 +17,20 @@ data:
|
||||
CLUSTER_ENV: "base" # dev | prod (overlays set this)
|
||||
|
||||
# --- DNS / TLS ---
|
||||
# THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
|
||||
# the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
|
||||
# ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
|
||||
#
|
||||
# THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
|
||||
# hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
|
||||
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
|
||||
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
|
||||
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
|
||||
# subdomain, so the two clusters can no longer describe the same name.
|
||||
#
|
||||
# dev -> home.keithjackson.dev (UNCHANGED — see below)
|
||||
# prod -> prod.home.keithjackson.dev
|
||||
#
|
||||
# DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
|
||||
# `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
|
||||
# relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
|
||||
# the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
|
||||
# (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
|
||||
# GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
|
||||
# it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
|
||||
# tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
|
||||
BASE_DOMAIN: "home.keithjackson.dev"
|
||||
|
||||
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
|
||||
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
|
||||
# so it cannot be created, updated or deleted regardless of TXT ownership.
|
||||
#
|
||||
# DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
|
||||
# apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
|
||||
# that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
|
||||
# record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
|
||||
# from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
|
||||
#
|
||||
# PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
|
||||
# (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
|
||||
# chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
|
||||
# — it can never delete another environment's records — and the fallback is to set prod's value
|
||||
# back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
|
||||
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
|
||||
|
||||
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
|
||||
# neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
|
||||
# record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
|
||||
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
|
||||
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
|
||||
#
|
||||
# DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
|
||||
# under owner `default`; changing dev's value would make dev's own external-dns stop recognising
|
||||
# them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
|
||||
#
|
||||
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
|
||||
# forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
|
||||
# this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
|
||||
# visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
|
||||
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
|
||||
|
||||
BASE_DOMAIN: "home.keithjackson.dev" # platform hostname suffix (same both envs; split-horizon)
|
||||
DNS_ZONE: "keithjackson.dev" # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
|
||||
# ExternalDNS domain-filter must be the zone, not the subdomain
|
||||
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
||||
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
||||
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
|
||||
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
|
||||
# takes the pool's first free IP, and coredns-custom answers
|
||||
# *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
|
||||
# hairpins through the Gateway (correct SNI -> wildcard cert).
|
||||
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
|
||||
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
|
||||
# forgets this knob at least lands on a REAL network). A prod overlay
|
||||
# MUST still set the prod LAN VIP alongside the prod L2 pool range.
|
||||
|
||||
# --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply -------------
|
||||
# These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint:
|
||||
# `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24)
|
||||
# — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values
|
||||
# (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the
|
||||
# dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block
|
||||
# before `flux bootstrap --path=clusters/prod`.
|
||||
#
|
||||
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
|
||||
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
|
||||
# (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
|
||||
# velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
|
||||
# ship the literal string into the cluster.
|
||||
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
|
||||
#
|
||||
# *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
|
||||
# 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
|
||||
# control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
|
||||
# live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
|
||||
# own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
|
||||
# Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
|
||||
# outage you would notice. A virtual address that follows the control plane is the fix, and it is
|
||||
# what kube-vip was stood up to provide.
|
||||
#
|
||||
# dev -> 192.168.1.49 (= kube_vip_address, group_vars/all.yml; base default, below)
|
||||
# prod -> 192.168.2.9 (= kube_vip_address, group_vars/k3s_prod/vars.yml)
|
||||
#
|
||||
# THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
|
||||
# unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
|
||||
# move. Read it as "the API host Cilium dials", which is what it always meant.
|
||||
#
|
||||
# [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
|
||||
# (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
|
||||
# already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
|
||||
# the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
|
||||
# already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
|
||||
# block at the top of group_vars/all.yml.
|
||||
#
|
||||
# Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
|
||||
# in both envs, so it is NOT a knob.
|
||||
#
|
||||
# TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
|
||||
# both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
|
||||
# https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
|
||||
K3S_API_HOST: "192.168.1.49"
|
||||
|
||||
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
|
||||
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
|
||||
# tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in
|
||||
# practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's
|
||||
# first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and
|
||||
# the flagged block in clusters/prod/kustomization.yaml.)
|
||||
CILIUM_LB_POOL_START: "192.168.1.53"
|
||||
CILIUM_LB_POOL_STOP: "192.168.1.60"
|
||||
|
||||
# Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the
|
||||
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
||||
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
||||
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
||||
|
||||
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05). Consumed by BOTH backup paths: every CNPG
|
||||
# Cluster's barmanObjectStore.endpointURL (keycloak-db, nextcloud-db, semaphore-db) and Velero's
|
||||
# backupStorageLocation s3Url. Full URL including scheme (and port, if not the scheme default).
|
||||
#
|
||||
# OPERATOR DECISION 2026-09-05: **Cloudflare R2**, replacing the on-Proxmox MinIO — "for the time
|
||||
# being until a better on-prem backup solution is created". Two reasons beyond cost:
|
||||
# 1. The k3s prod cluster is on the SOHO network (pfSense's WAN) and CANNOT reach the MinIO inside
|
||||
# pfSense's LAN — measured, from a pod: connection timed out. R2 needs no firewall change.
|
||||
# 2. That MinIO runs on the SAME Proxmox host as dev's VMs, so dev's "off-cluster" backup shared a
|
||||
# physical failure domain with dev. One host loss took the cluster AND its backups. R2 is
|
||||
# genuinely off-site.
|
||||
# Measured to justify the cost: dev's whole estate is 4.0 GiB Velero (6,264 objects, 7d retention) +
|
||||
# 211 MiB CNPG (1,755 objects) — pennies per month, and R2 egress is free, so restore REHEARSALS
|
||||
# stay free and R-STATE-03 has no cost disincentive to stay proven.
|
||||
#
|
||||
# The MinIO copy is NOT deleted — it stays as a historical/local copy (3-2-1). Restoring from it
|
||||
# means pointing this knob back at http://192.168.1.9:9000 temporarily.
|
||||
S3_BACKUP_ENDPOINT: "https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com"
|
||||
|
||||
# S3 region. Required by every S3 client, IGNORED by MinIO, and must be exactly `auto` for R2.
|
||||
# This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-accident for
|
||||
# MinIO and wrong for R2, which is why it is a knob now.
|
||||
S3_BACKUP_REGION: "auto"
|
||||
GATEWAY_VIP: "10.73.0.240" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
||||
# ingress VIP of the `shared` Gateway (first IP of the Cilium
|
||||
# LB IP pool 10.73.0.240-.250, platform-config/cilium/l2-ip-pool.yaml).
|
||||
# coredns-custom answers *.home.keithjackson.dev with THIS so pod->
|
||||
# public-hostname traffic hairpins through the Gateway (correct SNI ->
|
||||
# wildcard cert). Base default = dev VIP; a prod overlay must set the
|
||||
# prod LAN VIP alongside the prod L2 pool range when hardware is built.
|
||||
|
||||
# --- storage (ADR-0003) ---
|
||||
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
||||
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
|
||||
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
|
||||
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
|
||||
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
|
||||
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
|
||||
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
|
||||
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
|
||||
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
|
||||
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
|
||||
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
|
||||
LONGHORN_DATA_PATH: "/var/lib/longhorn"
|
||||
|
||||
# --- observability sizing (docs/10) ---
|
||||
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
|
||||
@@ -223,39 +70,6 @@ data:
|
||||
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
|
||||
VELERO_RETENTION: "168h" # dev short | prod long
|
||||
|
||||
# BACKUP TENANCY — the two knobs that keep two clusters sharing ONE MinIO from destroying each
|
||||
# other's backups. Added 2026-09-05 when the operator ruled prod uses the SAME endpoint as dev.
|
||||
#
|
||||
# Sharing an endpoint is fine. Sharing a PATH is not, and nothing here was per-environment before:
|
||||
#
|
||||
# VELERO — bucket `velero`, no prefix, and BOTH clusters run schedules named `daily` and
|
||||
# `daily-resources-only`. Each cluster's Velero SYNCS backups it finds in the BSL into
|
||||
# itself as Backup CRs and GARBAGE-COLLECTS them when their TTL expires. Prod would
|
||||
# therefore adopt dev's backups and delete them on schedule.
|
||||
# CNPG — destinationPath `s3://cnpg` with CNPG appending serverName, which defaults to the
|
||||
# CLUSTER NAME. Both environments have `keycloak-db`, `nextcloud-db`, `semaphore-db`,
|
||||
# so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers
|
||||
# interleaving WAL into one Barman stream CORRUPTS THE CHAIN and destroys PITR — and
|
||||
# dev's DR restore is PROVEN (R-STATE-03). This is the worst of the two.
|
||||
#
|
||||
# Both environments are now NAMESPACED SYMMETRICALLY, which was not possible an hour ago and is
|
||||
# worth explaining, because the earlier reasoning was the opposite:
|
||||
#
|
||||
# While the target was the MinIO, dev's base value HAD to be the bucket root — its existing
|
||||
# backups lived there and giving dev a prefix would have ORPHANED the very backups its proven
|
||||
# restore depends on (the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default`).
|
||||
#
|
||||
# Moving to R2 starts BOTH clusters on an EMPTY bucket, so there is no history to orphan and no
|
||||
# reason to keep the asymmetry. `dev` and `prod` is clearer, greppable, and leaves the bucket root
|
||||
# free — so an unset/misrendered prefix lands somewhere obviously wrong instead of silently
|
||||
# colliding with a real environment's backups.
|
||||
#
|
||||
# *** THESE TWO VALUES AND S3_BACKUP_ENDPOINT MUST CHANGE TOGETHER. *** Setting a `dev` prefix while
|
||||
# the endpoint is still the MinIO would orphan dev's existing MinIO backups for no benefit. They are
|
||||
# one change; do not split the commit.
|
||||
VELERO_BACKUP_PREFIX: "dev" # dev: "dev" | prod: "prod"
|
||||
CNPG_BACKUP_PATH: "s3://cnpg/dev" # dev: s3://cnpg/dev | prod: s3://cnpg/prod
|
||||
|
||||
# --- placement ---
|
||||
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
|
||||
PLATFORM_NODE_SELECTOR_VALUE: "linux"
|
||||
@@ -5,24 +5,8 @@ apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- flux-system.yaml # GitRepository (reconciliation source)
|
||||
# Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
|
||||
# of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
|
||||
# Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
|
||||
# `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
|
||||
# interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
|
||||
# freeze the other eleven platform components. See clusters/base/cilium.yaml.
|
||||
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
||||
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
||||
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
||||
# SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
|
||||
# manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
|
||||
# datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
|
||||
# platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
|
||||
# keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
|
||||
# CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
|
||||
# cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
|
||||
# as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
|
||||
- platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
|
||||
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
|
||||
# Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
|
||||
# (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
|
||||
|
||||
@@ -1,135 +0,0 @@
|
||||
---
|
||||
# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
|
||||
# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
|
||||
#
|
||||
# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
|
||||
#
|
||||
# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
|
||||
# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
|
||||
# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
|
||||
# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
|
||||
# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
|
||||
# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
|
||||
# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
|
||||
#
|
||||
# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
|
||||
# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
|
||||
# graph order itself, so a cold cluster needs no human in the loop.
|
||||
#
|
||||
# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
|
||||
# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
|
||||
# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
|
||||
# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
|
||||
# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
|
||||
# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
|
||||
# kind. One edge, no pre-apply.
|
||||
#
|
||||
# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
|
||||
# and everything else waits on: putting CDI's health on that critical path is the mistake the
|
||||
# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
|
||||
# StorageProfiles that were in the wrong pass.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: platform-config-cdi
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
# 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
|
||||
# very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
|
||||
# now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
|
||||
# about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
|
||||
# cache is one retryInterval. Keep it short.
|
||||
retryInterval: 2m
|
||||
# 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
|
||||
# image to pull and no controller to become Available, so a slow apply here means a real error, not
|
||||
# a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
|
||||
timeout: 2m
|
||||
# TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
|
||||
# component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
|
||||
# It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
|
||||
# second path also declares. See the transition note at the bottom of this file.
|
||||
prune: true
|
||||
# TRUE, but understand exactly what it buys and what it does NOT.
|
||||
#
|
||||
# CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
|
||||
# provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
|
||||
# `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
|
||||
# it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
|
||||
# is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
|
||||
# If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
|
||||
# `apps-vms` waits on it.
|
||||
#
|
||||
# WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
|
||||
# provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
|
||||
# storageprofile-controller computes that status from our spec override plus inference. So Ready
|
||||
# here means "the spec override is accepted by the API server", not "CDI has folded it into
|
||||
# status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
|
||||
# below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
|
||||
# a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
|
||||
wait: true
|
||||
dependsOn:
|
||||
# platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
|
||||
# causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
|
||||
# ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
|
||||
# flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
|
||||
- name: platform-config
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform-config-cdi
|
||||
# NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
|
||||
# no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
|
||||
# oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
|
||||
# clusters/dev or clusters/prod for this tier.
|
||||
#
|
||||
# substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
|
||||
# substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
|
||||
# 1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
|
||||
# whose Kustomization declares substituteFrom. Omitting it would move
|
||||
# cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
|
||||
# could ship as a literal string, precisely the class of bug that check exists to stop.
|
||||
# 2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
|
||||
# with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
|
||||
# empty. Without the stanza there is no substitution at all and the literal ships silently.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
|
||||
# Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
|
||||
# Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
|
||||
# StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
|
||||
# the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
|
||||
# dual-ownership fight — the objects are pruned before they are re-applied under this tier's
|
||||
# `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
|
||||
#
|
||||
# Why that is acceptable here — and it is NOT the platform/cilium case:
|
||||
# * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
|
||||
# nothing else in the repo declares a StorageProfile. The stale set is exactly two
|
||||
# cluster-scoped CDI CRs.
|
||||
# * The objects do not actually go missing: CDI's storageprofile-controller creates a
|
||||
# StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
|
||||
# inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
|
||||
# absent".
|
||||
# * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
|
||||
# existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
|
||||
# is disturbed and no data path is touched.
|
||||
# * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
|
||||
# instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
|
||||
# permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
|
||||
# dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
|
||||
# created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
|
||||
# could.
|
||||
#
|
||||
# PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
|
||||
# happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
|
||||
# config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
|
||||
# healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
|
||||
# --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
|
||||
# StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
|
||||
# `flux get kustomization platform-config` must be Ready BEFORE merging.
|
||||
# If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
|
||||
# temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
|
||||
# this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
|
||||
# `prune: true` in a second commit.
|
||||
@@ -11,53 +11,8 @@ spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
# Cilium was moved OUT of this tier into its own Kustomization (clusters/base/cilium.yaml)
|
||||
# on 2026-09-01. `prune: true` is restored here now that the ownership transfer is VERIFIED
|
||||
# live: Namespace/kube-system, the Cilium HelmRelease and the HelmRepository all carry
|
||||
# `kustomize.toolkit.fluxcd.io/name=cilium`, this Kustomization's inventory no longer lists
|
||||
# any of the three, and the `cilium` Kustomization's inventory holds exactly those three.
|
||||
#
|
||||
# LESSON WORTH KEEPING: `platform/cilium/namespace.yaml` was the ONLY declaration of
|
||||
# Namespace/kube-system under ./platform. Moving that path out of a prune:true Kustomization
|
||||
# made the stale set `delete namespace kube-system` — coredns, metrics-server, local-path AND
|
||||
# Cilium. Before moving ANY platform/* component in future, check whether it is the sole
|
||||
# declarer of a shared Namespace; that turns a component-scoped prune into a cluster-scoped one.
|
||||
prune: true
|
||||
prune: true # drift correction: delete resources removed from git (R-GITOPS-03)
|
||||
wait: true # block `apps` until platform is healthy
|
||||
# ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
|
||||
#
|
||||
# `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root
|
||||
# `flux-system` Kustomization (clusters/<env>/sync.yaml), not by this one. Without an edge here,
|
||||
# `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a
|
||||
# NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING.
|
||||
# That is how `k8sServiceHost: ""` reached the Cilium HelmRelease.
|
||||
#
|
||||
# `dependsOn` is REVISION-gated, not merely Ready-gated (verified against
|
||||
# kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version):
|
||||
# when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile
|
||||
# revision N until the dependency has APPLIED revision N. Applying revision N is exactly what
|
||||
# writes a newly-added `cluster-vars` knob — so this one edge closes the race.
|
||||
#
|
||||
# `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting
|
||||
# consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty,
|
||||
# velero) chains transitively through it. One edge therefore protects the whole graph.
|
||||
#
|
||||
# TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest
|
||||
# ANYWHERE under the root's path (clusters/<env>, which includes clusters/base) fails the root
|
||||
# Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge
|
||||
# the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken
|
||||
# entrypoint should stop the line, not let consumers render against a half-applied variable set.
|
||||
# No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never
|
||||
# health-gates its own children (see clusters/dev/sync.yaml).
|
||||
dependsOn:
|
||||
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
|
||||
# THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
|
||||
# (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
|
||||
# suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
|
||||
# nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
|
||||
# is healthy. It is also revision-gated, which is what makes the prune-safety argument above
|
||||
# deterministic rather than a race.
|
||||
- name: cilium
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
|
||||
@@ -148,108 +148,16 @@ patches:
|
||||
- op: replace
|
||||
path: /data/ACME_CLUSTER_ISSUER
|
||||
value: "letsencrypt-prod"
|
||||
# --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
|
||||
# DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
|
||||
# like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
|
||||
# Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
|
||||
# Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
|
||||
# (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
|
||||
# prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
|
||||
# not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
|
||||
# not a variable change: infra/PROD-PROVISIONING.md §10.
|
||||
- op: replace
|
||||
path: /data/BASE_DOMAIN
|
||||
value: "home.keithjackson.dev"
|
||||
# ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
|
||||
# and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
|
||||
# the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_DOMAIN_FILTER
|
||||
value: "keithjackson.dev"
|
||||
# DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
|
||||
# registry under owner `default` (the external-dns chart default, which is what both clusters
|
||||
# silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
|
||||
# BY this string: any other value and it stops recognising every record it manages, then
|
||||
# re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
|
||||
# whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_TXT_OWNER_ID
|
||||
value: "default"
|
||||
# --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
|
||||
# EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
|
||||
# equals the base default. These values are what the LIVE dev cluster is running right now, so
|
||||
# the substituted output is byte-identical to the pre-parameterization hardcoded manifests
|
||||
# (verified 2026-08-31). Never let this block drift from the live cluster.
|
||||
#
|
||||
# Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
|
||||
# (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
|
||||
# (base default 10.73.0.240 was the old libvirt VIP). coredns-custom answers
|
||||
# *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
|
||||
- op: replace
|
||||
path: /data/GATEWAY_VIP
|
||||
value: "192.168.1.53"
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials — DEV'S CONTROL-PLANE VIP,
|
||||
# the address kube-vip floats, NOT the primary node .50.
|
||||
# [parity] equals the Ansible inventory's k3s_apiserver_endpoint for dev
|
||||
# (= kube_vip_address, group_vars/all.yml). See clusters/base/cluster-vars.yaml for why.
|
||||
#
|
||||
# *** THE ONE EXCEPTION TO "NEVER LET THIS BLOCK DRIFT FROM THE LIVE CLUSTER" ***
|
||||
# Every other value in this block is what dev is running right now. This one is not:
|
||||
# the live dev cilium-agents dial 192.168.1.50 (k3s-1). Changing it here is a
|
||||
# DELIBERATE, OPERATOR-APPROVED move (2026-09-06) and reconciling it ROLLS THE CILIUM
|
||||
# AGENT DAEMONSET on dev — one node at a time (updateStrategy.maxUnavailable: 1, kept
|
||||
# in parity for exactly this reason). Do not push this to dev-live casually, and not
|
||||
# while dev's DNS migration is mid-flight; the L2 datapath survives an agent restart
|
||||
# but a wrong value here is the 2026-09-01 incident. .49 is in dev's serving-cert SANs
|
||||
# already (k3s_tls_sans) and dev's own kubeconfig validates against it via
|
||||
# tls-server-name, so TLS is not the risk — the risk is reachability, and .49 is on the
|
||||
# same LAN as the nodes and ARP-announced by kube-vip on every server.
|
||||
- op: replace
|
||||
path: /data/K3S_API_HOST
|
||||
value: "192.168.1.49"
|
||||
# Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
|
||||
# (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_START
|
||||
value: "192.168.1.53"
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_STOP
|
||||
value: "192.168.1.60"
|
||||
# Pinned L4 VIPs out of that pool (lbipam.cilium.io/ips). Changing either MOVES a live device-
|
||||
# facing endpoint: unifi is the inform/STUN target every adopted UniFi device holds.
|
||||
- op: replace
|
||||
path: /data/UNIFI_L4_VIP
|
||||
value: "192.168.1.54"
|
||||
- op: replace
|
||||
path: /data/CRAFTY_L4_VIP
|
||||
value: "192.168.1.55"
|
||||
# OFF-CLUSTER BACKUP TARGET — no op here any more, deliberately.
|
||||
#
|
||||
# This used to pin the on-Proxmox MinIO (http://192.168.1.9:9000). As of 2026-09-05 the operator
|
||||
# moved backups to Cloudflare R2 "for the time being until a better on-prem backup solution is
|
||||
# created", and R2 is a single environment-INDEPENDENT endpoint: both clusters write to the same
|
||||
# buckets and are isolated by VELERO_BACKUP_PREFIX / CNPG_BACKUP_PATH (dev vs prod), not by URL.
|
||||
# Under R-ENV-02 a value that does not differ per environment belongs in base and must NOT be
|
||||
# restated here — two copies of one endpoint is just a rotation waiting to drift.
|
||||
#
|
||||
# *** MERGING THIS RESETS DEV'S PITR WINDOW. *** Dev's R2 prefix starts EMPTY, so Velero writes a
|
||||
# fresh kopia repository and CNPG takes new base backups; the 30-day Barman window rebuilds from
|
||||
# zero. The existing MinIO copy is NOT deleted and stays restorable — point S3_BACKUP_ENDPOINT
|
||||
# back at http://192.168.1.9:9000 (and VELERO_BACKUP_PREFIX back to "") to reach it. Do not merge
|
||||
# this while dev's proven restore is the only recovery path you are relying on.
|
||||
# dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
||||
value: "3"
|
||||
# DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
|
||||
# on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
|
||||
# mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
|
||||
# else would orphan every one of them (they are not recoverable from the old path by
|
||||
# rescheduling). Restated explicitly even though it equals the base default, exactly like the
|
||||
# LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
|
||||
# its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DATA_PATH
|
||||
value: "/var/lib/longhorn"
|
||||
# smaller retention / volumes to keep the dev workstation footprint light.
|
||||
- op: replace
|
||||
path: /data/MONITORING_PROMETHEUS_RETENTION
|
||||
|
||||
@@ -5,191 +5,11 @@
|
||||
# When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
|
||||
# here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
|
||||
# values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
|
||||
#
|
||||
# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
|
||||
# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
|
||||
# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
|
||||
# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
|
||||
# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
|
||||
# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
|
||||
# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
|
||||
# done here: prod is design-only and its root object is the prod bring-up owner's call.)
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- ../base
|
||||
- sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
|
||||
patches:
|
||||
# --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
|
||||
# Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
|
||||
# The `sops-age` Secret is created OUT OF BAND on the prod cluster
|
||||
# (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
|
||||
# Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: platform
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-wave-0
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-wave-1
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-wave-2
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: headlamp
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: crafty
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: velero
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
|
||||
#
|
||||
# ############################################################################################
|
||||
# ### ###
|
||||
# ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ###
|
||||
# ### ###
|
||||
# ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ###
|
||||
# ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ###
|
||||
# ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ###
|
||||
# ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ###
|
||||
# ### a live single point of failure, not a settled design. See "CUTOVER" below. ###
|
||||
# ### ###
|
||||
# ############################################################################################
|
||||
#
|
||||
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
|
||||
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
|
||||
# Two independent reasons, both consequences of the split-horizon design:
|
||||
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
|
||||
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
|
||||
# prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
|
||||
# 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
|
||||
# answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
|
||||
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
|
||||
# gitea.home.keithjackson.dev to itself and never reach a repository.
|
||||
#
|
||||
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
|
||||
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
|
||||
# operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
|
||||
# reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
|
||||
# change was needed. This is the same "local bootstrap source" pattern dev itself used before
|
||||
# its Phase-6 cutover (docs/05 "Source-of-truth").
|
||||
#
|
||||
# PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
|
||||
# for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
|
||||
# the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
|
||||
# .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
|
||||
#
|
||||
# NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
|
||||
# works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
|
||||
# curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
|
||||
# git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main
|
||||
# If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
|
||||
# THAT FIRST: a private repo here fails exactly that way.
|
||||
#
|
||||
# *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
|
||||
# WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
|
||||
# through prod's own Gateway VIP with a publicly-trusted cert.
|
||||
# DO: 1. push/mirror this repo into the prod in-cluster Gitea;
|
||||
# 2. change the url below to
|
||||
# https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
||||
# and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
|
||||
# clusters/dev/kustomization.yaml, is what "done" looks like);
|
||||
# 3. confirm `flux get sources git -n flux-system` reconciles the new revision;
|
||||
# 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
|
||||
# exactly as dev retained its bootstrap Gitea.
|
||||
# UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
|
||||
# powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
|
||||
# it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
|
||||
#
|
||||
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
|
||||
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
|
||||
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
|
||||
# NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
|
||||
# Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
|
||||
# is the revision you intend prod to run (`git ls-remote` above, against local `main`).
|
||||
- patch: |-
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: GitRepository
|
||||
metadata:
|
||||
name: flux-system
|
||||
namespace: flux-system
|
||||
spec:
|
||||
# TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
|
||||
# above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
|
||||
url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
|
||||
ref:
|
||||
branch: main
|
||||
- target:
|
||||
kind: ConfigMap
|
||||
name: cluster-vars
|
||||
@@ -201,87 +21,9 @@ patches:
|
||||
- op: replace
|
||||
path: /data/ACME_CLUSTER_ISSUER
|
||||
value: "letsencrypt-prod"
|
||||
# ==========================================================================================
|
||||
# PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
|
||||
# ==========================================================================================
|
||||
# PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
|
||||
# home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
|
||||
# public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
|
||||
# of which already happened:
|
||||
# 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
|
||||
# default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
|
||||
# 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
|
||||
# wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
|
||||
# have raced each other's `_acme-challenge` TXT records on that one zone.
|
||||
# Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
|
||||
# record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
|
||||
#
|
||||
# dev -> home.keithjackson.dev (UNCHANGED — dev is live; phase 2 moves it)
|
||||
# prod -> prod.home.keithjackson.dev (this knob)
|
||||
#
|
||||
# CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
|
||||
# gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
|
||||
# on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
|
||||
# `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
|
||||
# would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
|
||||
#
|
||||
# THE APEX INVERSION IS DELIBERATELY NOT ON THIS BRANCH. Moving prod to the bare apex (and dev
|
||||
# to dev.home.) is a LIVE MIGRATION of the dev cluster, not a variable change — see
|
||||
# infra/PROD-PROVISIONING.md §10a. It is kept as a single commit on branch
|
||||
# `migration/dns-apex-inversion` (9137bbd) and is applied as step D3 of that runbook, with the
|
||||
# operator present. Everything else on this branch is independent of it, which is the whole
|
||||
# reason it was split out.
|
||||
#
|
||||
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
|
||||
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
|
||||
# set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
|
||||
# trusts a different Keycloak than the one prod actually runs.
|
||||
- op: replace
|
||||
path: /data/BASE_DOMAIN
|
||||
value: "prod.home.keithjackson.dev"
|
||||
# SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
|
||||
# OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
|
||||
# TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
|
||||
# ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
|
||||
# the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
|
||||
# matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
|
||||
# records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
|
||||
# EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_DOMAIN_FILTER
|
||||
# MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
|
||||
# 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
|
||||
# record then skipped it with "no hosted zone matching record DNS Name", so prod would
|
||||
# publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
|
||||
# (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
|
||||
# (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
|
||||
# That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
|
||||
value: "keithjackson.dev"
|
||||
# SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
|
||||
# both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
|
||||
# its own and, under policy: sync, deleted the records behind them. This value MUST differ from
|
||||
# dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
|
||||
# cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_TXT_OWNER_ID
|
||||
value: "prod"
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
||||
value: "3"
|
||||
# Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
|
||||
# the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
|
||||
# and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
|
||||
# (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
|
||||
# measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
|
||||
# its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
|
||||
#
|
||||
# [parity] MUST equal node_baseline_longhorn_data_path in
|
||||
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
|
||||
# directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
|
||||
# path on the OS filesystem instead of the data partition and /var fills up.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DATA_PATH
|
||||
value: "/data/longhorn"
|
||||
# longer retention / larger volumes on prod NVMe.
|
||||
- op: replace
|
||||
path: /data/MONITORING_PROMETHEUS_RETENTION
|
||||
@@ -307,114 +49,8 @@ patches:
|
||||
- op: replace
|
||||
path: /data/VELERO_RETENTION
|
||||
value: "720h"
|
||||
# ==========================================================================================
|
||||
# PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12.
|
||||
# ==========================================================================================
|
||||
# These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would
|
||||
# have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and
|
||||
# this overlay supplies every one of them. Anything the operator has NOT yet decided stays an
|
||||
# INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED
|
||||
# op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*`
|
||||
# is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY.
|
||||
#
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table.
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# nodes 192.168.2.10 / .11 / .12 (Ansible inventory)
|
||||
# control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here)
|
||||
# ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below)
|
||||
# Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below)
|
||||
#
|
||||
# CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev
|
||||
# (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml)
|
||||
# pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and
|
||||
# apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a
|
||||
# matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30
|
||||
# proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept
|
||||
# answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing
|
||||
# in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy.
|
||||
# KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either.
|
||||
# The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and
|
||||
# LB-IPAM must never be able to hand it to a Service.
|
||||
- op: replace
|
||||
path: /data/GATEWAY_VIP
|
||||
value: "192.168.2.20"
|
||||
# Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above).
|
||||
# Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one.
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_START
|
||||
value: "192.168.2.20"
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_STOP
|
||||
value: "192.168.2.30"
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
|
||||
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
|
||||
#
|
||||
# *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
|
||||
# read "Not yet decided" while shipping .10 — the founder's own address — and that is what
|
||||
# the LIVE prod cluster is running: all three cilium-agents carry
|
||||
# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
|
||||
# 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
|
||||
# the datapath keeps running off Cilium's cache while no service or policy update lands, with
|
||||
# nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
|
||||
# control plane is, which is the whole reason kube-vip exists here.)
|
||||
#
|
||||
# It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
|
||||
# this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
|
||||
# equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
|
||||
# that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
|
||||
# `--server` join needs a host that is up before any VIP is floated. Two addresses, two
|
||||
# variables, deliberately.
|
||||
#
|
||||
# .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
|
||||
# LB-IPAM can never be handed the same IP — asserted by
|
||||
# conformance/tests/l0-static/ha-control-plane-as-code.
|
||||
- op: replace
|
||||
path: /data/K3S_API_HOST
|
||||
value: "192.168.2.9"
|
||||
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
|
||||
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
|
||||
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
|
||||
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
|
||||
# collide with the Gateway VIP or the node IPs (.10/.11/.12).
|
||||
- op: replace
|
||||
path: /data/UNIFI_L4_VIP
|
||||
value: "192.168.2.21"
|
||||
- op: replace
|
||||
path: /data/CRAFTY_L4_VIP
|
||||
value: "192.168.2.22"
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05).
|
||||
#
|
||||
# THERE IS DELIBERATELY NO `S3_BACKUP_ENDPOINT` OP HERE, and that is a reversal of the rule
|
||||
# stated at the top of this overlay ("an OMITTED op means prod silently inherits a dev
|
||||
# address"). That rule exists because base defaults are dev's LAN addresses. R2 is not one: it
|
||||
# is a single shared, environment-independent endpoint, so under R-ENV-02 it belongs in base and
|
||||
# overriding it with an identical value would only create two places to change on a rotation —
|
||||
# the exact drift this repo has been bitten by before. Isolation does NOT come from the
|
||||
# endpoint; it comes from the two path knobs below.
|
||||
#
|
||||
# WHY R2 AND NOT THE MinIO the operator first named: prod is on the SOHO network
|
||||
# 192.168.2.0/24, which is pfSense's WAN, while the MinIO sits inside pfSense's LAN
|
||||
# 192.168.1.0/24. Measured from a pod INSIDE prod (the workstation proves nothing about pod
|
||||
# egress): `curl http://192.168.1.9:9000/minio/health/live` → connection timed out after 15s.
|
||||
# Reaching it would need a WAN→LAN rule that does not exist. R2 needs no firewall change, and
|
||||
# unlike that MinIO — which runs on the same Proxmox host as dev's VMs — it is genuinely
|
||||
# off-site. Verify prod can reach the new endpoint before resuming velero:
|
||||
# bootstrap/standup.sh ... --s3-endpoint <the R2 endpoint> s3check
|
||||
#
|
||||
# THE PATH KNOBS ARE WHAT MAKE ONE SHARED BUCKET SAFE. Nothing about the paths was
|
||||
# per-environment before 2026-09-05: Velero used one unprefixed bucket, and every CNPG Cluster
|
||||
# used `s3://cnpg` with serverName defaulting to the CLUSTER NAME — and both environments have
|
||||
# keycloak-db / nextcloud-db / semaphore-db. Without these knobs prod would (a) adopt and
|
||||
# TTL-delete dev's Velero backups and (b) interleave two Postgres servers' WAL into ONE Barman
|
||||
# stream, corrupting the chain and destroying PITR for both. See clusters/base/cluster-vars.yaml.
|
||||
# ------------------------------------------------------------------------------------------
|
||||
- op: replace
|
||||
path: /data/VELERO_BACKUP_PREFIX
|
||||
value: "prod"
|
||||
# Barman objects land at s3://cnpg/prod/<clusterName>/; dev's at s3://cnpg/dev/<clusterName>/.
|
||||
- op: replace
|
||||
path: /data/CNPG_BACKUP_PATH
|
||||
value: "s3://cnpg/prod"
|
||||
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
|
||||
# TODO(WS-NET, Phase 7): set GATEWAY_VIP to the prod LAN ingress VIP (add a `replace` op here)
|
||||
# at the same time the prod Cilium LB IP pool range is supplied — the split-horizon
|
||||
# coredns-custom (platform/coredns) answers *.home.keithjackson.dev with ${GATEWAY_VIP}, so
|
||||
# the prod value must match the prod pool's Gateway address. Base default is the dev VIP.
|
||||
@@ -1,32 +0,0 @@
|
||||
---
|
||||
# Self-managing ROOT Kustomization for PROD (mirrors clusters/dev/sync.yaml).
|
||||
#
|
||||
# Reconciles ./clusters/prod from the GitRepository so the ENTIRE prod graph — the cluster-vars
|
||||
# ConfigMap, the GitRepository source, platform / platform-config / the app waves, and THIS object
|
||||
# itself — is applied from git on every push. dev/sync.yaml carried the note "a prod overlay needs
|
||||
# its own sync.yaml -> ./clusters/prod"; this is it.
|
||||
#
|
||||
# WHY IT IS NOW LOAD-BEARING, not just symmetry: `clusters/base/platform.yaml` gained
|
||||
# `dependsOn: [flux-system]` (2026-09-01) to close the post-build substitution race — a dependent
|
||||
# cannot reconcile revision N until the ROOT has APPLIED revision N. Without this object there is no
|
||||
# `flux-system` Kustomization on prod, so `platform` would block forever on a dependency that does
|
||||
# not exist, and NOTHING downstream would ever reconcile.
|
||||
#
|
||||
# `wait: false` is deliberate and is what keeps the dependsOn edge deadlock-free: the root reaches
|
||||
# Ready on apply success without health-gating its children (platform does its own wait/health
|
||||
# gating). Do not "harden" this to true.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: flux-system
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./clusters/prod
|
||||
@@ -5,25 +5,3 @@ _artifacts/
|
||||
|
||||
# Downloaded pinned tool binaries (run.sh may fetch chainsaw/kubectl here).
|
||||
.bin/
|
||||
|
||||
# Python bytecode from lib/ (status.py imports coverage.py); never committed.
|
||||
__pycache__/
|
||||
|
||||
# --- THE ONE EXCEPTION: the committed run ledger (runs/) -------------------------------
|
||||
# `runs/<UTC>-<short-sha>/{report.junit.xml,run.log}` + `runs/ledger.jsonl` are the
|
||||
# machine-readable proof-of-pass record that lib/status.py's docstring asks for. They are
|
||||
# SOURCE, not regenerated artifacts: a run's evidence cannot be reconstructed after the
|
||||
# fact, so it must survive in git.
|
||||
#
|
||||
# The negations are needed because the ROOT .gitignore ignores `*.junit.xml` globally
|
||||
# (and `coverage/`). A deeper .gitignore overrides a shallower one, and nothing ignores
|
||||
# the `conformance/runs/` directory itself, so re-including the files here works (git
|
||||
# cannot re-include a file whose PARENT DIRECTORY is excluded — that is why this must not
|
||||
# live under _artifacts/).
|
||||
#
|
||||
# The run LOG is the load-bearing half: Chainsaw's JUnit carries no stdout, and the L2/L3
|
||||
# tests self-skip GREEN, so the sentinel token in run.log is the only thing separating
|
||||
# "proven" from "skipped". Never commit a report.junit.xml without its run.log.
|
||||
!runs/
|
||||
!runs/**
|
||||
!runs/**/*.junit.xml
|
||||
@@ -18,11 +18,9 @@ behavioral / L3 fault-injection** (this harness).
|
||||
./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||
./run.sh --layer l2 # one layer: l1 | l2 | l3 | meta | all
|
||||
./run.sh --coverage-only # regenerate the coverage report (no cluster)
|
||||
./run.sh --status # derived requirement-STATUS report (no cluster)
|
||||
./run.sh --lint # structural-validate test defs (no cluster)
|
||||
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
./run.sh --require-all # phase-exit gate: every requirement must be covered
|
||||
./run.sh --record # run AND capture the evidence (see "Proof of pass")
|
||||
```
|
||||
|
||||
**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
|
||||
@@ -39,33 +37,6 @@ behavioral / L3 fault-injection** (this harness).
|
||||
used to be here; it now has a real L3 test — `tests/l3-fault-injection/node-loss` — and
|
||||
is covered, not exempted.)
|
||||
|
||||
### Coverage vs status — two different questions
|
||||
|
||||
`coverage.md` answers **"does a requirement-tagged test EXIST for R-x?"** (R-TEST-04).
|
||||
`status.md` (`./run.sh --status`, generator `lib/status.py`) answers the operator's
|
||||
question — **"what is the real state of each requirement given what is actually in the
|
||||
repo?"** — deriving three orthogonal axes per requirement:
|
||||
|
||||
1. **implementation** — is there a tracked artifact naming the requirement?
|
||||
2. **reconciliation** — is that artifact reachable by walking `resources:` from
|
||||
`clusters/<env>/kustomization.yaml` and expanding every Flux `Kustomization`'s
|
||||
`spec.path`? This is what separates **authored** from **actually reconciled**: a
|
||||
commented-out `resources:` entry (e.g. `platform/image-automation`) is correctly
|
||||
reported as in-git-but-not-on-any-cluster.
|
||||
3. **verification** — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts
|
||||
both as "covered"; status does not. TODO markers in a body always outrank an
|
||||
`executable: "true"` annotation, so ambiguity never upgrades a test.
|
||||
|
||||
**It deliberately refuses to derive proof-of-pass.** Nothing tracked in git records a test
|
||||
result — `_artifacts/` and `results/` are git-ignored and DECISIONS-LOG.md prose is not a
|
||||
signal — so every requirement's proof state is reported as `not-derivable` rather than
|
||||
defaulting to green. A status report that overstates completion is worse than none
|
||||
(`agents/definition-of-done.md`). Closing that gap needs a **committed, machine-readable
|
||||
run ledger**; until then the strongest claim the report makes is *wired and testable*.
|
||||
`--strict-coverage` also applies status's hygiene gate (registry↔spec drift,
|
||||
annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold).
|
||||
Exit codes follow `ci/lib.sh`: `0` pass / `1` fail / `3` skipped — a skip is never green.
|
||||
|
||||
`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
|
||||
the *same* script with the same config, so results are identical.
|
||||
|
||||
@@ -75,8 +46,6 @@ the *same* script with the same config, so results are identical.
|
||||
| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
|
||||
| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
|
||||
| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
|
||||
| `_artifacts/status/status.md` | Human requirement-STATUS report derived from repo state | operator / PM |
|
||||
| `_artifacts/status/status.json` | Machine status: implementation / reconciliation / verification per requirement | tooling |
|
||||
|
||||
## Reproducibility contract (R-TEST-02 — what the harness guarantees)
|
||||
- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
|
||||
@@ -105,90 +74,13 @@ requirements/registry.yaml mirror of EVERY spec R-* ID (coverage enumerates fr
|
||||
lib/
|
||||
common.sh tagging convention + test discovery/selection helpers
|
||||
coverage.py requirement-coverage generator (stdlib only)
|
||||
status.py requirement-STATUS generator (stdlib only; --selftest)
|
||||
record_run.py run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
|
||||
emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster)
|
||||
runs/
|
||||
ledger.jsonl TRACKED append-only proof-of-pass ledger (one row per
|
||||
testcase per run) — the only source status.py trusts
|
||||
<UTC>-<short-sha>/ TRACKED per-run evidence: run.log + report.junit.xml
|
||||
tests/
|
||||
l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md)
|
||||
incluster/
|
||||
chainsaw-job.yaml rbac.yaml in-cluster L2/L3 Job + scoped test SA (ADR-0011)
|
||||
```
|
||||
|
||||
## Proof of pass — the run ledger (`runs/`)
|
||||
|
||||
`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
|
||||
proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
|
||||
root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
|
||||
only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
|
||||
asked for — **a committed, machine-readable run ledger**.
|
||||
|
||||
```bash
|
||||
./run.sh --record # runs normally, and captures the evidence
|
||||
# -> conformance/runs/<UTC>-<short-sha>/run.log (tee'd, carries the sentinels)
|
||||
# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml (that run's JUnit)
|
||||
# -> conformance/runs/ledger.jsonl (one appended row per testcase)
|
||||
python3 lib/record_run.py --run-dir runs/<id> [--dry-run] # or record an existing run dir
|
||||
python3 lib/record_run.py --selftest # deterministic, no cluster
|
||||
```
|
||||
|
||||
**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
|
||||
that silently records itself is how a bad run becomes a permanent false proof. **Commit
|
||||
the run dir and the ledger** — an uncommitted ledger proves nothing.
|
||||
|
||||
### Why a committed JUnit alone would be WORSE than nothing
|
||||
|
||||
The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
|
||||
`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
|
||||
and `exit 0`, and their `check:` accepts either outcome —
|
||||
|
||||
```yaml
|
||||
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
|
||||
|| contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
|
||||
```
|
||||
|
||||
So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
|
||||
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
|
||||
A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
|
||||
R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.
|
||||
|
||||
**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
|
||||
`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
|
||||
`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
|
||||
verdict token), so the table below is never hand-maintained:
|
||||
|
||||
| Test | pass token (proof) | skip token (never proof) |
|
||||
|---|---|---|
|
||||
| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
|
||||
| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
|
||||
| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
|
||||
| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
|
||||
| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
|
||||
| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
|
||||
| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |
|
||||
|
||||
A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
|
||||
token present in that test's log lines, and **no** skip token anywhere in them. A skip
|
||||
token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
|
||||
`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
|
||||
non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
|
||||
skips as JUnit `<skipped/>`, never as a green pass.
|
||||
|
||||
### Staleness — a proof is a claim about a tree, not a badge
|
||||
|
||||
`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
|
||||
last commit touching that requirement's implementing manifests or its verifying test**.
|
||||
Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
|
||||
dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
|
||||
it never counts. Re-run with `--record` to refresh.
|
||||
|
||||
A requirement reads `proven` only when **every** verifying test does; one skipped, failed
|
||||
or unrecorded test drags the whole requirement down. With an empty ledger `--status`
|
||||
behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.
|
||||
|
||||
## How CI invokes it
|
||||
CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
|
||||
|
||||
|
||||
@@ -57,13 +57,6 @@ rules:
|
||||
verbs: [create, delete, get, list, watch, patch]
|
||||
- apiGroups: [""]
|
||||
resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
|
||||
# LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
|
||||
# agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
|
||||
# oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
|
||||
# time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
|
||||
# chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
|
||||
# If this rule is ever narrowed to the test's own namespace, those bindings fail and the
|
||||
# tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
|
||||
verbs: [create, delete, get, list, watch, patch]
|
||||
- apiGroups: [""]
|
||||
resources: [pods/exec]
|
||||
@@ -71,22 +64,6 @@ rules:
|
||||
- apiGroups: [""]
|
||||
resources: [pods/log]
|
||||
verbs: [get] # read fault/probe Job logs on failure
|
||||
# ---- Storage-tier precondition gate (READ-ONLY). Every storage test self-skips on the
|
||||
# absence of the `longhorn` StorageClass rather than failing: data-survival (R-STATE-01),
|
||||
# node-loss (R-STATE-02), dr-restore (R-STATE-03), vm-data-survival / vm-lifecycle /
|
||||
# vm-default-deny (R-PLAT-03). Without this read the gate cannot tell "absent" from
|
||||
# "forbidden" and would skip for the wrong reason.
|
||||
- apiGroups: [storage.k8s.io]
|
||||
resources: [storageclasses]
|
||||
verbs: [get, list]
|
||||
# ---- Controller-readiness preconditions (READ-ONLY). dr-restore (R-STATE-03) skips cleanly
|
||||
# unless the Velero Deployment and its node-agent DaemonSet (kopia File System Backup, the
|
||||
# volume-DATA path) are actually ready — a backup path that is not up is an absent
|
||||
# precondition, not a failure. No write verbs: tests never scale or patch a platform
|
||||
# controller.
|
||||
- apiGroups: [apps]
|
||||
resources: [deployments, daemonsets]
|
||||
verbs: [get, list, watch]
|
||||
# ---- Probe / fault Jobs (R-TLS-01/03 tls-probe; R-NET-06 tunnel stop/start)
|
||||
# `create` cannot be constrained by resourceNames, and the fault Job names are
|
||||
# fixed per-test (tunnel-stop/tunnel-start/tunnel-start-finally); scoped by the
|
||||
@@ -126,29 +103,10 @@ rules:
|
||||
# resources: [imagepolicies, imagerepositories]
|
||||
# verbs: [get, patch]
|
||||
#
|
||||
# ---- R-STATE-03 (DR restore, tests/l3-fault-injection/dr-restore): the drill creates a
|
||||
# Velero Backup of its OWN scratch namespace, destroys that namespace + its PV, creates a
|
||||
# Restore into a fresh namespace, and (in `finally`) issues a DeleteBackupRequest so the
|
||||
# objects it wrote are removed from the off-cluster bucket again. `create` cannot be
|
||||
# constrained by resourceNames; the blast radius is bounded by the fact that every object
|
||||
# the test creates or deletes is named `<its-own-chainsaw-namespace>-dr-*`.
|
||||
- apiGroups: [velero.io]
|
||||
resources: [backups, restores, deletebackuprequests]
|
||||
verbs: [create, delete, get, list, watch]
|
||||
# Read-only evidence: the BSL is the off-cluster target whose Available phase + endpoint the
|
||||
# test asserts; schedules prove backups are SCHEDULED (not one-off); podvolumebackups /
|
||||
# podvolumerestores prove the kopia snapshot of the volume DATA reached/returned from the
|
||||
# remote repository, and are dumped on failure for diagnosis.
|
||||
- apiGroups: [velero.io]
|
||||
resources: [backupstoragelocations, schedules, podvolumebackups, podvolumerestores]
|
||||
verbs: [get, list, watch]
|
||||
# R-STATE-03 needs `delete` on the PV, not merely read: the proof that the restore came from
|
||||
# OFF-cluster is that no in-cluster copy of the data survived. The test asserts its own PV is
|
||||
# gone, and force-deletes it if a Retain policy or a stuck finalizer left it behind. It only
|
||||
# ever names the PV bound to the PVC it created itself.
|
||||
- apiGroups: [""]
|
||||
resources: [persistentvolumes]
|
||||
verbs: [get, list, delete]
|
||||
# ---- R-STATE-03 (DR restore): trigger a Velero restore
|
||||
# - apiGroups: [velero.io]
|
||||
# resources: [restores, backups]
|
||||
# verbs: [create, get, list, watch]
|
||||
#
|
||||
# ---- R-GITOPS-03 (drift): mutate a reconciled resource + observe Flux revert
|
||||
# - apiGroups: [kustomize.toolkit.fluxcd.io]
|
||||
|
||||
@@ -47,7 +47,7 @@ conf_dirs_for_requirements() {
|
||||
done < <(conf_all_test_files) | sort -u
|
||||
}
|
||||
|
||||
# Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
|
||||
# Directories for a given layer (subtree): l1 | l2 | l3 | meta | all.
|
||||
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
|
||||
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
|
||||
# this is what lets `run.sh --layer l2,l3` (the ci/conformance.sh L2/L3 stage) select both
|
||||
@@ -58,7 +58,6 @@ conf_dirs_for_layer() {
|
||||
for layer in ${layers//,/ }; do
|
||||
case "$layer" in
|
||||
all) roots=("${CONF_ROOT}/tests"); break ;;
|
||||
l0) roots+=("${CONF_ROOT}/tests/l0-static") ;;
|
||||
l1) roots+=("${CONF_ROOT}/tests/l1-rendering") ;;
|
||||
l2) roots+=("${CONF_ROOT}/tests/l2-behavioral") ;;
|
||||
l3) roots+=("${CONF_ROOT}/tests/l3-fault-injection") ;;
|
||||
|
||||
@@ -115,9 +115,7 @@ def parse_test(path: Path) -> dict:
|
||||
# Layer inferred from path as a fallback / cross-check.
|
||||
rel = str(path)
|
||||
if not layer:
|
||||
if "l0-static" in rel:
|
||||
layer = "L0"
|
||||
elif "l1-rendering" in rel:
|
||||
if "l1-rendering" in rel:
|
||||
layer = "L1"
|
||||
elif "l2-behavioral" in rel:
|
||||
layer = "L2"
|
||||
|
||||
@@ -33,8 +33,7 @@ def collect(path: Path) -> tuple[str, list[str], str]:
|
||||
reqs = sorted({m.group(1).upper() for m in
|
||||
(LABEL_LINE_RE.match(l) for l in text.splitlines()) if m})
|
||||
rel = str(path)
|
||||
layer = ("L0" if "l0-static" in rel
|
||||
else "L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
||||
layer = ("L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
||||
else "L3" if "l3-fault-injection" in rel else "META")
|
||||
return name, reqs, layer
|
||||
|
||||
|
||||
@@ -1,949 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Append a conformance RUN LEDGER row per testcase — the committed proof-of-pass record.
|
||||
|
||||
WHY THIS EXISTS
|
||||
---------------
|
||||
`lib/status.py` reports PROOF as `not-derivable` for every requirement, and names its own
|
||||
fix in its docstring: *"a committed, machine-readable run ledger"*. There was none, because
|
||||
`conformance/_artifacts/` is git-ignored and the root `.gitignore` kills `*.junit.xml` and
|
||||
`coverage/` globally, so nothing about a green run ever reached repo state. This tool is
|
||||
that ledger's writer: it reads one run's JUnit + run log out of `conformance/runs/<run-id>/`
|
||||
and appends one JSON object per testcase to `conformance/runs/ledger.jsonl`, which IS
|
||||
tracked (see conformance/.gitignore for the negations that let it survive).
|
||||
|
||||
THE TRAP THIS TOOL EXISTS TO AVOID — READ BEFORE CHANGING ANYTHING
|
||||
------------------------------------------------------------------
|
||||
The L2/L3 tests are RUNNABLE / SELF-SKIPPING: when a precondition is absent (no `longhorn`
|
||||
StorageClass, no Velero CRDs, <3 Ready nodes) they echo a skip token and `exit 0`, and their
|
||||
`check:` block is deliberately written to accept either outcome, e.g. tests/l3-fault-
|
||||
injection/dr-restore:
|
||||
|
||||
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
|
||||
|| contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
|
||||
|
||||
So a SELF-SKIP exits 0 and Chainsaw records `passed` EXACTLY as a real run does. And
|
||||
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`
|
||||
(verified against the committed reports). Therefore:
|
||||
|
||||
*** A COMMITTED JUnit ALONE WOULD RECORD `DR_RESTORE_SKIPPED_OK` AS PROOF OF R-STATE-03. ***
|
||||
|
||||
AND THE SECOND TRAP, WHICH IS THE MIRROR OF THE FIRST
|
||||
------------------------------------------------------
|
||||
Chainsaw echoes each script's SOURCE into an indented `=== COMMAND` block before running
|
||||
it, so `echo NODE_LOSS_SKIPPED_OK` — the skip branch that was NOT taken — is present in
|
||||
the log of every run of that test. A section-blind grep therefore reads a skip token out
|
||||
of the source of every skip-gated test and can never record `proven` for any of them: the
|
||||
ledger fails closed so completely that it records nothing and proves nothing. Tokens are
|
||||
consequently read from SECTIONS (see classify_sections): a pass token only from
|
||||
`=== STDOUT`, a skip token from any section that is not script source. Skip still beats
|
||||
pass — within genuine output, which is the only place either was ever evidence.
|
||||
|
||||
That is the fake-green this project refuses, and it would be worse than no ledger at all
|
||||
because it would look authoritative. THE SENTINEL TOKEN IS THE ONLY THING THAT SEPARATES
|
||||
PROVEN FROM SKIPPED, AND IT EXISTS ONLY IN THE RUN LOG. Hence: this tool requires BOTH
|
||||
files, and for the Chainsaw layers it will NEVER emit `proof: "proven"` on a JUnit `passed`
|
||||
unless the log carries that test's own pass token and carries NO skip token.
|
||||
|
||||
HOW THE TOKENS ARE OBTAINED (derived, never hand-listed)
|
||||
--------------------------------------------------------
|
||||
Hard-coding a token table would rot the moment a test is edited. Instead every token is
|
||||
parsed out of the test's own `chainsaw-test.yaml`: each `contains($stdout, 'TOKEN')` in a
|
||||
`check:` block is a declared sentinel. Classification is by the project's naming convention
|
||||
(`*_SKIP*_OK` is a skip token, anything else is a pass token), and the TERMINAL pass token
|
||||
— the last non-skip token in file order — is the test's verdict token. Edit a test's tokens
|
||||
and this tool follows automatically; delete them and the test simply becomes unprovable
|
||||
rather than silently proven.
|
||||
|
||||
node-loss pass NODE_LOSS_VOLUME_AVAILABLE_OK skip NODE_LOSS_SKIPPED_OK
|
||||
dr-restore pass DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK skip DR_RESTORE_SKIPPED_OK
|
||||
(+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK / DR_SCHEDULE_SKIPPED_OK)
|
||||
|
||||
WHY L0/L1 NEED NO SENTINEL (and why that is not a loophole)
|
||||
-----------------------------------------------------------
|
||||
The offline layers are not run by Chainsaw. `lib/run_l1_offline.py` shells out to each
|
||||
dir's `check.sh` / `render-check.sh` and maps exit 3 to a JUnit `<skipped/>` element and
|
||||
any other non-zero to `<failure>`. A self-skip there is ALREADY a JUnit skip, never a green
|
||||
pass, so the JUnit is a truthful record on its own. Only the Chainsaw layers (L2/L3/META)
|
||||
can express "skipped" as "passed", and only they require a sentinel — see
|
||||
SENTINEL_REQUIRED_LAYERS.
|
||||
|
||||
LEDGER ROW (conformance/runs/ledger.jsonl — append-only, one object per testcase per run)
|
||||
ts_utc run-dir timestamp, else the JUnit's mtime (when)
|
||||
git_sha `git rev-parse HEAD`, suffixed `-dirty` when tracked files were
|
||||
modified — a proof captured from a dirty tree is NOT honoured by
|
||||
status.py, because what ran is not what is committed (what)
|
||||
env --env / CONF_ENV (cluster overlay the run targeted)
|
||||
chainsaw_version CHAINSAW_VERSION from versions.env (the pin IS the truth)
|
||||
kube_version `kubectl version` server gitVersion, else "unknown"
|
||||
layer L0|L1|L2|L3|META, from the test's `layer` annotation / JUnit property
|
||||
test testcase name (== the test dir basename, asserted by convention)
|
||||
requirements[] the `conformance.home.k3s/R-*` labels on the test — the SAME
|
||||
authoritative source coverage.py uses, not the JUnit classname
|
||||
junit_result passed | failed | error | skipped
|
||||
sentinel the DECIDING token grepped from run.log, or null
|
||||
proof proven | skipped | failed | junit-skipped
|
||||
| unknown-no-sentinel-declared | unknown-sentinel-absent
|
||||
run_id the runs/<dir> name (provenance: which log backs this row)
|
||||
note one line saying WHY this row landed on that proof state
|
||||
|
||||
`proven` is emitted on exactly one path (see decide()); every other path is non-green.
|
||||
|
||||
RECORDING IS EXPLICIT, NEVER IMPLICIT
|
||||
-------------------------------------
|
||||
Invoke this tool by hand, or via `run.sh --record`. A run that silently records itself is
|
||||
how a bad run becomes a permanent false proof, so `run.sh` without `--record` writes no
|
||||
ledger row at all.
|
||||
|
||||
EXIT CODES (ci/lib.sh convention: 0 pass / 1 fail / 3 skipped-not-green)
|
||||
0 rows appended.
|
||||
1 a real failure: unparsable JUnit, unreadable test tree, unwritable ledger.
|
||||
3 the inputs a ledger row needs are absent (no report.junit.xml, or no run.log).
|
||||
NOTHING is recorded and this is NEVER green — a run we cannot evidence is not a run.
|
||||
|
||||
Pure standard library, matching the harness's no-pip-dependency posture (coverage.py,
|
||||
status.py). PyYAML lives in catalog/render.py; the harness does not depend on it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
|
||||
_LIB = Path(__file__).resolve().parent
|
||||
if str(_LIB) not in sys.path:
|
||||
sys.path.insert(0, str(_LIB))
|
||||
# Importing a sibling would otherwise litter conformance/lib/__pycache__/ into the working
|
||||
# tree, and the lint stage scans untracked files. Same guard as status.py.
|
||||
sys.dont_write_bytecode = True
|
||||
import coverage as cov # noqa: E402 — reuse the registry/test parsers, don't fork them
|
||||
|
||||
CONF_ROOT = _LIB.parent
|
||||
REPO_ROOT = CONF_ROOT.parent
|
||||
DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
|
||||
|
||||
# The ONLY layers exempt from needing a pass sentinel: the offline ones, whose skips
|
||||
# lib/run_l1_offline.py already renders as JUnit <skipped/> rather than as a green pass
|
||||
# (see docstring). Written as an EXEMPT set, not a REQUIRED set, so that an unrecognised
|
||||
# or new layer fails CLOSED — it needs a sentinel — instead of quietly becoming provable
|
||||
# by JUnit alone.
|
||||
SENTINEL_EXEMPT_LAYERS = frozenset({"L0", "L1"})
|
||||
|
||||
# Layer from the test's subtree. Authoritative over the annotation: coverage.py's layer
|
||||
# regex stops at the first non-[L/A/M/0-9] character, so `layer: "META"` parses as "M" —
|
||||
# harmless for its own report, but here a mis-read layer decides whether a sentinel is
|
||||
# required at all, so it is derived from the path exactly as emit_skipped_junit.py does.
|
||||
_LAYER_BY_SUBTREE = (("/l0-static/", "L0"), ("/l1-rendering/", "L1"),
|
||||
("/l2-behavioral/", "L2"), ("/l3-fault-injection/", "L3"),
|
||||
("/meta/", "META"))
|
||||
|
||||
|
||||
def layer_of(path: Path) -> str:
|
||||
rel = str(path)
|
||||
for frag, layer in _LAYER_BY_SUBTREE:
|
||||
if frag in rel:
|
||||
return layer
|
||||
return ""
|
||||
|
||||
# A declared sentinel: `contains($stdout, 'TOKEN')` inside a Chainsaw `check:` block.
|
||||
# Both quote styles are accepted; the repo uses single quotes throughout.
|
||||
_SENTINEL_RE = re.compile(r"contains\(\s*\$stdout\s*,\s*['\"]([A-Za-z0-9_]+)['\"]\s*\)")
|
||||
|
||||
# Naming convention for a SKIP token (tests/l3-fault-injection/*: NODE_LOSS_SKIPPED_OK,
|
||||
# DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK, VM_DATA_SURVIVAL_SKIPPED_OK, ...).
|
||||
# Deliberately broad: mis-classifying a PASS token as a skip only ever loses a proof;
|
||||
# mis-classifying a SKIP token as a pass would manufacture one.
|
||||
_SKIP_TOKEN_RE = re.compile(r"_SKIP")
|
||||
|
||||
# A Chainsaw log line header, as it really appears (Go `t.Log` prefixes the source site and
|
||||
# pads it with backspaces, and the fields are ANSI-coloured — both are stripped first):
|
||||
# l.go:52: | 23:51:18 | node-loss | volume-stays-available... | SCRIPT | LOG |
|
||||
# Field 2 is the test name; it is how output is attributed back to a testcase. The
|
||||
# `<file>.go:<line>:` prefix is optional so a redirected/plain log still attributes.
|
||||
_LOG_HEADER_RE = re.compile(
|
||||
r"^\s*(?:[\w./-]+\.go:\d+:\s*)?"
|
||||
r"\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
|
||||
|
||||
# ANSI colour + the backspace padding Go writes ahead of a `t.Log` continuation. Stripped
|
||||
# before ANY parsing; neither ever appears at the head of a line, so indentation survives.
|
||||
_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
|
||||
|
||||
|
||||
def _strip_noise(line: str) -> str:
|
||||
return _ANSI_RE.sub("", line).replace("\b", "")
|
||||
|
||||
|
||||
# ===================================================================== output sections
|
||||
# THE BUG THIS SECTION MACHINERY EXISTS TO KILL — READ BEFORE CHANGING ANYTHING
|
||||
# -----------------------------------------------------------------------------
|
||||
# Chainsaw echoes the script it is ABOUT TO RUN, verbatim, in an indented `=== COMMAND`
|
||||
# block, and only afterwards prints what the script actually wrote in `=== STDOUT`. Every
|
||||
# skip-gated test contains its own skip token as SOURCE TEXT (`echo NODE_LOSS_SKIPPED_OK`),
|
||||
# so a raw whole-log grep finds that token in EVERY run of that test — including runs that
|
||||
# never took the skip branch. Combined with decide()'s (correct) rule that a skip token
|
||||
# outranks a pass token, a section-blind scan can never record `proven` for ANY test that
|
||||
# has a skip gate: the ledger fails closed so hard it records nothing. Observed on
|
||||
# runs/20260901T235117Z-e9a4b66, where NODE_LOSS_SKIPPED_OK appears twice as source at
|
||||
# lines 51/62 (`=== COMMAND`) and never in output, yet the run genuinely proved R-STATE-02.
|
||||
#
|
||||
# So tokens are attributed to SECTIONS, never to raw lines:
|
||||
# l.go:52: | 23:51:18 | node-loss | step | CMD | RUN | <- header, no section
|
||||
# === COMMAND <- marker (indent 8)
|
||||
# /usr/bin/sh -c set -euo pipefail <- source, indent >= 8
|
||||
# echo NODE_LOSS_SKIPPED_OK <- SOURCE, not output
|
||||
# l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG | <- indent 4 => section ends
|
||||
# === STDOUT <- marker
|
||||
# NODE_LOSS_VOLUME_AVAILABLE_OK <- the REAL emission
|
||||
#
|
||||
# A marker is an INDENTED `=== NAME` on a line of its own. Go's own test markers
|
||||
# (`=== RUN chainsaw`, `=== PAUSE`, `=== CONT`) are at column 0 and carry a test name
|
||||
# after the verb, so they match neither the indent nor the end-of-line anchor; they are
|
||||
# additionally named in _GO_TEST_MARKERS so a future indented variant still cannot open a
|
||||
# section that would swallow the whole log.
|
||||
_SECTION_MARKER_RE = re.compile(r"^(\s+)=== ([A-Z][A-Z0-9_]*)\s*$")
|
||||
_GO_TEST_MARKERS = frozenset({"RUN", "PAUSE", "CONT"})
|
||||
|
||||
# Pseudo-section for a log that contains NO section markers at all (the plain/redirected
|
||||
# logs the earlier ledger rows were written from, and the selftest fixtures). Such a log
|
||||
# never echoes script source, so there is nothing to be confused by and the legacy
|
||||
# whole-log scan is exactly as safe as it was — narrowing is only needed where a COMMAND
|
||||
# echo exists. Lines OUTSIDE a section in a log that DOES use sections are attributed to
|
||||
# "" and are evidence for nothing.
|
||||
UNSECTIONED = "*"
|
||||
|
||||
# A pass token counts ONLY where Chainsaw's own `contains($stdout, ...)` check looks:
|
||||
# real stdout. Anything else — source echo, stderr, an unrecognised future section — can
|
||||
# never manufacture a proof.
|
||||
PASS_SECTIONS = frozenset({"STDOUT", UNSECTIONED})
|
||||
|
||||
# Sections that carry SOURCE rather than output. A skip token is honoured in every OTHER
|
||||
# section (STDOUT, STDERR, any section name added by a later Chainsaw release), because
|
||||
# over-reading a skip only ever COSTS a proof, while over-reading a pass would fabricate
|
||||
# one. Unknown section names therefore fail closed in both directions.
|
||||
SOURCE_SECTIONS = frozenset({"COMMAND"})
|
||||
|
||||
|
||||
def classify_sections(log_text: str) -> list[tuple[str, str]]:
|
||||
"""[(section, line)] for every line of a run log.
|
||||
|
||||
`section` is the enclosing `=== NAME` block, "" for a line inside no block, or
|
||||
UNSECTIONED for every line when the log declares no sections at all. A block runs from
|
||||
its marker until the first non-blank line indented LESS than the marker (blank lines do
|
||||
not end it) or until the next marker; the marker line itself is content of nothing.
|
||||
"""
|
||||
rows: list[tuple[str, str]] = []
|
||||
cur, cur_indent, sectioned = "", 0, False
|
||||
for raw in log_text.splitlines():
|
||||
line = _strip_noise(raw)
|
||||
m = _SECTION_MARKER_RE.match(line)
|
||||
if m and m.group(2) not in _GO_TEST_MARKERS:
|
||||
sectioned = True
|
||||
cur, cur_indent = m.group(2), len(m.group(1))
|
||||
rows.append(("", line))
|
||||
continue
|
||||
if cur:
|
||||
if not line.strip() or len(line) - len(line.lstrip()) >= cur_indent:
|
||||
rows.append((cur, line))
|
||||
continue
|
||||
cur = ""
|
||||
rows.append(("", line))
|
||||
if not sectioned:
|
||||
return [(UNSECTIONED, line) for _, line in rows]
|
||||
return rows
|
||||
|
||||
|
||||
def scan_texts(lines) -> tuple[str, str]:
|
||||
"""(text a PASS token may be read from, text a SKIP token may be read from).
|
||||
|
||||
Accepts either [(section, line)] pairs from classify_sections() or a plain list of
|
||||
strings, which is treated as an unsectioned log (legacy whole-log scanning).
|
||||
"""
|
||||
pass_lines, skip_lines = [], []
|
||||
for item in lines:
|
||||
section, line = item if isinstance(item, tuple) else (UNSECTIONED, item)
|
||||
if section in PASS_SECTIONS:
|
||||
pass_lines.append(line)
|
||||
if section and section not in SOURCE_SECTIONS:
|
||||
skip_lines.append(line)
|
||||
return "\n".join(pass_lines), "\n".join(skip_lines)
|
||||
|
||||
_RUN_ID_TS_RE = re.compile(r"^(\d{8}T\d{6}Z)-")
|
||||
|
||||
|
||||
# ========================================================================= sentinels
|
||||
def parse_sentinels(test_yaml: Path) -> dict:
|
||||
"""Declared sentinel tokens of one Chainsaw test, derived from the test file itself.
|
||||
|
||||
Returns {"pass": [...], "skip": [...], "terminal_pass": str|None}. `terminal_pass` is
|
||||
the LAST non-skip token in file order — the test's verdict token, i.e. the one its
|
||||
final `check:` block asserts (dr-restore -> DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK).
|
||||
A test with no `contains($stdout, ...)` assertion declares no sentinel and is reported
|
||||
unprovable rather than proven; see decide().
|
||||
"""
|
||||
try:
|
||||
text = test_yaml.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return {"pass": [], "skip": [], "terminal_pass": None}
|
||||
toks = _SENTINEL_RE.findall(text)
|
||||
passes, skips = [], []
|
||||
for t in toks:
|
||||
(skips if _SKIP_TOKEN_RE.search(t) else passes).append(t)
|
||||
# De-duplicate, preserving order (dr-restore names DR_RESTORE_SKIPPED_OK twice).
|
||||
def uniq(xs: list[str]) -> list[str]:
|
||||
return list(dict.fromkeys(xs))
|
||||
passes, skips = uniq(passes), uniq(skips)
|
||||
return {"pass": passes, "skip": skips,
|
||||
"terminal_pass": passes[-1] if passes else None}
|
||||
|
||||
|
||||
def sentinel_map(tests_dir: Path) -> dict[str, dict]:
|
||||
"""{test-name: sentinels} for every test in the tree.
|
||||
|
||||
Keyed on the Chainsaw `metadata.name`, which is what lands in the JUnit `<testcase
|
||||
name=...>`. This repo keeps that equal to the test's directory basename; the mapping
|
||||
is by name, so it stays correct if that ever drifts.
|
||||
"""
|
||||
out: dict[str, dict] = {}
|
||||
for f in sorted(tests_dir.rglob("chainsaw-test.yaml")):
|
||||
meta = cov.parse_test(f)
|
||||
name = meta["name"] or f.parent.name
|
||||
s = parse_sentinels(f)
|
||||
s["dir"] = str(f.parent)
|
||||
s["layer"] = layer_of(f) or meta["layer"]
|
||||
s["requirements"] = meta["labels"]
|
||||
out[name] = s
|
||||
return out
|
||||
|
||||
|
||||
# =============================================================================== log
|
||||
def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[tuple[str, str]]],
|
||||
bool]:
|
||||
"""Split a run log into per-testcase buckets of (section, line) pairs.
|
||||
|
||||
Chainsaw prefixes its log lines with `| HH:MM:SS | <test> | <step> | ...` and prints a
|
||||
script's sections underneath that header, so the test name is carried by a state
|
||||
machine: a header naming a KNOWN test switches the current bucket, every following line
|
||||
lands in it. Requiring the name to be known is what stops run.sh's own `[run.sh]`
|
||||
chatter (and any other pipe-shaped line) from opening a bogus bucket.
|
||||
|
||||
Section labels come from classify_sections() and travel WITH the lines, so a bucket
|
||||
still knows which of its lines were script source and which were real output.
|
||||
|
||||
Returns (buckets, attributed) where `attributed` is False when the log contained no
|
||||
recognisable header at all — the signal for the degraded global scan in decide().
|
||||
"""
|
||||
buckets: dict[str, list[tuple[str, str]]] = {}
|
||||
cur: str | None = None
|
||||
attributed = False
|
||||
for section, line in classify_sections(log_text):
|
||||
m = _LOG_HEADER_RE.match(line)
|
||||
if m and m.group(1) in known:
|
||||
cur = m.group(1)
|
||||
attributed = True
|
||||
if cur is not None:
|
||||
buckets.setdefault(cur, []).append((section, line))
|
||||
return buckets, attributed
|
||||
|
||||
|
||||
def unique_tokens(smap: dict[str, dict]) -> set[str]:
|
||||
"""Tokens claimed by EXACTLY ONE test in the tree.
|
||||
|
||||
Only these may be used by the degraded global scan: `DNS_OK` is echoed by two different
|
||||
tests, so in an unattributable log it can never be evidence for either; but
|
||||
`NODE_LOSS_VOLUME_AVAILABLE_OK` names precisely one test and still can.
|
||||
"""
|
||||
owners: dict[str, set[str]] = {}
|
||||
for name, s in smap.items():
|
||||
for t in s["pass"] + s["skip"]:
|
||||
owners.setdefault(t, set()).add(name)
|
||||
return {t for t, o in owners.items() if len(o) == 1}
|
||||
|
||||
|
||||
# ============================================================================ decide
|
||||
def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
|
||||
scope: str) -> tuple[str | None, str, str]:
|
||||
"""The ONE place a testcase becomes `proven` — or, far more often, does not.
|
||||
|
||||
Returns (sentinel, proof, note).
|
||||
|
||||
Ordering is worst-first ON PURPOSE, so that ambiguity can only ever cost a proof:
|
||||
1. JUnit failure/error -> failed (a red run is recorded as red)
|
||||
2. JUnit <skipped/> -> junit-skipped (never green; ci/lib.sh rule)
|
||||
3. ANY declared skip token seen -> skipped (BEATS a green JUnit and beats
|
||||
IN REAL OUTPUT the pass token: if any gated
|
||||
step self-skipped, the run did
|
||||
not prove the requirement)
|
||||
4. no sentinel declared -> proven only for L0/L1, whose skips are already
|
||||
JUnit skips; unknown-no-sentinel-declared for
|
||||
the Chainsaw layers
|
||||
5. EVERY declared pass token seen -> proven
|
||||
6. anything else -> unknown-sentinel-absent
|
||||
Step 5 requires ALL pass tokens, not just the terminal one: every Chainsaw step here
|
||||
asserts `pass || skip`, so a missing pass token means a step did not reach its proven
|
||||
branch, whatever the JUnit says.
|
||||
|
||||
Steps 3 and 5 read from SECTION-SCOPED text, never from raw lines: a pass token counts
|
||||
only in `=== STDOUT`, a skip token counts in any section that is not script source.
|
||||
That narrows WHERE a token is read from and changes nothing about what it means — skip
|
||||
still outranks pass, and it outranks it within genuine output, which is the only place
|
||||
either of them was ever evidence. `lines` may be [(section, line)] pairs or plain
|
||||
strings (an unsectioned log; see scan_texts / UNSECTIONED).
|
||||
"""
|
||||
if junit_result in ("failed", "error"):
|
||||
return (None, "failed", f"JUnit recorded {junit_result}")
|
||||
if junit_result == "skipped":
|
||||
return (None, "junit-skipped",
|
||||
"JUnit <skipped/> — the test did not execute; a skip is never green")
|
||||
|
||||
pass_text, skip_text = scan_texts(lines)
|
||||
seen_skip = [t for t in sent["skip"] if t in skip_text]
|
||||
if seen_skip:
|
||||
return (seen_skip[0], "skipped",
|
||||
f"run log OUTPUT carries the SELF-SKIP token {seen_skip[0]!r} — the test "
|
||||
f"exited 0 without exercising the behaviour, so this is NOT proof")
|
||||
|
||||
if not sent["pass"] and not sent["skip"]:
|
||||
if layer not in SENTINEL_EXEMPT_LAYERS:
|
||||
return (None, "unknown-no-sentinel-declared",
|
||||
f"{layer or 'unknown-layer'} test declares no contains($stdout, ...) "
|
||||
f"sentinel, so a green JUnit cannot be told apart from a self-skip — "
|
||||
f"refusing to call it proven")
|
||||
return (None, "proven",
|
||||
f"{layer} offline check exited 0; run_l1_offline.py renders its skips as "
|
||||
f"JUnit <skipped/>, so a green JUnit is truthful without a sentinel")
|
||||
|
||||
missing = [t for t in sent["pass"] if t not in pass_text]
|
||||
if not missing:
|
||||
return (sent["terminal_pass"], "proven",
|
||||
f"run log OUTPUT carries every declared pass token "
|
||||
f"({', '.join(sent['pass'])}) and no skip token [{scope}]")
|
||||
return (None, "unknown-sentinel-absent",
|
||||
f"run log OUTPUT lacks pass token(s) {', '.join(missing)} for this test "
|
||||
f"[{scope}] — refusing to infer proof from the JUnit alone")
|
||||
|
||||
|
||||
# ============================================================================= junit
|
||||
def junit_cases(path: Path) -> list[dict]:
|
||||
"""[{name, layer, requirements, result}] for every <testcase> in a JUnit report."""
|
||||
root = ET.parse(path).getroot()
|
||||
cases: list[dict] = []
|
||||
for c in root.iter("testcase"):
|
||||
if c.find("failure") is not None:
|
||||
result = "failed"
|
||||
elif c.find("error") is not None:
|
||||
result = "error"
|
||||
elif c.find("skipped") is not None:
|
||||
result = "skipped"
|
||||
else:
|
||||
result = "passed"
|
||||
props = {p.get("name"): (p.get("value") or "")
|
||||
for p in c.iter("property") if p.get("name")}
|
||||
reqs = sorted(set(cov.RID_RE.findall(props.get("requirements", ""))))
|
||||
layer = (props.get("layer") or "").upper()
|
||||
if not layer:
|
||||
layer = (c.get("classname") or "").split(":", 1)[0].upper()
|
||||
cases.append({"name": c.get("name") or "?", "layer": layer,
|
||||
"requirements": reqs, "result": result})
|
||||
return cases
|
||||
|
||||
|
||||
# =============================================================================== git
|
||||
def git_sha(repo: Path) -> str:
|
||||
"""HEAD, suffixed `-dirty` when TRACKED files differ from it.
|
||||
|
||||
Untracked files are deliberately ignored, matching status.py's `tracked_files()`
|
||||
posture (untracked scratch is not repo state) — and because a `--record` run creates
|
||||
untracked `runs/<id>/` files as it goes, which would otherwise mark every run dirty.
|
||||
A `-dirty` sha is not a valid rev, so status.py's ancestry check cannot resolve it and
|
||||
the proof degrades to `stale-proof`: correct, since what ran is not what is committed.
|
||||
"""
|
||||
try:
|
||||
sha = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
|
||||
capture_output=True, text=True, check=True).stdout.strip()
|
||||
except (subprocess.CalledProcessError, OSError):
|
||||
return "unknown"
|
||||
dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--"],
|
||||
capture_output=True).returncode != 0
|
||||
return f"{sha}-dirty" if dirty else sha
|
||||
|
||||
|
||||
def pinned_chainsaw_version(conf_root: Path) -> str:
|
||||
"""CHAINSAW_VERSION from versions.env — the pin is the single source of truth."""
|
||||
f = conf_root / "versions.env"
|
||||
if f.is_file():
|
||||
m = re.search(r'^CHAINSAW_VERSION="?([^"\n]+)"?', f.read_text(), re.M)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return "unknown"
|
||||
|
||||
|
||||
def server_kube_version() -> str:
|
||||
"""Server gitVersion via `kubectl version` (read-only), else "unknown"."""
|
||||
for args in (["kubectl", "version", "-o", "json"],
|
||||
["kubectl", "version", "--output=json"]):
|
||||
try:
|
||||
p = subprocess.run(args, capture_output=True, text=True, timeout=15)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
continue
|
||||
if p.returncode != 0 or not p.stdout.strip():
|
||||
continue
|
||||
try:
|
||||
v = json.loads(p.stdout).get("serverVersion") or {}
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if v.get("gitVersion"):
|
||||
return v["gitVersion"]
|
||||
return "unknown"
|
||||
|
||||
|
||||
def run_timestamp(run_dir: Path, junit: Path) -> str:
|
||||
"""UTC stamp: the run-dir name's own `<UTC>-<sha>` prefix, else the JUnit's mtime."""
|
||||
m = _RUN_ID_TS_RE.match(run_dir.name)
|
||||
if m:
|
||||
t = m.group(1)
|
||||
return f"{t[0:4]}-{t[4:6]}-{t[6:8]}T{t[9:11]}:{t[11:13]}:{t[13:15]}Z"
|
||||
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(junit.stat().st_mtime))
|
||||
|
||||
|
||||
# ============================================================================ record
|
||||
def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
|
||||
kube_version: str, sha: str) -> list[dict]:
|
||||
junit = run_dir / "report.junit.xml"
|
||||
log = run_dir / "run.log"
|
||||
smap = sentinel_map(tests_dir)
|
||||
log_text = log.read_text(errors="replace")
|
||||
buckets, attributed = attribute_log(log_text, set(smap))
|
||||
uniq = unique_tokens(smap)
|
||||
all_pairs = classify_sections(log_text)
|
||||
sectioned = any(sec not in ("", UNSECTIONED) for sec, _ in all_pairs)
|
||||
section_note = ("tokens read from real-output sections only"
|
||||
if sectioned else "log declares no === sections; whole-log scan")
|
||||
ts = run_timestamp(run_dir, junit)
|
||||
|
||||
rows: list[dict] = []
|
||||
for case in junit_cases(junit):
|
||||
name = case["name"]
|
||||
sent = smap.get(name, {"pass": [], "skip": [], "terminal_pass": None,
|
||||
"layer": case["layer"], "requirements": []})
|
||||
layer = sent.get("layer") or case["layer"] or "?"
|
||||
# Requirements come from the test's LABELS (coverage.py's authoritative source);
|
||||
# the JUnit property is the fallback for a testcase with no matching test file.
|
||||
reqs = sent.get("requirements") or case["requirements"]
|
||||
|
||||
if name in buckets:
|
||||
lines, scope = buckets[name], f"log scoped to this testcase; {section_note}"
|
||||
elif not attributed:
|
||||
# Degraded mode: the log had no recognisable Chainsaw header (a redirected or
|
||||
# reformatted log). Fall back to a whole-log scan, but ONLY for tokens owned by
|
||||
# exactly one test, so a token another test emitted can never prove this one.
|
||||
lines = all_pairs
|
||||
sent = dict(sent,
|
||||
**{"pass": [t for t in sent["pass"] if t in uniq],
|
||||
"skip": [t for t in sent["skip"] if t in uniq]})
|
||||
sent["terminal_pass"] = sent["pass"][-1] if sent["pass"] else None
|
||||
scope = ("DEGRADED: unattributable log, globally-unique tokens only; "
|
||||
+ section_note)
|
||||
else:
|
||||
lines, scope = [], "no log lines attributed to this testcase"
|
||||
|
||||
sentinel, proof, note = decide(layer, case["result"], sent, lines, scope)
|
||||
rows.append({
|
||||
"ts_utc": ts,
|
||||
"git_sha": sha,
|
||||
"env": env,
|
||||
"chainsaw_version": chainsaw_version,
|
||||
"kube_version": kube_version,
|
||||
"layer": layer,
|
||||
"test": name,
|
||||
"requirements": reqs,
|
||||
"junit_result": case["result"],
|
||||
"sentinel": sentinel,
|
||||
"proof": proof,
|
||||
"run_id": run_dir.name,
|
||||
"note": note,
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
def append_rows(ledger: Path, rows: list[dict]) -> None:
|
||||
"""Append-only, one compact JSON object per line. Never rewrites existing rows."""
|
||||
ledger.parent.mkdir(parents=True, exist_ok=True)
|
||||
with ledger.open("a", encoding="utf-8") as fh:
|
||||
for r in rows:
|
||||
fh.write(json.dumps(r, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
# ========================================================================== selftest
|
||||
def _selftest() -> int:
|
||||
"""Deterministic, no-cluster self-test of every decision rule (house pattern:
|
||||
`status.py --selftest`, `catalog/render.py --selftest`). Throwaway tree only.
|
||||
|
||||
The load-bearing case is `SKIP sentinel does NOT produce proven`: it is the single
|
||||
assertion standing between this ledger and a permanent false proof of R-STATE-03.
|
||||
"""
|
||||
import tempfile
|
||||
ok = True
|
||||
|
||||
def check(label: str, cond: bool) -> None:
|
||||
nonlocal ok
|
||||
if not cond:
|
||||
sys.stderr.write(f"record_run.py selftest: FAIL — {label}\n")
|
||||
ok = False
|
||||
|
||||
# --- token derivation, against the dr-restore/node-loss shape ---------------------
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
root = Path(td)
|
||||
tdir = root / "tests" / "l3-fault-injection" / "dr-restore"
|
||||
tdir.mkdir(parents=True)
|
||||
(tdir / "chainsaw-test.yaml").write_text(
|
||||
"apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n"
|
||||
" name: dr-restore\n labels:\n"
|
||||
" conformance.home.k3s/R-STATE-03: \"true\"\n annotations:\n"
|
||||
" conformance.home.k3s/requirements: \"R-STATE-03\"\n"
|
||||
" conformance.home.k3s/layer: \"L3\"\nspec:\n steps:\n"
|
||||
" - script:\n content: 'echo OFF_CLUSTER_TARGET_OK'\n"
|
||||
" check:\n"
|
||||
" (contains($stdout, 'OFF_CLUSTER_TARGET_OK')"
|
||||
" || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n"
|
||||
" - script:\n content: 'echo x'\n check:\n"
|
||||
" (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')"
|
||||
" || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n")
|
||||
s = parse_sentinels(tdir / "chainsaw-test.yaml")
|
||||
check("pass tokens derived from the test file",
|
||||
s["pass"] == ["OFF_CLUSTER_TARGET_OK",
|
||||
"DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"])
|
||||
check("skip token classified by the *_SKIP*_OK convention, de-duplicated",
|
||||
s["skip"] == ["DR_RESTORE_SKIPPED_OK"])
|
||||
check("terminal pass token is the LAST non-skip token",
|
||||
s["terminal_pass"] == "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK")
|
||||
|
||||
smap = sentinel_map(root / "tests")
|
||||
check("sentinel_map keys on metadata.name and carries layer + labels",
|
||||
smap["dr-restore"]["layer"] == "L3"
|
||||
and smap["dr-restore"]["requirements"] == ["R-STATE-03"])
|
||||
|
||||
# --- decide(): the SKIP-is-never-proven contract ----------------------------------
|
||||
l3 = {"pass": ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"],
|
||||
"skip": ["DR_RESTORE_SKIPPED_OK"],
|
||||
"terminal_pass": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"}
|
||||
|
||||
# *** THE CASE THIS WHOLE FILE EXISTS FOR ***
|
||||
sent, proof, _ = decide("L3", "passed", l3, ["DR_RESTORE_SKIPPED_OK"], "t")
|
||||
check("SKIP sentinel on a GREEN JUnit is 'skipped', NEVER 'proven'",
|
||||
proof == "skipped" and sent == "DR_RESTORE_SKIPPED_OK")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
|
||||
check("pass sentinel on a green JUnit is 'proven'", proof == "proven")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK",
|
||||
"DR_RESTORE_SKIPPED_OK"], "t")
|
||||
check("a skip token BEATS a pass token in the same log (worst wins)",
|
||||
proof == "skipped")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3, ["nothing relevant here"], "t")
|
||||
check("green JUnit with no sentinel in the log is unknown, not proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3, [], "t")
|
||||
check("green JUnit with NO log lines attributed is unknown, not proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
_, proof, _ = decide("L3", "failed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
|
||||
check("a failing JUnit outranks a pass token in the log", proof == "failed")
|
||||
|
||||
_, proof, _ = decide("L3", "skipped", l3, [], "t")
|
||||
check("JUnit <skipped/> records junit-skipped, never green",
|
||||
proof == "junit-skipped")
|
||||
|
||||
multi = {"pass": ["A_OK", "B_OK"], "skip": [], "terminal_pass": "B_OK"}
|
||||
_, proof, _ = decide("L3", "passed", multi, ["B_OK"], "t")
|
||||
check("every declared pass token must be present, not just the terminal one",
|
||||
proof == "unknown-sentinel-absent")
|
||||
_, proof, _ = decide("L3", "passed", multi, ["A_OK", "B_OK"], "t")
|
||||
check("all pass tokens present == proven", proof == "proven")
|
||||
|
||||
none = {"pass": [], "skip": [], "terminal_pass": None}
|
||||
_, proof, _ = decide("L2", "passed", none, ["whatever"], "t")
|
||||
check("a Chainsaw test declaring no sentinel is NOT proven by a green JUnit",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
_, proof, _ = decide("L1", "passed", none, [], "t")
|
||||
check("L0/L1 offline checks are proven by JUnit (their skips are JUnit skips)",
|
||||
proof == "proven")
|
||||
_, proof, _ = decide("L0", "skipped", none, [], "t")
|
||||
check("an L0 JUnit skip is still never green", proof == "junit-skipped")
|
||||
_, proof, _ = decide("META", "passed", none, [], "t")
|
||||
check("META (a Chainsaw layer) needs a sentinel like L2/L3",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
_, proof, _ = decide("", "passed", none, [], "t")
|
||||
check("an unknown/blank layer fails CLOSED — sentinel required",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
|
||||
# --- SECTION SCOPING: the COMMAND echo is SOURCE, never evidence -------------------
|
||||
# Regression cases for the ledger's first real L3 run, which recorded a genuine PASS as
|
||||
# `skipped` because `echo NODE_LOSS_SKIPPED_OK` — the skip branch NOT taken — is echoed
|
||||
# inside `=== COMMAND` on every run of a skip-gated test.
|
||||
nl = {"pass": ["NODE_LOSS_VOLUME_AVAILABLE_OK"],
|
||||
"skip": ["NODE_LOSS_SKIPPED_OK"],
|
||||
"terminal_pass": "NODE_LOSS_VOLUME_AVAILABLE_OK"}
|
||||
_CMD = (" l.go:52: | 23:51:18 | node-loss | step | CMD | RUN |\n"
|
||||
" === COMMAND\n"
|
||||
" /usr/bin/sh -c set -euo pipefail\n"
|
||||
" if ! kubectl get storageclass longhorn >/dev/null 2>&1; then\n"
|
||||
" echo NODE_LOSS_SKIPPED_OK\n"
|
||||
"\n"
|
||||
" exit 0\n"
|
||||
" fi\n"
|
||||
" echo NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |\n")
|
||||
|
||||
ran = classify_sections(_CMD + " === STDOUT\n"
|
||||
" ready nodes: 3\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" l.go:52: | 23:52:42 | node-loss | step | SCRIPT | DONE |\n")
|
||||
sent, proof, _ = decide("L3", "passed", nl, ran, "t")
|
||||
check("skip token ONLY in the COMMAND echo + pass token in STDOUT is PROVEN",
|
||||
proof == "proven" and sent == "NODE_LOSS_VOLUME_AVAILABLE_OK")
|
||||
|
||||
selfskipped = classify_sections(_CMD + " === STDOUT\n"
|
||||
" SKIP: 'longhorn' StorageClass absent\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
sent, proof, _ = decide("L3", "passed", nl, selfskipped, "t")
|
||||
check("skip token GENUINELY in STDOUT is still 'skipped' (the guarantee holds)",
|
||||
proof == "skipped" and sent == "NODE_LOSS_SKIPPED_OK")
|
||||
|
||||
neither = classify_sections(_CMD + " === STDOUT\n ready nodes: 3\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, neither, "t")
|
||||
check("both tokens in COMMAND and neither in STDOUT is NOT proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
both_out = classify_sections(" === STDOUT\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, both_out, "t")
|
||||
check("skip still BEATS pass when both are in genuine output", proof == "skipped")
|
||||
|
||||
stderr_skip = classify_sections(" === STDERR\n NODE_LOSS_SKIPPED_OK\n"
|
||||
" === STDOUT\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, stderr_skip, "t")
|
||||
check("a skip token in a non-source section (STDERR) still costs the proof",
|
||||
proof == "skipped")
|
||||
|
||||
stderr_pass = classify_sections(" === STDERR\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, stderr_pass, "t")
|
||||
check("a pass token OUTSIDE stdout cannot prove anything",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
# --- classify_sections: markers, extents, and Go's own `=== RUN` --------------------
|
||||
gorows = classify_sections("=== RUN chainsaw\n=== PAUSE chainsaw\n"
|
||||
"=== CONT chainsaw/node-loss\n"
|
||||
" === COMMAND\n echo NODE_LOSS_SKIPPED_OK\n"
|
||||
" l.go:52: | 1:00:00 | node-loss | s | SCRIPT | DONE |\n"
|
||||
"--- PASS: chainsaw/node-loss (131.09s)\n")
|
||||
secs = [sec for sec, _ in gorows]
|
||||
check("Go's column-0 `=== RUN/PAUSE/CONT` never open a section",
|
||||
secs[:3] == ["", "", ""])
|
||||
check("a section covers its indented body and ends at a less-indented line",
|
||||
secs[3] == "" and secs[4] == "COMMAND" and secs[5] == "" and secs[6] == "")
|
||||
check("a log with no === markers stays wholly scannable (UNSECTIONED)",
|
||||
{sec for sec, _ in classify_sections("plain line\n another\n")} == {UNSECTIONED})
|
||||
|
||||
# --- the preserved run itself, as a fixture ----------------------------------------
|
||||
# runs/20260901T235117Z-e9a4b66 IS the bug: NODE_LOSS_SKIPPED_OK twice in `=== COMMAND`
|
||||
# (source), never in output; NODE_LOSS_VOLUME_AVAILABLE_OK once in `=== STDOUT`.
|
||||
fixture = CONF_ROOT / "runs" / "20260901T235117Z-e9a4b66"
|
||||
if (fixture / "run.log").is_file() and (fixture / "report.junit.xml").is_file():
|
||||
rows = build_rows(fixture, CONF_ROOT / "tests", "dev", "v0.2.12", "v1.36.4", "sha")
|
||||
row = next((r for r in rows if r["test"] == "node-loss"), None)
|
||||
check("preserved real L3 run records node-loss/R-STATE-02 as proven",
|
||||
row is not None and row["proof"] == "proven"
|
||||
and row["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK"
|
||||
and row["requirements"] == ["R-STATE-02"])
|
||||
check("preserved real L3 run attributes (no DEGRADED fallback needed)",
|
||||
row is not None and "DEGRADED" not in row["note"])
|
||||
pairs = classify_sections((fixture / "run.log").read_text(errors="replace"))
|
||||
secs = {sec for sec, line in pairs if "NODE_LOSS_SKIPPED_OK" in line}
|
||||
check("every NODE_LOSS_SKIPPED_OK in the preserved log is COMMAND source",
|
||||
secs == {"COMMAND"})
|
||||
secs = {sec for sec, line in pairs
|
||||
if line.strip() == "NODE_LOSS_VOLUME_AVAILABLE_OK"}
|
||||
check("the real emission in the preserved log is in STDOUT", secs == {"STDOUT"})
|
||||
|
||||
# --- log attribution --------------------------------------------------------------
|
||||
log = ("[run.sh] Coverage report first\n"
|
||||
" | 20:41:32 | node-loss | step | SCRIPT | RUN |\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n"
|
||||
" | 20:44:01 | dr-restore | step | SCRIPT | RUN |\n"
|
||||
" DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK\n")
|
||||
b, attributed = attribute_log(log, {"node-loss", "dr-restore"})
|
||||
check("log lines are attributed to the testcase whose header preceded them",
|
||||
attributed
|
||||
and any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["node-loss"])
|
||||
and not any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["dr-restore"]))
|
||||
check("a pass token does NOT leak across testcase boundaries",
|
||||
not any("DR_RESTORED" in l for _, l in b["node-loss"]))
|
||||
check("an unsectioned log keeps whole-log scanning (nothing narrowed away)",
|
||||
all(sec == UNSECTIONED for sec, _ in b["node-loss"]))
|
||||
|
||||
# Chainsaw's real headers carry a Go `t.Log` source prefix and ANSI colour; the earlier
|
||||
# regex matched neither, so every genuine Chainsaw log silently fell into DEGRADED mode.
|
||||
real_hdr = (" l.go:52: \x08\x08\x08| 23:51:18 | \x1b[32;1mnode-loss\x1b[0;22m | "
|
||||
"\x1b[32;1mstep\x1b[0;22m | SCRIPT | LOG |\n"
|
||||
" === STDOUT\n NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
b2, attributed_real = attribute_log(real_hdr, {"node-loss"})
|
||||
check("a REAL Chainsaw header (go source prefix + ANSI) attributes, not degrades",
|
||||
attributed_real and "node-loss" in b2
|
||||
and ("STDOUT", " NODE_LOSS_VOLUME_AVAILABLE_OK") in b2["node-loss"])
|
||||
_, attributed2 = attribute_log("no chainsaw headers at all\n", {"node-loss"})
|
||||
check("an unattributable log is reported as such (degraded mode)", not attributed2)
|
||||
|
||||
smap2 = {"a": {"pass": ["DNS_OK", "A_ONLY_OK"], "skip": []},
|
||||
"b": {"pass": ["DNS_OK"], "skip": []}}
|
||||
check("only tokens owned by exactly one test may be used in degraded mode",
|
||||
unique_tokens(smap2) == {"A_ONLY_OK"})
|
||||
|
||||
# --- end-to-end: a self-skipped L3 run must land in the ledger as `skipped` --------
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
root = Path(td)
|
||||
tdir = root / "tests" / "l3-fault-injection" / "node-loss"
|
||||
tdir.mkdir(parents=True)
|
||||
(tdir / "chainsaw-test.yaml").write_text(
|
||||
"kind: Test\nmetadata:\n name: node-loss\n labels:\n"
|
||||
" conformance.home.k3s/R-STATE-02: \"true\"\n annotations:\n"
|
||||
" conformance.home.k3s/requirements: \"R-STATE-02\"\n"
|
||||
" conformance.home.k3s/layer: \"L3\"\nspec:\n steps:\n"
|
||||
" - script:\n content: 'echo hi'\n check:\n"
|
||||
" (contains($stdout, 'NODE_LOSS_VOLUME_AVAILABLE_OK')"
|
||||
" || contains($stdout, 'NODE_LOSS_SKIPPED_OK')): true\n")
|
||||
rd = root / "runs" / "20260901T120000Z-abc1234"
|
||||
rd.mkdir(parents=True)
|
||||
(rd / "report.junit.xml").write_text(
|
||||
'<testsuites><testsuite><testcase classname="L3:R-STATE-02" '
|
||||
'name="node-loss"><properties>'
|
||||
'<property name="requirements" value="R-STATE-02"/>'
|
||||
'<property name="layer" value="L3"/></properties></testcase>'
|
||||
"</testsuite></testsuites>")
|
||||
(rd / "run.log").write_text(
|
||||
" | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
|
||||
" SKIP: 'longhorn' StorageClass absent\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
|
||||
check("end-to-end: Chainsaw 'passed' + skip token => proof 'skipped'",
|
||||
len(rows) == 1 and rows[0]["junit_result"] == "passed"
|
||||
and rows[0]["proof"] == "skipped"
|
||||
and rows[0]["sentinel"] == "NODE_LOSS_SKIPPED_OK")
|
||||
check("end-to-end: row carries ts from the run-dir name and the R-ID label",
|
||||
rows[0]["ts_utc"] == "2026-09-01T12:00:00Z"
|
||||
and rows[0]["requirements"] == ["R-STATE-02"])
|
||||
|
||||
(rd / "run.log").write_text(
|
||||
" | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
|
||||
check("end-to-end: Chainsaw 'passed' + pass token => proof 'proven'",
|
||||
rows[0]["proof"] == "proven"
|
||||
and rows[0]["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK")
|
||||
|
||||
ledger = root / "ledger.jsonl"
|
||||
append_rows(ledger, rows)
|
||||
append_rows(ledger, rows)
|
||||
check("ledger is append-only JSONL (one parsable object per line)",
|
||||
len(ledger.read_text().strip().splitlines()) == 2
|
||||
and json.loads(ledger.read_text().splitlines()[0])["test"] == "node-loss")
|
||||
|
||||
if not ok:
|
||||
return 1
|
||||
sys.stderr.write("record_run.py selftest: PASS\n")
|
||||
return 0
|
||||
|
||||
|
||||
# ============================================================================== main
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--run-dir", type=Path,
|
||||
help="conformance/runs/<UTC>-<sha>/ holding report.junit.xml + run.log")
|
||||
ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
|
||||
help=f"append-only ledger (default {DEFAULT_LEDGER})")
|
||||
ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
|
||||
ap.add_argument("--repo", type=Path, default=REPO_ROOT)
|
||||
ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
|
||||
ap.add_argument("--chainsaw-version", default=None,
|
||||
help="override the versions.env pin (recorded verbatim)")
|
||||
ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
|
||||
help="override the `kubectl version` probe (recorded verbatim)")
|
||||
ap.add_argument("--dry-run", action="store_true",
|
||||
help="print the rows that WOULD be appended; write nothing")
|
||||
ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.selftest:
|
||||
return _selftest()
|
||||
if args.run_dir is None:
|
||||
sys.stderr.write("[record] --run-dir is required (or use --selftest)\n")
|
||||
return 1
|
||||
|
||||
run_dir = args.run_dir.resolve()
|
||||
junit, log = run_dir / "report.junit.xml", run_dir / "run.log"
|
||||
# BOTH files or nothing. The JUnit alone cannot tell a self-skip from a real pass, so
|
||||
# recording from it alone is exactly the false proof this tool exists to prevent.
|
||||
missing = [p.name for p in (junit, log) if not p.is_file()]
|
||||
if missing:
|
||||
sys.stderr.write(
|
||||
f"[record] {run_dir} is missing {', '.join(missing)} — nothing recorded. "
|
||||
f"The JUnit alone cannot distinguish a self-skip from a real pass "
|
||||
f"(see module docstring). SKIPPED (not green).\n")
|
||||
return 3
|
||||
|
||||
try:
|
||||
rows = build_rows(
|
||||
run_dir, args.tests, args.env,
|
||||
args.chainsaw_version or pinned_chainsaw_version(CONF_ROOT),
|
||||
args.kube_version or server_kube_version(),
|
||||
git_sha(args.repo))
|
||||
except ET.ParseError as e:
|
||||
sys.stderr.write(f"[record] {junit} is not parsable JUnit: {e}\n")
|
||||
return 1
|
||||
except OSError as e:
|
||||
sys.stderr.write(f"[record] cannot read the run inputs: {e}\n")
|
||||
return 1
|
||||
|
||||
if not rows:
|
||||
sys.stderr.write(f"[record] {junit} contains no <testcase> — nothing to record. "
|
||||
f"SKIPPED (not green).\n")
|
||||
return 3
|
||||
|
||||
for r in rows:
|
||||
print(f"[record] {r['layer']:4s} {r['proof']:28s} {r['test']} "
|
||||
f"({','.join(r['requirements']) or 'untagged'}) "
|
||||
f"junit={r['junit_result']} sentinel={r['sentinel'] or '—'}")
|
||||
counts: dict[str, int] = {}
|
||||
for r in rows:
|
||||
counts[r["proof"]] = counts.get(r["proof"], 0) + 1
|
||||
print(f"[record] {len(rows)} testcase(s): "
|
||||
+ ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
|
||||
|
||||
if args.dry_run:
|
||||
print("[record] --dry-run: ledger NOT written.")
|
||||
return 0
|
||||
try:
|
||||
append_rows(args.ledger, rows)
|
||||
except OSError as e:
|
||||
sys.stderr.write(f"[record] cannot append to {args.ledger}: {e}\n")
|
||||
return 1
|
||||
print(f"[record] appended {len(rows)} row(s) -> {args.ledger}")
|
||||
print("[record] COMMIT the run dir and the ledger; an uncommitted ledger proves nothing.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,24 +1,17 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the CLUSTER-INDEPENDENT conformance checks OFFLINE and emit JUnit.
|
||||
"""Run the L1 *rendering* conformance checks OFFLINE (no cluster) and emit JUnit.
|
||||
|
||||
docs/14 puts two layers in CI with no cluster:
|
||||
* L0 static/policy -- "manifests/ServiceDefinitions obey invariants" (tests/l0-static/,
|
||||
each dir carrying a `check.sh`).
|
||||
* L1 rendering -- ServiceDefinition -> rendered resources vs golden expectations
|
||||
(tests/l1-rendering/, each dir carrying a `render-check.sh`).
|
||||
Both are executed here so they report a REAL pass/fail instead of being skipped with the
|
||||
cluster-gated layers. run.sh invokes this for `--layer l0`, `--layer l1`, and as the
|
||||
offline leg of a full run.
|
||||
docs/14: "L1 rendering assertions run in CI without a cluster." L1 verifies the
|
||||
ServiceDefinition -> rendered-resources mapping against golden expectations; unlike L2/L3
|
||||
it needs no live cluster. Each l1-rendering test dir carries a `render-check.sh` (renderer
|
||||
+ golden diff + structural asserts). run.sh invokes this for `--layer l1` so those tests
|
||||
report a REAL pass/fail instead of being skipped with the cluster-gated layers.
|
||||
|
||||
Pure standard library (matches the harness's no-pip-dependency posture); the checks
|
||||
themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives
|
||||
there; L0 checks to git + PyYAML).
|
||||
Pure standard library (matches the harness's no-pip-dependency posture); the render checks
|
||||
themselves shell out to catalog/render.py (the renderer's PyYAML dependency lives there).
|
||||
|
||||
Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] <test-dir> [<test-dir> ...]
|
||||
Exit 0 iff every selected offline check passes.
|
||||
|
||||
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
|
||||
carries; a dir with neither is reported SKIPPED (never a pass).
|
||||
Exit 0 iff every selected render check passes.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -32,19 +25,6 @@ from xml.dom import minidom
|
||||
|
||||
LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
|
||||
|
||||
# The offline check script a test dir may carry, in preference order.
|
||||
CHECK_SCRIPTS = ("render-check.sh", "check.sh")
|
||||
|
||||
|
||||
def layer_of(d: Path) -> str:
|
||||
"""Layer for the JUnit properties, from the subtree the test lives in."""
|
||||
rel = str(d)
|
||||
if "l0-static" in rel:
|
||||
return "L0"
|
||||
if "l1-rendering" in rel:
|
||||
return "L1"
|
||||
return "OFFLINE"
|
||||
|
||||
|
||||
def collect_meta(test_yaml: Path, prefix: str):
|
||||
text = test_yaml.read_text() if test_yaml.exists() else ""
|
||||
@@ -64,11 +44,10 @@ def main() -> int:
|
||||
|
||||
cases = [] # (name, reqs, dir, rc, output)
|
||||
for d in args.dirs:
|
||||
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
|
||||
check = d / "render-check.sh"
|
||||
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
|
||||
if check is None:
|
||||
cases.append((name, reqs, d, 3,
|
||||
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
|
||||
if not check.exists():
|
||||
cases.append((name, reqs, d, 3, f"no render-check.sh in {d}"))
|
||||
continue
|
||||
proc = subprocess.run(
|
||||
["bash", str(check)], capture_output=True, text=True
|
||||
@@ -82,7 +61,7 @@ def main() -> int:
|
||||
suite = ET.Element(
|
||||
"testsuite",
|
||||
{
|
||||
"name": "home-k3s-conformance L0/L1 (offline)",
|
||||
"name": "home-k3s-conformance L1 (offline render)",
|
||||
"tests": str(len(cases)),
|
||||
"failures": str(failures),
|
||||
"errors": "0",
|
||||
@@ -93,21 +72,19 @@ def main() -> int:
|
||||
tc = ET.SubElement(
|
||||
suite,
|
||||
"testcase",
|
||||
{"classname": f"{layer_of(d)}:{','.join(reqs) if reqs else 'UNTAGGED'}",
|
||||
"name": name},
|
||||
{"classname": f"L1:{','.join(reqs) if reqs else 'UNTAGGED'}", "name": name},
|
||||
)
|
||||
props = ET.SubElement(tc, "properties")
|
||||
ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
|
||||
ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
|
||||
ET.SubElement(props, "property", {"name": "layer", "value": "L1"})
|
||||
ET.SubElement(props, "property", {"name": "path", "value": str(d)})
|
||||
if rc == 3:
|
||||
ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
|
||||
ET.SubElement(tc, "skipped", {"message": out or "no render-check.sh"})
|
||||
elif rc != 0:
|
||||
fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
|
||||
fail = ET.SubElement(tc, "failure", {"message": f"render-check exit {rc}"})
|
||||
fail.text = out
|
||||
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
|
||||
print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
|
||||
f"({','.join(reqs) or 'untagged'})",
|
||||
print(f"[l1-offline] {status:4s} {name} ({','.join(reqs) or 'untagged'})",
|
||||
file=sys.stderr)
|
||||
if rc not in (0, 3):
|
||||
print(out, file=sys.stderr)
|
||||
@@ -126,7 +103,7 @@ def main() -> int:
|
||||
args.junit.parent.mkdir(parents=True, exist_ok=True)
|
||||
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
|
||||
args.junit.write_text(xml)
|
||||
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
|
||||
print(f"[l1-offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
|
||||
f"{args.junit}", file=sys.stderr)
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
+46
-190
@@ -8,51 +8,18 @@
|
||||
#
|
||||
# ./run.sh # full suite; skips L1/L2/L3 if no cluster
|
||||
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
|
||||
# ./run.sh --layer l2 # one layer
|
||||
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
|
||||
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
|
||||
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
|
||||
# # requirement, is it implemented, is it actually
|
||||
# # reconciled by Flux, is its test executable or a
|
||||
# # scaffold, and whether the committed run ledger
|
||||
# # (runs/ledger.jsonl) actually proves it — see
|
||||
# # lib/status.py. No ledger row => not-derivable.
|
||||
# ./run.sh --lint # structural-validate test defs (no cluster)
|
||||
# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate)
|
||||
# ./run.sh --require-all # phase-exit: every requirement must be covered
|
||||
# ./run.sh --record # RUN AND CAPTURE: tee the whole run to
|
||||
# # runs/<UTC>-<short-sha>/run.log, copy the JUnit
|
||||
# # beside it, and append one ledger row per
|
||||
# # testcase to runs/ledger.jsonl (lib/record_run.py)
|
||||
#
|
||||
# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
|
||||
# --------------------------------------------------------------------------
|
||||
# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
|
||||
# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
|
||||
# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
|
||||
# that — `conformance/runs/` (see conformance/.gitignore for the negations).
|
||||
#
|
||||
# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
|
||||
# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
|
||||
#
|
||||
# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
|
||||
# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
|
||||
# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
|
||||
# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
|
||||
# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
|
||||
#
|
||||
# Artifacts (always written under conformance/_artifacts/, gitignored):
|
||||
# _artifacts/report.junit.xml JUnit (Chainsaw output, or skipped-scaffold shim)
|
||||
# _artifacts/coverage/coverage.md human coverage report
|
||||
# _artifacts/coverage/coverage.json machine coverage report
|
||||
# _artifacts/status/status.md human requirement-status report (--status)
|
||||
# _artifacts/status/status.json machine requirement-status report (--status)
|
||||
#
|
||||
# Run ledger (TRACKED — commit these; --record only):
|
||||
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
|
||||
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
|
||||
# runs/ledger.jsonl append-only, one row per testcase per run
|
||||
#
|
||||
# Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
|
||||
# execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
|
||||
@@ -69,34 +36,30 @@ source "${CONF_ROOT}/versions.env"
|
||||
|
||||
ARTIFACTS="${CONF_ROOT}/_artifacts"
|
||||
COVERAGE_OUT="${ARTIFACTS}/coverage"
|
||||
STATUS_OUT="${ARTIFACTS}/status"
|
||||
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
|
||||
# under `tee`. Captured BEFORE parsing so nothing is normalized away.
|
||||
ORIG_ARGS=("$@")
|
||||
|
||||
LAYER="all"
|
||||
REQS=""
|
||||
MODE="run" # run | coverage-only | status | lint
|
||||
MODE="run" # run | coverage-only | lint
|
||||
STRICT_COVERAGE=0
|
||||
REQUIRE_ALL=0
|
||||
FORCE_NO_CLUSTER=0
|
||||
RECORD=0
|
||||
|
||||
# Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
|
||||
# owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
|
||||
# exemption is explicit and reviewable, not silent.
|
||||
#
|
||||
# EMPTY as of 2026-09-01. R-ENV-02 and R-GITOPS-01 were the last two entries — exempted
|
||||
# because they were unscaffolded while the coarse path->requirement map keys clusters/*
|
||||
# and ci/* to them, so any overlay/CI edit would have false-blocked. They now have a real
|
||||
# verifying L0 test (tests/l0-static/flux-postbuild-substitution), which asserts the
|
||||
# substitution mechanism that IS the "authored once, promoted" contract, so they are
|
||||
# covered rather than exempt and the completeness gate can hold them to it. That test
|
||||
# names its own limits in its header: it proves the STATIC property (no manifest renders
|
||||
# an undefined variable), not the live-cluster half of either requirement.
|
||||
# R-ENV-02 (dev->prod promote) — owned by WS-INFRA/WS-GITOPS; unscaffolded (a GAP in
|
||||
# coverage.md). The coarse path->requirement map keys
|
||||
# clusters/* AND ci/* to it and the R-ENV-02 doc token
|
||||
# appears in platform/kustomization.yaml's header, so ANY
|
||||
# infra/overlay/CI edit (e.g. enabling a component, bumping
|
||||
# SPEC_PIN) would otherwise false-block on it.
|
||||
# R-GITOPS-01 (state in git/Flux) — same: owned by WS-GITOPS, unscaffolded, and the ci/*
|
||||
# path-map keys to it, so editing any ci/ file (SPEC_PIN
|
||||
# adoption included) would false-block.
|
||||
# These are visible GAPs, not hidden — removing them from this list is the right move the
|
||||
# moment WS-TEST adds their verifying tests.
|
||||
#
|
||||
# FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header
|
||||
# tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in
|
||||
@@ -112,95 +75,31 @@ RECORD=0
|
||||
# R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra")
|
||||
# but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no
|
||||
# longer a gap and has been removed from this list; the coverage report counts it as covered.
|
||||
DOCUMENTED_GAPS=""
|
||||
DOCUMENTED_GAPS="R-ENV-02 R-GITOPS-01"
|
||||
|
||||
# Print the header block (line 2 through the closing `####` rule, which is dropped) as
|
||||
# help. Bounded by the rule rather than a fixed line number so the header can grow.
|
||||
usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
|
||||
usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--layer) LAYER="$2"; shift 2 ;;
|
||||
--requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
|
||||
--coverage-only) MODE="coverage-only"; shift ;;
|
||||
--status) MODE="status"; shift ;;
|
||||
--lint) MODE="lint"; shift ;;
|
||||
--strict-coverage) STRICT_COVERAGE=1; shift ;;
|
||||
--require-all) REQUIRE_ALL=1; shift ;;
|
||||
--no-cluster) FORCE_NO_CLUSTER=1; shift ;;
|
||||
--record) RECORD=1; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "unknown arg: $1" >&2; usage; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
|
||||
mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}"
|
||||
|
||||
log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
|
||||
|
||||
###############################################################################
|
||||
# --record: capture this run as committed evidence, then hand it to the ledger.
|
||||
#
|
||||
# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
|
||||
# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
|
||||
# operator-visible output — no second, differently-behaving "recording mode" to drift.
|
||||
# CONF_RECORD_CHILD guards against recursion.
|
||||
#
|
||||
# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
|
||||
# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
|
||||
# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
|
||||
###############################################################################
|
||||
if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
|
||||
case "${MODE}" in
|
||||
run) ;;
|
||||
# --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
|
||||
# test, so a ledger row from one of them would be a row about nothing.
|
||||
*) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
|
||||
esac
|
||||
RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
|
||||
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
|
||||
RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
|
||||
mkdir -p "${RUN_DIR}"
|
||||
# Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
|
||||
# ever capture a report THIS run produced. Copying a stale artifact would pair someone
|
||||
# else's result with this run's log — a fabricated pairing, which is precisely the class
|
||||
# of false proof the ledger exists to prevent.
|
||||
rm -f "${JUNIT_OUT}"
|
||||
# Strip --record from the argv handed to the child; everything else passes through.
|
||||
CHILD_ARGS=()
|
||||
for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
|
||||
[ "$a" = "--record" ] || CHILD_ARGS+=("$a")
|
||||
done
|
||||
log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
|
||||
set +e
|
||||
CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
|
||||
| tee "${RUN_DIR}/run.log"
|
||||
RUN_RC=${PIPESTATUS[0]}
|
||||
set -e
|
||||
if [ -f "${JUNIT_OUT}" ]; then
|
||||
cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
|
||||
else
|
||||
log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
|
||||
fi
|
||||
# Explicit, separate step: read the JUnit + the log and append one row per testcase.
|
||||
# rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
|
||||
# CONF_LEDGER exists so the harness's own tests can exercise this path without writing
|
||||
# to the real ledger; operators should never set it.
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
|
||||
--run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
|
||||
--ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
|
||||
log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
|
||||
log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
|
||||
exit "${RUN_RC}"
|
||||
fi
|
||||
|
||||
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
|
||||
gen_coverage() {
|
||||
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
|
||||
# requirement is exempt, and handing the flag an empty string would be a silent no-op
|
||||
# that reads like an exemption in the CLI log.
|
||||
local flags=()
|
||||
[ -n "${DOCUMENTED_GAPS}" ] && flags+=(--allow-gap "${DOCUMENTED_GAPS}")
|
||||
local flags=(--allow-gap "${DOCUMENTED_GAPS}")
|
||||
[ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict)
|
||||
# A tagged subset run is also the merge gate: the touched requirements MUST be covered.
|
||||
[ -n "${REQS}" ] && flags+=(--require "${REQS}")
|
||||
@@ -212,23 +111,6 @@ gen_coverage() {
|
||||
"${flags[@]}"
|
||||
}
|
||||
|
||||
# --- Derived STATUS report (no cluster) ------------------------------------------------
|
||||
# Coverage answers "does a tagged test EXIST?". Status answers the operator's question —
|
||||
# "what is the real state of each requirement given what is actually in the repo?" — by
|
||||
# deriving, per requirement: implementing artifact? reachable from a Flux Kustomization
|
||||
# (reconciled, not merely authored)? tagged test that is EXECUTABLE rather than a TODO
|
||||
# scaffold? It REFUSES to derive whether a test passed: nothing tracked in git records a
|
||||
# run result, and a status report that defaults to green is worse than none.
|
||||
gen_status() {
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/status.py" \
|
||||
--repo "$(cd "${CONF_ROOT}/.." && pwd)" \
|
||||
--registry "${CONF_ROOT}/requirements/registry.yaml" \
|
||||
--tests "${CONF_ROOT}/tests" \
|
||||
--out "${STATUS_OUT}" \
|
||||
--env "${CONF_ENV:-dev}" \
|
||||
"$@"
|
||||
}
|
||||
|
||||
# --- Structural lint (no cluster): parse each test def and check the tag convention ----
|
||||
lint_tests() {
|
||||
log "Linting test definitions (structural, no cluster)"
|
||||
@@ -372,40 +254,26 @@ run_chainsaw() {
|
||||
return 0
|
||||
}
|
||||
|
||||
# --- L0/L1: the cluster-INDEPENDENT layers (docs/14) ------------------------------------
|
||||
# docs/14 puts TWO layers in CI without a cluster:
|
||||
# L0 static/policy tests/l0-static/** each dir carries a `check.sh` (repo invariants)
|
||||
# L1 rendering tests/l1-rendering/** each dir carries a `render-check.sh` (render == golden)
|
||||
# Both are executed directly here and emit a REAL pass/fail JUnit, instead of being skipped
|
||||
# with the cluster-gated layers. Honors the same --layer/--requirements selection as the rest
|
||||
# of the suite (select_dirs), filtered to those two subtrees.
|
||||
OFFLINE_SUBTREES='/l0-static/|/l1-rendering/'
|
||||
|
||||
offline_selected_dirs() {
|
||||
select_dirs | grep -E "${OFFLINE_SUBTREES}" || true
|
||||
# --- L1 rendering: cluster-INDEPENDENT render checks (docs/14) --------------------------
|
||||
# L1 verifies ServiceDefinition -> rendered-resources against golden expectations; it needs
|
||||
# no live cluster. Each l1-rendering test dir carries a `render-check.sh` (renderer + golden
|
||||
# diff + structural asserts). We run those directly and emit a REAL pass/fail JUnit, instead
|
||||
# of skipping L1 with the cluster-gated layers (T-0.3). Honors the same --layer/--requirements
|
||||
# selection as the rest of the suite (select_dirs), filtered to the l1-rendering subtree.
|
||||
l1_selected_dirs() {
|
||||
select_dirs | grep '/l1-rendering/' || true
|
||||
}
|
||||
|
||||
# True when EVERY requested layer is cluster-independent, so the offline result is the
|
||||
# whole result and run.sh must exit on it (the ci/l0.sh + ci/l1.sh merge-gate paths).
|
||||
layers_are_offline_only() {
|
||||
local l
|
||||
[ "${LAYER}" = all ] && return 1
|
||||
for l in ${LAYER//,/ }; do
|
||||
case "$l" in l0|l1) ;; *) return 1 ;; esac
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
run_offline_checks() {
|
||||
local -a dirs
|
||||
mapfile -t dirs < <(offline_selected_dirs)
|
||||
if [ "${#dirs[@]}" -eq 0 ]; then
|
||||
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
|
||||
run_l1_offline() {
|
||||
local -a l1dirs
|
||||
mapfile -t l1dirs < <(l1_selected_dirs)
|
||||
if [ "${#l1dirs[@]}" -eq 0 ]; then
|
||||
log "No L1 render tests match the selection (layer=${LAYER} reqs='${REQS}')."
|
||||
return 0
|
||||
fi
|
||||
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
|
||||
log "Running ${#l1dirs[@]} L1 render check(s) OFFLINE (no cluster; docs/14)."
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
|
||||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
|
||||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${l1dirs[@]}"
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
@@ -419,17 +287,6 @@ case "${MODE}" in
|
||||
log "coverage-only: done. See ${COVERAGE_OUT}/coverage.md"
|
||||
exit 0
|
||||
;;
|
||||
status)
|
||||
# rc: 0 report written, 1 a derivation/strict failure, 3 a check could not run
|
||||
# (SKIPPED, never counted as green — ci/lib.sh convention). Propagated as-is.
|
||||
rc=0
|
||||
# --strict-coverage is the harness's hygiene switch; apply it to status too.
|
||||
status_flags=()
|
||||
[ "${STRICT_COVERAGE}" = 1 ] && status_flags+=(--strict)
|
||||
gen_status ${status_flags[@]+"${status_flags[@]}"} || rc=$?
|
||||
log "status: done. See ${STATUS_OUT}/status.md"
|
||||
exit "${rc}"
|
||||
;;
|
||||
lint)
|
||||
lint_tests
|
||||
log "lint: done."
|
||||
@@ -437,30 +294,29 @@ case "${MODE}" in
|
||||
;;
|
||||
esac
|
||||
|
||||
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
|
||||
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
|
||||
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
||||
OFFLINE_RC=0
|
||||
run_offline_checks || OFFLINE_RC=$?
|
||||
if layers_are_offline_only; then
|
||||
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
|
||||
if [ "${OFFLINE_RC}" -eq 0 ]; then
|
||||
log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||||
# L1 rendering is cluster-independent (docs/14): run the selected L1 render checks offline
|
||||
# and treat their result as authoritative REGARDLESS of cluster availability. A render-check
|
||||
# failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
||||
L1_RC=0
|
||||
run_l1_offline || L1_RC=$?
|
||||
if [ "${LAYER}" = l1 ]; then
|
||||
# L1-only run (the ci/l1.sh merge-gate path): the offline render result IS the result.
|
||||
if [ "${L1_RC}" -eq 0 ]; then
|
||||
log "Done (L1 render). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||||
else
|
||||
log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
|
||||
log "L1 render checks FAILED (rc=${L1_RC}). JUnit: ${JUNIT_OUT}"
|
||||
fi
|
||||
exit "${OFFLINE_RC}"
|
||||
exit "${L1_RC}"
|
||||
fi
|
||||
if [ "${OFFLINE_RC}" -ne 0 ]; then
|
||||
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
|
||||
exit "${OFFLINE_RC}"
|
||||
if [ "${L1_RC}" -ne 0 ]; then
|
||||
log "L1 render checks FAILED (rc=${L1_RC}) — blocking (render failure is not cluster-gated)."
|
||||
exit "${L1_RC}"
|
||||
fi
|
||||
|
||||
# run mode: need a cluster for L2/L3. Without one (Phase 0), emit skipped JUnit for the
|
||||
# cluster-gated layers. (L1 already ran offline above and passed.)
|
||||
if [ "${FORCE_NO_CLUSTER}" = 1 ] || ! conf_cluster_reachable; then
|
||||
log "No reachable cluster — emitting SKIPPED JUnit scaffold for L2/L3 (Phase 0 behavior)."
|
||||
log "NOTE: the offline L0/L1 results above already ran for real; this shim covers L2/L3 only."
|
||||
log "Also running structural lint so scaffolds stay valid."
|
||||
lint_tests || { log "lint failed"; exit 1; }
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/emit_skipped_junit.py" \
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="node-loss" classname="" time="131.090121" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -1,232 +0,0 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/node-loss
|
||||
=== PAUSE chainsaw/node-loss
|
||||
=== CONT chainsaw/node-loss
|
||||
l.go:52: | 23:51:18 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-fond-marlin
|
||||
l.go:52: | 23:51:18 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 23:51:18 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -euo pipefail
|
||||
NS="$NAMESPACE"
|
||||
|
||||
# -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
|
||||
READY="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
| grep -c '^True$' || true)"
|
||||
echo "ready nodes: ${READY}"
|
||||
if [ "${READY:-0}" -lt 3 ]; then
|
||||
echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
CANARY="node-loss-canary-$(date +%s%N)"
|
||||
|
||||
# -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
|
||||
# writer pod that writes the canary and flushes it to the replicated volume. --
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: survivor
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
apply_pod() {
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: $1
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 3600"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: survivor
|
||||
EOF
|
||||
}
|
||||
|
||||
apply_pod writer
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
|
||||
printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
|
||||
echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
|
||||
|
||||
# The victim node is the one the attached volume currently lives on.
|
||||
VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
|
||||
[ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
|
||||
echo "victim node (currently hosts the attached volume): $VICTIM"
|
||||
|
||||
# Best-effort precondition log: if the Longhorn Volume CR is readable, show the
|
||||
# volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
|
||||
# runner is least-privilege and may not have longhorn.io read rights) — the
|
||||
# data-availability assertion below is the authoritative proof, not this.
|
||||
PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
if [ -n "$PVNAME" ]; then
|
||||
kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
|
||||
2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
|
||||
fi
|
||||
|
||||
# -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
|
||||
# Label it with THIS test's unique namespace so `finally` can always find and
|
||||
# uncordon exactly this node; then cordon it; then evict ONLY our own writer
|
||||
# pod, forcing the volume to detach from the victim. (No `drain` — unrelated
|
||||
# pods on the node are never touched.)
|
||||
kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
|
||||
kubectl cordon "$VICTIM"
|
||||
echo "cordoned + labelled victim node: $VICTIM"
|
||||
kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
|
||||
echo "evicted writer off the victim node; volume detached from $VICTIM"
|
||||
|
||||
# -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
|
||||
# so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
|
||||
# the volume there from its replicas on the other nodes. --
|
||||
apply_pod reader
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
|
||||
READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
|
||||
echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
|
||||
if [ "$READER_NODE" = "$VICTIM" ]; then
|
||||
echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
[ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
|
||||
echo "volume available + data intact from a surviving node"
|
||||
|
||||
# Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
|
||||
# (bounded) for robustness to settle back to available (healthy|degraded — both
|
||||
# serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
|
||||
# Non-fatal for the same least-privilege reason as the pre-fault log.
|
||||
if [ -n "$PVNAME" ]; then
|
||||
for i in $(seq 1 30); do
|
||||
ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='{.status.robustness}' 2>/dev/null || true)"
|
||||
[ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
|
||||
echo "post-fault robustness[$i]=$ROB"
|
||||
case "$ROB" in healthy|degraded) break ;; esac
|
||||
sleep 10
|
||||
done
|
||||
fi
|
||||
|
||||
echo NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
ready nodes: 3
|
||||
persistentvolumeclaim/survivor created
|
||||
pod/writer created
|
||||
pod/writer condition met
|
||||
wrote canary: node-loss-canary-1788306678898809444
|
||||
victim node (currently hosts the attached volume): k3s-dev-2
|
||||
pre-fault robustness=healthy replicas=3
|
||||
node/k3s-dev-2 labeled
|
||||
node/k3s-dev-2 cordoned
|
||||
cordoned + labelled victim node: k3s-dev-2
|
||||
pod "writer" deleted from chainsaw-fond-marlin namespace
|
||||
evicted writer off the victim node; volume detached from k3s-dev-2
|
||||
pod/reader created
|
||||
pod/reader condition met
|
||||
reader scheduled on surviving node: k3s-dev-1 (victim was k3s-dev-2)
|
||||
recovered='node-loss-canary-1788306678898809444' expected='node-loss-canary-1788306678898809444'
|
||||
volume available + data intact from a surviving node
|
||||
post-fault robustness[1]=healthy
|
||||
NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 23:52:42 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -uo pipefail
|
||||
for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
|
||||
kubectl uncordon "${n#node/}" || true
|
||||
kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
|
||||
echo "restored node to service: ${n#node/}"
|
||||
done
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
node/k3s-dev-2 uncordoned
|
||||
node/k3s-dev-2 unlabeled
|
||||
restored node to service: k3s-dev-2
|
||||
l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
|
||||
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
pod "reader" deleted from chainsaw-fond-marlin namespace
|
||||
persistentvolumeclaim "survivor" deleted from chainsaw-fond-marlin namespace
|
||||
cleanup best-effort; chainsaw namespace teardown finalizes
|
||||
l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-fond-marlin
|
||||
l.go:52: | 23:53:29 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/node-loss (131.09s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -1,4 +0,0 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="dr-restore" classname="" time="149.053701" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -1,671 +0,0 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/dr-restore
|
||||
=== PAUSE chainsaw/dr-restore
|
||||
=== CONT chainsaw/dr-restore
|
||||
l.go:52: | 00:19:07 | [32;1mdr-restore[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-cunning-hen
|
||||
l.go:52: | 00:19:07 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 00:19:07 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
|
||||
# ---- helpers -------------------------------------------------------
|
||||
# Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
|
||||
# fail loudly; an absent API is a genuine precondition gap and may skip.
|
||||
die_or_skip() {
|
||||
case "$2" in
|
||||
*orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
|
||||
echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
|
||||
echo "$2" >&2
|
||||
exit 1 ;;
|
||||
*)
|
||||
echo "SKIP: $1 not available on this cluster -- $2"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
ip2int() {
|
||||
_q="$(echo "$1" | tr '.' ' ')"
|
||||
set -- $_q
|
||||
[ $# -eq 4 ] || { echo -1; return 0; }
|
||||
echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
|
||||
}
|
||||
|
||||
# in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
|
||||
in_cidr() {
|
||||
case "$2" in *"/"*) ;; *) return 1 ;; esac
|
||||
_net="${2%/*}"; _len="${2#*/}"
|
||||
case "$_len" in ''|*[!0-9]*) return 1 ;; esac
|
||||
[ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
|
||||
_i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
|
||||
[ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
|
||||
[ "$_len" -eq 0 ] && return 0
|
||||
_d=1; _s=0
|
||||
while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
|
||||
[ $(( _i / _d )) -eq $(( _n / _d )) ]
|
||||
}
|
||||
|
||||
# ---- Gate 1: the replicated storage tier the DR-critical data lives on --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
|
||||
2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
|
||||
if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
|
||||
echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "backup storage locations:"
|
||||
echo "$BSL_LINES"
|
||||
|
||||
VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
|
||||
|
||||
# ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
|
||||
# File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
|
||||
# deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
|
||||
# ready node-agent no volume data can be captured, so the drill is not runnable.
|
||||
VREADY=0; NAREADY=0
|
||||
for i in $(seq 1 12); do
|
||||
VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
|
||||
NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
|
||||
echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
|
||||
if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
|
||||
sleep 10
|
||||
done
|
||||
if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
|
||||
echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- Pick the BSL this drill will use: prefer the default, and it MUST be
|
||||
# Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
|
||||
# gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
|
||||
# wait for first-validation on a cold start), it does not skip. --------
|
||||
BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
|
||||
for i in $(seq 1 12); do
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
|
||||
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
|
||||
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
|
||||
if [ -n "$CAND" ]; then
|
||||
BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
|
||||
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
|
||||
BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
|
||||
BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
|
||||
break
|
||||
fi
|
||||
echo "poll $i: no Available BackupStorageLocation yet"
|
||||
sleep 10
|
||||
done
|
||||
if [ -z "$BSL_NAME" ]; then
|
||||
echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
|
||||
echo " The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
|
||||
echo "$BSL_LINES" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "using BSL ${BSL_NS}/${BSL_NAME} bucket=${BSL_BUCKET} endpoint=${BSL_URL}"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
|
||||
if [ -z "$BSL_URL" ]; then
|
||||
echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
|
||||
echo " the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
|
||||
echo " test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
|
||||
exit 1
|
||||
fi
|
||||
HOST="${BSL_URL#*://}" # strip scheme
|
||||
HOST="${HOST%%/*}" # strip path
|
||||
HOST="${HOST%%:*}" # strip port
|
||||
echo "backup endpoint host: $HOST"
|
||||
|
||||
case "$HOST" in
|
||||
localhost|*.svc|*.svc.cluster.local|*.cluster.local)
|
||||
echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
|
||||
echo " An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
|
||||
exit 1 ;;
|
||||
esac
|
||||
|
||||
# Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
|
||||
# off-cluster property -- that is a failure, never a silent pass.
|
||||
case "$HOST" in
|
||||
*[!0-9.]*)
|
||||
# Endpoint given as a name. Try the resolvers that may exist in the
|
||||
# runner image, in order; if none is available or none resolves, the
|
||||
# test FAILS below rather than guessing.
|
||||
EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
|
||||
[ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
|
||||
[ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
|
||||
*) EPIP="$HOST" ;;
|
||||
esac
|
||||
if [ -z "$EPIP" ]; then
|
||||
echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
|
||||
echo " the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "backup endpoint resolves to: $EPIP"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
|
||||
case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
|
||||
|
||||
NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
for a in $NODE_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
|
||||
|
||||
SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
|
||||
for a in $SVC_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
|
||||
|
||||
POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
|
||||
for a in $POD_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
|
||||
|
||||
POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
for c in $POD_CIDRS; do
|
||||
case "$c" in *:*) continue ;; esac # IPv6 prefixes are out of scope here
|
||||
if in_cidr "$EPIP" "$c"; then
|
||||
echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
|
||||
|
||||
# Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
|
||||
# runner may not read): the checks above are the authoritative ones.
|
||||
SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
if [ -n "$SVC_CIDRS" ]; then
|
||||
for c in $SVC_CIDRS; do
|
||||
case "$c" in *:*) continue ;; esac
|
||||
if in_cidr "$EPIP" "$c"; then
|
||||
echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
|
||||
else
|
||||
echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
|
||||
fi
|
||||
|
||||
echo "OFF_CLUSTER_TARGET_OK bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
backup storage locations:
|
||||
velero default Available aws true http://192.168.1.9:9000 velero
|
||||
poll 1: velero readyReplicas=1 node-agent numberReady=3
|
||||
using BSL velero/default bucket=velero endpoint=http://192.168.1.9:9000
|
||||
backup endpoint host: 192.168.1.9
|
||||
backup endpoint resolves to: 192.168.1.9
|
||||
endpoint is not any node address (checked 6 node addresses)
|
||||
endpoint is not any Service ClusterIP (checked 81)
|
||||
endpoint is not any Pod IP (checked 186)
|
||||
endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
|
||||
endpoint is outside every ServiceCIDR (10.43.0.0/16 )
|
||||
OFF_CLUSTER_TARGET_OK bsl=velero/default endpoint=192.168.1.9 bucket=velero
|
||||
l.go:52: | 00:19:11 | [32;1mdr-restore[0;22m | [32;1massert-backup-target-is-off-cluster [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backup-target-is-off-cluster [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
SCHED="$(kubectl get schedules.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
|
||||
if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
|
||||
# Distinguish "Velero absent" (skip, in lock-step with step 1) from
|
||||
# "Velero present but nothing scheduled" (a real R-STATE-03 defect).
|
||||
if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
|
||||
if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
|
||||
echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
|
||||
echo DR_SCHEDULE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
|
||||
echo " R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
|
||||
echo DR_SCHEDULE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "velero schedules:"
|
||||
echo "$SCHED"
|
||||
# A Schedule that exists but is paused/disabled backs nothing up.
|
||||
if echo "$SCHED" | grep -q 'phase=Enabled'; then
|
||||
echo BACKUPS_ARE_SCHEDULED_OK
|
||||
else
|
||||
echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
|
||||
exit 1
|
||||
fi
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
velero schedules:
|
||||
velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
|
||||
velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
|
||||
BACKUPS_ARE_SCHEDULED_OK
|
||||
l.go:52: | 00:19:11 | [32;1mdr-restore[0;22m | [32;1massert-backups-are-scheduled [0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1massert-backups-are-scheduled [0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 00:19:11 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -eu
|
||||
NS="$NAMESPACE"
|
||||
SRC="${NS}-dr-src" # scratch source namespace (destroyed by design)
|
||||
DST="${NS}-dr-dst" # clean restore target (must not pre-exist)
|
||||
BK="${NS}-dr-backup"
|
||||
RS="${NS}-dr-restore"
|
||||
OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
|
||||
|
||||
# ---- Re-derive the preconditions (steps do not share shell state). Every
|
||||
# skip below mirrors step 1 exactly, so the whole test skips or runs
|
||||
# as a unit -- it can never half-run. -----------------------------
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
|
||||
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
|
||||
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
|
||||
if [ -z "$CAND" ]; then
|
||||
echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
|
||||
echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
VNS="$(echo "$CAND" | cut -d' ' -f1)"
|
||||
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
|
||||
if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
|
||||
echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
|
||||
echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
|
||||
|
||||
# A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
|
||||
# be free. If either is lying around from an aborted run, remove it before we
|
||||
# claim "clean" -- and fail loudly if it will not go, because a half-deleted
|
||||
# namespace would silently invalidate the destroy proof below.
|
||||
kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
|
||||
|
||||
# ============================ GIVEN ==============================
|
||||
# A DR-critical stateful workload with known data on the replicated tier.
|
||||
CANARY="dr-canary-$(date +%s%N)"
|
||||
kubectl create namespace "$SRC"
|
||||
kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
|
||||
|
||||
kubectl -n "$SRC" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: dr-data
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
kubectl -n "$SRC" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: dr-workload
|
||||
labels:
|
||||
app: dr-workload
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 7200"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: dr-data
|
||||
EOF
|
||||
|
||||
kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
|
||||
printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
|
||||
echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
|
||||
|
||||
ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
|
||||
[ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
|
||||
echo "original in-cluster PersistentVolume: $ORIG_PV"
|
||||
|
||||
# ======================= WHEN (part 1): back up off-cluster =======
|
||||
# Created as a Velero Backup CR -- exactly what `velero backup create` does.
|
||||
# defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
|
||||
# the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
|
||||
# ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
|
||||
# own GC still removes this drill's objects from the bucket.
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: velero.io/v1
|
||||
kind: Backup
|
||||
metadata:
|
||||
name: ${BK}
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
includedNamespaces: [${SRC}]
|
||||
storageLocation: ${BSL_NAME}
|
||||
snapshotVolumes: false
|
||||
defaultVolumesToFsBackup: true
|
||||
ttl: 2h0m0s
|
||||
EOF
|
||||
|
||||
BPHASE=""
|
||||
for i in $(seq 1 90); do
|
||||
BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
echo "poll $i: backup phase=${BPHASE:-<pending>}"
|
||||
case "${BPHASE:-}" in
|
||||
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
|
||||
esac
|
||||
sleep 10
|
||||
done
|
||||
if [ "${BPHASE:-}" != "Completed" ]; then
|
||||
echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
|
||||
kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
echo "backup $BK Completed to ${BSL_NAME}"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
|
||||
# kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
|
||||
PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
|
||||
-l velero.io/backup-name="$BK" \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
|
||||
echo "pod volume backups:"
|
||||
echo "$PVB"
|
||||
if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
|
||||
echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
|
||||
echo " Resource-only backups cannot restore DR-critical DATA." >&2
|
||||
exit 1
|
||||
fi
|
||||
# A line is "<name> <phase> <snapshotID>"; require Completed AND a
|
||||
# non-empty snapshot ID (an empty ID means nothing reached the repository).
|
||||
GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
|
||||
if [ "${GOOD_PVB:-0}" -lt 1 ]; then
|
||||
echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
|
||||
echo " the volume data did not reach the off-cluster repository." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
|
||||
|
||||
# ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
|
||||
# Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
|
||||
# underlying PersistentVolume (and with it the Longhorn replicas). After this
|
||||
# there is no in-cluster source the restore could quietly read from.
|
||||
kubectl delete namespace "$SRC" --wait=true --timeout=420s
|
||||
for i in $(seq 1 30); do
|
||||
if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
|
||||
echo "poll $i: waiting for namespace $SRC to disappear"
|
||||
sleep 10
|
||||
done
|
||||
if kubectl get namespace "$SRC" >/dev/null 2>&1; then
|
||||
echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "source namespace destroyed"
|
||||
|
||||
# The longhorn StorageClass reclaims Delete, so the PV should already be gone;
|
||||
# force it if a Retain policy (or a stuck finalizer) left it behind, because a
|
||||
# surviving PV would make the restore prove nothing about off-cluster recovery.
|
||||
kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
|
||||
for i in $(seq 1 30); do
|
||||
if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
|
||||
echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
|
||||
sleep 10
|
||||
done
|
||||
if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
|
||||
echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
|
||||
echo " An in-cluster copy of the data still exists, so a successful restore" >&2
|
||||
echo " would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
|
||||
|
||||
# Best-effort corroboration (the least-privilege runner may not read longhorn.io):
|
||||
# the Longhorn Volume CR and its replicas should be gone too.
|
||||
if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
|
||||
echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
|
||||
|
||||
# ============================ RESTORE ============================
|
||||
# Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
|
||||
# the catalog's "restored into a clean state".
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: velero.io/v1
|
||||
kind: Restore
|
||||
metadata:
|
||||
name: ${RS}
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
backupName: ${BK}
|
||||
includedNamespaces: [${SRC}]
|
||||
namespaceMapping:
|
||||
${SRC}: ${DST}
|
||||
existingResourcePolicy: none
|
||||
EOF
|
||||
|
||||
RPHASE=""
|
||||
for i in $(seq 1 90); do
|
||||
RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
echo "poll $i: restore phase=${RPHASE:-<pending>}"
|
||||
case "${RPHASE:-}" in
|
||||
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
|
||||
esac
|
||||
sleep 10
|
||||
done
|
||||
if [ "${RPHASE:-}" != "Completed" ]; then
|
||||
echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
|
||||
kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
|
||||
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
echo "restore $RS Completed into $DST"
|
||||
|
||||
# ============================= THEN ==============================
|
||||
# The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
|
||||
# `restore-wait` init container that blocks until the node-agent has written the
|
||||
# volume contents back, so Ready here already implies the data landed.
|
||||
if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
|
||||
echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
|
||||
kubectl -n "$DST" describe pod dr-workload >&2 || true
|
||||
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
|
||||
[ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
|
||||
if [ "$NEW_PV" = "$ORIG_PV" ]; then
|
||||
echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
|
||||
echo " That is an in-cluster survival, not an off-cluster restore." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
|
||||
|
||||
GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
if [ "$GOT" != "$CANARY" ]; then
|
||||
echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
|
||||
l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
velero ns=velero bsl=default src=chainsaw-cunning-hen-dr-src dst=chainsaw-cunning-hen-dr-dst
|
||||
namespace/chainsaw-cunning-hen-dr-src created
|
||||
namespace/chainsaw-cunning-hen-dr-src labeled
|
||||
persistentvolumeclaim/dr-data created
|
||||
pod/dr-workload created
|
||||
pod/dr-workload condition met
|
||||
seeded canary: dr-canary-1788308352223263220
|
||||
original in-cluster PersistentVolume: pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745
|
||||
backup.velero.io/chainsaw-cunning-hen-dr-backup created
|
||||
poll 1: backup phase=InProgress
|
||||
poll 2: backup phase=InProgress
|
||||
poll 3: backup phase=Completed
|
||||
backup chainsaw-cunning-hen-dr-backup Completed to default
|
||||
pod volume backups:
|
||||
chainsaw-cunning-hen-dr-backup-l95lw Completed 47fa2737115e17142fb41857cef4e779
|
||||
volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
|
||||
namespace "chainsaw-cunning-hen-dr-src" deleted
|
||||
source namespace destroyed
|
||||
original PersistentVolume pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 is GONE -- no in-cluster copy of the data remains
|
||||
(longhorn Volume CR pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 not present -- either deleted or not readable by this identity)
|
||||
restore.velero.io/chainsaw-cunning-hen-dr-restore created
|
||||
poll 1: restore phase=InProgress
|
||||
poll 2: restore phase=InProgress
|
||||
poll 3: restore phase=InProgress
|
||||
poll 4: restore phase=Completed
|
||||
restore chainsaw-cunning-hen-dr-restore Completed into chainsaw-cunning-hen-dr-dst
|
||||
pod/dr-workload condition met
|
||||
restored PVC bound to PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
|
||||
restored onto a NEW volume (pvc-63c47973-9c7f-4423-a27e-caa008026e49 != pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745) -- the data came from outside the cluster
|
||||
recovered='dr-canary-1788308352223263220' expected='dr-canary-1788308352223263220'
|
||||
DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
|
||||
=== STDERR
|
||||
Defaulted container "app" out of: app, restore-wait (init)
|
||||
l.go:52: | 00:21:16 | [32;1mdr-restore[0;22m | [32;1mdata-restores-from-the-off-cluster-backup[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 00:21:16 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -u
|
||||
NS="$NAMESPACE"
|
||||
SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
|
||||
|
||||
VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
|
||||
|
||||
# Velero objects first: a DeleteBackupRequest makes Velero remove the backup
|
||||
# DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
|
||||
if [ -n "$VNS" ]; then
|
||||
kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
|
||||
if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
|
||||
kubectl apply -f - <<EOF || true
|
||||
apiVersion: velero.io/v1
|
||||
kind: DeleteBackupRequest
|
||||
metadata:
|
||||
name: ${BK}-delete
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
backupName: ${BK}
|
||||
EOF
|
||||
for i in $(seq 1 18); do
|
||||
kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
|
||||
sleep 10
|
||||
done
|
||||
# Backstop if Velero could not process the request (the 2h ttl also GCs it).
|
||||
kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
|
||||
kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
|
||||
fi
|
||||
echo "velero objects for $NS cleaned up (ns $VNS)"
|
||||
fi
|
||||
|
||||
# Scratch namespaces (both, in case the run aborted at any point).
|
||||
for n in "$DST" "$SRC"; do
|
||||
kubectl delete namespace "$n" --ignore-not-found --wait=false || true
|
||||
done
|
||||
# Anything left labelled as ours (defensive; the two names above are the only
|
||||
# namespaces this test ever creates).
|
||||
kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
|
||||
|
||||
# Released PVs from either namespace's PVCs would otherwise leak storage.
|
||||
for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
|
||||
kubectl delete pv "$pv" --ignore-not-found --wait=false || true
|
||||
echo "released leftover PersistentVolume: $pv"
|
||||
done
|
||||
|
||||
echo "dr-restore teardown complete (best-effort; every object was created by this test)"
|
||||
l.go:52: | 00:21:28 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
restore.velero.io "chainsaw-cunning-hen-dr-restore" deleted from velero namespace
|
||||
deletebackuprequest.velero.io/chainsaw-cunning-hen-dr-backup-delete created
|
||||
velero objects for chainsaw-cunning-hen cleaned up (ns velero)
|
||||
namespace "chainsaw-cunning-hen-dr-dst" deleted
|
||||
namespace "chainsaw-cunning-hen-dr-dst" deleted
|
||||
persistentvolume "pvc-63c47973-9c7f-4423-a27e-caa008026e49" deleted
|
||||
released leftover PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
|
||||
dr-restore teardown complete (best-effort; every object was created by this test)
|
||||
l.go:52: | 00:21:28 | [32;1mdr-restore[0;22m | [32;1mdata-restores-from-the-off-cluster-backup[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 00:21:28 | [36;1mdr-restore[0;22m | [36;1mdata-restores-from-the-off-cluster-backup[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 00:21:28 | [36;1mdr-restore[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 00:21:28 | [32;1mdr-restore[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-cunning-hen
|
||||
l.go:52: | 00:21:36 | [36;1mdr-restore[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/dr-restore (149.05s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -1,2 +0,0 @@
|
||||
{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "57d4df8c4f992732a49f3e2daddcace866778ab0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260901T235117Z-e9a4b66", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-01T23:51:17Z"}
|
||||
{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "fdc3265585f4b775c0a6ae3fb33c837b0c643655", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260902T001907Z-fdc3265", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-02T00:19:07Z"}
|
||||
@@ -4,7 +4,6 @@ Every test is a **Chainsaw `Test`** in its own directory:
|
||||
|
||||
```
|
||||
tests/
|
||||
l0-static/ L0 — whole-repo invariants over the tracked tree (no cluster)
|
||||
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
|
||||
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
|
||||
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
|
||||
@@ -15,35 +14,6 @@ tests/
|
||||
The layer a test belongs to is its subtree **and** its `.../layer` annotation; the
|
||||
coverage generator cross-checks the two.
|
||||
|
||||
### L0 static (`l0-static/`) — what belongs here, and what does not
|
||||
|
||||
docs/14's L0 row is *"Pre-apply, in CI · manifests/ServiceDefinitions obey invariants"*.
|
||||
Kyverno (`platform/kyverno/`) covers the invariants expressible over a **single
|
||||
resource**. This subtree covers the L0 invariants that are **properties of the repository
|
||||
as a whole** and that no per-resource policy can state — e.g. *no plaintext secret exists
|
||||
anywhere in the tracked tree*, *no tracked RoleBinding grants a workload identity a
|
||||
privileged role*, *the spec-immutability gate actually blocks a modified spec*.
|
||||
|
||||
Each `l0-static/<name>/` dir carries a **`check.sh`**: a self-contained offline assertion
|
||||
script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:
|
||||
|
||||
- by `conformance/run.sh --layer l0` (and as the offline leg of a full run), via
|
||||
`lib/run_l1_offline.py`, producing a REAL pass/fail JUnit entry — never a skip; and
|
||||
- by the merge gate's **`ci/l0.sh`** stage, so these assertions run on every PR; and
|
||||
- by the wrapping Chainsaw `Test`, which runs the SAME `check.sh`, so an in-cluster run
|
||||
executes byte-identical assertions.
|
||||
|
||||
Two authoring rules specific to this layer:
|
||||
|
||||
1. **State the gap.** A statically-provable half of a requirement is still only half.
|
||||
Every `check.sh` header ends with an explicit *"what this does NOT prove"* list naming
|
||||
the cluster-layer remainder. Covering a requirement at L0 does not finish it.
|
||||
2. **Exemptions must be self-policing.** Where a check must tolerate something (a
|
||||
deliberately-plaintext policy fixture, an `.example` template), the exemption is
|
||||
conditioned on a property that would break if the exemption were abused — e.g. the
|
||||
plaintext Secret fixture is exempt *only while* a `kyverno-test.yaml` asserts the
|
||||
policy rejects it. Never a bare path allowlist.
|
||||
|
||||
## Requirement-tagging convention (authoritative)
|
||||
|
||||
A test declares the requirement(s) it verifies **three** ways; the label is the source
|
||||
@@ -58,7 +28,7 @@ metadata:
|
||||
annotations:
|
||||
# Human-readable canonical list — MUST equal the label set.
|
||||
conformance.home.k3s/requirements: "R-NET-01"
|
||||
# Declared layer (L0|L1|L2|L3|META) — must match the subtree.
|
||||
# Declared layer (L1|L2|L3|META) — must match the subtree.
|
||||
conformance.home.k3s/layer: "L2"
|
||||
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
@@ -93,52 +63,6 @@ and coverage extraction exact and robust.
|
||||
break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
|
||||
- **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
|
||||
the PM — do not relax the assertion or the requirement.
|
||||
- **Never write a per-environment value into a test (R-ENV-02).** No hostname suffix, no
|
||||
Gateway VIP, no LB-pool range. One authored suite must run green against dev *and* prod,
|
||||
so a test **asks the cluster it is pointed at** what it is. The mechanism is a chainsaw
|
||||
binding over the same ConfigMap Flux substitutes from:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
bindings:
|
||||
- name: baseDomain
|
||||
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
||||
```
|
||||
|
||||
Declare it at **test level** (step `outputs`/`bindings` are step-scoped), then consume it:
|
||||
`($baseDomain)` for a pod `env:` value, `(join('.', ['myhost', $baseDomain]))` where a
|
||||
full hostname is needed — including inside a `fixtures/*.yaml` applied with `apply: file:`,
|
||||
which chainsaw templates with the same bindings. Prefer reading the *actual* observed value
|
||||
over restating a range (e.g. assert the DNS answer equals `GATEWAY_VIP`, or equals the LB IP
|
||||
the Service was assigned) — it is both stricter and environment-independent. Never take the
|
||||
environment from a CLI flag or an exported variable: a flag someone forgets is how a suite
|
||||
ends up silently asserting the other cluster's expectations. L1 goldens are the one
|
||||
exception: they are rendering fixtures and carry the literal `${BASE_DOMAIN}` token.
|
||||
|
||||
### How a scaffold declares itself (and how `status.py` tells one from a verifier)
|
||||
|
||||
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
|
||||
later roadmap phase delivers the capability under test. `lib/status.py` classifies each
|
||||
test EXECUTABLE or SCAFFOLD from the test's own files — `chainsaw-test.yaml`,
|
||||
`render-check.sh`, `check.sh` — and either signal below **outranks**
|
||||
`conformance.home.k3s/executable: "true"`, so a placeholder body can never be reported as
|
||||
an executable test. Declare a scaffold in **both** positions:
|
||||
|
||||
1. **`# STATUS: scaffold — <what is missing>`** in the file's header comment block. The
|
||||
declared state must be the first token after `STATUS:` (executable tests say
|
||||
`# STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`).
|
||||
2. **A `TODO`/`FIXME` marker carrying a message, in the body** — the file from its first
|
||||
non-comment line onward. `echo "TODO(T-4.2): inject a fault to fire a rule"`,
|
||||
or an inline comment on a missing fixture:
|
||||
`file: fixtures/smtp-sink.yaml # TODO(T-4.1): hermetic SMTP capture sink`.
|
||||
|
||||
What is **not** a scaffold marker, and why the distinction is load-bearing: a marker
|
||||
mentioned in the **header prose** (the Given/When/Then and the mandatory "what this does
|
||||
NOT prove" list), and a marker that is the **entire content of a quoted literal** — a
|
||||
placeholder *detector*'s own vocabulary, e.g. `SEED = ["TBD", "TODO", "FIXME", ...]` in
|
||||
`l0-static/placeholder-guard-coupling/check.sh`. A whole-file `"TODO" in text` test used to
|
||||
classify that 32-assertion check as a scaffold and, through it, report the safety-critical
|
||||
R-STATE-03 as `exec=1 scaffold=1`. If your check must hold a marker as data, keep each
|
||||
idiom a **standalone quoted literal in a list**; a marker with text attached (a regex
|
||||
alternation, several idioms in one string) still classifies as a scaffold, because
|
||||
ambiguity here only ever downgrades.
|
||||
later roadmap phase delivers the capability under test.
|
||||
@@ -1,106 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the five auditability proofs can be produced on demand (R-SEC-03) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
|
||||
# operator can prove isolation, egress path, cert validity, and backup integrity)"),
|
||||
# made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
|
||||
# must be able to prove": Isolation (denied-probe test), Egress path (egress IP
|
||||
# check), Kill-switch (fault-injection test), TLS (cert status + external check),
|
||||
# Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
|
||||
# one-time checks", with the acceptance gate "The five auditability proofs above can
|
||||
# be produced on demand."
|
||||
#
|
||||
# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
|
||||
# -----------------------------------------------------------------------------------
|
||||
# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
|
||||
# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
|
||||
# failure modes are asserted instead, each by DRIVING the real machinery:
|
||||
# (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
|
||||
# the spec's own words require (fault injection / a DR drill => L3), taken from
|
||||
# lib/status.py's own verdict so this check cannot disagree with the status report;
|
||||
# (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
|
||||
# must resolve to real test dirs AND pass the merge gate's coverage assertion;
|
||||
# (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
|
||||
# attribute — a proof that passes but records nothing leaves the operator with a green
|
||||
# terminal and no evidence;
|
||||
# (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
|
||||
# real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
|
||||
# token, and a ledger row claiming `proven` at the repository's ROOT commit — and
|
||||
# requires record_run.py to report not-proven and status.py to report `stale-proof`. It
|
||||
# also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
|
||||
# ci/ or run.sh ran before this test existed.
|
||||
# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
|
||||
# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
|
||||
# so a sixth proof added to the spec fails here instead of being silently unverified.
|
||||
#
|
||||
# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
|
||||
# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
|
||||
# anyone needs it.
|
||||
#
|
||||
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
||||
# WHEN lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
|
||||
# lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
|
||||
# and two synthetic forgeries are presented to them,
|
||||
# THEN each of docs/07's five proofs maps to an executable test at the required layer, the
|
||||
# documented one-command invocation selects and asserts them, each is capturable as a
|
||||
# ledger row, and neither a self-skip nor a stale row can be recorded as proof.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
|
||||
# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
|
||||
# it directly, and this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
|
||||
# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
|
||||
# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
|
||||
# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
|
||||
# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
|
||||
# assertion here was NOT relaxed to accommodate them (R-TEST-06).
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * That the five proofs PASS. This asserts they are present, strong, runnable by one
|
||||
# command, capturable, and that the chain refuses forgeries — not that isolation holds
|
||||
# today (R-NET-01's test) or that the restore works (R-STATE-03's).
|
||||
# * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
|
||||
# suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
|
||||
# * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
|
||||
# would need signed history.
|
||||
# * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
|
||||
# requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
|
||||
# not re-asserted here.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: auditability-proofs-producible
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree and the harness's own tooling, when status.py, the requirement
|
||||
resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
|
||||
tools' self-tests are driven and two synthetic forgeries are presented to them, then each
|
||||
of docs/07's five auditability proofs has an executable test at the layer the spec
|
||||
requires, is selectable and asserted by the documented one-command invocation, is
|
||||
capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
|
||||
evidence.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
|
||||
# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
|
||||
# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
|
||||
# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
|
||||
# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
|
||||
# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
|
||||
# the wrong trade (R-TEST-06).
|
||||
@@ -1,431 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — auditability-proofs-producible
|
||||
# Serves: R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an operator can
|
||||
# prove isolation, egress path, cert validity, and backup integrity)."
|
||||
# — spec docs/00-vision-and-scope.md).
|
||||
#
|
||||
# The spec makes this meta-requirement concrete. docs/07-security-and-secrets.md
|
||||
# §"Auditability requirements (R-SEC-03) — what an operator must be able to prove":
|
||||
# - **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a
|
||||
# denied-probe test.
|
||||
# - **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
|
||||
# - **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
|
||||
# - **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert
|
||||
# status + external check.
|
||||
# - **Backups**: "irreplaceable data is restorable" — via a DR drill.
|
||||
# "These proofs are part of the platform acceptance suite, not one-time checks."
|
||||
# and its acceptance gate: "The five auditability proofs above can be produced on demand."
|
||||
#
|
||||
# HOW A META-REQUIREMENT IS TESTED WITHOUT BECOMING A TAUTOLOGY
|
||||
# ------------------------------------------------------------
|
||||
# The trap is obvious and this check is built to avoid it: "a test exists for R-NET-01,
|
||||
# therefore isolation is auditable" is true of any suite that compiles, proves nothing, and
|
||||
# would keep passing while every proof rotted into a scaffold. Four separate failure modes
|
||||
# have to be caught for "an operator can PROVE it" to mean anything, and each one is asserted
|
||||
# by DRIVING the real machinery rather than by reading files:
|
||||
#
|
||||
# (a) THE PROOF EXISTS, AT THE RIGHT STRENGTH. Each of the five named proofs maps to a
|
||||
# requirement that must have an EXECUTABLE verifying test — not a placeholder scaffold — and,
|
||||
# where the spec's own words name fault injection ("with VPN down"; "a DR drill"), an
|
||||
# executable L3 test specifically. Executable-vs-scaffold is not re-derived here: this
|
||||
# check RUNS lib/status.py and reads its verdict, so it cannot disagree with the
|
||||
# repository's own status report.
|
||||
# (b) THE PROOF IS PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND. `run.sh
|
||||
# --requirements <the five IDs>` must resolve to a non-empty test set and must ASSERT
|
||||
# their coverage (that flag is the merge gate's own path: it passes --require to
|
||||
# coverage.py). This check drives run.sh itself, so if the resolver, the labels or the
|
||||
# gate wiring break, R-SEC-03 fails.
|
||||
# (c) THE PROOF IS CAPTURABLE. A test that passes but records nothing leaves an operator
|
||||
# with a green terminal and no evidence. For each Chainsaw-layer proof, the test must
|
||||
# declare a PASS SENTINEL that lib/record_run.py can actually attribute — computed with
|
||||
# record_run.py's OWN parser, so deleting a test's sentinel makes this check fail
|
||||
# instead of silently making the requirement unprovable.
|
||||
# (d) THE EVIDENCE CANNOT BE FORGED, AND IS NOT HONOURED WHEN STALE. This is the clause
|
||||
# that makes the check bite. Two INDEPENDENT forgery attempts are mounted against the
|
||||
# real tools, in a throwaway directory:
|
||||
# * a Chainsaw run in which the JUnit says a real proof test PASSED while the run log
|
||||
# carries only that test's SKIP token — the exact shape of a self-skipping L2/L3
|
||||
# test. record_run.py must refuse to call it `proven`.
|
||||
# * a ledger row claiming `proven` at a commit that PREDATES the requirement's
|
||||
# implementing code — status.py must degrade it to `stale-proof`, never green.
|
||||
# Both tools' own built-in self-tests (`--selftest`) are also driven, because nothing
|
||||
# in ci/ or run.sh runs them today: the guards that keep a skip from becoming a proof
|
||||
# were themselves unexercised in the merge gate until this check.
|
||||
#
|
||||
# The FIVE PROOFS ARE NOT HAND-LISTED where the spec can be read: when the read-only spec
|
||||
# repo is mounted, the bullet labels are parsed out of docs/07 and must match this check's
|
||||
# mapping EXACTLY. Add a sixth proof to the spec and this check fails as unmapped rather
|
||||
# than quietly ignoring it.
|
||||
#
|
||||
# WHY L0: every clause above is a property of the repository and its own tooling, provable
|
||||
# offline, and it belongs in the merge gate — the point of an audit chain is that it is intact
|
||||
# BEFORE anyone needs it. Nothing here needs a cluster, and nothing here would be more true
|
||||
# with one.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
|
||||
# - THAT THE FIVE PROOFS PASS. This asserts the proofs are present, strong, runnable by one
|
||||
# documented command, capturable as evidence, and that the evidence chain refuses
|
||||
# forgeries. Whether isolation actually holds today is R-NET-01's test; whether the DR
|
||||
# restore works is R-STATE-03's. R-SEC-03 is about the operator's ABILITY to produce the
|
||||
# proof, which is what is asserted.
|
||||
# - HUBBLE. docs/07 names "Hubble flow logs" as part of the isolation proof. No test in this
|
||||
# suite queries Hubble, and this check does not pretend otherwise: it asserts the
|
||||
# denied-probe half, which is the half that is automated. The Hubble half remains an
|
||||
# operator-interactive capability — a real, named gap in R-SEC-03's coverage.
|
||||
# - THE OPERATOR-FACING NARRATIVE. That a human can find and read the evidence (the status
|
||||
# report's prose, the ledger's provenance fields) is a usability property, not a testable
|
||||
# one.
|
||||
# - LEDGER COMPLETENESS OVER TIME. The ledger is append-only by convention and by review;
|
||||
# nothing here can prove a row was never removed from it (that would need signed history).
|
||||
# - THAT THE PROOFS COVER "ALL THE ABOVE PROPERTIES". R-SEC-03's own wording is broader than
|
||||
# the five bullets docs/07 enumerates. The five are what the spec made concrete and are
|
||||
# what is asserted; suite-wide coverage of every requirement is R-TEST-01/04's claim,
|
||||
# tested by tests/meta/harness-selfcheck, and is deliberately NOT re-asserted here.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
|
||||
export SPEC_REPO
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
CONF = ROOT / "conformance"
|
||||
LIB = CONF / "lib"
|
||||
FAILURES: list[str] = []
|
||||
NOTES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def note(msg):
|
||||
NOTES.append(msg)
|
||||
|
||||
|
||||
def die():
|
||||
print(f"CHECK FAIL (auditability-proofs-producible): {len(FAILURES)} of {CHECKS} "
|
||||
"assertions failed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def run(cmd, **kw) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(cmd, capture_output=True, text=True, cwd=kw.pop("cwd", CONF), **kw)
|
||||
|
||||
|
||||
# ================================================= THE FIVE PROOFS docs/07 ENUMERATES
|
||||
# key = the bullet label docs/07 uses (verified against the spec below when it is mounted)
|
||||
# reqs = the requirement(s) whose verifying test IS that proof
|
||||
# layer= the WEAKEST layer the spec's own words allow. Where docs/07 says "a fault-injection
|
||||
# test" / "a DR drill", an L2 happy-path test does not discharge the proof.
|
||||
PROOFS = {
|
||||
"Isolation": {"reqs": ["R-NET-01"], "layers": {"L2", "L3"},
|
||||
"why": '"service X cannot reach service Y" — via Hubble flow logs + a '
|
||||
"denied-probe test"},
|
||||
"Egress path": {"reqs": ["R-NET-05"], "layers": {"L2", "L3"},
|
||||
"why": '"service X\'s public IP is the VPN\'s" — via an egress IP check'},
|
||||
"Kill-switch": {"reqs": ["R-NET-06"], "layers": {"L3"},
|
||||
"why": '"with VPN down, X leaks nothing" — via a fault-injection test'},
|
||||
"TLS": {"reqs": ["R-TLS-01"], "layers": {"L2", "L3"},
|
||||
"why": '"every endpoint serves a valid, non-self-signed, unexpired cert" '
|
||||
"— via cert status + external check"},
|
||||
"Backups": {"reqs": ["R-STATE-03"], "layers": {"L3"},
|
||||
"why": '"irreplaceable data is restorable" — via a DR drill'},
|
||||
}
|
||||
PROOF_REQS = sorted({r for p in PROOFS.values() for r in p["reqs"]})
|
||||
|
||||
# ---- SELF-POLICING: the proof list must still be the SPEC's proof list ---------------
|
||||
spec = Path(os.environ.get("SPEC_REPO", ""))
|
||||
sec_doc = spec / "docs" / "07-security-and-secrets.md"
|
||||
if sec_doc.is_file():
|
||||
text = sec_doc.read_text()
|
||||
m = re.search(r"^##\s+Auditability requirements \(R-SEC-03\).*?$(.*?)^##\s", text,
|
||||
re.M | re.S)
|
||||
check(m is not None,
|
||||
"docs/07 no longer carries an '## Auditability requirements (R-SEC-03)' section — "
|
||||
"the proof list this check is coupled to has moved or been removed, so the mapping "
|
||||
"below can no longer be trusted (R-TEST-06: follow the spec, never the other way)")
|
||||
if m:
|
||||
labels = re.findall(r"^\s*-\s+\*\*([^*]+)\*\*:", m.group(1), re.M)
|
||||
check(set(labels) == set(PROOFS),
|
||||
f"docs/07 enumerates the auditability proofs {sorted(labels)} but this check "
|
||||
f"maps {sorted(PROOFS)} — every proof the spec names must map to a verifying "
|
||||
"test, and a proof this check does not know about would be silently unverified")
|
||||
check(len(labels) == 5,
|
||||
f"docs/07 enumerates {len(labels)} auditability proof(s); its own acceptance "
|
||||
"gate says 'The five auditability proofs above can be produced on demand' — the "
|
||||
"prose and the list have drifted apart")
|
||||
else:
|
||||
note(f"the read-only spec repo is not mounted at {spec} — the proof list could not be "
|
||||
"re-derived from docs/07, so the mapping below was used as written. Every other "
|
||||
"assertion still ran. (ci/l0.sh mounts the spec; a bare local run may not.)")
|
||||
|
||||
# ============================================ (a) EACH PROOF EXISTS, AT THE RIGHT STRENGTH
|
||||
# Driven through lib/status.py so this check and the repository's own status report cannot
|
||||
# disagree about what is executable and what is a scaffold.
|
||||
tmp = Path(tempfile.mkdtemp(prefix="r-sec-03-"))
|
||||
try:
|
||||
out = tmp / "status"
|
||||
st = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
|
||||
"--registry", str(CONF / "requirements" / "registry.yaml"),
|
||||
"--tests", str(CONF / "tests"), "--out", str(out)])
|
||||
check(st.returncode == 0,
|
||||
f"lib/status.py exited {st.returncode} — the report an operator would use to find "
|
||||
f"the audit evidence does not generate: {(st.stderr or st.stdout).strip()[:400]}")
|
||||
status = {}
|
||||
sj = out / "status.json"
|
||||
if sj.is_file():
|
||||
status = json.loads(sj.read_text())
|
||||
check(bool(status.get("by_requirement")),
|
||||
"lib/status.py produced no per-requirement report — the audit posture of each "
|
||||
"requirement is not derivable, so R-SEC-03 cannot be assessed at all")
|
||||
by_req = status.get("by_requirement", {})
|
||||
|
||||
for label, spec_proof in sorted(PROOFS.items()):
|
||||
for rid in spec_proof["reqs"]:
|
||||
rec = by_req.get(rid)
|
||||
check(rec is not None,
|
||||
f"proof '{label}' maps to {rid}, which the status report does not know — "
|
||||
"either the registry lost the requirement or this mapping is stale")
|
||||
if not rec:
|
||||
continue
|
||||
tests = (rec.get("verification") or {}).get("tests") or []
|
||||
executable = [t for t in tests if t.get("state", "").startswith("executable")]
|
||||
check(bool(executable),
|
||||
f"proof '{label}' ({rid}) has no EXECUTABLE verifying test — its tests are "
|
||||
f"{[t.get('name') + ':' + t.get('state', '?') for t in tests]}. docs/07: "
|
||||
f"{spec_proof['why']}. A placeholder scaffold cannot be produced on demand, so the "
|
||||
"operator cannot prove this property")
|
||||
at_layer = [t for t in executable if t.get("layer") in spec_proof["layers"]]
|
||||
check(bool(at_layer),
|
||||
f"proof '{label}' ({rid}) has no executable test at layer(s) "
|
||||
f"{sorted(spec_proof['layers'])} — it has "
|
||||
f"{[t.get('name') + '@' + str(t.get('layer')) for t in executable]}. "
|
||||
f"docs/07 requires this proof to be produced {spec_proof['why']}, and a "
|
||||
"weaker layer does not discharge it (a rendering check is not a drill)")
|
||||
|
||||
# ==================================== (b) PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND
|
||||
# The resolver: --requirements must translate the five R-IDs into real test dirs.
|
||||
resolver = run(["bash", "-c",
|
||||
f'export CONF_ROOT="{CONF}"; . "{LIB}/common.sh"; '
|
||||
f'conf_dirs_for_requirements "{" ".join(PROOF_REQS)}"'])
|
||||
dirs = [l for l in resolver.stdout.splitlines() if l.strip()]
|
||||
check(resolver.returncode == 0 and len(dirs) >= len(PROOF_REQS),
|
||||
f"`run.sh --requirements {','.join(PROOF_REQS)}` resolves to {len(dirs)} test dir(s) "
|
||||
f"for {len(PROOF_REQS)} proof requirement(s) — the single command docs/07's "
|
||||
"acceptance gate depends on ('produced on demand') cannot select the proofs")
|
||||
|
||||
# The gate: the same flag ASSERTS coverage (run.sh passes --require to coverage.py), so a
|
||||
# zero exit here is the repository's own statement that all five proofs are covered.
|
||||
gate = run(["./run.sh", "--coverage-only", "--requirements", ",".join(PROOF_REQS)])
|
||||
check(gate.returncode == 0,
|
||||
f"`./run.sh --coverage-only --requirements {','.join(PROOF_REQS)}` exited "
|
||||
f"{gate.returncode} — the merge gate's own coverage assertion rejects the audit "
|
||||
f"proof set: {(gate.stderr or gate.stdout).strip()[:400]}")
|
||||
|
||||
# ================================================ (c) THE PROOF IS CAPTURABLE AS EVIDENCE
|
||||
# Use record_run.py's OWN sentinel parser: a Chainsaw-layer proof with no declared pass
|
||||
# token can pass forever and never be recorded as proof of anything.
|
||||
sys.path.insert(0, str(LIB))
|
||||
os.environ.setdefault("PYTHONDONTWRITEBYTECODE", "1")
|
||||
sys.dont_write_bytecode = True
|
||||
import record_run as rr # noqa: E402
|
||||
|
||||
smap = rr.sentinel_map(CONF / "tests")
|
||||
check(len(smap) >= 10,
|
||||
f"record_run.py's sentinel map contains {len(smap)} test(s) — the parser lost its "
|
||||
"grip on the test tree, so the assertions below would assert nothing")
|
||||
for label, spec_proof in sorted(PROOFS.items()):
|
||||
for rid in spec_proof["reqs"]:
|
||||
rec = by_req.get(rid) or {}
|
||||
tests = (rec.get("verification") or {}).get("tests") or []
|
||||
names = [t.get("name") for t in tests
|
||||
if t.get("state", "").startswith("executable")
|
||||
and t.get("layer") in spec_proof["layers"]]
|
||||
for name in names:
|
||||
ent = smap.get(name)
|
||||
check(ent is not None,
|
||||
f"proof '{label}': record_run.py does not know the test {name!r} — its "
|
||||
"result could never be attributed to a ledger row")
|
||||
if not ent:
|
||||
continue
|
||||
if ent.get("layer") in rr.SENTINEL_EXEMPT_LAYERS:
|
||||
continue # offline layers are truthful by JUnit alone
|
||||
check(bool(ent.get("pass")),
|
||||
f"proof '{label}': the Chainsaw test {name!r} declares NO pass sentinel, "
|
||||
"so lib/record_run.py can never record it as proven. It would go green "
|
||||
"in the terminal and leave the operator with no evidence — exactly the "
|
||||
"gap the ledger exists to close (docs/07: the proofs are 'part of the "
|
||||
"platform acceptance suite, not one-time checks')")
|
||||
check(bool(ent.get("terminal_pass")),
|
||||
f"proof '{label}': the test {name!r} has no TERMINAL pass token, so "
|
||||
"record_run.py cannot decide its verdict")
|
||||
|
||||
# =============================== (d) THE EVIDENCE CANNOT BE FORGED, NOR HONOURED WHEN STALE
|
||||
# Both tools' own guards, which NOTHING in ci/ or run.sh drives today.
|
||||
for tool in ("record_run.py", "status.py"):
|
||||
r = run([sys.executable, str(LIB / tool), "--selftest"])
|
||||
check(r.returncode == 0,
|
||||
f"`lib/{tool} --selftest` exited {r.returncode} — the guards that keep a "
|
||||
"self-skipped or stale run from being recorded as proof are broken, so nothing "
|
||||
f"in the evidence chain can be trusted: {(r.stderr or r.stdout).strip()[:500]}")
|
||||
|
||||
# ---- FORGERY 1: a green JUnit for a REAL proof test whose log holds only a SKIP token.
|
||||
# This is the precise shape of a self-skipping L2/L3 test, and it must NOT become proof.
|
||||
forge_target, forge_skip = None, None
|
||||
for label, spec_proof in sorted(PROOFS.items()):
|
||||
for rid in spec_proof["reqs"]:
|
||||
for name, ent in smap.items():
|
||||
if rid in (ent.get("requirements") or []) and ent.get("skip") \
|
||||
and ent.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS:
|
||||
forge_target, forge_skip = name, ent["skip"][0]
|
||||
break
|
||||
if forge_target:
|
||||
break
|
||||
if forge_target:
|
||||
break
|
||||
check(forge_target is not None,
|
||||
"no Chainsaw-layer proof test declares a skip token, so the forgery attempt below "
|
||||
"could not be mounted. That is not a pass: it means this check could not verify "
|
||||
"that a self-skip is refused as proof, which is the whole point of the ledger")
|
||||
if forge_target:
|
||||
run_dir = tmp / "20260101T000000Z-forged"
|
||||
run_dir.mkdir(parents=True)
|
||||
(run_dir / "report.junit.xml").write_text(
|
||||
'<?xml version="1.0" encoding="UTF-8"?>\n'
|
||||
'<testsuites><testsuite name="chainsaw" tests="1" failures="0" errors="0">'
|
||||
f'<testcase name="{forge_target}" classname="forged"/>'
|
||||
"</testsuite></testsuites>\n")
|
||||
(run_dir / "run.log").write_text(
|
||||
f"=== RUN chainsaw/{forge_target}\n"
|
||||
f" === STDOUT\n {forge_skip}\n")
|
||||
forged = run([sys.executable, str(LIB / "record_run.py"), "--run-dir", str(run_dir),
|
||||
"--dry-run", "--ledger", str(tmp / "never-written.jsonl"),
|
||||
"--tests", str(CONF / "tests"), "--repo", str(ROOT)])
|
||||
verdicts = re.findall(rf"^\[record\]\s+\S+\s+(\S+)\s+{re.escape(forge_target)}\b",
|
||||
forged.stdout, re.M)
|
||||
check(verdicts and all(v != "proven" for v in verdicts),
|
||||
f"lib/record_run.py recorded a FORGED run as {verdicts!r} for {forge_target!r}: "
|
||||
"a JUnit saying `passed` with nothing in the log but that test's own SKIP token "
|
||||
f"({forge_skip}). A self-skip must never become proof — with this hole every "
|
||||
"audit proof in docs/07 could be produced by a cluster that skipped it")
|
||||
check(not (tmp / "never-written.jsonl").exists(),
|
||||
"record_run.py --dry-run WROTE to the ledger — a tool that records while "
|
||||
"claiming not to cannot be trusted to record only real runs")
|
||||
|
||||
# ---- FORGERY 2: a ledger row claiming `proven` at a commit that predates the code.
|
||||
# status.py must degrade it to stale-proof (not green), or an old green run would keep
|
||||
# vouching for code it never executed.
|
||||
root_sha = run(["git", "-C", str(ROOT), "rev-list", "--max-parents=0", "HEAD"])
|
||||
old_sha = (root_sha.stdout.strip().splitlines() or [""])[0]
|
||||
check(len(old_sha) == 40,
|
||||
"the repository's root commit could not be resolved, so the stale-proof guard could "
|
||||
"not be exercised (that is a gap in this check, not a pass)")
|
||||
if len(old_sha) == 40 and forge_target:
|
||||
stale_ledger = tmp / "stale.jsonl"
|
||||
rid = sorted(smap[forge_target].get("requirements") or ["R-NET-06"])[0]
|
||||
stale_ledger.write_text(json.dumps({
|
||||
"ts_utc": "2026-01-01T00:00:00Z", "git_sha": old_sha, "env": "dev",
|
||||
"chainsaw_version": "v0.0.0", "kube_version": "unknown",
|
||||
"layer": smap[forge_target].get("layer", "L3"), "test": forge_target,
|
||||
"requirements": [rid], "junit_result": "passed",
|
||||
"sentinel": (smap[forge_target].get("terminal_pass") or "X"),
|
||||
"proof": "proven", "run_id": "20260101T000000Z-forged",
|
||||
"note": "synthetic row written by the R-SEC-03 conformance check",
|
||||
}) + "\n")
|
||||
out2 = tmp / "status-stale"
|
||||
st2 = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
|
||||
"--registry", str(CONF / "requirements" / "registry.yaml"),
|
||||
"--tests", str(CONF / "tests"), "--out", str(out2),
|
||||
"--ledger", str(stale_ledger)])
|
||||
check(st2.returncode == 0,
|
||||
f"lib/status.py failed against a synthetic ledger: "
|
||||
f"{(st2.stderr or st2.stdout).strip()[:300]}")
|
||||
s2 = json.loads((out2 / "status.json").read_text()) if (out2 / "status.json").is_file() else {}
|
||||
rec2 = (s2.get("by_requirement") or {}).get(rid) or {}
|
||||
per = {p.get("test"): p.get("state") for p in ((rec2.get("proof") or {}).get("per_test") or [])}
|
||||
check(per.get(forge_target) == "stale-proof",
|
||||
f"a ledger row claiming `proven` for {forge_target!r} at the repository's ROOT "
|
||||
f"commit was reported as {per.get(forge_target)!r} instead of 'stale-proof'. A "
|
||||
"proof from a run that predates the code it vouches for is not evidence, and "
|
||||
"honouring it would let one old green run certify every later change")
|
||||
check((rec2.get("proof") or {}).get("state") != "proven",
|
||||
f"{rid} is reported PROVEN on the strength of a row from the root commit — the "
|
||||
"freshness guard is not holding")
|
||||
|
||||
# ---- the ledger itself is machine-readable, and no row proves a test that is gone ----
|
||||
ledger = CONF / "runs" / "ledger.jsonl"
|
||||
if ledger.is_file():
|
||||
rows, bad = [], []
|
||||
for i, line in enumerate(ledger.read_text().splitlines(), 1):
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
rows.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
bad.append(i)
|
||||
check(not bad,
|
||||
f"conformance/runs/ledger.jsonl has unparsable line(s) {bad} — the audit record "
|
||||
"is not machine-readable, so no proof can be derived from it")
|
||||
for r in rows:
|
||||
for key in ("ts_utc", "git_sha", "test", "requirements", "junit_result", "proof",
|
||||
"run_id"):
|
||||
check(key in r,
|
||||
f"a ledger row for {r.get('test', '?')!r} lacks {key!r} — a row without "
|
||||
"provenance cannot be audited")
|
||||
if r.get("proof") == "proven":
|
||||
check(bool(r.get("sentinel")),
|
||||
f"a ledger row claims `proven` for {r.get('test')!r} with NO sentinel — "
|
||||
"the sentinel is the only thing separating a real pass from a self-skip")
|
||||
check(r.get("test") in smap,
|
||||
f"a ledger row claims `proven` for {r.get('test')!r}, which is not a "
|
||||
"test in the suite any more — a proof of something that no longer exists")
|
||||
check("-dirty" not in str(r.get("git_sha", "")),
|
||||
f"a ledger row claims `proven` for {r.get('test')!r} from a DIRTY tree "
|
||||
f"({r.get('git_sha')}) — what ran is not what is committed")
|
||||
else:
|
||||
note("conformance/runs/ledger.jsonl does not exist yet — the ledger-shape assertions "
|
||||
"were skipped; every other assertion ran, and the forgery probes above use "
|
||||
"synthetic ledgers, not this file")
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
||||
check(CHECKS >= 30,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
die()
|
||||
for n in NOTES:
|
||||
print(f" note: {n}")
|
||||
print(f"check OK: auditability-proofs-producible — {CHECKS} assertions; the five docs/07 "
|
||||
f"proofs ({', '.join(sorted(PROOFS))}) each map to an executable test at the layer the "
|
||||
f"spec's own wording requires, `run.sh --requirements {','.join(PROOF_REQS)}` resolves "
|
||||
"and asserts them, each Chainsaw-layer proof declares a sentinel record_run.py can "
|
||||
"attribute, both evidence tools' self-tests pass, and two independent forgeries — a "
|
||||
"self-skip presented as a pass, and a `proven` row from the root commit — were REFUSED. "
|
||||
"R-SEC-03's producibility half only: whether each proof PASSES is its own "
|
||||
"requirement's test, and the Hubble half of the isolation proof is automated nowhere "
|
||||
"(see the header's gap list).")
|
||||
PY
|
||||
@@ -1,75 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — dev/test nodes are provisioned as code, end to end (R-ENV-01) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC:
|
||||
# created, baselined, cluster-joined, and managed as code"); the four stages
|
||||
# are enumerated in docs/02 §"Node lifecycle stages".
|
||||
#
|
||||
# WHY L0: R-ENV-01 is a requirement about HOW the dev nodes came to exist, and a live
|
||||
# Node object cannot answer it — a node built by OpenTofu and one built by hand in a
|
||||
# hypervisor UI are indistinguishable through the Kubernetes API. What is provable, with
|
||||
# no cluster, is that a COMPLETE and EXECUTABLE provisioning path exists for all four
|
||||
# stages and covers EVERY dev node. For the Ansible half that is proven by DRIVING the
|
||||
# real toolchain offline (`--syntax-check`, `--list-hosts` — neither opens a connection
|
||||
# to any node), not by reading files: a chain that does not resolve fails the check.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count appears in the test.
|
||||
# The dev group names are read from the playbooks' own default host patterns, the node
|
||||
# names are generated from the IaC module's prefix/count variables, and the two sets are
|
||||
# compared against each other — so the assertion survives a rename or a resize.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the provisioning chain is parsed and (where offline execution is possible) driven,
|
||||
# THEN the IaC module creates the dev nodes on a LOCAL KVM hypervisor (allowlisted
|
||||
# provider — a cloud provider fails), from a for_each map rather than a count index,
|
||||
# each with CPU/RAM/OS disk/dedicated Longhorn data disk/NIC/cloud-init seed; the
|
||||
# node set it creates is EXACTLY the set Ansible manages; the baseline stage applies
|
||||
# the package floor, kernel modules, sysctls, data-disk mount, swap-off and iscsid
|
||||
# declaratively; one k3s role founds the cluster with --cluster-init and joins the
|
||||
# rest with --server while disabling flannel/kube-proxy/traefik/servicelb at
|
||||
# install; and site.yml's chain (baseline -> k3s -> Cilium -> verify -> Flux)
|
||||
# resolves and targets every dev node at every stage.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
|
||||
# where present — ansible-playbook in no-connection modes; no cluster, no network);
|
||||
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
|
||||
# wraps the SAME script.
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * That the RUNNING dev VMs were created by this module — needs the OpenTofu state or
|
||||
# hypervisor access, neither available offline.
|
||||
# * `tofu validate` — needs `tofu init` (provider registry), i.e. network. The module is
|
||||
# parsed, not evaluated; a network-allowed CI stage would close this.
|
||||
# * "Never hand-edited on the box" (docs/02 stage 4's second half) — a live drift check.
|
||||
# * REBUILD/re-converge (docs/02's own proof line) — exercised at L3 by
|
||||
# tests/l3-fault-injection/node-loss (R-STATE-02).
|
||||
# * docs/02 prose puts the baseline in cloud-init; the implementation moved it into the
|
||||
# Ansible node_baseline role. R-ENV-01 requires the node to be "baselined ... as code",
|
||||
# not that cloud-init be the tool, so the check asserts the STAGE and RECORDS the
|
||||
# divergence rather than hiding it.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: dev-nodes-provisioned-as-code
|
||||
labels:
|
||||
conformance.home.k3s/R-ENV-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree, when the dev provisioning chain is parsed and driven
|
||||
offline, then every dev node is created by local-KVM IaC, baselined, cluster-joined
|
||||
and handed to Flux by one code path that covers all of them — with no node outside
|
||||
it and no stage missing.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-every-dev-node-is-created-baselined-joined-and-managed-as-code
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,442 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — dev-nodes-provisioned-as-code
|
||||
# Serves: R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: created,
|
||||
# baselined, cluster-joined, and managed as code." — spec
|
||||
# docs/00-vision-and-scope.md; the four stages are enumerated in
|
||||
# docs/02-environments-and-provisioning.md §"Node lifecycle stages
|
||||
# (R-ENV-01 — 'created, baselined, cluster-joined, managed as code')":
|
||||
# 1. Create — OpenTofu instantiates the VM
|
||||
# 2. Baseline — OS baseline and prerequisites
|
||||
# 3. Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables
|
||||
# k3s bundled flannel/traefik/servicelb/kube-proxy
|
||||
# 4. Manage — GitOps-managed by Flux; node config changes are made via IaC
|
||||
# and re-applied, never hand-edited on the box.)
|
||||
#
|
||||
# WHY L0 IS THE RIGHT LAYER, AND WHAT IT MEANS TO TEST THIS BEHAVIOURALLY
|
||||
# ----------------------------------------------------------------------
|
||||
# R-ENV-01 is a requirement about HOW the dev nodes come to exist. The live cluster
|
||||
# cannot answer it: a Kubernetes node object looks identical whether it was built by
|
||||
# OpenTofu or by hand in a hypervisor UI (there is no API field for "was I created by
|
||||
# IaC"). What CAN be proven, with no cluster, is that a COMPLETE, EXECUTABLE
|
||||
# provisioning path exists for all four stages and covers every dev node — and, for the
|
||||
# Ansible half, that is proven by RUNNING the real toolchain in a no-connection mode
|
||||
# (`--syntax-check`, `--list-hosts`) rather than by reading the files, so a chain that
|
||||
# does not actually resolve fails here. The IaC half is asserted by parsing the module:
|
||||
# `tofu init/validate` needs the provider registry, and L0 is offline by contract.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the provisioning chain is parsed and (where offline execution is possible)
|
||||
# actually driven,
|
||||
# THEN CREATE: the IaC module targets a LOCAL KVM hypervisor (every declared provider
|
||||
# is a KVM-family hypervisor provider — no cloud provider may appear),
|
||||
# instantiates its nodes from a for_each map (never a count index, so one
|
||||
# node can be replaced without disturbing its siblings), and gives each
|
||||
# node CPU, RAM, an OS disk, a DEDICATED Longhorn data disk, a NIC and a
|
||||
# cloud-init seed carrying the login user + SSH key;
|
||||
# AND the node set it creates is EXACTLY the dev node set Ansible manages —
|
||||
# names derived from the module's own prefix/count, so no node can be
|
||||
# hand-added to the inventory or hand-created in the hypervisor;
|
||||
# BASELINE: a baseline stage exists as declarative Ansible (package floor, kernel
|
||||
# modules persisted AND loaded, sysctls, the Longhorn data disk formatted
|
||||
# and mounted, swap off, iscsid enabled), driven by variables with
|
||||
# defaults rather than per-host special cases;
|
||||
# JOIN: one role installs k3s for both roles of every node — `--cluster-init`
|
||||
# on the primary, `--server` join on the secondaries — with flannel,
|
||||
# kube-proxy, traefik and servicelb disabled AT INSTALL, and Cilium
|
||||
# installed as the CNI afterwards;
|
||||
# MANAGE: site.yml chains baseline -> k3s -> Cilium -> verify -> Flux bootstrap;
|
||||
# `ansible-playbook --syntax-check` on that chain SUCCEEDS; and
|
||||
# `--list-hosts` shows every one of those stages targeting EVERY dev
|
||||
# node, so no node is outside the code path.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count is written down here.
|
||||
# The dev group names come from the playbooks' own default host patterns, the node names
|
||||
# from the IaC module's prefix/count variables, and the two sets are compared to each
|
||||
# other. This check therefore keeps working when the estate is renamed or resized.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
|
||||
# - That the RUNNING dev VMs were in fact created by this module. Nothing offline can:
|
||||
# it would need the OpenTofu state (a remote R2 backend + credentials) or hypervisor
|
||||
# access. `tofu plan` against the live estate is the operator's own check.
|
||||
# - That the module APPLIES cleanly. `tofu validate` needs `tofu init`, which downloads
|
||||
# the provider from the registry; L0 is offline by contract, so the module is parsed,
|
||||
# not evaluated. ci/lint.sh does not cover .tf either — a gap worth closing in CI
|
||||
# with a network-allowed stage, not here.
|
||||
# - "Never hand-edited on the box" (stage 4's second half) — a drift check against the
|
||||
# live nodes, which is an operator/Ansible-check-mode exercise, not a repo property.
|
||||
# - REBUILD/re-converge ("destroy a dev node VM and rebuild it", docs/02's own proof
|
||||
# line) — that is exercised at L3 by tests/l3-fault-injection/node-loss (R-STATE-02).
|
||||
# - That the baseline actually ran on a node (cluster/host state), and that the
|
||||
# hypervisor is on the operator's own LAN rather than remote — "local" is asserted as
|
||||
# "a KVM hypervisor provider driving a self-hosted endpoint variable", not as a
|
||||
# network topology fact.
|
||||
# - DEVIATION FROM docs/02 PROSE, ASSERTED DELIBERATELY: docs/02 describes stage 2 as
|
||||
# cloud-init applying the OS baseline. The implementation moved the rich baseline into
|
||||
# the Ansible `node_baseline` role (cloud-init keeps hostname/user/SSH/network),
|
||||
# because Proxmox provisioning is API-only and a custom cloud-init snippet needs
|
||||
# SSH/SFTP to the hypervisor. R-ENV-01's own wording requires the node to be
|
||||
# "baselined ... as code", not that cloud-init be the tool, so this check asserts the
|
||||
# STAGE, wherever it lives, and records the divergence rather than hiding it.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
NOTES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def note(msg):
|
||||
NOTES.append(msg)
|
||||
|
||||
|
||||
# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
|
||||
# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
|
||||
# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
|
||||
# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
|
||||
# narrow: every other non-zero exit is a real failure. The parse assertions always run.
|
||||
ENV_ONLY_FAILURE = re.compile(
|
||||
r"vault password|vault-id|Attempting to decrypt"
|
||||
r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
|
||||
|
||||
TOFU = ROOT / "infra" / "tofu"
|
||||
ANS = ROOT / "infra" / "ansible"
|
||||
check(TOFU.is_dir(), "infra/tofu/ is missing — the CREATE stage has no IaC at all (R-ENV-01)")
|
||||
check(ANS.is_dir(), "infra/ansible/ is missing — the BASELINE/JOIN/MANAGE stages have no code")
|
||||
if not TOFU.is_dir() or not ANS.is_dir():
|
||||
print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
TF = "\n".join((p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))))
|
||||
check(len(TF) > 500, "infra/tofu/*.tf is empty or a stub — nothing creates the dev nodes")
|
||||
|
||||
# ============================================================ STAGE 1 — CREATE (IaC)
|
||||
# A LOCAL KVM hypervisor, by allowlist: any provider outside this set fails, so moving
|
||||
# dev onto a cloud (or anything that is not a KVM hypervisor the operator runs) is caught
|
||||
# here rather than being noticed later. Each entry is justified, not just listed:
|
||||
# bpg/proxmox Proxmox VE — a KVM/QEMU hypervisor the operator hosts.
|
||||
# dmacvicar/libvirt libvirt/KVM directly (the pre-Proxmox substrate).
|
||||
KVM_PROVIDERS = {"bpg/proxmox", "dmacvicar/libvirt"}
|
||||
providers = dict(re.findall(r'source\s*=\s*"([^"]+)"', TF) and
|
||||
[(m.group(1), m.group(2)) for m in
|
||||
re.finditer(r'(\w[\w-]*)\s*=\s*\{[^}]*?source\s*=\s*"([^"]+)"', TF, re.S)])
|
||||
check(len(providers) > 0,
|
||||
"infra/tofu declares no required_providers — the substrate is unpinned, so 'via IaC' "
|
||||
"cannot be checked at all")
|
||||
for alias, source in sorted(providers.items()):
|
||||
check(source.lower() in KVM_PROVIDERS,
|
||||
f"infra/tofu declares provider {alias}={source!r}, which is not a local KVM hypervisor "
|
||||
f"provider (allowed: {sorted(KVM_PROVIDERS)}) — R-ENV-01 requires dev/test to be LOCAL KVMs")
|
||||
|
||||
# The VM resource: for_each-keyed, and carrying the hardware + seed docs/02 enumerates.
|
||||
vm_blocks = re.findall(r'resource\s+"([\w]*(?:_vm|_domain))"\s+"(\w+)"\s*\{(.*?)\n\}',
|
||||
TF, re.S)
|
||||
check(len(vm_blocks) >= 1,
|
||||
"no VM/domain resource found in infra/tofu — nothing in IaC creates a dev node")
|
||||
for rtype, rname, body in vm_blocks:
|
||||
check("for_each" in body,
|
||||
f'resource "{rtype}" "{rname}" is not for_each-keyed. A count index makes single-node '
|
||||
"replace unsafe (a -target'ed destroy cascaded to sibling nodes — DECISIONS-LOG F1), so "
|
||||
"'created ... as code' would not survive a rebuild")
|
||||
check(not re.search(r"^\s*count\s*=", body, re.M),
|
||||
f'resource "{rtype}" "{rname}" uses a positional count index')
|
||||
for want, why in (
|
||||
(r"\bcpu\b", "CPU"),
|
||||
(r"\bmemory\b", "RAM"),
|
||||
(r"network_device|network_interface", "a NIC"),
|
||||
(r"initialization|cloud_init|cloudinit", "a cloud-init seed"),
|
||||
):
|
||||
check(re.search(want, body) is not None,
|
||||
f'resource "{rtype}" "{rname}" declares no {why} — docs/02 stage 1 requires CPU, RAM, '
|
||||
"disks, NICs and a cloud-init seed to come from IaC")
|
||||
disks = len(re.findall(r"^\s*disk\s*\{", body, re.M))
|
||||
check(disks >= 2,
|
||||
f'resource "{rtype}" "{rname}" declares {disks} disk block(s); docs/02 requires the OS '
|
||||
"disk PLUS a dedicated data disk for Longhorn")
|
||||
check(re.search(r"user_account|ssh_authorized_key|ssh_keys", body) is not None,
|
||||
f'resource "{rtype}" "{rname}" seeds no login user / SSH key — the node could not be '
|
||||
"reached by the BASELINE stage without a hand step")
|
||||
|
||||
# Node identity comes from variables, and those variables define the whole node set.
|
||||
def tf_default(var: str):
|
||||
m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
|
||||
if not m:
|
||||
return None
|
||||
d = re.search(r'default\s*=\s*(.+)', m.group(1))
|
||||
return d.group(1).strip() if d else None
|
||||
|
||||
|
||||
prefix_raw = tf_default("node_name_prefix")
|
||||
count_raw = tf_default("node_count")
|
||||
ips_raw = tf_default("node_ips")
|
||||
check(prefix_raw is not None and count_raw is not None,
|
||||
"infra/tofu has no node_name_prefix/node_count variables — the node set is not derived "
|
||||
"from code, so the created nodes cannot be compared with the managed ones")
|
||||
prefix = (prefix_raw or "").strip('"')
|
||||
try:
|
||||
node_count = int((count_raw or "0").strip())
|
||||
except ValueError:
|
||||
node_count = 0
|
||||
check(node_count >= 1, f"node_count default is {count_raw!r} — no node would be created")
|
||||
if ips_raw is not None:
|
||||
n_ips = len(re.findall(r'"[^"]+"', ips_raw))
|
||||
check(n_ips >= node_count,
|
||||
f"node_ips default has {n_ips} address(es) for node_count={node_count} — the module "
|
||||
"cannot address every node it creates")
|
||||
check((TOFU / "outputs.tf").is_file() and
|
||||
re.search(r'output\s+"(ansible_inventory|nodes)"', (TOFU / "outputs.tf").read_text()) is not None,
|
||||
"infra/tofu/outputs.tf surfaces no node inventory output — the JOIN stage would have to be "
|
||||
"told the node facts by hand instead of consuming them from IaC")
|
||||
|
||||
# ------------------------------------------------- the dev node set, derived twice over
|
||||
# Group names are taken from the PLAYBOOKS' own default host patterns, so nothing here
|
||||
# restates the estate's naming.
|
||||
PLAYBOOKS = ANS / "playbooks"
|
||||
|
||||
|
||||
def default_patterns(playbook: str) -> list[str]:
|
||||
p = PLAYBOOKS / playbook
|
||||
if not p.is_file():
|
||||
return []
|
||||
return re.findall(r"default\('([^']+)'\)", p.read_text())
|
||||
|
||||
|
||||
dev_groups = set(default_patterns("node-baseline.yml") + default_patterns("k3s.yml"))
|
||||
check(len(dev_groups) > 0,
|
||||
"no playbook declares a default host pattern — the dev target is not expressed in code")
|
||||
|
||||
INV = ANS / "inventory" / "hosts.ini"
|
||||
check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — no node is managed as code")
|
||||
ini_groups: dict[str, list[str]] = {}
|
||||
children: dict[str, list[str]] = {}
|
||||
cur = None
|
||||
for line in (INV.read_text().splitlines() if INV.is_file() else []):
|
||||
s = line.strip()
|
||||
if not s or s.startswith("#"):
|
||||
continue
|
||||
m = re.match(r"^\[([^\]]+)\]$", s)
|
||||
if m:
|
||||
cur = m.group(1)
|
||||
(children if cur.endswith(":children") else ini_groups).setdefault(
|
||||
cur.split(":")[0] if cur.endswith(":children") else cur, [])
|
||||
continue
|
||||
if cur is None or cur.endswith(":vars"):
|
||||
continue
|
||||
host = s.split()[0]
|
||||
if cur.endswith(":children"):
|
||||
children.setdefault(cur.split(":")[0], []).append(host)
|
||||
else:
|
||||
ini_groups.setdefault(cur, []).append(host)
|
||||
|
||||
|
||||
def hosts_in(group: str, seen=None) -> set[str]:
|
||||
seen = seen or set()
|
||||
if group in seen:
|
||||
return set()
|
||||
seen.add(group)
|
||||
out = set(ini_groups.get(group, []))
|
||||
for child in children.get(group, []):
|
||||
out |= hosts_in(child, seen)
|
||||
return out
|
||||
|
||||
|
||||
dev_hosts: set[str] = set()
|
||||
for g in sorted(dev_groups):
|
||||
dev_hosts |= hosts_in(g)
|
||||
check(len(dev_hosts) > 0,
|
||||
f"the dev groups {sorted(dev_groups)} resolve to NO hosts in hosts.ini — nothing is managed")
|
||||
expected = {f"{prefix}{i + 1}" for i in range(node_count)}
|
||||
check(dev_hosts == expected,
|
||||
f"the dev nodes Ansible manages ({sorted(dev_hosts)}) are not exactly the nodes the IaC "
|
||||
f"module creates ({sorted(expected)}) — a node outside IaC is a hand-built node (R-ENV-01)")
|
||||
|
||||
# ============================================================ STAGE 2 — BASELINE (code)
|
||||
BASE_ROLE = ANS / "roles" / "node_baseline"
|
||||
check(BASE_ROLE.is_dir(),
|
||||
"no node_baseline role — the BASELINE stage has no code (docs/02 stage 2)")
|
||||
base_tasks = "\n".join(p.read_text(errors="replace")
|
||||
for p in sorted((BASE_ROLE / "tasks").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
|
||||
base_defaults = "\n".join(p.read_text(errors="replace")
|
||||
for p in sorted((BASE_ROLE / "defaults").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
|
||||
# Each item docs/02 names for stage 2, matched on the ANSIBLE MODULE that would do it —
|
||||
# a declarative, idempotent task, not a shell one-liner.
|
||||
for pattern, what in (
|
||||
(r"ansible\.builtin\.apt:", "a package baseline (apt module)"),
|
||||
(r"modprobe:", "kernel modules loaded (modprobe module)"),
|
||||
(r"modules-load|/etc/modules", "kernel modules PERSISTED across reboot"),
|
||||
(r"sysctl", "sysctl tuning"),
|
||||
(r"filesystem:|mkfs", "a filesystem on the Longhorn data disk"),
|
||||
(r"mount:|ansible\.posix\.mount", "the Longhorn data disk MOUNTED"),
|
||||
(r"swap", "swap disabled"),
|
||||
(r"iscsid", "iscsid enabled (Longhorn prerequisite)"),
|
||||
):
|
||||
check(re.search(pattern, base_tasks) is not None,
|
||||
f"node_baseline does not declare {what} — docs/02 stage 2 requires the OS baseline and "
|
||||
"k3s/Longhorn/KubeVirt prerequisites to be applied as code")
|
||||
check(re.search(r"^node_baseline_packages:", base_defaults, re.M) is not None and
|
||||
re.search(r"^node_baseline_kernel_modules:", base_defaults, re.M) is not None,
|
||||
"node_baseline's package/module sets are not variables with defaults — a per-host special "
|
||||
"case is not 'managed as code' (and could not be scoped per environment, R-ENV-02)")
|
||||
|
||||
# ========================================================== STAGE 3 — CLUSTER-JOIN (k3s)
|
||||
K3S_ROLE = ANS / "roles" / "k3s_server"
|
||||
k3s_text = "\n".join(p.read_text(errors="replace")
|
||||
for p in sorted((K3S_ROLE).rglob("*.yml"))) if K3S_ROLE.is_dir() else ""
|
||||
check(K3S_ROLE.is_dir() and len(k3s_text) > 200,
|
||||
"no k3s_server role — nothing joins the nodes into a cluster (docs/02 stage 3)")
|
||||
check("--cluster-init" in k3s_text,
|
||||
"the k3s role never passes --cluster-init — the HA embedded-etcd cluster is never founded")
|
||||
check(re.search(r"--server\s+https", k3s_text) is not None,
|
||||
"the k3s role never joins secondaries with --server — nodes would not form one cluster")
|
||||
check(re.search(r"K3S_TOKEN|--token", k3s_text) is not None,
|
||||
"the k3s role never supplies a join token — the join could only be completed by hand")
|
||||
# The bundled components docs/02 requires to be disabled AT INSTALL, each checked by the
|
||||
# flag that actually disables it.
|
||||
for pattern, what in (
|
||||
(r"--flannel-backend=none", "flannel (Cilium is the CNI)"),
|
||||
(r"--disable-kube-proxy", "kube-proxy (Cilium kube-proxy replacement)"),
|
||||
(r"traefik", "traefik (Gateway API via Cilium)"),
|
||||
(r"servicelb", "servicelb"),
|
||||
):
|
||||
check(re.search(pattern, k3s_text) is not None,
|
||||
f"the k3s install does not disable {what} — docs/02 stage 3 requires it disabled at "
|
||||
"install, or the bundled component fights the platform's own")
|
||||
check((ANS / "roles" / "cilium").is_dir(),
|
||||
"no cilium role — stage 3 requires Cilium to be installed as the CNI at bootstrap")
|
||||
check((ANS / "roles" / "flux_bootstrap").is_dir(),
|
||||
"no flux_bootstrap role — stage 4's hand-off to GitOps has no code")
|
||||
|
||||
# ============================================== STAGE 4 — MANAGE + the chain is REAL
|
||||
SITE = PLAYBOOKS / "site.yml"
|
||||
check(SITE.is_file(), "no playbooks/site.yml — the provisioning chain is not expressed as one path")
|
||||
site_txt = SITE.read_text() if SITE.is_file() else ""
|
||||
order = [m for m in re.findall(r"import_playbook:\s*(\S+)", site_txt)]
|
||||
check(order[:1] == ["node-baseline.yml"],
|
||||
f"site.yml imports {order} — the BASELINE stage must come first (docs/02 sequencing)")
|
||||
for want in ("k3s.yml", "cilium.yml", "flux-bootstrap.yml"):
|
||||
check(want in order, f"site.yml does not import {want} — a lifecycle stage is missing from the chain")
|
||||
if "k3s.yml" in order and "cilium.yml" in order:
|
||||
check(order.index("k3s.yml") < order.index("cilium.yml"),
|
||||
"site.yml installs Cilium before k3s — the CNI cannot precede the cluster")
|
||||
|
||||
# --- DRIVE THE REAL TOOLCHAIN, OFFLINE ------------------------------------------------
|
||||
# --syntax-check and --list-hosts open NO connection to any node: they resolve the
|
||||
# playbook, roles, collections and inventory only. Running them is the difference between
|
||||
# "the chain is written down" and "the chain resolves".
|
||||
if shutil.which("ansible-playbook") is None:
|
||||
note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
|
||||
"chain was PARSED but not DRIVEN; the merge gate's container has it and does drive it")
|
||||
else:
|
||||
# cwd=infra/ansible so the tracked ansible.cfg (roles_path, collections_path, inventory)
|
||||
# is the one in force — the same resolution a real run gets.
|
||||
def ansible(*args) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(["ansible-playbook", *args], cwd=ANS,
|
||||
capture_output=True, text=True)
|
||||
|
||||
syn = ansible("--syntax-check", "playbooks/site.yml")
|
||||
syn_out = (syn.stderr or "") + (syn.stdout or "")
|
||||
lh = None
|
||||
if syn.returncode != 0 and ENV_ONLY_FAILURE.search(syn_out):
|
||||
# NARROW, SELF-POLICING SKIP — see ENV_ONLY_FAILURE above.
|
||||
note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
|
||||
"passphrase and/or the gitignored Galaxy collections), so --syntax-check/--list-hosts "
|
||||
"were not driven; every parse assertion above still ran")
|
||||
else:
|
||||
check(syn.returncode == 0,
|
||||
"`ansible-playbook --syntax-check playbooks/site.yml` FAILED — the provisioning chain "
|
||||
f"does not even load: {syn_out.strip()[:400]}")
|
||||
lh = ansible("--list-hosts", "playbooks/site.yml")
|
||||
check(lh.returncode == 0,
|
||||
"`ansible-playbook --list-hosts playbooks/site.yml` FAILED — the chain does not "
|
||||
f"resolve against the tracked inventory: "
|
||||
f"{((lh.stderr or '') + (lh.stdout or '')).strip()[:400]}")
|
||||
if lh is not None and lh.returncode == 0:
|
||||
# Parse plays -> hosts. NEVER `ansible-inventory --list`: it prints DECRYPTED vault values.
|
||||
plays: list[tuple[str, set[str]]] = []
|
||||
name, hosts, in_hosts = None, set(), False
|
||||
for line in lh.stdout.splitlines():
|
||||
m = re.match(r"\s*play #\d+ \([^)]*\):\s*(.*?)\s*TAGS:", line)
|
||||
if m:
|
||||
if name is not None:
|
||||
plays.append((name, hosts))
|
||||
name, hosts, in_hosts = m.group(1), set(), False
|
||||
continue
|
||||
if re.match(r"\s*hosts \(\d+\):", line):
|
||||
in_hosts = True
|
||||
continue
|
||||
if in_hosts and line.strip() and not line.strip().startswith("pattern:"):
|
||||
hosts.add(line.strip())
|
||||
if name is not None:
|
||||
plays.append((name, hosts))
|
||||
check(len(plays) >= 4,
|
||||
f"--list-hosts resolved {len(plays)} play(s) — the four-stage chain did not expand")
|
||||
# Every lifecycle stage must cover EVERY dev node. A stage that reaches two of three
|
||||
# nodes means the third was finished by hand.
|
||||
STAGES = {
|
||||
"baseline": r"baseline",
|
||||
"cluster-join": r"^k3s (primary|secondar)",
|
||||
"cni": r"cilium",
|
||||
"manage (flux hand-off)": r"flux",
|
||||
}
|
||||
for stage, rx in STAGES.items():
|
||||
covered: set[str] = set()
|
||||
matched = 0
|
||||
for pname, phosts in plays:
|
||||
if re.search(rx, pname, re.I):
|
||||
matched += 1
|
||||
covered |= phosts
|
||||
check(matched >= 1, f"no play in site.yml implements the {stage} stage")
|
||||
check(covered == dev_hosts,
|
||||
f"the {stage} stage targets {sorted(covered)}, not every dev node "
|
||||
f"{sorted(dev_hosts)} — a node outside the code path is a hand-provisioned "
|
||||
"node (R-ENV-01)")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
||||
check(CHECKS >= 30,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
for n in NOTES:
|
||||
print(f" note: {n}")
|
||||
print(f"check OK: dev-nodes-provisioned-as-code — {CHECKS} assertions; "
|
||||
f"{len(dev_hosts)} dev node(s) {sorted(dev_hosts)} created by IaC ({sorted(providers.values())}), "
|
||||
f"baselined, cluster-joined and Flux-managed by the same chain. R-ENV-01's authored-path "
|
||||
"clause only — the live VMs' provenance, `tofu validate`, and rebuild/re-converge are NOT "
|
||||
"proven here (see the header's gap list; rebuild is L3 node-loss).")
|
||||
PY
|
||||
@@ -1,94 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — two egress paths, and the rule set that selects between them (R-NET-04) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
|
||||
# routing rules ... realized as two paths — the local `internet` path and the
|
||||
# in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress
|
||||
# routing: two paths, one physical WAN": "the `internet` path is simply the default
|
||||
# route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and
|
||||
# the nodes themselves — egress the local/private WAN by default and **never** the
|
||||
# VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the
|
||||
# kill-switch blackholes **only** that class."
|
||||
#
|
||||
# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here)
|
||||
# ------------------------------------------------------------------------
|
||||
# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class
|
||||
# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and
|
||||
# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES
|
||||
# that select among them, which nothing else asserts:
|
||||
# * exactly TWO paths, the same closed set in the schema, the admission policy and the
|
||||
# renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope);
|
||||
# * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one
|
||||
# Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one
|
||||
# `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because
|
||||
# absence of a matching rule IS the default path;
|
||||
# * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a
|
||||
# vpn-class pod's cluster/LAN traffic still takes the local path;
|
||||
# * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing
|
||||
# table, `route-noexec` is set and a pushed `redirect-gateway` is ignored;
|
||||
# * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every
|
||||
# DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN.
|
||||
# (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.)
|
||||
#
|
||||
# WHY L0: every clause above is a statement about ALL rules — including rules that must be
|
||||
# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default
|
||||
# route in the main table). A cluster can show which path two pods took; it cannot show that
|
||||
# no third rule exists or that the vocabulary is closed.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are
|
||||
# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod
|
||||
# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked
|
||||
# against its own values.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy
|
||||
# routing / OpenVPN / nftables templates, and every environment's overlay addresses are
|
||||
# parsed and cross-checked,
|
||||
# THEN there are exactly two paths with `internet` as the default; exactly one steering rule
|
||||
# selects exactly the vpn class by label; the gateway is selected by node label with a
|
||||
# named egress interface; all RFC1918 space and every address any environment assigns
|
||||
# itself is excluded from the tunnel; the tunnel's default route and its fail-closed
|
||||
# blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN
|
||||
# cannot take the main table; and no nftables DROP is unscoped.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
||||
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
||||
# Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by
|
||||
# tests/l2-behavioral/egress-default-path (same requirement label); the two labelled
|
||||
# classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test.
|
||||
# * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates
|
||||
# that generate them; a hand-added rule on the box is invisible here.
|
||||
# * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no
|
||||
# honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default".
|
||||
# * IPv6 egress.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: egress-paths-selected-by-rules
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree, when the egress vocabulary, every steering rule and the gateway
|
||||
node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers
|
||||
exactly two egress paths with the WAN as the default, exactly one rule selects exactly
|
||||
the vpn class, local destinations stay on the local path in every environment, the
|
||||
tunnel never becomes the default route, and nothing unscoped can blackhole the default
|
||||
path.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,513 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — egress-paths-selected-by-rules
|
||||
# Serves: R-NET-04 ("The platform MUST support **multiple egress paths** selected by routing
|
||||
# rules. Per OQ-1 the environment has **one physical commercial WAN and no
|
||||
# multi-ISP plans**, so this is realized as two paths — the local `internet` path and
|
||||
# the in-cluster `vpn` path — not multi-ISP routing. Physical multi-WAN (multiple ISP
|
||||
# uplinks) is **out of scope** unless a future ADR reopens it."
|
||||
# — spec docs/00-vision-and-scope.md).
|
||||
#
|
||||
# The elaborating spec text this check quotes and asserts against — docs/03
|
||||
# §"Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)":
|
||||
# | `internet` | Out the single commercial WAN | Normal pod egress via node -> SOHO edge
|
||||
# router -> internet |
|
||||
# | `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN
|
||||
# egress gateway (ADR-0013) |
|
||||
# "**Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing
|
||||
# to policy-route across; the `internet` path is simply the default route out."
|
||||
# "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes
|
||||
# themselves — egress the local/private WAN by default and **never** the VPN. Only
|
||||
# services labeled `egress-class=vpn` are steered to the tunnel, and the kill-switch
|
||||
# blackholes **only** that class."
|
||||
# "It is **split-tunnel**: it creates `tun0` but does **not** set the node's default
|
||||
# route (`route-noexec` + policy routing), so the node's own traffic stays on the WAN —
|
||||
# only what Cilium points at `tun0` uses the VPN."
|
||||
#
|
||||
# WHAT R-NET-04 ADDS OVER R-NET-05/07 (which are ALREADY covered) — READ THIS FIRST
|
||||
# ---------------------------------------------------------------------------------
|
||||
# R-NET-05/07 are about a SERVICE: it can declare its class, enforcement happens outside it,
|
||||
# and the class can be flipped via GitOps. Those are proven by
|
||||
# tests/l1-rendering/egress-class-render (the render) and tests/l2-behavioral/egress-class
|
||||
# (two labelled pods observe two different public IPs, and a relabel re-routes the same pod).
|
||||
# Re-tagging either of those with R-NET-04 would be a duplicate wearing a new label.
|
||||
#
|
||||
# R-NET-04 is about the PLATFORM's PATH SET and the RULES THAT SELECT AMONG THEM. Its
|
||||
# distinctive content — none of which any existing test asserts — is:
|
||||
#
|
||||
# (1) THERE ARE EXACTLY TWO PATHS, and the same two everywhere. The class vocabulary is
|
||||
# one closed set of two values agreed by the schema, the admission policy and the
|
||||
# renderer. A third value appearing in one of them but not the others would be a path
|
||||
# nothing routes (or a path nothing validates); "multi-ISP" reappearing as a third
|
||||
# class would contradict OQ-1's resolution.
|
||||
# (2) SELECTION IS BY A ROUTING RULE, and the rule set is TOTAL AND DISJOINT. The `vpn`
|
||||
# path is entered only via rules that select the vpn class — a Cilium
|
||||
# CiliumEgressGatewayPolicy pod-selector, and on the gateway node an `ip rule` scoped
|
||||
# to the tunnel source address. Nothing selects the `internet` class, because
|
||||
# "the `internet` path is simply the default route out": ABSENCE of a matching rule IS
|
||||
# the default path. So an unlabelled pod, and an `internet`-class pod, take the WAN.
|
||||
# (3) THE PATHS COEXIST — the choice is per-destination, not per-pod-lifetime. The vpn
|
||||
# rule tunnels public destinations while EXCLUDING all private space, so a vpn-class
|
||||
# pod's cluster/LAN traffic still takes the local path at the same time as its
|
||||
# internet traffic takes the tunnel.
|
||||
# (4) THE VPN PATH NEVER BECOMES THE DEFAULT. Split tunnel: the tunnel's default route
|
||||
# lives in a DEDICATED routing table reached only through the scoped rule, the
|
||||
# provider's pushed `redirect-gateway` is ignored, and `route-noexec` keeps OpenVPN
|
||||
# from touching the main table. Otherwise there would be one path (the VPN), not two.
|
||||
# (5) THE KILL-SWITCH BLACKHOLES ONLY THAT CLASS. The blackhole default is inside the
|
||||
# dedicated table; the nftables chain's policy is ACCEPT and every DROP is qualified
|
||||
# by the vpn class's own source set / fwmark. The `internet` path therefore fails
|
||||
# OPEN and can never be blackholed. (R-NET-06's opposite claim — that `vpn` fails
|
||||
# CLOSED — is proven at L3 by tests/l3-fault-injection/vpn-killswitch. This check
|
||||
# asserts the OTHER direction, which that test does not.)
|
||||
#
|
||||
# WHY L0. Every clause above is a property of the RULE SET, i.e. of authored configuration:
|
||||
# totality ("nothing selects the default path"), disjointness ("exactly one rule, selecting
|
||||
# exactly one class"), and scoping ("no unqualified drop, no default route in the main
|
||||
# table") are statements about ALL rules, including the ones that would have to be absent.
|
||||
# A running cluster can show you which path two pods took; it cannot show you that no third
|
||||
# rule exists, that no rule is unscoped, or that the vocabulary is closed. The behavioural
|
||||
# complement — that an UNLABELLED pod really does take the WAN path — is at L2 in
|
||||
# tests/l2-behavioral/egress-default-path (same requirement label).
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no address is written down. The private ranges the vpn rule must
|
||||
# exclude are checked as CONTAINMENT of the addresses each environment's own Flux overlay
|
||||
# declares (pod CIDR, Gateway VIP, LB pool, API host) — so a new environment on a different
|
||||
# LAN is checked against its own addresses, and the check keeps working when they change.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
|
||||
# - Behaviour. That a vpn-class pod's packets really leave via tun0, that an unlabelled
|
||||
# pod's really do not, and that the tunnel is up at all. L2 covers the two labelled
|
||||
# classes (tests/l2-behavioral/egress-class, R-NET-05) and the unlabelled default
|
||||
# (tests/l2-behavioral/egress-default-path, R-NET-04); the kill-switch firing is L3.
|
||||
# - THE RUNNING NODE's routing tables and nftables ruleset. This reads the templates that
|
||||
# generate them, driven by no node. A hand-added `ip rule` on the gateway node is
|
||||
# invisible here (and to every other test); drift on the box is an Ansible check-mode
|
||||
# exercise for the operator.
|
||||
# - PHYSICAL MULTI-WAN. Out of scope per OQ-1, so its absence is not asserted: there is no
|
||||
# honest repository signal that would distinguish "one WAN, correctly" from "one WAN,
|
||||
# because nobody tried" — and inventing one would be a hollow assertion.
|
||||
# - IPv6. The policy's CIDRs and the routing rules are IPv4-only, matching the platform.
|
||||
# If IPv6 egress is ever enabled, the `internet` path would need the same analysis and
|
||||
# this check would NOT catch its absence.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import ipaddress
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
LBL = "homelab.keithjackson.dev"
|
||||
CLASS_LABEL = f"{LBL}/egress-class"
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def die():
|
||||
print(f"CHECK FAIL (egress-paths-selected-by-rules): {len(FAILURES)} of {CHECKS} "
|
||||
"assertions failed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def docs(path: Path):
|
||||
"""Every YAML document in a file, skipping empties."""
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml")
|
||||
if ".git" not in p.parts and "_artifacts" not in p.parts)
|
||||
SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json"
|
||||
RENDER = ROOT / "catalog" / "render.py"
|
||||
VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway"
|
||||
CLUSTERS = ROOT / "clusters"
|
||||
|
||||
for p in (SCHEMA, RENDER, VPN_ROLE, CLUSTERS):
|
||||
check(p.exists(), f"{p.relative_to(ROOT)} is missing — the egress path model cannot be "
|
||||
"read at all (R-NET-04)")
|
||||
if FAILURES:
|
||||
die()
|
||||
|
||||
|
||||
# ======================================================= (1) EXACTLY TWO PATHS, AGREED
|
||||
import json # noqa: E402 — only needed for the schema
|
||||
|
||||
schema = json.loads(SCHEMA.read_text())
|
||||
egress_prop = ((schema.get("properties") or {}).get("egress") or {})
|
||||
enum = egress_prop.get("enum")
|
||||
check(isinstance(enum, list) and len(enum) == 2,
|
||||
f"schemas/servicedefinition.schema.json declares egress enum {enum!r} — R-NET-04 is "
|
||||
"realized as EXACTLY TWO paths (OQ-1: one WAN, no multi-ISP). A third value is a path "
|
||||
"nothing routes; one value is not 'multiple egress paths'")
|
||||
PATHS = set(enum or [])
|
||||
check(PATHS == {"internet", "vpn"},
|
||||
f"the declared egress paths are {sorted(PATHS)} — docs/03's table fixes them as "
|
||||
"`internet` (out the single commercial WAN) and `vpn` (the in-cluster tunnel)")
|
||||
check(egress_prop.get("default") == "internet",
|
||||
f"the schema's default egress class is {egress_prop.get('default')!r} — docs/03: "
|
||||
"'Default is the WAN; the VPN is strictly opt-in', and 'the `internet` path is simply "
|
||||
"the default route out'. A service that says nothing must get the WAN")
|
||||
|
||||
# The admission policy must constrain the pod label to the SAME closed set, or a workload
|
||||
# could carry a class the schema never allowed (and that no rule routes).
|
||||
pol_files = sorted((ROOT / "platform" / "kyverno").rglob("*egress*.yaml"))
|
||||
check(len(pol_files) >= 1,
|
||||
"no Kyverno policy file mentions egress under platform/kyverno — nothing stops a pod "
|
||||
"carrying an egress class outside the declared path set")
|
||||
pol_text = "\n".join(p.read_text(errors="replace") for p in pol_files)
|
||||
for cls in sorted(PATHS):
|
||||
check(re.search(rf"\b{re.escape(cls)}\b", pol_text) is not None,
|
||||
f"the egress admission policy never names the {cls!r} path — a class the policy "
|
||||
"does not know cannot be validated against the path set")
|
||||
# No tracked object may carry a class outside the closed set — with ONE self-policing
|
||||
# exemption. A Kyverno policy-test fixture deliberately carries an INVALID class so the
|
||||
# admission test can assert the policy REJECTS it; that pod is exempt only while a
|
||||
# kyverno-test.yaml in the same tests tree still expects `result: fail` for it. Delete the
|
||||
# expectation (or flip it to pass) and the exemption evaporates — never a bare path allowlist.
|
||||
def rejected_by_a_policy_test(resource_name: str, fixture: Path) -> bool:
|
||||
tests_root = fixture.parent
|
||||
while tests_root != ROOT and tests_root.name != "tests":
|
||||
tests_root = tests_root.parent
|
||||
if tests_root == ROOT:
|
||||
return False
|
||||
for tf in sorted(tests_root.rglob("kyverno-test.yaml")):
|
||||
for d in docs(tf):
|
||||
for case in (d.get("results") or []):
|
||||
if case.get("result") == "fail" and resource_name in (case.get("resources") or []):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
stray: list[str] = []
|
||||
_STRAY_RE = re.compile(rf"{re.escape(CLASS_LABEL)}:\s*[\"']?([A-Za-z][A-Za-z0-9-]*)")
|
||||
for f in TRACKED_YAML:
|
||||
for d in docs(f):
|
||||
labels = ((d.get("metadata") or {}).get("labels") or {})
|
||||
cls = labels.get(CLASS_LABEL)
|
||||
# Also catch pod-template labels (Deployments/DaemonSets/Jobs/VMs).
|
||||
tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}).get("labels") or {}
|
||||
for value, owner in ((cls, d), (tmpl.get(CLASS_LABEL), d)):
|
||||
if value is None or value in PATHS or value == "CLASS":
|
||||
continue
|
||||
name = ((owner.get("metadata") or {}).get("name") or "?")
|
||||
if rejected_by_a_policy_test(name, f):
|
||||
continue
|
||||
stray.append(f"{f.relative_to(ROOT)}:{name}={value}")
|
||||
# Values written as raw text (templates, docs-as-code) are caught too.
|
||||
for value in set(_STRAY_RE.findall(f.read_text(errors="replace"))):
|
||||
if value in PATHS or value == "CLASS":
|
||||
continue
|
||||
if any(value in s for s in stray):
|
||||
continue
|
||||
# Raw-text hit with no parsed owner: exempt only if EVERY object in the file that
|
||||
# carries this value is a policy-test fixture proven to be rejected.
|
||||
owners = [((d.get("metadata") or {}).get("name") or "?") for d in docs(f)
|
||||
if ((d.get("metadata") or {}).get("labels") or {}).get(CLASS_LABEL) == value]
|
||||
if owners and all(rejected_by_a_policy_test(o, f) for o in owners):
|
||||
continue
|
||||
stray.append(f"{f.relative_to(ROOT)}:<text>={value}")
|
||||
check(not stray,
|
||||
f"tracked YAML assigns egress-class value(s) outside the declared path set "
|
||||
f"{sorted(PATHS)}: {sorted(stray)} — a value no routing rule selects and no admission "
|
||||
"policy allows is a workload on no path at all (the only exemption is a Kyverno "
|
||||
"policy-test fixture that a kyverno-test.yaml still expects to FAIL)")
|
||||
|
||||
# The renderer must project the DECLARED class onto the pod label, unmodified: that label is
|
||||
# what the routing rule selects on, so a renderer that rewrote it would break the coupling
|
||||
# between the declaration and the path.
|
||||
rtext = RENDER.read_text()
|
||||
check(re.search(rf'f?"\{{LBL\}}/egress-class":\s*s\["egress"\]', rtext) is not None
|
||||
or re.search(r'egress-class\"\]?\s*:\s*s\[\"egress\"\]', rtext) is not None,
|
||||
"catalog/render.py does not project the ServiceDefinition's own `egress` value onto the "
|
||||
f"{CLASS_LABEL} pod label — the declaration and the routing rule's selector would be "
|
||||
"two independent facts that could disagree")
|
||||
|
||||
|
||||
# ============================== (2) SELECTION IS BY RULE; THE RULE SET IS TOTAL & DISJOINT
|
||||
policies: list[tuple[Path, dict]] = []
|
||||
for p in TRACKED_YAML:
|
||||
for d in docs(p):
|
||||
if d.get("kind") == "CiliumEgressGatewayPolicy":
|
||||
policies.append((p, d))
|
||||
check(len(policies) == 1,
|
||||
f"{len(policies)} CiliumEgressGatewayPolicy object(s) are authored "
|
||||
f"({[str(p.relative_to(ROOT)) for p, _ in policies]}) — R-NET-04's steering must be ONE "
|
||||
"rule. Two rules can overlap, and which one wins for an overlapping pod is not "
|
||||
"something the repository states")
|
||||
|
||||
for path, pol in policies:
|
||||
rel = str(path.relative_to(ROOT))
|
||||
spec = pol.get("spec") or {}
|
||||
selectors = spec.get("selectors") or []
|
||||
check(len(selectors) == 1,
|
||||
f"{rel}: the steering rule has {len(selectors)} selector(s) — every additional "
|
||||
"selector widens the set of pods steered off the default path, and R-NET-04's "
|
||||
"selection must be exactly 'the vpn class'")
|
||||
for sel in selectors:
|
||||
pod_sel = sel.get("podSelector")
|
||||
check(isinstance(pod_sel, dict) and pod_sel,
|
||||
f"{rel}: a selector has no podSelector — an empty selector in Cilium matches "
|
||||
"EVERY pod, which would put the whole cluster on the VPN path and leave no "
|
||||
"default path at all")
|
||||
ml = (pod_sel or {}).get("matchLabels") or {}
|
||||
me = (pod_sel or {}).get("matchExpressions") or []
|
||||
check(not me,
|
||||
f"{rel}: the podSelector uses matchExpressions {me!r}. The class contract is a "
|
||||
"single label equality; expressions (In/NotIn/Exists) can select pods that "
|
||||
"never declared a class, so the rule set would no longer be disjoint from the "
|
||||
"default path")
|
||||
check(set(ml) == {CLASS_LABEL},
|
||||
f"{rel}: the podSelector matches on {sorted(ml)} — it must select on exactly "
|
||||
f"{CLASS_LABEL}. Any other key makes path selection depend on something other "
|
||||
"than the declared class (and a namespace/app key would silently exempt pods)")
|
||||
check(ml.get(CLASS_LABEL) == "vpn",
|
||||
f"{rel}: the steering rule selects {CLASS_LABEL}={ml.get(CLASS_LABEL)!r}. Only "
|
||||
"the `vpn` class may be steered: docs/03 — 'Only services labeled "
|
||||
"egress-class=vpn are steered to the tunnel'. A rule selecting `internet` would "
|
||||
"route the default path through the tunnel, i.e. leave one path, not two")
|
||||
|
||||
# The gateway is chosen by a NODE LABEL, not a hostname: the path's endpoint is itself a
|
||||
# rule, so the gateway can move (or gain a standby) with no rule rewrite.
|
||||
gw = spec.get("egressGateway") or {}
|
||||
node_sel = (gw.get("nodeSelector") or {}).get("matchLabels") or {}
|
||||
check(bool(node_sel),
|
||||
f"{rel}: the egressGateway selects no node by label — pinning the path's exit to a "
|
||||
"named host means the vpn path cannot fail over (docs/03 requires two gateway "
|
||||
"nodes + Egress Gateway HA)")
|
||||
check("nodeName" not in gw,
|
||||
f"{rel}: the egressGateway pins a nodeName — the exit node must be selected by "
|
||||
"label so the path survives losing that node")
|
||||
check(isinstance(gw.get("interface"), str) and gw.get("interface"),
|
||||
f"{rel}: the egressGateway names no egress interface — without it the steered class "
|
||||
"is not actually sent down a different path from the default")
|
||||
|
||||
# ------------------- (3) THE PATHS COEXIST: the choice is per DESTINATION -------------
|
||||
dests = spec.get("destinationCIDRs") or []
|
||||
excl = spec.get("excludedCIDRs") or []
|
||||
check([str(c) for c in dests] == ["0.0.0.0/0"],
|
||||
f"{rel}: destinationCIDRs is {dests!r}. The vpn path is for PUBLIC destinations "
|
||||
"('Routed through the in-cluster VPN tunnel'), expressed as the whole internet "
|
||||
"minus the exclusions below — narrowing it here would silently leave public "
|
||||
"destinations on the WAN for a vpn-class service")
|
||||
try:
|
||||
excluded = [ipaddress.ip_network(str(c)) for c in excl]
|
||||
except ValueError as e:
|
||||
excluded = []
|
||||
check(False, f"{rel}: excludedCIDRs contains an unparsable network ({e})")
|
||||
check(len(excluded) >= 3,
|
||||
f"{rel}: only {len(excluded)} excluded CIDR(s) — docs/03 requires intra-cluster, "
|
||||
"pod->service and LAN traffic to stay on the LOCAL path, which is all of RFC1918")
|
||||
|
||||
def covered(net: ipaddress.IPv4Network | ipaddress.IPv6Network) -> bool:
|
||||
return any(net.subnet_of(e) for e in excluded if e.version == net.version)
|
||||
|
||||
# RFC1918 in full — computed, not string-matched, so a typo'd mask is caught.
|
||||
for rfc in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"):
|
||||
net = ipaddress.ip_network(rfc)
|
||||
check(covered(net),
|
||||
f"{rel}: the private range {rfc} is NOT excluded from the tunnel — a vpn-class "
|
||||
"pod's cluster/LAN traffic would be SNAT'd out tun0, so the two paths would not "
|
||||
"coexist for the same pod and east-west traffic would break")
|
||||
|
||||
# Every address each ENVIRONMENT declares for itself must fall in the excluded set: that
|
||||
# is what makes 'local stays local' true per environment rather than by coincidence.
|
||||
def overlay_addresses() -> list[tuple[str, str]]:
|
||||
found: list[tuple[str, str]] = []
|
||||
base = CLUSTERS / "base" / "cluster-vars.yaml"
|
||||
data = {}
|
||||
if base.is_file():
|
||||
for d in docs(base):
|
||||
if d.get("kind") == "ConfigMap":
|
||||
data = {k: str(v) for k, v in (d.get("data") or {}).items()}
|
||||
for f in sorted(CLUSTERS.glob("*/kustomization.yaml")):
|
||||
env = {**data}
|
||||
for m in re.finditer(
|
||||
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n"
|
||||
r"\s*value:\s*(?P<v>\S.*?)\s*$", f.read_text(), re.M):
|
||||
env[m.group("k")] = m.group("v").strip().strip('"').strip("'")
|
||||
label = f.parent.name
|
||||
for key in ("CLUSTER_POD_CIDR", "GATEWAY_VIP", "CILIUM_LB_POOL_START",
|
||||
"CILIUM_LB_POOL_STOP", "K3S_API_HOST"):
|
||||
v = env.get(key, "")
|
||||
if v and not v.startswith("TBD"):
|
||||
found.append((f"{label}/{key}", v))
|
||||
return found
|
||||
|
||||
addrs = overlay_addresses()
|
||||
check(len(addrs) >= 5,
|
||||
f"only {len(addrs)} environment address(es) could be read from the Flux overlays — "
|
||||
"the 'local traffic stays local' assertion would be checking almost nothing")
|
||||
for label, value in addrs:
|
||||
try:
|
||||
net = ipaddress.ip_network(value, strict=False)
|
||||
except ValueError:
|
||||
continue
|
||||
check(covered(net),
|
||||
f"{rel}: {label}={value}, an address THIS repository assigns to a cluster, is "
|
||||
"NOT inside the tunnel's excludedCIDRs — a vpn-class pod in that environment "
|
||||
"would try to reach its own cluster/LAN through the VPN")
|
||||
|
||||
|
||||
# ============ (4) THE VPN PATH IS A SECOND TABLE, NEVER THE DEFAULT (split tunnel) ========
|
||||
TPL = VPN_ROLE / "templates"
|
||||
up = (TPL / "vpn-egress-up.sh.j2")
|
||||
down = (TPL / "vpn-egress-down.sh.j2")
|
||||
conf = (TPL / "vpn-egress.conf.j2")
|
||||
nft = (TPL / "vpn-egress-killswitch.nft.j2")
|
||||
for f in (up, down, conf, nft):
|
||||
check(f.is_file(), f"{f.relative_to(ROOT)} is missing — the second egress path's routing "
|
||||
"rules are not authored")
|
||||
if FAILURES:
|
||||
die()
|
||||
|
||||
defaults = yaml.safe_load((VPN_ROLE / "defaults" / "main.yml").read_text()) or {}
|
||||
|
||||
|
||||
def render(text: str) -> str:
|
||||
"""Substitute `{{ var }}` with the role's own default value, for every default.
|
||||
|
||||
The templates reference their knobs by NAME, so an analysis that looked for the VALUE
|
||||
(e.g. the nft set name) would silently match nothing and pass vacuously — which is
|
||||
exactly what an earlier draft of this check did. Rendering first means the assertions
|
||||
below examine the text the node actually gets. Only simple `{{ name }}` expressions are
|
||||
substituted; anything else is left as-is (and the assertions are written so an
|
||||
unsubstituted expression fails rather than passes).
|
||||
"""
|
||||
out = text
|
||||
for k, v in defaults.items():
|
||||
if isinstance(v, (str, int)):
|
||||
out = re.sub(r"\{\{\s*" + re.escape(str(k)) + r"\s*\}\}", str(v), out)
|
||||
return out
|
||||
|
||||
|
||||
up_t, conf_t, nft_t = render(up.read_text()), render(conf.read_text()), render(nft.read_text())
|
||||
table_id = str(defaults.get("vpn_egress_gateway_route_table_id", ""))
|
||||
check(table_id.isdigit() and table_id not in ("0", "253", "254", "255"),
|
||||
f"the tunnel's routing table id is {table_id!r} — the second path must live in a "
|
||||
"DEDICATED table. 254 (main), 255 (local) and 253 (default) are the kernel's own, and "
|
||||
"putting the tunnel's default route in any of them makes the VPN the node's default "
|
||||
"route instead of a selectable second path")
|
||||
check(str(defaults.get("vpn_egress_gateway_rule_priority", "")).isdigit(),
|
||||
"no ip-rule priority is declared for the tunnel table — 'selected by routing rules' "
|
||||
"needs a rule, at a defined precedence")
|
||||
|
||||
# Every `ip route ... default` the up-script installs must name the dedicated table.
|
||||
route_lines = [l.strip() for l in up_t.splitlines()
|
||||
if re.search(r"^\s*ip\s+route\s+(replace|add)\b", l)]
|
||||
check(len(route_lines) >= 2,
|
||||
f"the up-script installs {len(route_lines)} route(s) — the vpn path needs its own "
|
||||
"default route via the tunnel AND the fail-closed blackhole behind it")
|
||||
for l in route_lines:
|
||||
check(re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
|
||||
f"the up-script installs a route WITHOUT naming the dedicated table: {l!r}. A route "
|
||||
"landing in the main table changes the path every other pod and the node itself "
|
||||
"uses — the VPN would stop being opt-in")
|
||||
check("blackhole" not in l or re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
|
||||
f"a blackhole route is installed outside the tunnel table: {l!r}. docs/03: the "
|
||||
"kill-switch 'blackholes **only** that class'; `internet` fails OPEN and must never "
|
||||
"be blackholed")
|
||||
|
||||
# Every `ip rule` must be SCOPED. An unscoped rule (`ip rule add lookup <table>`) sends
|
||||
# everything into the tunnel table, which would leave one path instead of two.
|
||||
rule_lines = [l.strip() for l in up_t.splitlines() if re.search(r"^\s*ip\s+rule\s+add\b", l)]
|
||||
check(len(rule_lines) >= 1,
|
||||
"the up-script adds no `ip rule` — nothing SELECTS the tunnel path, so the second "
|
||||
"routing table would never be consulted (R-NET-04's 'selected by routing rules')")
|
||||
for l in rule_lines:
|
||||
check(re.search(r"\bfrom\s+\S|\bfwmark\s+\S|\biif\s+\S", l) is not None,
|
||||
f"an unscoped ip rule is installed: {l!r}. Without a `from`/`fwmark`/`iif` selector "
|
||||
"EVERY packet is looked up in the tunnel table, so the WAN path disappears and the "
|
||||
"VPN is no longer opt-in")
|
||||
check(re.search(r'lookup\s+"?\$\{?TABLE_ID', l) is not None,
|
||||
f"an ip rule looks up a table other than the tunnel's: {l!r}")
|
||||
check(re.search(r'ip rule add from\s+"\$\{SRC\}"', up_t) is not None,
|
||||
"the steering rule is not scoped to ${SRC}, the tunnel's own local address — that "
|
||||
"address is what Cilium SNATs vpn-class pods to, and it is the ONLY thing that "
|
||||
"distinguishes the steered class from every other flow on the gateway node")
|
||||
check(re.search(r"^\s*if \[ -z \"\$\{SRC\}\" \]", up_t, re.M) is not None,
|
||||
"the up-script does not refuse to run when the tunnel address is unknown — installing "
|
||||
"a partial rule set is how a path ends up half-selected")
|
||||
|
||||
# The provider must not be able to make the tunnel the default route either.
|
||||
check(re.search(r"^\s*route-noexec\s*$", conf_t, re.M) is not None,
|
||||
"the OpenVPN wrapper config lacks `route-noexec` — OpenVPN would install the pushed "
|
||||
"routes itself, in the MAIN table, and the VPN would become the node's default route "
|
||||
"(docs/03: 'does **not** set the node's default route (`route-noexec` + policy "
|
||||
"routing)')")
|
||||
check(re.search(r'pull-filter ignore "redirect-gateway"', conf_t) is not None,
|
||||
"the OpenVPN wrapper config does not ignore a pushed `redirect-gateway` — the VPN "
|
||||
"provider could redirect the node's whole default route into the tunnel, collapsing "
|
||||
"the two paths into one")
|
||||
check(re.search(r"^\s*redirect-gateway", conf_t, re.M) is None,
|
||||
"the OpenVPN wrapper config itself sets `redirect-gateway` — that IS making the VPN the "
|
||||
"default route")
|
||||
|
||||
|
||||
# ================= (5) THE KILL-SWITCH BLACKHOLES ONLY THE VPN CLASS (internet fails OPEN)
|
||||
src_set = str(defaults.get("vpn_egress_gateway_nft_src_set", ""))
|
||||
fwmark = str(defaults.get("vpn_egress_gateway_fwmark", ""))
|
||||
check(bool(src_set),
|
||||
"the role declares no nftables source set — the kill-switch would have no way to name "
|
||||
"'the steered class' and could only match broadly")
|
||||
policies_lines = [l.strip() for l in nft_t.splitlines() if re.search(r"^\s*type filter hook", l)]
|
||||
check(len(policies_lines) >= 1,
|
||||
"the kill-switch nft template declares no base chain — nothing would be enforced")
|
||||
for l in policies_lines:
|
||||
check(re.search(r"policy\s+accept\s*;", l) is not None,
|
||||
f"an nft base chain is not `policy accept`: {l!r}. A default-drop chain here "
|
||||
"blackholes the `internet` path too, and docs/03 requires the two classes to fail "
|
||||
"in OPPOSITE directions — 'internet fails **open** (WAN, never blackholed)'")
|
||||
drop_lines = [l.strip() for l in nft_t.splitlines()
|
||||
if re.search(r"\bdrop\b", l) and not l.lstrip().startswith("#")]
|
||||
check(len(drop_lines) >= 1,
|
||||
"the kill-switch nft template contains no drop rule at all — there is no kill-switch, "
|
||||
"and R-NET-04's 'the kill-switch blackholes only that class' has nothing to scope")
|
||||
for l in drop_lines:
|
||||
scoped = (src_set and src_set in l) or (fwmark and "mark" in l)
|
||||
check(scoped,
|
||||
f"an nft DROP rule is not scoped to the vpn class: {l!r}. It must match the tunnel "
|
||||
f"source set (@{src_set}) or the class's fwmark; an unqualified drop would blackhole "
|
||||
"the default `internet` path, which must never be blackholed")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
||||
check(CHECKS >= 35,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
die()
|
||||
print(f"check OK: egress-paths-selected-by-rules — {CHECKS} assertions; "
|
||||
f"paths={sorted(PATHS)} (default={egress_prop.get('default')}), "
|
||||
f"{len(policies)} steering rule selecting {CLASS_LABEL}=vpn only, tunnel default route "
|
||||
f"in dedicated table {table_id} behind a scoped ip rule, blackhole and every nft DROP "
|
||||
f"scoped to the vpn class. R-NET-04's RULE-SET half only: the paths' actual BEHAVIOUR "
|
||||
"is L2 (egress-class for the two labelled classes, egress-default-path for the "
|
||||
"unlabelled default) and the kill-switch firing is L3 (vpn-killswitch).")
|
||||
PY
|
||||
@@ -1,60 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — Flux postBuild substitution is complete and fail-closed-safe
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# R-ENV-02 ("authored once, promoted dev -> prod") is implemented by ONE mechanism:
|
||||
# `postBuild.substituteFrom cluster-vars` injecting `${TOKEN}`s into the manifests each
|
||||
# Flux Kustomization renders. That mechanism FAILS OPEN — an undefined variable becomes
|
||||
# the empty string while the Kustomization still reports Ready — which emptied Cilium's
|
||||
# k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01).
|
||||
#
|
||||
# GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom,
|
||||
# WHEN each `spec.path` is resolved to the exact file set it renders (following
|
||||
# kustomization.yaml `resources:` transitively, not globbing the repo),
|
||||
# THEN every `${...}` in those files is either DEFINED in clusters/base/cluster-vars.yaml
|
||||
# or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default
|
||||
# operator on an undefined name (the silent-empty class strict mode does NOT catch);
|
||||
# no unescaped expansion is unparseable by envsubst (which fails the WHOLE
|
||||
# Kustomization); every path rendering a knob actually substitutes cluster-vars;
|
||||
# every overlay-patched knob has a base default; and the check is wired into the
|
||||
# merge gate as a blocking, unconditionally-run stage.
|
||||
#
|
||||
# This is the prerequisite proof for enabling kustomize-controller's
|
||||
# StrictPostBuildSubstitutions feature gate, and the permanent guard that keeps it safe.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library
|
||||
# only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
|
||||
# Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests
|
||||
# a path includes — a deliberate superset of the post-`kustomize build` stream. It does
|
||||
# not render, does not prove the feature gate is enabled on the live kustomize-controller,
|
||||
# and does not validate knob VALUES. R-GITOPS-01 is tagged for the narrow static slice
|
||||
# only: what git says a Kustomization renders is fully determined, with no variable
|
||||
# silently emptied on the way in. The live-state half is L2, untested today.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: flux-postbuild-substitution
|
||||
labels:
|
||||
conformance.home.k3s/R-ENV-02: "true"
|
||||
conformance.home.k3s/R-GITOPS-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every variable a substituted Flux path renders is defined in cluster-vars or
|
||||
escaped, so post-build substitution cannot silently empty a value and enabling
|
||||
StrictPostBuildSubstitutions cannot fail a legitimate manifest.
|
||||
spec:
|
||||
steps:
|
||||
- name: audit-every-substituted-flux-path
|
||||
description: >-
|
||||
Resolve each postBuild.substituteFrom Kustomization to the files it renders and
|
||||
assert every ${...} is defined or escaped, then assert the gate wires it blocking.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,369 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — flux-postbuild-substitution
|
||||
# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
|
||||
# — spec docs/02-environments-and-provisioning.md)
|
||||
# R-GITOPS-01 ("All desired state in git, reconciled by Flux."
|
||||
# — spec docs/05-gitops-and-delivery.md)
|
||||
#
|
||||
# WHY THIS EXISTS — Flux post-build substitution FAILS OPEN
|
||||
# --------------------------------------------------------
|
||||
# R-ENV-02 is implemented by exactly one mechanism: `clusters/base/cluster-vars.yaml`
|
||||
# is injected into every `${TOKEN}` of the manifests each Flux Kustomization renders,
|
||||
# via `postBuild.substituteFrom`. That mechanism is silently lossy by default: an
|
||||
# UNDEFINED variable becomes the EMPTY STRING and the Kustomization still reports
|
||||
# Ready. On 2026-09-01 that emptied `k8sServiceHost` in the Cilium HelmRelease and put
|
||||
# cilium into Init:Error on 2 of 3 dev nodes (DECISIONS-LOG "substitution race that
|
||||
# fails OPEN"). "Authored once, promoted" is therefore only true while every token a
|
||||
# substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
|
||||
# the repo asserted that. This check does.
|
||||
#
|
||||
# ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
|
||||
# `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
|
||||
# ${DEFINED} -> substituted with the cluster-vars value.
|
||||
# ${UNDEFINED} -> EMPTY STRING, silently. `--strict` / the cluster-wide
|
||||
# StrictPostBuildSubstitutions feature gate turns this into a
|
||||
# loud Kustomization failure ("variable not set (strict mode)").
|
||||
# ${UNDEFINED:-d} -> the DEFAULT is applied AT RECONCILE TIME. Strict mode does
|
||||
# NOT catch this (it exempts default-operator forms by design),
|
||||
# so a shell guard written `[ -z "${X:-}" ]` inside a rendered
|
||||
# manifest is pre-resolved to `[ -z "" ]` and its else-branch
|
||||
# becomes unreachable in the container. Silent, and permanent.
|
||||
# $${VAR}, $${V:-d}, $${ARR[0]} -> emitted as the LITERAL ${...} for a downstream
|
||||
# shell/cloud-init consumer. Escaping is total: it neutralises
|
||||
# even the indexed form.
|
||||
# $VAR, $(cmd) -> left alone by envsubst; they need no escaping.
|
||||
# ${...}, ${a b}, ${ARR[0]} (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
|
||||
# variable name" / "missing closing brace") which fails the
|
||||
# WHOLE Kustomization — every resource in the path, not just
|
||||
# the offending one.
|
||||
# The escape contract this enforces is the one already documented in
|
||||
# `clusters/base/apps-vms.yaml`; this check is that comment made executable.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# WHEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom
|
||||
# is resolved to the exact file set its `spec.path` renders (by following
|
||||
# kustomization.yaml `resources:` transitively — a path, not a glob),
|
||||
# THEN (1) every `${...}` in those files is DEFINED in cluster-vars, or ESCAPED `$${...}`;
|
||||
# (2) no UNESCAPED expansion carries a default/length/substitution operator on an
|
||||
# undefined name (the silent-empty class strict mode does not catch);
|
||||
# (3) no UNESCAPED expansion is unparseable by envsubst (`${...}`, `${ARR[0]}`) —
|
||||
# that fails the entire Kustomization, so it is fatal wherever it appears;
|
||||
# (4) every Kustomization that renders a cluster-vars token actually declares
|
||||
# `substituteFrom` a ConfigMap named cluster-vars (else the literal token
|
||||
# ships into the cluster);
|
||||
# (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
|
||||
# the "add the knob with a base default BEFORE referencing it" rule that the
|
||||
# incident's interim discipline depends on;
|
||||
# AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and
|
||||
# ci/gate.sh sets the merge block when the l0 stage fails.
|
||||
# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
|
||||
# preserves body comments into the stream envsubst reads (verified: a `$`-carrying
|
||||
# comment survives into the built apps/wave-1 output), so a braced expression in a
|
||||
# comment is a live hazard, not prose.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
|
||||
# - It does NOT render the manifests. It scans the SOURCE files a path includes,
|
||||
# which is a deliberate SUPERSET of the post-`kustomize build` stream (kustomize
|
||||
# drops some comments). A finding in a comment may not reach envsubst TODAY; it is
|
||||
# still reported, because whether a given comment survives the build is a kustomize
|
||||
# implementation detail that changes with the pinned version, and the failure it
|
||||
# causes is total (assertion 3). The end-to-end proof — `kustomize build <path> |
|
||||
# flux envsubst --strict` for every substituted path — needs kustomize + flux,
|
||||
# which the L0 CI toolchain does not install (.gitea/workflows/scripts/install-tools.sh
|
||||
# l0), so it cannot run here without permanently SKIPPING. It was run by hand and
|
||||
# recorded in DECISIONS-LOG; this check is the reproducible static half.
|
||||
# - It does NOT prove the cluster's live state matches git, nor that the
|
||||
# StrictPostBuildSubstitutions feature gate is actually enabled on
|
||||
# kustomize-controller. Both are cluster facts (L2), untested here. Tagging
|
||||
# R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
|
||||
# is fully determined, with no variable silently emptied on the way in.
|
||||
# - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
|
||||
# that an overlay's LAN block is self-consistent). Presence, not correctness.
|
||||
# - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
|
||||
# several are deliberately reference-only and say so in cluster-vars.yaml
|
||||
# (PIA_REGION "retained, not substituted"; CLUSTER_POD_CIDR "Documents ..."), so
|
||||
# failing on them would be a false block on documented intent.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green) if the thing under test is not present at all.
|
||||
if [ ! -d "${ROOT}/clusters" ] || [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then
|
||||
echo "check SKIPPED: clusters/ or clusters/base/cluster-vars.yaml absent — nothing to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit of Flux postBuild substitution safety. Pure standard library.
|
||||
|
||||
No PyYAML: the file shapes parsed here are line-constrained and the parse contracts are
|
||||
documented at each parser, matching lib/coverage.py + lib/status.py. The kustomization
|
||||
`resources:` parser is IMPORTED from lib/status.py rather than forked, so there is one
|
||||
definition of "what a Flux path actually renders" in the harness.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
sys.path.insert(0, str(ROOT / "conformance" / "lib"))
|
||||
import status as st # noqa: E402 — reuse the walk, don't fork it
|
||||
|
||||
FAILURES: list[str] = []
|
||||
ADVISORIES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ cluster-vars
|
||||
# Parse contract: `data:` block of a single ConfigMap, one ` KEY: "value"` per line
|
||||
# (every entry in clusters/base/cluster-vars.yaml is a quoted scalar; comments and
|
||||
# continuation-comment lines are indented further and never match).
|
||||
_DATA_KEY_RE = re.compile(r'^ ([A-Za-z_][A-Za-z0-9_]*):\s*(?:"|\')')
|
||||
|
||||
|
||||
def cluster_vars_keys(path: Path) -> set[str]:
|
||||
keys: set[str] = set()
|
||||
in_data = False
|
||||
for raw in path.read_text().splitlines():
|
||||
if re.match(r"^data:\s*$", raw):
|
||||
in_data = True
|
||||
continue
|
||||
if not in_data:
|
||||
continue
|
||||
if st._TOP_KEY_RE.match(raw): # a new top-level key ends the data block
|
||||
break
|
||||
m = _DATA_KEY_RE.match(raw)
|
||||
if m:
|
||||
keys.add(m.group(1))
|
||||
return keys
|
||||
|
||||
|
||||
VARS_FILE = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
||||
KEYS = cluster_vars_keys(VARS_FILE)
|
||||
check(len(KEYS) > 0, f"no data keys parsed from {VARS_FILE} — the parser or the file shape changed")
|
||||
|
||||
# ------------------------------------------- Flux Kustomizations with substituteFrom
|
||||
# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`, a doc
|
||||
# is a Flux entrypoint only if it carries the kustomize.toolkit apiVersion AND
|
||||
# `kind: Kustomization` AND a spec-level `path:`. Inline `patch: |-` bodies in the
|
||||
# overlays carry Flux Kustomizations WITHOUT a path and are correctly skipped.
|
||||
_SUBFROM_RE = re.compile(r"^(\s*)substituteFrom:\s*$", re.M)
|
||||
_SUBITEM_KIND_RE = re.compile(r"^\s*-\s*kind:\s*(\S+)")
|
||||
_SUBITEM_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)")
|
||||
|
||||
|
||||
def substitute_sources(doc: str) -> list[tuple[str, str]]:
|
||||
"""[(kind, name)] declared under postBuild.substituteFrom in this doc."""
|
||||
m = _SUBFROM_RE.search(doc)
|
||||
if not m:
|
||||
return []
|
||||
indent = len(m.group(1))
|
||||
out: list[tuple[str, str]] = []
|
||||
kind = ""
|
||||
for raw in doc[m.end():].splitlines():
|
||||
if raw.strip() == "" or raw.lstrip().startswith("#"):
|
||||
continue
|
||||
if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"):
|
||||
break # dedented out of the list
|
||||
km = _SUBITEM_KIND_RE.match(raw)
|
||||
if km:
|
||||
kind = km.group(1)
|
||||
continue
|
||||
nm = _SUBITEM_NAME_RE.match(raw)
|
||||
if nm and kind:
|
||||
out.append((kind, nm.group(1)))
|
||||
kind = ""
|
||||
return out
|
||||
|
||||
|
||||
kustomizations: list[dict] = []
|
||||
for yfile in sorted((ROOT / "clusters").rglob("*.yaml")):
|
||||
text = yfile.read_text()
|
||||
if st.FLUX_API not in text:
|
||||
continue
|
||||
for doc in st._DOC_SPLIT_RE.split(text):
|
||||
if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M):
|
||||
continue
|
||||
pm = st._PATH_RE.search(doc)
|
||||
if not pm:
|
||||
continue
|
||||
nm = st._NAME_RE.search(doc)
|
||||
srcs = substitute_sources(doc)
|
||||
if not srcs:
|
||||
continue # no substitution => nothing to audit
|
||||
kustomizations.append({
|
||||
"file": str(yfile.relative_to(ROOT)),
|
||||
"name": nm.group(1) if nm else "?",
|
||||
"path": pm.group(1),
|
||||
"sources": srcs,
|
||||
})
|
||||
|
||||
check(len(kustomizations) > 0,
|
||||
"no Flux Kustomization with postBuild.substituteFrom found under clusters/ — "
|
||||
"the parser or the repo layout changed; this check would silently audit nothing")
|
||||
|
||||
# ------------------------------------------------------------------ token scanning
|
||||
# One `$`-run followed by a braced expression. `[^{}]*` is exhaustive for this tree:
|
||||
# a repo-wide sweep found no nested/unclosed brace expression in any manifest (only
|
||||
# Gitea Actions `${{ ... }}` in .gitea/, which no Flux Kustomization renders).
|
||||
TOKEN_RE = re.compile(r"(\$+)\{([^{}]*)\}")
|
||||
NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
# An expansion whose name is followed by a POSIX parameter operator. envsubst EVALUATES
|
||||
# these, so an undefined name yields the default/empty AT RECONCILE TIME.
|
||||
OP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)(:[-=?+].*|:[0-9].*|/.*|#.*|%.*|\^.*|,.*)$")
|
||||
LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")
|
||||
|
||||
|
||||
def classify(dollars: int, inner: str) -> tuple[str, str] | None:
|
||||
"""Return (severity-tag, remediation) for a hazardous expansion, else None."""
|
||||
if dollars >= 2:
|
||||
return None # escaped: emitted as a literal
|
||||
if NAME_RE.match(inner):
|
||||
if inner in KEYS:
|
||||
return None # defined knob — the happy path
|
||||
return ("UNDEFINED",
|
||||
f"strict mode fails the Kustomization; today it renders EMPTY. Either add "
|
||||
f"'{inner}' to clusters/base/cluster-vars.yaml with a base default, or "
|
||||
f"escape it as $${{{inner}}} if it is meant for a shell/cloud-init consumer")
|
||||
m = OP_RE.match(inner) or LEN_RE.match(inner)
|
||||
if m:
|
||||
base = m.group(1)
|
||||
if base in KEYS:
|
||||
return None # defined knob with an operator
|
||||
return ("SILENT-EMPTY",
|
||||
f"envsubst resolves this AT RECONCILE TIME because '{base}' is not a "
|
||||
f"cluster-var, so the rendered manifest carries the default, not the "
|
||||
f"expression. StrictPostBuildSubstitutions does NOT catch this form. "
|
||||
f"Escape it as $${{{inner}}} if the CONTAINER's shell is meant to expand it")
|
||||
return ("FATAL-PARSE",
|
||||
"envsubst cannot parse this and fails the ENTIRE Kustomization (every resource "
|
||||
"in the path). Escape it as $${...} or rewrite the prose without braces")
|
||||
|
||||
|
||||
def rendered_files(path_spec: str) -> tuple[list[str], bool]:
|
||||
"""Repo-relative files the Flux `spec.path` renders, resolving `resources:` transitively."""
|
||||
rel = os.path.normpath(path_spec.lstrip("./")) if path_spec not in (".", "./") else "."
|
||||
if not (ROOT / rel).is_dir():
|
||||
return [], False
|
||||
applied, _ = st.reconciled_files(ROOT, rel)
|
||||
return sorted(applied), True
|
||||
|
||||
|
||||
findings: list[str] = []
|
||||
referenced: dict[str, set[str]] = {}
|
||||
inventory: list[str] = []
|
||||
|
||||
for k in kustomizations:
|
||||
files, exists = rendered_files(k["path"])
|
||||
check(exists, f"{k['file']}: Kustomization '{k['name']}' renders path '{k['path']}' "
|
||||
f"which is not a directory in this repo")
|
||||
has_cluster_vars = ("ConfigMap", "cluster-vars") in k["sources"]
|
||||
n_def = n_esc = n_bad = 0
|
||||
for rel in files:
|
||||
f = ROOT / rel
|
||||
try:
|
||||
text = f.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
continue
|
||||
for lineno, line in enumerate(text.splitlines(), 1):
|
||||
for m in TOKEN_RE.finditer(line):
|
||||
dollars, inner = len(m.group(1)), m.group(2)
|
||||
verdict = classify(dollars, inner)
|
||||
if verdict is None:
|
||||
if dollars >= 2:
|
||||
n_esc += 1
|
||||
else:
|
||||
n_def += 1
|
||||
referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
|
||||
set()).add(k["name"])
|
||||
# (4) a DEFINED knob is only substituted if this Kustomization
|
||||
# actually pulls cluster-vars in; otherwise the literal ships.
|
||||
check(has_cluster_vars,
|
||||
f"{rel}:{lineno}: renders cluster-var ${{{inner}}} but "
|
||||
f"Kustomization '{k['name']}' ({k['file']}) does not "
|
||||
f"substituteFrom ConfigMap/cluster-vars — the literal token "
|
||||
f"would ship into the cluster")
|
||||
continue
|
||||
n_bad += 1
|
||||
tag, fix = verdict
|
||||
findings.append(f" [{tag}] {rel}:{lineno} {m.group(0)}\n"
|
||||
f" rendered by Flux Kustomization '{k['name']}' "
|
||||
f"({k['file']}, path {k['path']})\n"
|
||||
f" {fix}\n"
|
||||
f" | {line.strip()[:120]}")
|
||||
inventory.append(f" {k['name']:<16} path={k['path']:<22} files={len(files):<3} "
|
||||
f"defined={n_def:<3} escaped={n_esc:<3} hazards={n_bad}")
|
||||
|
||||
check(not findings,
|
||||
f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")
|
||||
|
||||
# --------------------------------------------- (5) overlay knobs exist in the base
|
||||
# Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
|
||||
# `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
|
||||
# header) is that a knob is ADDED TO THE BASE with a default BEFORE any overlay or
|
||||
# manifest references it — that ordering is what closes the define-then-use race.
|
||||
_OP_PATH_RE = re.compile(r"^\s*(?:-\s*)?path:\s*/data/([A-Za-z_][A-Za-z0-9_]*)\s*$", re.M)
|
||||
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
||||
for key in set(_OP_PATH_RE.findall(overlay.read_text())):
|
||||
check(key in KEYS,
|
||||
f"{overlay.relative_to(ROOT)}: patches /data/{key} but '{key}' has no BASE "
|
||||
f"default in clusters/base/cluster-vars.yaml — a second overlay would render "
|
||||
f"${{{key}}} empty (R-ENV-02: author once, supply per env)")
|
||||
|
||||
# ------------------------------------------------------- (6) the check is load-bearing
|
||||
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
check("flux-postbuild-substitution/check.sh" in L0,
|
||||
"ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped "
|
||||
"conformance selection can filter this check out of a gate run, so l0.sh must invoke "
|
||||
"it directly and unconditionally or the gate is not actually enforcing it")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
# ------------------------------------------------------------- advisory: dead knobs
|
||||
for key in sorted(KEYS - set(referenced)):
|
||||
ADVISORIES.append(f"cluster-vars knob '{key}' is referenced by NO reconciled manifest "
|
||||
f"(dead knob, or documentation-only — advisory, not a failure)")
|
||||
|
||||
# ------------------------------------------------------------------------- report
|
||||
print("Flux postBuild substitution inventory (per Kustomization):")
|
||||
for line in inventory:
|
||||
print(line)
|
||||
print(f"\ncluster-vars knobs defined: {len(KEYS)} referenced: {len(referenced)}")
|
||||
|
||||
if ADVISORIES:
|
||||
print("\nADVISORIES (not failures):")
|
||||
for a in ADVISORIES:
|
||||
print(f" - {a}")
|
||||
|
||||
if findings:
|
||||
print("\nHAZARDS — unescaped expansions that are not cluster-vars knobs:", file=sys.stderr)
|
||||
for f in findings:
|
||||
print(f, file=sys.stderr)
|
||||
|
||||
print()
|
||||
if not FAILURES:
|
||||
print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "
|
||||
f"(R-ENV-02, R-GITOPS-01)")
|
||||
raise SystemExit(0)
|
||||
print(f"CHECK FAIL (flux-postbuild-substitution): {len(FAILURES)} failed, "
|
||||
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" FAIL {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
@@ -1,88 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the HA control plane is authored, for every environment (R-PLAT-01) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
|
||||
# production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
|
||||
# exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA,
|
||||
# embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)",
|
||||
# "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs").
|
||||
#
|
||||
# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the
|
||||
# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments
|
||||
# from one code path, with exactly one --cluster-init founder, no external datastore, and a
|
||||
# floating API endpoint that is SAN'd into the serving cert and cannot collide with the
|
||||
# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live
|
||||
# cluster cannot tell you what the NEXT environment brought up from this repo would look
|
||||
# like. The observed half is asserted separately at L2 by
|
||||
# tests/l2-behavioral/ha-control-plane, which carries the same requirement label.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down.
|
||||
# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group /
|
||||
# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay
|
||||
# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the inventory, the k3s_server and kube_vip roles, their plays and each
|
||||
# environment's Flux overlay are parsed and cross-checked against each other,
|
||||
# THEN k3s is the pinned distro and nothing is installed as an agent; every environment
|
||||
# resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY
|
||||
# ONE --cluster-init founder and every other server joining it with --server;
|
||||
# the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd);
|
||||
# the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip
|
||||
# DaemonSet on every server whose address is the environment-scoped VIP, that VIP is
|
||||
# in the apiserver serving cert's SANs together with every server's own address, and
|
||||
# it is neither a node address nor the ingress Gateway VIP nor inside that
|
||||
# environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME
|
||||
# roles and plays retargeted by variables only.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
||||
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
||||
# Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating,
|
||||
# API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane.
|
||||
# * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves)
|
||||
# is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss
|
||||
# drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
|
||||
# * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
|
||||
# infra/PROD-PROVISIONING.md).
|
||||
#
|
||||
# TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible):
|
||||
# * `K3S_API_HOST` (Cilium's k8sServiceHost) was the PRIMARY NODE's address in both
|
||||
# environments rather than the floating VIP, contradicting the prod overlay's own comment.
|
||||
# It was ESCALATED, not asserted, because the workstream could not decide it (R-TEST-06).
|
||||
# The operator decided the VIP, so it is now ASSERTED — on both sides of the [parity]
|
||||
# contract and per environment.
|
||||
# * kube-vip was NOT imported by playbooks/site.yml. It is now, BETWEEN k3s and cilium, and
|
||||
# the order is asserted — not as a tidiness preference but because the CNI's endpoint is
|
||||
# the VIP, so an unfloated VIP means a cold build brings up a CNI that cannot reach its
|
||||
# API server.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: ha-control-plane-as-code
|
||||
labels:
|
||||
conformance.home.k3s/R-PLAT-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-PLAT-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each
|
||||
environment's Flux overlay are parsed and cross-checked, then every environment is
|
||||
authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one
|
||||
embedded-etcd founder, no external datastore, and a leader-elected control-plane-only
|
||||
VIP that is in the serving cert's SANs and collides with no other allocator — built by
|
||||
one code path for both environments.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,729 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — ha-control-plane-as-code
|
||||
# Serves: R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with production-grade
|
||||
# HA control plane in prod; dev/test MAY run fewer nodes but MUST exercise the
|
||||
# same multi-node topology." — spec docs/00-vision-and-scope.md).
|
||||
#
|
||||
# The elaborating spec text this check quotes and asserts against:
|
||||
# docs/01-architecture-overview.md "Kubernetes distro | **k3s** (HA, embedded etcd)"
|
||||
# docs/02-environments-and-provisioning.md, table:
|
||||
# "Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd
|
||||
# (k3s server on all 3)" (the SAME cell for dev/test and prod)
|
||||
# "Node count | 3 VMs (mirror prod topology) | 3 physical nodes"
|
||||
# docs/02 §"k3s topology decisions":
|
||||
# "HA control plane with **embedded etcd** across all 3 nodes (no external datastore)."
|
||||
# "Nodes are both control-plane and workload (3-node converged cluster); with only 3
|
||||
# nodes we do not isolate control-plane nodes."
|
||||
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
|
||||
# step differs"
|
||||
#
|
||||
# WHY L0 IS THE RIGHT LAYER FOR THESE CLAUSES (and where the rest lives)
|
||||
# ---------------------------------------------------------------------
|
||||
# R-PLAT-01 splits cleanly in two, and the halves are provable at different layers:
|
||||
#
|
||||
# AUTHORED half — "the configuration in git builds a 3-server HA embedded-etcd control
|
||||
# plane in BOTH environments, from ONE code path": provable with no cluster, and
|
||||
# ONLY provable here. A live cluster cannot tell you whether the *next* environment
|
||||
# brought up from this repo will also be HA, whether a second `--cluster-init`
|
||||
# founder could exist, or whether an external datastore is configured anywhere. That
|
||||
# is this check.
|
||||
# OBSERVED half — "the cluster this suite is pointed at really is 3 Ready servers, all
|
||||
# etcd members, with the API answering through the floating VIP": that is cluster
|
||||
# state and lives in tests/l2-behavioral/ha-control-plane (same requirement label).
|
||||
#
|
||||
# Neither half is sufficient alone, and this file claims only the first.
|
||||
#
|
||||
# ENVIRONMENTS ARE DERIVED, NOT LISTED. The check does not know the words "dev" or "prod"
|
||||
# as inventory names. It enumerates environments from the R-ENV-02 seam itself — the
|
||||
# `k3s_primary_group` / `k3s_servers_group` pair in group_vars/all.yml plus every
|
||||
# group_vars/<group>/vars.yml that overrides them — and identifies WHICH environment is
|
||||
# prod by the Flux overlay that environment bootstraps (`flux_bootstrap_env_path`), not by
|
||||
# its group name. Add a fourth prod node, rename a group, or add a third environment and
|
||||
# the assertions follow; nothing here restates an address, a hostname or a node count.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the Ansible inventory, the k3s/kube-vip roles and plays, and each environment's
|
||||
# Flux overlay are parsed and cross-checked against each other,
|
||||
# THEN DISTRO: k3s is the distro, pinned, installed by the one role, and no node is
|
||||
# installed as an `agent` (an agent is not a control-plane member);
|
||||
# SERVERS: every environment's server group resolves to a quorum-capable, ODD
|
||||
# number of hosts (>= 3, tolerating >= 1 loss), the PROD environment to
|
||||
# EXACTLY three, and each has EXACTLY ONE `--cluster-init` founder while
|
||||
# every other server joins it with `--server` (so a second founder — the
|
||||
# split-brain shape — cannot be authored);
|
||||
# ETCD: the datastore is k3s's EMBEDDED etcd: no `--datastore-endpoint` and no
|
||||
# `--disable-etcd` anywhere in the provisioning code, so every server is
|
||||
# an etcd member (docs/02's "no external datastore", "on all 3");
|
||||
# ENDPOINT: the API has a floating endpoint that survives losing the founder — the
|
||||
# kube-vip VIP is control-plane-only (never Service LB, or it would fight
|
||||
# Cilium LB-IPAM for ARP), leader-elected, deployed to EVERY server, and
|
||||
# baked into the apiserver serving cert's SAN list ALONG WITH every
|
||||
# server's own address (so TLS validates whichever node answers);
|
||||
# ADDRESSES: each environment's control-plane VIP is not one of its node addresses,
|
||||
# is not that environment's ingress Gateway VIP, and lies OUTSIDE that
|
||||
# environment's Cilium LoadBalancer pool — the two allocators can never be
|
||||
# handed the same address;
|
||||
# CNI PATH: the CNI dials the FLOATING endpoint, not a machine (added 2026-09-06). The
|
||||
# two API addresses stay SPLIT — k3s_api_host is the founder and feeds only
|
||||
# the cold `--server` join, k3s_apiserver_endpoint derives from
|
||||
# kube_vip_address and feeds Cilium's k8sServiceHost — the Flux overlay's
|
||||
# K3S_API_HOST equals that environment's VIP and is none of its node
|
||||
# addresses ([parity], both halves), the HelmRelease keeps a per-environment
|
||||
# token rather than a literal, and site.yml floats the VIP BETWEEN k3s and
|
||||
# Cilium with roles/kube_vip blocking until it answers;
|
||||
# ONE PATH: dev/test and prod are brought up by the SAME role and the SAME plays,
|
||||
# retargeted by variables only (docs/02 "the *same* Ansible bootstrap
|
||||
# runs"), so "exercise the same multi-node topology" is a property of the
|
||||
# code rather than a coincidence of two hand-written inventories.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
|
||||
# - THE LIVE CLUSTERS. That three nodes are Ready, that all three are etcd members with
|
||||
# a healthy quorum, that kube-vip is actually floating the VIP, and that the API
|
||||
# answers through it. All of that is asserted at L2 by
|
||||
# tests/l2-behavioral/ha-control-plane, which reads it from the cluster under test.
|
||||
# - FAILOVER. That the control plane SURVIVES the loss of a server (etcd quorum holds,
|
||||
# the VIP moves, the API stays up) is a fault-injection claim. The nearest existing
|
||||
# coverage is tests/l3-fault-injection/node-loss, which drains a node for R-STATE-02's
|
||||
# data-availability claim and does NOT assert control-plane survival. A dedicated
|
||||
# "kill the VIP leader / kill the founder" L3 test is the remaining R-PLAT-01 gap and
|
||||
# is NOT claimed here or at L2.
|
||||
# - "PRODUCTION-GRADE" AS A HARDWARE/OPERATIONS JUDGEMENT. Whether three converged
|
||||
# mini-PCs with etcd sharing a spindle with Longhorn is production-grade is an operator
|
||||
# decision recorded in infra/PROD-PROVISIONING.md §2, not a repository property. This
|
||||
# check asserts the control-plane TOPOLOGY the spec fixes, not a performance verdict.
|
||||
# - IPv6 / dual-stack, and any claim about the edge network.
|
||||
#
|
||||
# TWO FORMER GAPS ARE NOW ASSERTED, and the reasoning is worth keeping (2026-09-06):
|
||||
# * CILIUM'S OWN API ENDPOINT used to be listed here as an ESCALATION — `K3S_API_HOST`
|
||||
# (clusters/*/kustomization.yaml), which becomes Cilium's `k8sServiceHost`, was each
|
||||
# environment's PRIMARY NODE address rather than the floating VIP, contradicting the prod
|
||||
# overlay's own comment. Making the check fail over it would have turned the merge gate
|
||||
# red on a design question this workstream may not decide (R-TEST-06), so it was reported
|
||||
# instead. THE OPERATOR DECIDED THE VIP on 2026-09-06, which turns the question into a
|
||||
# repository invariant, and invariants belong in assertions — see the "THE CNI DIALS THE
|
||||
# VIP" section. Note what made it assertable: the fix SPLIT one variable in two, so there
|
||||
# is finally an HA-valued name (`k3s_apiserver_endpoint`) to point the contract at.
|
||||
# * kube-vip WAS NOT PART OF THE ONE-COMMAND BRING-UP (site.yml chained node-baseline ->
|
||||
# k3s -> cilium -> verify -> flux-bootstrap), which was recorded here as an observation
|
||||
# and not asserted, because R-PLAT-01 does not fix the bring-up order. It is asserted now
|
||||
# for a different reason that IS in scope: with the CNI's endpoint being the VIP, an
|
||||
# unfloated VIP means a cold build installs a CNI that cannot reach its API server. The
|
||||
# ordering is no longer a preference about tidiness, it is a dependency of the control
|
||||
# plane coming up at all.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import ipaddress
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
NOTES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def note(msg):
|
||||
NOTES.append(msg)
|
||||
|
||||
|
||||
def die():
|
||||
print(f"CHECK FAIL (ha-control-plane-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
ANS = ROOT / "infra" / "ansible"
|
||||
INV = ANS / "inventory" / "hosts.ini"
|
||||
GV = ANS / "inventory" / "group_vars"
|
||||
K3S_ROLE = ANS / "roles" / "k3s_server"
|
||||
VIP_ROLE = ANS / "roles" / "kube_vip"
|
||||
PLAYBOOKS = ANS / "playbooks"
|
||||
CLUSTERS = ROOT / "clusters"
|
||||
|
||||
for p, why in ((INV, "the inventory that defines the node set"),
|
||||
(K3S_ROLE, "the role that installs the k3s control plane"),
|
||||
(VIP_ROLE, "the role that floats the control-plane VIP"),
|
||||
(CLUSTERS, "the Flux overlays that identify each environment")):
|
||||
check(p.exists(), f"{p.relative_to(ROOT)} is missing — {why} does not exist, so no HA "
|
||||
"control plane is authored at all (R-PLAT-01)")
|
||||
if FAILURES:
|
||||
die()
|
||||
|
||||
K3S_TEXT = "\n".join(p.read_text(errors="replace") for p in sorted(K3S_ROLE.rglob("*.yml")))
|
||||
VIP_TEXT = "\n".join(p.read_text(errors="replace")
|
||||
for p in sorted(list(VIP_ROLE.rglob("*.yml")) + list(VIP_ROLE.rglob("*.j2"))))
|
||||
|
||||
|
||||
# ===================================================================== inventory parse
|
||||
# Parser contract (same shape lib/coverage.py-style parsers use elsewhere): hosts.ini is a
|
||||
# plain INI inventory — `[group]` sections of `host key=value ...` lines, `[group:children]`
|
||||
# sections naming child groups, `[group:vars]` sections ignored. Comments start with '#'.
|
||||
ini_hosts: dict[str, list[str]] = {}
|
||||
ini_children: dict[str, list[str]] = {}
|
||||
host_vars: dict[str, dict[str, str]] = {}
|
||||
cur = None
|
||||
for line in INV.read_text().splitlines():
|
||||
s = line.strip()
|
||||
if not s or s.startswith("#"):
|
||||
continue
|
||||
m = re.match(r"^\[([^\]]+)\]$", s)
|
||||
if m:
|
||||
cur = m.group(1)
|
||||
continue
|
||||
if cur is None or cur.endswith(":vars"):
|
||||
continue
|
||||
host = s.split()[0]
|
||||
if cur.endswith(":children"):
|
||||
ini_children.setdefault(cur.split(":")[0], []).append(host)
|
||||
else:
|
||||
ini_hosts.setdefault(cur, []).append(host)
|
||||
hv = host_vars.setdefault(host, {})
|
||||
for kv in s.split()[1:]:
|
||||
if "=" in kv:
|
||||
k, v = kv.split("=", 1)
|
||||
hv[k] = v
|
||||
|
||||
|
||||
def hosts_in(group: str, seen=None) -> set[str]:
|
||||
seen = seen or set()
|
||||
if group in seen:
|
||||
return set()
|
||||
seen.add(group)
|
||||
out = set(ini_hosts.get(group, []))
|
||||
for child in ini_children.get(group, []):
|
||||
out |= hosts_in(child, seen)
|
||||
return out
|
||||
|
||||
|
||||
def load_yaml(p: Path) -> dict:
|
||||
try:
|
||||
d = yaml.safe_load(p.read_text()) or {}
|
||||
except yaml.YAMLError:
|
||||
return {}
|
||||
return d if isinstance(d, dict) else {}
|
||||
|
||||
|
||||
ALL_VARS = load_yaml(GV / "all.yml")
|
||||
|
||||
|
||||
# =============================================== ENVIRONMENTS, DERIVED FROM THE SEAM
|
||||
# An "environment" here is a (primary group, servers group) pair. group_vars/all.yml holds
|
||||
# the default pair; every group_vars/<g>/vars.yml that overrides `k3s_primary_group` is
|
||||
# another environment. Nothing below knows the words "dev" or "prod" as group names.
|
||||
class Env:
|
||||
def __init__(self, label, vars_, path):
|
||||
self.label = label
|
||||
self.vars = vars_
|
||||
self.path = path
|
||||
|
||||
def var(self, name, default=None):
|
||||
# group-scoped value wins, else the cluster-wide default from all.yml.
|
||||
if name in self.vars:
|
||||
return self.vars[name]
|
||||
return ALL_VARS.get(name, default)
|
||||
|
||||
|
||||
envs: list[Env] = [Env("group_vars/all.yml (default environment)", ALL_VARS, GV / "all.yml")]
|
||||
for vf in sorted(GV.glob("*/vars.yml")):
|
||||
v = load_yaml(vf)
|
||||
if "k3s_primary_group" in v or "k3s_servers_group" in v:
|
||||
envs.append(Env(str(vf.relative_to(ROOT)), v, vf))
|
||||
|
||||
check(len(envs) >= 2,
|
||||
"only ONE environment is derivable from the inventory's R-ENV-02 seam "
|
||||
"(k3s_primary_group / k3s_servers_group). R-PLAT-01 constrains prod AND requires "
|
||||
"dev/test to exercise the same multi-node topology, so two must be authored")
|
||||
|
||||
# WHICH environment is prod: the one that bootstraps the clusters/prod Flux overlay.
|
||||
# roles/flux_bootstrap's own default supplies the value for an environment that does not
|
||||
# override it, so this is read from code, never guessed from a group name.
|
||||
fb_default = None
|
||||
fb_defaults_file = ANS / "roles" / "flux_bootstrap" / "defaults" / "main.yml"
|
||||
if fb_defaults_file.is_file():
|
||||
fb_default = load_yaml(fb_defaults_file).get("flux_bootstrap_env_path")
|
||||
check(isinstance(fb_default, str) and fb_default.startswith("clusters/"),
|
||||
"roles/flux_bootstrap declares no flux_bootstrap_env_path default — the mapping from "
|
||||
"an inventory environment to its Flux overlay is not expressed in code, so this check "
|
||||
"cannot tell which environment is prod")
|
||||
|
||||
prod_envs = []
|
||||
for e in envs:
|
||||
overlay = e.var("flux_bootstrap_env_path", fb_default)
|
||||
e.overlay = overlay if isinstance(overlay, str) else ""
|
||||
if e.overlay.rstrip("/").endswith("/prod"):
|
||||
prod_envs.append(e)
|
||||
check(len(prod_envs) == 1,
|
||||
f"{len(prod_envs)} environment(s) bootstrap the clusters/prod overlay "
|
||||
f"({[e.label for e in prod_envs]}) — exactly one must be prod, or 'in prod' in "
|
||||
"R-PLAT-01 has no unambiguous referent")
|
||||
|
||||
|
||||
# ============================================== overlay knob values (per environment)
|
||||
# Parser contract: clusters/base/cluster-vars.yaml holds the base ConfigMap `data:` map;
|
||||
# each clusters/<env>/kustomization.yaml overrides values with JSON6902 patches written as
|
||||
# - op: replace
|
||||
# path: /data/<KEY>
|
||||
# value: "<v>"
|
||||
# (verified shape across both overlays). A key not patched keeps the base value.
|
||||
BASE_VARS: dict[str, str] = {}
|
||||
base_cm = CLUSTERS / "base" / "cluster-vars.yaml"
|
||||
if base_cm.is_file():
|
||||
d = load_yaml(base_cm)
|
||||
BASE_VARS = {k: str(v) for k, v in (d.get("data") or {}).items()}
|
||||
check(len(BASE_VARS) > 5,
|
||||
"clusters/base/cluster-vars.yaml exposes no data map — the overlay knob surface "
|
||||
"cannot be read, so the VIP/pool separation cannot be checked")
|
||||
|
||||
_PATCH_RE = re.compile(
|
||||
r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n\s*value:\s*(?P<v>\S.*?)\s*$",
|
||||
re.M)
|
||||
|
||||
|
||||
def overlay_vars(overlay: str) -> dict[str, str]:
|
||||
out = dict(BASE_VARS)
|
||||
f = ROOT / overlay / "kustomization.yaml"
|
||||
if not f.is_file():
|
||||
return out
|
||||
for m in _PATCH_RE.finditer(f.read_text()):
|
||||
out[m.group("k")] = m.group("v").strip().strip('"').strip("'")
|
||||
return out
|
||||
|
||||
|
||||
# ============================================================ PER-ENVIRONMENT TOPOLOGY
|
||||
def as_ip(s):
|
||||
try:
|
||||
return ipaddress.ip_address(str(s).strip())
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
for e in envs:
|
||||
L = e.label
|
||||
pg = e.var("k3s_primary_group")
|
||||
sg = e.var("k3s_servers_group")
|
||||
check(isinstance(pg, str) and isinstance(sg, str) and pg and sg,
|
||||
f"{L}: k3s_primary_group / k3s_servers_group are not both set — the roles cannot "
|
||||
"resolve 'my environment's primary/servers' and would only ever build dev")
|
||||
if not (isinstance(pg, str) and isinstance(sg, str)):
|
||||
continue
|
||||
primaries = hosts_in(pg)
|
||||
servers = hosts_in(sg)
|
||||
|
||||
# --- SERVERS: quorum-capable, one founder, prod exactly three ---------------------
|
||||
check(len(primaries) == 1,
|
||||
f"{L}: group {pg!r} resolves to {sorted(primaries)} — EXACTLY ONE host may run "
|
||||
"`--cluster-init`. Two founders is two etcd clusters (split brain), zero is no "
|
||||
"cluster at all")
|
||||
check(primaries and primaries <= servers,
|
||||
f"{L}: the primary {sorted(primaries)} is not a member of the servers group "
|
||||
f"{sg!r} ({sorted(servers)}) — the founder must itself be one of the servers, or "
|
||||
"it is not part of the HA control plane it founds")
|
||||
n = len(servers)
|
||||
check(n >= 3,
|
||||
f"{L}: group {sg!r} resolves to {n} server(s) {sorted(servers)}. docs/02 fixes "
|
||||
"'HA embedded etcd (k3s server on all 3)' for dev/test AND prod, and an etcd "
|
||||
"cluster of fewer than 3 members tolerates ZERO losses — it is not HA")
|
||||
check(n % 2 == 1,
|
||||
f"{L}: group {sg!r} has an EVEN number of servers ({n}). An even embedded-etcd "
|
||||
f"member count adds a member without adding fault tolerance and introduces a tie: "
|
||||
f"quorum for {n} is {n // 2 + 1}, the same as for {n - 1}")
|
||||
if n:
|
||||
tolerance = n - (n // 2 + 1)
|
||||
check(tolerance >= 1,
|
||||
f"{L}: {n} server(s) give a quorum of {n // 2 + 1} and therefore tolerate "
|
||||
f"{tolerance} loss(es) — 'production-grade HA' requires surviving at least one")
|
||||
if e in prod_envs:
|
||||
check(n == 3,
|
||||
f"{L} is the PROD environment and resolves to {n} server(s). R-PLAT-01: "
|
||||
"'k3s across 3 nodes ... in prod'; docs/02: 'Node count | ... | 3 physical nodes'")
|
||||
else:
|
||||
check(n >= 2,
|
||||
f"{L}: a non-prod environment resolves to {n} host(s). R-PLAT-01 allows fewer "
|
||||
"nodes than prod but requires dev/test to 'exercise the same multi-node "
|
||||
"topology' — one node exercises no topology")
|
||||
|
||||
# --- ADDRESSES: the VIP can collide with nothing ----------------------------------
|
||||
vip_raw = e.var("kube_vip_address")
|
||||
vip = as_ip(vip_raw)
|
||||
check(vip is not None,
|
||||
f"{L}: kube_vip_address is {vip_raw!r}, not an IP address — the control plane has "
|
||||
"no floating endpoint, so losing the founder takes the API endpoint with it")
|
||||
node_ips = {}
|
||||
for h in sorted(servers):
|
||||
ip = as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
|
||||
check(ip is not None,
|
||||
f"{L}: host {h} declares no usable k3s_node_ip — k3s --node-ip, the join "
|
||||
"target and the serving-cert SAN list all derive from it")
|
||||
if ip is not None:
|
||||
node_ips[h] = ip
|
||||
if vip is not None:
|
||||
check(vip not in set(node_ips.values()),
|
||||
f"{L}: kube_vip_address {vip} is ALSO a node address "
|
||||
f"({[h for h, i in node_ips.items() if i == vip]}) — a floating VIP that is a "
|
||||
"node's own address cannot float off that node")
|
||||
ov = overlay_vars(getattr(e, "overlay", ""))
|
||||
gw = as_ip(ov.get("GATEWAY_VIP", ""))
|
||||
lo = as_ip(ov.get("CILIUM_LB_POOL_START", ""))
|
||||
hi = as_ip(ov.get("CILIUM_LB_POOL_STOP", ""))
|
||||
check(gw is not None and lo is not None and hi is not None,
|
||||
f"{L}: overlay {getattr(e, 'overlay', '')!r} does not resolve GATEWAY_VIP + "
|
||||
"CILIUM_LB_POOL_START/STOP — the separation between the control-plane VIP and "
|
||||
"the Service-LB allocator cannot be checked")
|
||||
if gw is not None:
|
||||
check(vip != gw,
|
||||
f"{L}: the control-plane VIP and the ingress Gateway VIP are the SAME "
|
||||
f"address ({vip}) — kube-vip and Cilium would both ARP for it")
|
||||
if lo is not None and hi is not None:
|
||||
check(not (lo <= vip <= hi),
|
||||
f"{L}: the control-plane VIP {vip} lies INSIDE that environment's Cilium "
|
||||
f"LoadBalancer pool {lo}-{hi} — LB-IPAM could hand the kube-API's VIP to a "
|
||||
"Service, and the two allocators would fight over ARP for it")
|
||||
check(lo <= hi,
|
||||
f"{L}: Cilium LB pool {lo}-{hi} is inverted — the pool bound cannot be "
|
||||
"used to prove the VIP is outside it")
|
||||
|
||||
# ==================================================================== DISTRO: k3s only
|
||||
check(re.search(r"^k3s_version:\s*\"?v\d+\.\d+", (K3S_ROLE / "defaults" / "main.yml").read_text(), re.M)
|
||||
is not None,
|
||||
"roles/k3s_server pins no concrete k3s_version (vX.Y...) — an unpinned control-plane "
|
||||
"version is not a reproducible platform (docs/01: the distro is k3s, ADR-0001)")
|
||||
check("get.k3s.io" in K3S_TEXT,
|
||||
"roles/k3s_server does not install from the k3s installer — R-PLAT-01 names k3s as the "
|
||||
"distro (ADR-0001) and no other distro may stand in for it")
|
||||
check(re.search(r"^\s*server\s*$", K3S_TEXT, re.M) is not None,
|
||||
"roles/k3s_server never composes the `server` subcommand — without it k3s installs an "
|
||||
"AGENT, which is not a control-plane member (docs/02: 'k3s server on all 3')")
|
||||
for bad, why in (
|
||||
(r"--datastore-endpoint", "an EXTERNAL datastore (docs/02: 'embedded etcd across all 3 "
|
||||
"nodes (no external datastore)')"),
|
||||
(r"datastore-endpoint\s*:", "an external datastore via config.yaml"),
|
||||
(r"--disable-etcd", "a server with etcd DISABLED, i.e. a control-plane node that is not "
|
||||
"an etcd member (docs/02: etcd 'across all 3 nodes')"),
|
||||
(r"INSTALL_K3S_EXEC\s*[:=]\s*[\"']?agent", "an AGENT-only node"),
|
||||
):
|
||||
hits = [str(p.relative_to(ROOT)) for p in sorted(ANS.rglob("*"))
|
||||
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh", ".ini")
|
||||
and re.search(bad, p.read_text(errors="replace"))]
|
||||
check(not hits,
|
||||
f"{hits} configure {why} — R-PLAT-01's control plane is k3s's own embedded-etcd HA "
|
||||
f"mode, and {bad!r} contradicts it")
|
||||
|
||||
# ============================================== FOUNDER / JOINER, EXACTLY ONE OF EACH
|
||||
check(re.search(r"--cluster-init", K3S_TEXT) is not None,
|
||||
"roles/k3s_server never passes --cluster-init — the embedded-etcd cluster is never "
|
||||
"founded, so there is no HA datastore to join")
|
||||
init_when = re.search(r"--cluster-init.*?\n(?:.*\n){0,8}?\s*when:\s*k3s_server_mode\s*==\s*'primary'",
|
||||
K3S_TEXT)
|
||||
check(init_when is not None,
|
||||
"the --cluster-init install task is not gated on k3s_server_mode == 'primary' — an "
|
||||
"ungated founder flag would re-found the cluster on every server (split brain)")
|
||||
join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
|
||||
check(join is not None,
|
||||
"roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
|
||||
"— without a join target the servers are separate single-node clusters")
|
||||
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
|
||||
K3S_TEXT, re.S)
|
||||
check(join_when is not None,
|
||||
"the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
|
||||
"would also try to join itself")
|
||||
k3s_play = PLAYBOOKS / "k3s.yml"
|
||||
check(k3s_play.is_file(), "playbooks/k3s.yml is missing — the control-plane install has no entrypoint")
|
||||
if k3s_play.is_file():
|
||||
play_txt = k3s_play.read_text()
|
||||
modes = re.findall(r"k3s_server_mode:\s*(\w+)", play_txt)
|
||||
check(sorted(modes) == ["primary", "secondary"],
|
||||
f"playbooks/k3s.yml declares server modes {modes} — exactly one primary play and "
|
||||
"one secondary play must exist, or the founder/joiner split is not authored")
|
||||
check(re.search(r"serial:\s*1", play_txt) is not None,
|
||||
"playbooks/k3s.yml does not restart/join the secondaries with `serial: 1` — "
|
||||
"converging two of three servers at once drops embedded etcd below quorum, which "
|
||||
"is exactly the outage an HA control plane exists to avoid")
|
||||
hosts_patterns = re.findall(r"^\s*hosts:\s*(.+)$", play_txt, re.M)
|
||||
check(len(hosts_patterns) >= 2 and all("{{" in h for h in hosts_patterns),
|
||||
f"playbooks/k3s.yml targets literal host patterns {hosts_patterns} — a literal "
|
||||
"group name means this play can only ever build ONE environment, so prod cannot "
|
||||
"be brought up by the same code (docs/02: 'the *same* Ansible bootstrap runs')")
|
||||
|
||||
# =========================================== SERVING-CERT SANs: VIP + every server
|
||||
sans = str(load_yaml(K3S_ROLE / "defaults" / "main.yml").get("k3s_tls_sans", ""))
|
||||
check("kube_vip_address" in sans,
|
||||
"roles/k3s_server's k3s_tls_sans does not include kube_vip_address — the apiserver's "
|
||||
"serving cert would carry no SAN for the floating VIP, so every client reaching the "
|
||||
"API through it fails TLS validation and the HA endpoint is unusable")
|
||||
check("k3s_servers_group" in sans and "k3s_node_ip" in sans,
|
||||
"k3s_tls_sans does not extract k3s_node_ip for every host of k3s_servers_group — the "
|
||||
"cert must be valid for WHICHEVER server answers behind the VIP, and per-environment "
|
||||
"resolution of that set is what keeps one role serving both environments (R-ENV-02)")
|
||||
check(re.search(r"tls-san:", K3S_TEXT) is not None,
|
||||
"the SAN list is passed only as an install flag and never persisted to "
|
||||
"/etc/rancher/k3s/config.yaml — on an ALREADY-installed cluster the install task is a "
|
||||
"no-op, so a changed SAN set (e.g. a new server) would never reach the serving cert")
|
||||
|
||||
# ============================ THE CNI DIALS THE VIP, NOT THE FOUNDER (added 2026-09-06)
|
||||
# This section used to be an ESCALATION in the header's gap list, not an assertion: Cilium's
|
||||
# k8sServiceHost was each environment's PRIMARY NODE address, so on a 3-server HA control
|
||||
# plane the CNI's control-plane path depended on one machine. Measured on live prod
|
||||
# 2026-09-06: all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10
|
||||
# (k3s-prod-1's own address) while kube-vip held 192.168.2.9. That failure is SILENT — the
|
||||
# datapath keeps serving from Cilium's cache, so nothing looks broken while no service or
|
||||
# policy update lands at all.
|
||||
#
|
||||
# The operator decided the VIP (2026-09-06), so this is now a repository property and belongs
|
||||
# here rather than in the gap list. THIS IS THE RIGHT LAYER because the claim is about what
|
||||
# the configuration in git WOULD build, in BOTH environments, including the next one — a live
|
||||
# cluster can only tell you about itself.
|
||||
#
|
||||
# WHAT MAKES THIS CHECKABLE AT ALL is that the fix split one variable into two. Asserting
|
||||
# "Cilium uses the HA address" requires an HA address to name; before the split there was
|
||||
# only k3s_api_host, which had to be the founder for the cold join. So the assertions below
|
||||
# check the SPLIT as much as the value: the join keeps the founder, the CNI gets the VIP, and
|
||||
# the Flux half agrees with the Ansible half per environment.
|
||||
CIL_ROLE = ANS / "roles" / "cilium"
|
||||
cil_values_tpl = CIL_ROLE / "templates" / "cilium-values.yaml.j2"
|
||||
check(cil_values_tpl.is_file(),
|
||||
"roles/cilium ships no cilium-values.yaml.j2 — the bootstrap's CNI values, including the "
|
||||
"API endpoint its kube-proxy replacement dials, are not authored")
|
||||
if cil_values_tpl.is_file():
|
||||
# Parse the rendered-shape template, never its prose: strip comment lines first, exactly
|
||||
# as the kube-vip parser above does, so a reassuring comment cannot satisfy the check.
|
||||
cil_body = "\n".join(l for l in cil_values_tpl.read_text(errors="replace").splitlines()
|
||||
if not l.lstrip().startswith("#"))
|
||||
m = re.search(r"^k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", cil_body, re.M)
|
||||
check(m is not None,
|
||||
"roles/cilium's values template sets no k8sServiceHost — with kube-proxy disabled "
|
||||
"(ADR-0001) the agent has no API endpoint to dial and the CNI cannot come up")
|
||||
if m is not None:
|
||||
host_expr = m.group(1).strip()
|
||||
check("k3s_apiserver_endpoint" in host_expr,
|
||||
f"roles/cilium renders k8sServiceHost={host_expr!r}. It MUST be the HA endpoint "
|
||||
"variable {{ k3s_apiserver_endpoint }} (= kube_vip_address). Any node-valued "
|
||||
"expression — k3s_api_host, a primary hostvar, a literal IP — makes the CNI's "
|
||||
"control-plane path depend on ONE server of an HA control plane, and the failure "
|
||||
"is silent: the datapath keeps running from Cilium's cache while no service or "
|
||||
"policy update ever lands (measured on prod 2026-09-06)")
|
||||
check("k3s_api_host" not in host_expr,
|
||||
f"roles/cilium renders k8sServiceHost={host_expr!r}, which references "
|
||||
"k3s_api_host — the FOUNDER's own address. That variable exists only for the "
|
||||
"cold-bootstrap `--server` join and one tls-san entry; it is a single point of "
|
||||
"failure by construction and must never be a steady-state endpoint")
|
||||
|
||||
# The HA endpoint variable must BE the VIP, and must be DERIVED from kube_vip_address rather
|
||||
# than restated: an environment overrides one line (kube_vip_address) and both the floating
|
||||
# address and the CNI's endpoint move together. A literal here could disagree with the VIP the
|
||||
# same environment actually floats, which is the two-places-one-fact bug this repo keeps
|
||||
# hitting.
|
||||
endpoint_def = str(ALL_VARS.get("k3s_apiserver_endpoint", ""))
|
||||
check("kube_vip_address" in endpoint_def,
|
||||
f"group_vars/all.yml defines k3s_apiserver_endpoint as {endpoint_def!r}. It MUST derive "
|
||||
"from kube_vip_address, so that overriding the VIP for an environment also moves what "
|
||||
"that environment's CNI dials. A restated literal can silently disagree with the VIP "
|
||||
"kube-vip is actually floating")
|
||||
check("kube_vip_address" not in str(ALL_VARS.get("k3s_api_host", "")),
|
||||
"group_vars/all.yml's k3s_api_host resolves from kube_vip_address — the two API "
|
||||
"addresses have been re-merged. k3s_api_host is the secondary join target and on a COLD "
|
||||
"build no VIP exists yet, so a VIP-valued join target makes a cold bootstrap unjoinable")
|
||||
|
||||
# THE [parity] CONTRACT, now machine-checked instead of asserted in a comment: the Flux
|
||||
# overlay's K3S_API_HOST becomes Cilium's k8sServiceHost under the HelmRelease, and the
|
||||
# Ansible bootstrap renders k3s_apiserver_endpoint for the same field. If the two disagree,
|
||||
# Flux and the bootstrap describe different control-plane paths and fight over the CNI.
|
||||
for e in envs:
|
||||
L = e.label
|
||||
ov = overlay_vars(getattr(e, "overlay", ""))
|
||||
k3s_api_host_knob = as_ip(ov.get("K3S_API_HOST", ""))
|
||||
vip = as_ip(e.var("kube_vip_address"))
|
||||
check(k3s_api_host_knob is not None,
|
||||
f"{L}: overlay {getattr(e, 'overlay', '')!r} resolves no usable K3S_API_HOST — that "
|
||||
"token becomes Cilium's k8sServiceHost via platform/cilium/helmrelease.yaml, and "
|
||||
"kustomize-controller runs with StrictPostBuildSubstitutions so an unresolvable one "
|
||||
"fails the whole platform Kustomization")
|
||||
if k3s_api_host_knob is not None and vip is not None:
|
||||
check(k3s_api_host_knob == vip,
|
||||
f"{L}: [parity] BROKEN — the Flux overlay sets K3S_API_HOST={k3s_api_host_knob} "
|
||||
f"but this environment's control-plane VIP (kube_vip_address, which is what "
|
||||
f"k3s_apiserver_endpoint and therefore the Ansible-side k8sServiceHost resolve "
|
||||
f"to) is {vip}. Flux and the Ansible bootstrap would render DIFFERENT API "
|
||||
"endpoints into the same CNI and fight over it. If they differ because the "
|
||||
"overlay still holds a NODE address, that is the 2026-09-06 defect: the CNI of "
|
||||
"an HA control plane must not dial one server")
|
||||
check(k3s_api_host_knob not in set(
|
||||
as_ip(host_vars.get(h, {}).get("k3s_node_ip", ""))
|
||||
for h in hosts_in(str(e.var("k3s_servers_group")))),
|
||||
f"{L}: the overlay's K3S_API_HOST {k3s_api_host_knob} is one of this "
|
||||
"environment's own NODE addresses. Cilium's kube-proxy replacement would dial a "
|
||||
"single machine on a control plane built specifically to survive losing one")
|
||||
|
||||
# The Flux half must stay a per-environment TOKEN. A literal IP in the shared HelmRelease
|
||||
# would ship one environment's control-plane address to every environment (R-ENV-02), and an
|
||||
# empty value is the 2026-09-01 incident verbatim (k8sServiceHost: "" made the agent's config
|
||||
# init container fall back to the in-cluster ClusterIP 10.43.0.1:443, which only the agent
|
||||
# being replaced could translate: deadlocked on 2 of 3 nodes).
|
||||
hr = ROOT / "platform" / "cilium" / "helmrelease.yaml"
|
||||
check(hr.is_file(), "platform/cilium/helmrelease.yaml is missing — nothing adopts the CNI "
|
||||
"config after the bootstrap, so the parity contract has only one side")
|
||||
if hr.is_file():
|
||||
hr_body = "\n".join(l for l in hr.read_text(errors="replace").splitlines()
|
||||
if not l.lstrip().startswith("#"))
|
||||
hm = re.search(r"^\s*k8sServiceHost:\s*\"?([^\"\n]*)\"?\s*$", hr_body, re.M)
|
||||
check(hm is not None and hm.group(1).strip() == "${K3S_API_HOST}",
|
||||
f"platform/cilium/helmrelease.yaml sets k8sServiceHost="
|
||||
f"{(hm.group(1) if hm else None)!r} instead of the cluster-vars token "
|
||||
"\"${K3S_API_HOST}\" — a literal there would ship ONE environment's control-plane "
|
||||
"address to every environment (R-ENV-02), and an empty value is the 2026-09-01 "
|
||||
"incident verbatim")
|
||||
|
||||
# ORDERING: the VIP must be floating BEFORE the CNI is installed against it. site.yml is the
|
||||
# one-command bring-up, so this is a property of the authored sequence, not of an operator's
|
||||
# memory. (kube-vip can precede the CNI because its DaemonSet is hostNetwork with
|
||||
# KUBERNETES_SERVICE_HOST=127.0.0.1 — asserted separately below/above via the manifest parse.)
|
||||
site_play = PLAYBOOKS / "site.yml"
|
||||
check(site_play.is_file(), "playbooks/site.yml is missing — there is no one-command bring-up "
|
||||
"whose ORDER can be checked")
|
||||
if site_play.is_file():
|
||||
imports = re.findall(r"import_playbook:\s*(\S+)", site_play.read_text())
|
||||
for needed in ("k3s.yml", "kube-vip.yml", "cilium.yml"):
|
||||
check(needed in imports,
|
||||
f"playbooks/site.yml does not import {needed} (imports: {imports}). Cilium's "
|
||||
"k8sServiceHost is the control-plane VIP, so kube-vip is a PREREQUISITE of the "
|
||||
"CNI, not an optional extra step — leaving it out of the bring-up means a cold "
|
||||
"build installs a CNI pointed at an address nothing announces")
|
||||
if all(n in imports for n in ("k3s.yml", "kube-vip.yml", "cilium.yml")):
|
||||
check(imports.index("k3s.yml") < imports.index("kube-vip.yml") < imports.index("cilium.yml"),
|
||||
f"playbooks/site.yml imports {imports} — the order MUST be k3s.yml -> "
|
||||
"kube-vip.yml -> cilium.yml. kube-vip after cilium installs the CNI against an "
|
||||
"unannounced VIP (the 2026-09-01 shape: agents that cannot reach the API server "
|
||||
"on a cold cluster); kube-vip before k3s cannot work at all, because the VIP "
|
||||
"ships as a k3s auto-deploy manifest and needs a cluster to apply it")
|
||||
check(re.search(r"wait_for", VIP_TEXT) is not None
|
||||
and re.search(r"kube_vip_address", VIP_TEXT) is not None,
|
||||
"roles/kube_vip never WAITS for the VIP to answer — templating the manifest is not the "
|
||||
"same as having a floating endpoint (k3s still has to apply it, the image still has to "
|
||||
"pull, leader election still has to settle). Since the CNI is installed immediately "
|
||||
"afterward against this exact address, an unverified VIP turns a missing endpoint into "
|
||||
"unreachable cilium-agents diagnosed from the far side")
|
||||
|
||||
# ============================================ FLOATING ENDPOINT: kube-vip on every server
|
||||
# Read the DaemonSet's ACTUAL container env out of the template, never the role's prose.
|
||||
# (An earlier draft matched `svc_enable=false` anywhere in the role and passed happily with
|
||||
# the template flipped to true — the comment in defaults/main.yml satisfied it. Parse the
|
||||
# thing that ships.) Parser contract: the template writes each env var as an adjacent
|
||||
# `- name: X` / `value: "Y"` pair; comment lines are dropped first.
|
||||
VIP_TEMPLATES = sorted(VIP_ROLE.rglob("*.j2"))
|
||||
check(len(VIP_TEMPLATES) >= 1,
|
||||
"roles/kube_vip ships no manifest template — nothing deploys the API VIP")
|
||||
vip_manifest = "\n".join(
|
||||
"\n".join(l for l in p.read_text(errors="replace").splitlines()
|
||||
if not l.lstrip().startswith("#"))
|
||||
for p in VIP_TEMPLATES)
|
||||
vip_env = dict(re.findall(r"-\s*name:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\n\s*value:\s*\"?([^\"\n]*)\"?",
|
||||
vip_manifest))
|
||||
check(len(vip_env) >= 6,
|
||||
f"only {len(vip_env)} env var(s) parsed out of the kube-vip manifest template — the "
|
||||
"parser lost its grip on the file, so the assertions below would assert nothing")
|
||||
check(vip_env.get("cp_enable") == "true",
|
||||
f"the kube-vip manifest sets cp_enable={vip_env.get('cp_enable')!r} — control-plane "
|
||||
"mode must be ON or it floats no API VIP at all")
|
||||
check(vip_env.get("svc_enable") == "false",
|
||||
f"the kube-vip manifest sets svc_enable={vip_env.get('svc_enable')!r} — it must be "
|
||||
"false, or kube-vip ALSO serves Service LoadBalancers and contends with Cilium LB-IPAM "
|
||||
"for the same addresses (ADR-0002 gives Service LB to Cilium)")
|
||||
check(vip_env.get("vip_leaderelection") == "true",
|
||||
f"the kube-vip manifest sets vip_leaderelection={vip_env.get('vip_leaderelection')!r} "
|
||||
"— without leader election the VIP has no defined owner to fail over FROM, which is "
|
||||
"the whole HA mechanism")
|
||||
check(vip_env.get("address", "").strip() == "{{ kube_vip_address }}",
|
||||
f"the kube-vip manifest announces address={vip_env.get('address')!r} instead of the "
|
||||
"environment-scoped {{ kube_vip_address }} — a literal here would float DEV's VIP on "
|
||||
"prod (R-ENV-02) and would not match the serving-cert SAN list")
|
||||
check(re.search(r"^kind:\s*DaemonSet\s*$", vip_manifest, re.M) is not None,
|
||||
"the kube-vip manifest is not a DaemonSet — a single-replica VIP holder dies with its "
|
||||
"node, which is the opposite of a floating endpoint")
|
||||
vip_play = PLAYBOOKS / "kube-vip.yml"
|
||||
check(vip_play.is_file(), "playbooks/kube-vip.yml is missing — nothing deploys the API VIP")
|
||||
if vip_play.is_file():
|
||||
vp = vip_play.read_text()
|
||||
vip_hosts = re.findall(r"^\s*hosts:\s*(.+)$", vp, re.M)
|
||||
check(len(vip_hosts) == 1 and "k3s_hosts" in vip_hosts[0],
|
||||
f"playbooks/kube-vip.yml targets {vip_hosts} — it must target the whole servers "
|
||||
"group through the same environment-scoped variable the other plays use, so the "
|
||||
"manifest exists on EVERY server and the VIP survives losing any one of them")
|
||||
check(re.search(r"kube_vip_manifest_path", VIP_TEXT) is not None
|
||||
and re.search(r"k3s_data_dir", VIP_TEXT) is not None,
|
||||
"roles/kube_vip's manifest path is not derived from k3s_data_dir — k3s only auto-applies "
|
||||
"<data-dir>/server/manifests, so an environment that moves --data-dir would write the "
|
||||
"manifest somewhere k3s never reads and come up with no VIP at all")
|
||||
|
||||
# ================================================== ONE CODE PATH FOR BOTH ENVIRONMENTS
|
||||
# docs/02: "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
|
||||
# step differs". A prod-only role or playbook would be a fork of the control-plane path.
|
||||
role_names = {p.name for p in (ANS / "roles").iterdir() if p.is_dir()}
|
||||
play_names = {p.name for p in PLAYBOOKS.glob("*.yml")}
|
||||
env_words = re.compile(r"prod|dev", re.I)
|
||||
forked = sorted(n for n in (role_names | play_names)
|
||||
if env_words.search(n) and n not in ("memattr_survey", "memattr-survey.yml"))
|
||||
check(not forked,
|
||||
f"environment-named roles/playbooks exist ({forked}) — dev and prod must be built by "
|
||||
"the SAME code retargeted by variables (docs/02), not by a per-environment fork")
|
||||
for e in envs:
|
||||
if e.path.name != "vars.yml":
|
||||
continue
|
||||
# Every key a non-default environment sets must be a variable something already reads,
|
||||
# or it is an invented knob nothing consumes (a silent no-op that looks like config).
|
||||
consumers = "\n".join(p.read_text(errors="replace") for p in sorted(ANS.rglob("*"))
|
||||
if p.is_file() and p.suffix in (".yml", ".yaml", ".j2", ".sh"))
|
||||
for k in sorted(e.vars):
|
||||
if k.startswith("ansible_"):
|
||||
continue
|
||||
check(consumers.count(k) > 1,
|
||||
f"{e.label} sets {k!r}, which appears nowhere else under infra/ansible — an "
|
||||
"environment override that nothing reads is not a scoped override, it is dead "
|
||||
"configuration (and would make prod silently inherit a dev default)")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
|
||||
check(CHECKS >= 40,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
die()
|
||||
for n in NOTES:
|
||||
print(f" note: {n}")
|
||||
summary = "; ".join(
|
||||
f"{e.label.split('/')[-1]}[{'PROD' if e in prod_envs else 'non-prod'}]="
|
||||
f"{len(hosts_in(str(e.var('k3s_servers_group'))))} servers"
|
||||
for e in envs)
|
||||
print(f"check OK: ha-control-plane-as-code — {CHECKS} assertions; {summary}. "
|
||||
"R-PLAT-01's AUTHORED half only: k3s, one --cluster-init founder per environment, "
|
||||
"embedded etcd on every server with no external datastore, a leader-elected "
|
||||
"control-plane-only VIP on every server that is SAN'd into the serving cert and "
|
||||
"collides with no other allocator, floated BEFORE the CNI that dials it (and the CNI "
|
||||
"dials that VIP, not the founder, in both environments and on both sides of the "
|
||||
"[parity] contract), from one code path for both environments. The LIVE "
|
||||
"cluster shape is proven at L2 (tests/l2-behavioral/ha-control-plane); control-plane "
|
||||
"SURVIVAL of a node loss is proven NOWHERE yet (see the header's gap list).")
|
||||
PY
|
||||
@@ -1,44 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — least privilege for workload identities and network reach (R-SEC-02)
|
||||
# Catalog: docs/14 L0 row, "Manifests/ServiceDefinitions obey invariants" — runs in
|
||||
# CI WITHOUT a cluster.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked RBAC object and every least-network-reach policy is inspected,
|
||||
# THEN no in-repo role grants a wildcard or a privilege-escalation verb, no workload
|
||||
# ServiceAccount is bound to a privileged role, the conformance runner's own
|
||||
# privileged test identity stays scoped to its authored ClusterRole, and each
|
||||
# least-network-reach policy both Enforces and is proven by an L0 negative row
|
||||
# (or a generated-resource row) to actually close reach.
|
||||
#
|
||||
# STATUS: EXECUTABLE — and DELIBERATELY PARTIAL. `check.sh` is a pure offline assertion
|
||||
# script (git ls-files + PyYAML); conformance/run.sh --layer l0 and ci/l0.sh run it
|
||||
# directly, and this Chainsaw Test wraps the SAME script. It proves the STATIC half of
|
||||
# both clauses. The RUNTIME half — that the API server actually denies an unrequested
|
||||
# verb, and that pod-to-pod traffic is actually dropped — needs a cluster and is NOT
|
||||
# asserted under this requirement; see the gap list in check.sh's header. R-SEC-02 is
|
||||
# therefore covered, not finished.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: least-privilege-rbac
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Workload identities carry no wildcard, no escalation verb and no privileged
|
||||
binding; the least-network-reach policies are enforced and proven to reject.
|
||||
spec:
|
||||
steps:
|
||||
- name: scan-rbac-and-least-reach-policies
|
||||
description: >-
|
||||
Assert the authored least-privilege posture of every tracked RBAC object and
|
||||
the enforced, test-proven status of the least-network-reach policies.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,269 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — least-privilege-rbac
|
||||
# Serves: R-SEC-02 ("The platform MUST follow least-privilege for workload
|
||||
# identities and network reach." — spec docs/00-vision-and-scope.md;
|
||||
# detail in docs/07 "least privilege ... least network reach").
|
||||
#
|
||||
# The requirement has TWO halves and this check covers the STATIC half of each,
|
||||
# at the layer docs/14 puts them ("L0 Static/policy | Pre-apply, in CI").
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked RBAC object and every least-network-reach policy is inspected,
|
||||
# THEN — workload identity —
|
||||
# (1) no in-repo Role/ClusterRole grants a wildcard apiGroup, resource or verb,
|
||||
# and none grants escalate/bind/impersonate (the privilege-escalation verbs);
|
||||
# (2) no (Cluster)RoleBinding binds a ServiceAccount — a WORKLOAD identity — to
|
||||
# `cluster-admin` or to any in-repo role holding wildcard rules. Human/Group
|
||||
# subjects are out of scope: R-SEC-02 constrains workload identities, and the
|
||||
# operator's own admin path is an OIDC group binding, deliberately;
|
||||
# (3) every ServiceAccount subject is namespaced (no cluster-wide ambiguity);
|
||||
# (4) the conformance runner's own SA — this workstream's PRIVILEGED TEST
|
||||
# IDENTITY, whose least-privilege authoring docs/14 explicitly delegates to
|
||||
# WS-TEST — is bound only to its own scoped ClusterRole, carries no wildcard
|
||||
# and no privilege-escalation verb, and grants nothing over RBAC objects
|
||||
# themselves (no rbac.authorization.k8s.io rules);
|
||||
# — network reach —
|
||||
# (5) the policies that implement least network reach exist, and each is Enforce
|
||||
# (or carries the Flux token that substitutes to Enforce): the default-deny
|
||||
# baseline generator, the baseline SHAPE validator, the host-namespace/
|
||||
# privilege restriction, and the egress-class label requirement;
|
||||
# (6) each of those policies is actually EXERCISED by the L0 Kyverno suite with a
|
||||
# NEGATIVE row (result: fail) — a policy nobody proves rejects anything is a
|
||||
# declaration, not a gate.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gap, cluster layer — R-SEC-02 is NOT fully covered):
|
||||
# - RUNTIME least privilege of workload identities: that the API server actually
|
||||
# denies the conformance SA (or any workload SA) the verbs it was not granted.
|
||||
# That needs a live cluster (L2) and no test asserts it today.
|
||||
# - RUNTIME least network reach: that pod-to-pod traffic is actually denied. The
|
||||
# behaviour is exercised by tests/l2-behavioral/{default-deny,deny-undeclared},
|
||||
# but those are tagged R-NET-01/R-NET-02 and are cluster-gated (skipped in Phase 0);
|
||||
# nothing proves it under an R-SEC-02 tag.
|
||||
# - Helm-chart-generated RBAC (e.g. what a HelmRelease's chart creates in-cluster) is
|
||||
# not in the repo, so it is outside this static scan entirely.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import os, re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def tracked() -> list[str]:
|
||||
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout
|
||||
return [p for p in out.split("\0") if p]
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
|
||||
if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
FILES = tracked()
|
||||
YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
|
||||
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
|
||||
|
||||
ESCALATION_VERBS = {"escalate", "bind", "impersonate"}
|
||||
PRIVILEGED_ROLES = {"cluster-admin", "admin", "edit"}
|
||||
|
||||
roles: dict[tuple[str, str], dict] = {} # (kind, name) -> {"file":…, "wildcard":bool, …}
|
||||
bindings: list[dict] = []
|
||||
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
kind = d.get("kind")
|
||||
name = (d.get("metadata") or {}).get("name")
|
||||
if kind in ("Role", "ClusterRole"):
|
||||
wildcard, escalating, rbac_over_rbac = False, set(), False
|
||||
for r in d.get("rules") or []:
|
||||
if not isinstance(r, dict):
|
||||
continue
|
||||
for field in ("apiGroups", "resources", "verbs"):
|
||||
if "*" in (r.get(field) or []):
|
||||
wildcard = True
|
||||
escalating |= set(v for v in (r.get("verbs") or [])) & ESCALATION_VERBS
|
||||
if "rbac.authorization.k8s.io" in (r.get("apiGroups") or []):
|
||||
rbac_over_rbac = True
|
||||
roles[(kind, name)] = {"file": rel, "wildcard": wildcard,
|
||||
"escalating": escalating, "rbac": rbac_over_rbac,
|
||||
"doc": d}
|
||||
# (1) no wildcards, no privilege-escalation verbs.
|
||||
check(not wildcard,
|
||||
f"{rel}: {kind}/{name} grants a wildcard apiGroup/resource/verb — that is "
|
||||
"not least-privilege (R-SEC-02)")
|
||||
check(not escalating,
|
||||
f"{rel}: {kind}/{name} grants privilege-escalation verb(s) "
|
||||
f"{sorted(escalating)} (R-SEC-02)")
|
||||
elif kind in ("RoleBinding", "ClusterRoleBinding"):
|
||||
bindings.append({"file": rel, "name": name, "doc": d})
|
||||
|
||||
check(len(roles) > 0, "no Role/ClusterRole found in the repo — the RBAC scan is not looking at it")
|
||||
check(len(bindings) > 0, "no RoleBinding/ClusterRoleBinding found — the RBAC scan is not looking at it")
|
||||
|
||||
for b in bindings:
|
||||
d, rel, name = b["doc"], b["file"], b["name"]
|
||||
ref = d.get("roleRef") or {}
|
||||
ref_kind, ref_name = ref.get("kind"), ref.get("name")
|
||||
sa_subjects = [s for s in (d.get("subjects") or [])
|
||||
if isinstance(s, dict) and s.get("kind") == "ServiceAccount"]
|
||||
local = roles.get((ref_kind, ref_name))
|
||||
ref_is_privileged = (ref_name in PRIVILEGED_ROLES) or bool(local and local["wildcard"])
|
||||
# (2) a WORKLOAD identity must never hold a privileged role.
|
||||
check(not (sa_subjects and ref_is_privileged),
|
||||
f"{rel}: {d.get('kind')}/{name} binds ServiceAccount(s) "
|
||||
f"{[s.get('name') for s in sa_subjects]} to the privileged role {ref_name!r} — "
|
||||
"workload identities must be least-privilege (R-SEC-02)")
|
||||
# (3) SA subjects must name their namespace (a ClusterRoleBinding without one is
|
||||
# ambiguous and, in a RoleBinding, silently defaults to the binding's namespace).
|
||||
for s in sa_subjects:
|
||||
check(bool(s.get("namespace")) or d.get("kind") == "RoleBinding",
|
||||
f"{rel}: {d.get('kind')}/{name} has a ServiceAccount subject "
|
||||
f"{s.get('name')!r} with no namespace (R-SEC-02)")
|
||||
|
||||
# --------------------------------- (4) our own privileged test identity, in detail
|
||||
RBAC_FILE = "conformance/incluster/rbac.yaml"
|
||||
check(RBAC_FILE in FILES, f"{RBAC_FILE} is missing — the conformance runner has no authored RBAC")
|
||||
runner_role = roles.get(("ClusterRole", "conformance-runner"))
|
||||
check(runner_role is not None,
|
||||
f"{RBAC_FILE}: no ClusterRole/conformance-runner — the test identity's grants are undefined")
|
||||
if runner_role:
|
||||
check(runner_role["file"] == RBAC_FILE,
|
||||
f"ClusterRole/conformance-runner is defined in {runner_role['file']}, not {RBAC_FILE}")
|
||||
check(not runner_role["wildcard"], "ClusterRole/conformance-runner uses a wildcard (R-SEC-02)")
|
||||
check(not runner_role["escalating"],
|
||||
"ClusterRole/conformance-runner grants a privilege-escalation verb (R-SEC-02)")
|
||||
check(not runner_role["rbac"],
|
||||
"ClusterRole/conformance-runner grants rights over RBAC objects themselves — a test "
|
||||
"identity that can rewrite RBAC is not least-privilege (R-SEC-02)")
|
||||
|
||||
runner_bindings = [b for b in bindings
|
||||
if any(s.get("kind") == "ServiceAccount" and s.get("name") == "conformance-runner"
|
||||
for s in (b["doc"].get("subjects") or []))]
|
||||
check(len(runner_bindings) >= 1, "the conformance-runner ServiceAccount has no binding at all")
|
||||
for b in runner_bindings:
|
||||
ref = b["doc"].get("roleRef") or {}
|
||||
check(ref.get("name") == "conformance-runner",
|
||||
f"{b['file']}: conformance-runner is bound to {ref.get('name')!r} instead of its own "
|
||||
"scoped ClusterRole (R-SEC-02)")
|
||||
for s in b["doc"].get("subjects") or []:
|
||||
if s.get("kind") == "ServiceAccount":
|
||||
check(s.get("namespace") == "conformance",
|
||||
f"{b['file']}: conformance-runner subject is in namespace "
|
||||
f"{s.get('namespace')!r}, expected 'conformance' (R-SEC-02)")
|
||||
|
||||
# --------------------- (5)/(6) least NETWORK REACH: the policies exist, Enforce, and
|
||||
# are proven to reject something at L0.
|
||||
# Validating policies: must Enforce AND be proven to REJECT something at L0.
|
||||
REQUIRED_POLICIES = {
|
||||
"validate-default-deny-shape": "a baseline that leaves a direction open is rejected",
|
||||
"restrict-host-namespaces": "host namespaces / privileged containers are rejected",
|
||||
"require-egress-class-label": "workloads must declare their egress class",
|
||||
}
|
||||
# Generating policies have no validationFailureAction and no negative row: the proof
|
||||
# that they close network reach is that the L0 suite asserts the resource they GENERATE.
|
||||
REQUIRED_GENERATE_POLICIES = {
|
||||
"require-default-deny-baseline": "every namespace gets a default-deny baseline",
|
||||
}
|
||||
policies: dict[str, dict] = {}
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") in ("ClusterPolicy", "Policy"):
|
||||
policies[(d.get("metadata") or {}).get("name")] = {"file": rel, "doc": d}
|
||||
|
||||
for pname, why in REQUIRED_POLICIES.items():
|
||||
p = policies.get(pname)
|
||||
check(p is not None, f"policy {pname!r} is missing — nothing enforces: {why} (R-SEC-02)")
|
||||
if not p:
|
||||
continue
|
||||
action = str((p["doc"].get("spec") or {}).get("validationFailureAction", ""))
|
||||
check(action == "Enforce" or action.startswith("${"),
|
||||
f"{p['file']}: policy {pname!r} validationFailureAction is {action!r} — a least-reach "
|
||||
"policy in Audit mode blocks nothing (R-SEC-02)")
|
||||
|
||||
negatives: dict[str, int] = {}
|
||||
suites = [f for f in FILES if os.path.basename(f) == "kyverno-test.yaml"]
|
||||
check(len(suites) > 0, "no kyverno-test.yaml — the L0 policy gate is not exercised at all")
|
||||
for suite in suites:
|
||||
for d in docs_of(suite):
|
||||
for r in d.get("results") or []:
|
||||
if isinstance(r, dict) and r.get("result") == "fail":
|
||||
negatives[r.get("policy")] = negatives.get(r.get("policy"), 0) + 1
|
||||
|
||||
for pname, why in REQUIRED_POLICIES.items():
|
||||
check(negatives.get(pname, 0) > 0,
|
||||
f"no L0 test row asserts that policy {pname!r} REJECTS a violating resource "
|
||||
f"(result: fail) — unproven gate for: {why} (R-SEC-02)")
|
||||
|
||||
# The generating half: the policy must exist, and the L0 suite must assert the resource
|
||||
# it generates (a `generatedResource` row that passes) — otherwise nothing proves a new
|
||||
# namespace actually arrives with its reach closed.
|
||||
for pname, why in REQUIRED_GENERATE_POLICIES.items():
|
||||
p = policies.get(pname)
|
||||
check(p is not None, f"policy {pname!r} is missing — nothing enforces: {why} (R-SEC-02)")
|
||||
generated_rows = []
|
||||
for suite in suites:
|
||||
for d in docs_of(suite):
|
||||
for r in d.get("results") or []:
|
||||
if isinstance(r, dict) and r.get("policy") == pname \
|
||||
and r.get("generatedResource") and r.get("result") == "pass":
|
||||
generated_rows.append((suite, r.get("generatedResource")))
|
||||
check(len(generated_rows) > 0,
|
||||
f"no L0 test row asserts the resource generated by {pname!r} — unproven gate for: "
|
||||
f"{why} (R-SEC-02)")
|
||||
for suite, gen in generated_rows:
|
||||
gp = os.path.normpath(os.path.join(os.path.dirname(suite), str(gen)))
|
||||
check(gp in FILES, f"{suite}: generatedResource {gen!r} does not resolve to a tracked file")
|
||||
golden = "\n".join(l for l in (ROOT / gp).read_text(errors="replace").splitlines()
|
||||
if not l.lstrip().startswith("#")) if gp in FILES else ""
|
||||
check("NetworkPolicy" in golden and re.search(r"policyTypes:", golden),
|
||||
f"{gp}: the asserted default-deny baseline is not a shaped NetworkPolicy (R-SEC-02)")
|
||||
|
||||
# The host-namespace policy must prove BOTH of its rules reject, not just one.
|
||||
host_rules = set()
|
||||
for suite in suites:
|
||||
for d in docs_of(suite):
|
||||
for r in d.get("results") or []:
|
||||
if isinstance(r, dict) and r.get("policy") == "restrict-host-namespaces" \
|
||||
and r.get("result") == "fail":
|
||||
host_rules.add(r.get("rule"))
|
||||
check({"disallow-host-namespaces", "disallow-privileged-containers"} <= host_rules,
|
||||
"the L0 suite does not prove BOTH restrict-host-namespaces rules reject a violating pod "
|
||||
f"(negative rows seen for: {sorted(host_rules)}) — R-SEC-02")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (least-privilege-rbac): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: least-privilege-rbac — {CHECKS} assertions over {len(roles)} role(s), "
|
||||
f"{len(bindings)} binding(s), {len(REQUIRED_POLICIES)} least-reach policies")
|
||||
PY
|
||||
@@ -1,43 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — no plaintext secrets in git (R-SEC-01)
|
||||
# Catalog: docs/14 L0 row, "Manifests/ServiceDefinitions obey invariants ...
|
||||
# no plaintext secrets" — runs in CI WITHOUT a cluster.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked file is inspected,
|
||||
# THEN every committed Secret is SOPS ciphertext, every ansible group_vars vault is
|
||||
# ansible-vault encrypted, no private key material is tracked, decryption is
|
||||
# delegated to an in-cluster SOPS key, and the L0 policy that rejects plaintext
|
||||
# Secrets exists, Enforces, and is proven by a negative test row to reject one.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files +
|
||||
# PyYAML, no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it
|
||||
# directly, and this Chainsaw Test wraps the SAME script so an in-cluster run executes
|
||||
# the identical assertions. Encryption at rest is necessary but NOT sufficient: the
|
||||
# runtime half of "decrypted only in-cluster", and a scan of git HISTORY, are named as
|
||||
# explicit gaps in check.sh's header and are NOT claimed here.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: no-plaintext-secrets
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
No secret is stored in plaintext in git: committed Secrets are SOPS ciphertext,
|
||||
ansible vaults are encrypted, no private key material is tracked, and the
|
||||
no-plaintext-secrets policy gate is enforced and proven to reject.
|
||||
spec:
|
||||
steps:
|
||||
- name: scan-tracked-tree-for-plaintext-secrets
|
||||
description: >-
|
||||
Assert over the whole tracked tree that no secret material is stored in
|
||||
plaintext and that decryption is delegated to the in-cluster SOPS key.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,287 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — no-plaintext-secrets
|
||||
# Serves: R-SEC-01 ("Secrets MUST NOT be stored in plaintext in git; they MUST be
|
||||
# encrypted at rest in the repo and decrypted only in-cluster."
|
||||
# — spec docs/00-vision-and-scope.md; detail in docs/07.)
|
||||
#
|
||||
# docs/14 puts exactly this invariant at L0 ("Static/policy | Pre-apply, in CI |
|
||||
# ... no plaintext secrets"), so it is asserted here over the WHOLE tracked tree,
|
||||
# with no cluster. Run by `conformance/run.sh --layer l0` and by ci/l0.sh.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked file is inspected,
|
||||
# THEN (1) SOPS is configured at the repo root with a PUBLIC age recipient;
|
||||
# (2) every committed Kubernetes Secret is SOPS-encrypted — a top-level
|
||||
# `sops:` block with a mac AND every data/stringData value is ENC[...];
|
||||
# (3) the only two exempt classes are narrow and self-policing:
|
||||
# a. `*.example` templates, which must NOT be referenced by any
|
||||
# kustomization and whose secret-named values must be placeholders;
|
||||
# b. the Kyverno L0 negative fixture, which is exempt ONLY while it is
|
||||
# actually wired into a kyverno-test.yaml that asserts
|
||||
# `disallow-plaintext-secrets` REJECTS it (result: fail);
|
||||
# (4) every ansible-vault group_vars file is `$ANSIBLE_VAULT;`-encrypted, and
|
||||
# no secret-valued ansible key carries a literal outside a vault;
|
||||
# (5) no private key material of any kind is tracked (PEM private keys, an
|
||||
# age SECRET key, kubeconfig client-key-data);
|
||||
# (6) decryption is delegated to the cluster: a Flux Kustomization declares
|
||||
# `decryption.provider: sops` with a secretRef, and the private half of
|
||||
# that key pair is nowhere in the repo (5).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gap, cluster layer):
|
||||
# - that the decrypted Secret only ever materialises in-cluster (the runtime half
|
||||
# of "decrypted only in-cluster") — needs a live cluster (L2), untested today.
|
||||
# - that no plaintext secret exists in git HISTORY. This walks the WORKING TREE at
|
||||
# HEAD only; docs/07's history-scan acceptance gate needs a secret scanner over
|
||||
# all revisions and is NOT implemented here.
|
||||
# - value-level entropy: a placeholder-looking string that is in fact a real
|
||||
# credential is not detectable statically.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import os, re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def tracked() -> list[str]:
|
||||
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout
|
||||
return [p for p in out.split("\0") if p]
|
||||
|
||||
|
||||
FILES = tracked()
|
||||
check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout")
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
"""Parsed YAML documents of a tracked file, or [] when it is not parseable YAML."""
|
||||
p = ROOT / rel
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(p.read_text(errors="replace")) if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
YAMLISH = re.compile(r"\.(ya?ml)(\.example)?$")
|
||||
YAML_FILES = [f for f in FILES if YAMLISH.search(f)]
|
||||
|
||||
# ---------------------------------------------------------------- (1) SOPS config
|
||||
sops_cfg = ROOT / ".sops.yaml"
|
||||
check(sops_cfg.is_file(), ".sops.yaml is missing from the repository root (R-SEC-01)")
|
||||
if sops_cfg.is_file():
|
||||
cfg = yaml.safe_load(sops_cfg.read_text()) or {}
|
||||
rules = cfg.get("creation_rules") or []
|
||||
check(len(rules) > 0, ".sops.yaml declares no creation_rules — new secrets would be committed unencrypted")
|
||||
for r in rules:
|
||||
pr = r.get("path_regex", "<none>")
|
||||
recips = str(r.get("age", ""))
|
||||
check(bool(recips), f".sops.yaml creation_rule {pr!r} has no age recipient")
|
||||
check(all(x.startswith("age1") for x in re.split(r"[,\s]+", recips) if x),
|
||||
f".sops.yaml creation_rule {pr!r} recipient is not a PUBLIC age1... key")
|
||||
check("AGE-SECRET-KEY" not in recips.upper(),
|
||||
f".sops.yaml creation_rule {pr!r} contains an age PRIVATE key")
|
||||
|
||||
# --------------------------------------------- Kyverno negative-fixture exemption
|
||||
# A plaintext Secret is tolerated ONLY where it is the negative case that proves the
|
||||
# L0 policy rejects plaintext. We do not trust the path: the file must be listed in a
|
||||
# kyverno-test.yaml whose results assert disallow-plaintext-secrets returns `fail`.
|
||||
POLICY_NAME = "disallow-plaintext-secrets"
|
||||
negative_fixtures: set[str] = set()
|
||||
kyverno_suites = [f for f in FILES if os.path.basename(f) == "kyverno-test.yaml"]
|
||||
check(len(kyverno_suites) > 0, "no kyverno-test.yaml found — the L0 policy gate is not exercised at all")
|
||||
for suite in kyverno_suites:
|
||||
docs = docs_of(suite)
|
||||
if not docs:
|
||||
continue
|
||||
d = docs[0]
|
||||
results = d.get("results") or []
|
||||
rejects_plaintext = any(r.get("policy") == POLICY_NAME and r.get("result") == "fail"
|
||||
for r in results if isinstance(r, dict))
|
||||
if not rejects_plaintext:
|
||||
continue
|
||||
base = os.path.dirname(suite)
|
||||
for res in d.get("resources") or []:
|
||||
negative_fixtures.add(os.path.normpath(os.path.join(base, str(res))))
|
||||
|
||||
check(len(negative_fixtures) > 0,
|
||||
f"no kyverno-test.yaml asserts that {POLICY_NAME} REJECTS a plaintext Secret "
|
||||
"(result: fail) — the plaintext-secret gate is unproven, so nothing may be exempt")
|
||||
|
||||
# --------------------------------------------------- (2)/(3) committed Secret scan
|
||||
SECRET_KEY_RE = re.compile(r"(pass|secret|token|key|credential|cookie|cert)", re.I)
|
||||
PLACEHOLDER_RE = re.compile(r"^\s*(|<[^>]*>|replace[-_ ]?me.*|change[-_ ]?me.*|placeholder.*|x{3,})\s*$", re.I)
|
||||
|
||||
secrets_seen = 0
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") != "Secret":
|
||||
continue
|
||||
secrets_seen += 1
|
||||
values = []
|
||||
for field in ("data", "stringData"):
|
||||
for k, v in (d.get(field) or {}).items():
|
||||
values.append((k, "" if v is None else str(v)))
|
||||
|
||||
is_example = rel.endswith(".example")
|
||||
is_negative_fixture = os.path.normpath(rel) in negative_fixtures
|
||||
|
||||
if is_example:
|
||||
# Template: never applied, and its secret-named values must be placeholders.
|
||||
refs = [k for k in YAML_FILES
|
||||
if os.path.basename(k) == "kustomization.yaml"
|
||||
and os.path.basename(rel) in (ROOT / k).read_text(errors="replace")]
|
||||
check(not refs,
|
||||
f"{rel}: an unencrypted .example template is referenced by a kustomization "
|
||||
f"({refs}) — it would be applied as a plaintext Secret")
|
||||
for k, v in values:
|
||||
if SECRET_KEY_RE.search(k):
|
||||
check(bool(PLACEHOLDER_RE.match(v)),
|
||||
f"{rel}: template key {k!r} holds {v[:24]!r}, not a placeholder — "
|
||||
"a real credential may have been committed (R-SEC-01)")
|
||||
continue
|
||||
|
||||
if is_negative_fixture:
|
||||
# Exempt, but only as the proven negative case; assert it really is plaintext
|
||||
# (an encrypted 'bad' fixture would silently stop testing anything).
|
||||
continue
|
||||
|
||||
# Everything else: must be SOPS-encrypted at rest.
|
||||
sops_block = d.get("sops")
|
||||
check(isinstance(sops_block, dict) and bool(sops_block.get("mac")),
|
||||
f"{rel}: committed Secret {d.get('metadata', {}).get('name')!r} has no top-level "
|
||||
"`sops:` block with a mac — it is stored in plaintext (R-SEC-01)")
|
||||
for k, v in values:
|
||||
check(v.startswith("ENC["),
|
||||
f"{rel}: Secret value {k!r} is not SOPS ciphertext (expected ENC[...]) — "
|
||||
"plaintext secret material in git (R-SEC-01)")
|
||||
|
||||
check(secrets_seen > 0, "no Kubernetes Secret manifests found at all — the scan is not looking at the repo")
|
||||
|
||||
# ------------------------------------------------------------ (4) ansible vaults
|
||||
VAULT_HEADER = "$ANSIBLE_VAULT;"
|
||||
vault_files = [f for f in FILES
|
||||
if os.path.basename(f) in ("vault.yml", "vault.yaml")]
|
||||
check(len(vault_files) > 0, "no ansible vault.yml files found — the vault scan is not looking at the repo")
|
||||
for rel in vault_files:
|
||||
head = (ROOT / rel).read_text(errors="replace")[:len(VAULT_HEADER)]
|
||||
check(head == VAULT_HEADER,
|
||||
f"{rel}: an ansible group_vars vault file is NOT ansible-vault encrypted "
|
||||
"(must start with the $ANSIBLE_VAULT header) — R-SEC-01")
|
||||
|
||||
SECRET_ANSIBLE_KEYS = ("ansible_become_password", "ansible_become_pass", "ansible_password",
|
||||
"ansible_ssh_pass", "ansible_sudo_pass", "ansible_ssh_private_key_file")
|
||||
ASSIGN_RE = re.compile(
|
||||
r"^\s*(" + "|".join(SECRET_ANSIBLE_KEYS) + r")\s*:\s*(.*?)\s*$")
|
||||
for rel in FILES:
|
||||
p = ROOT / rel
|
||||
if not p.is_file() or p.is_symlink():
|
||||
continue
|
||||
try:
|
||||
text = p.read_text(errors="replace")
|
||||
except Exception:
|
||||
continue
|
||||
if text.startswith(VAULT_HEADER):
|
||||
continue # encrypted at rest: opaque, by design
|
||||
for lineno, line in enumerate(text.splitlines(), 1):
|
||||
if line.lstrip().startswith("#"):
|
||||
continue
|
||||
m = ASSIGN_RE.match(line)
|
||||
if not m:
|
||||
continue
|
||||
key, val = m.group(1), m.group(2).strip().strip("'\"")
|
||||
ok = (val.startswith("{{") or val.startswith("$")
|
||||
or bool(PLACEHOLDER_RE.match(val)) or val == "")
|
||||
check(ok, f"{rel}:{lineno}: {key} is assigned the literal {val[:24]!r} outside an "
|
||||
"ansible-vault file — a plaintext secret in git (R-SEC-01)")
|
||||
|
||||
# ------------------------------------------------------ (5) no private key material
|
||||
# NOTE: each marker is assembled from fragments so that THIS file never contains the
|
||||
# literal string it hunts for — otherwise the scanner would flag itself the moment it
|
||||
# is tracked, and the only way to silence it would be to exempt the scanner (which is
|
||||
# exactly the kind of self-exemption this suite exists to prevent).
|
||||
_DASH5 = "-" * 5
|
||||
PRIVKEY_PATTERNS = [
|
||||
(re.compile(_DASH5 + r"BEGIN (?:[A-Z ]+ )?PRIVATE" + r" KEY" + _DASH5),
|
||||
"a PEM private key block"),
|
||||
(re.compile("AGE" + "-SECRET" + "-KEY-1"), "an age SECRET (decryption) key"),
|
||||
(re.compile(r"^\s*client" + r"-key-data\s*:\s*\S"), "kubeconfig client-key-data"),
|
||||
]
|
||||
for rel in FILES:
|
||||
p = ROOT / rel
|
||||
if not p.is_file() or p.is_symlink():
|
||||
continue
|
||||
try:
|
||||
text = p.read_text(errors="replace")
|
||||
except Exception:
|
||||
continue
|
||||
if text.startswith(VAULT_HEADER):
|
||||
continue
|
||||
for rx, what in PRIVKEY_PATTERNS:
|
||||
for lineno, line in enumerate(text.splitlines(), 1):
|
||||
if line.lstrip().startswith("#"):
|
||||
continue
|
||||
if rx.search(line):
|
||||
check(False, f"{rel}:{lineno}: tracked file contains {what} (R-SEC-01)")
|
||||
|
||||
# ----------------------------------------- (6) decryption is delegated to the cluster
|
||||
sops_decryption = []
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") != "Kustomization":
|
||||
continue
|
||||
dec = d.get("spec", {}).get("decryption") if isinstance(d.get("spec"), dict) else None
|
||||
if isinstance(dec, dict) and dec.get("provider") == "sops" and dec.get("secretRef"):
|
||||
sops_decryption.append(rel)
|
||||
# Flux overlay patches carry the same block inside a patch body.
|
||||
for rel in YAML_FILES:
|
||||
text = (ROOT / rel).read_text(errors="replace")
|
||||
if re.search(r"^\s*decryption:\s*$", text, re.M) and re.search(r"^\s*provider:\s*sops\s*$", text, re.M) \
|
||||
and re.search(r"^\s*secretRef:\s*$", text, re.M):
|
||||
sops_decryption.append(rel)
|
||||
check(len(sops_decryption) > 0,
|
||||
"no Flux Kustomization declares `decryption: {provider: sops, secretRef: ...}` — nothing "
|
||||
"proves decryption happens in-cluster rather than from committed key material (R-SEC-01)")
|
||||
|
||||
# ------------------------------------------- (7) the L0 policy gate is real and strict
|
||||
policy_files = [f for f in FILES if f.endswith(f"{POLICY_NAME}.yaml")]
|
||||
check(len(policy_files) == 1, f"expected exactly one {POLICY_NAME} ClusterPolicy, found {policy_files}")
|
||||
for rel in policy_files:
|
||||
d = (docs_of(rel) or [{}])[0]
|
||||
action = str(d.get("spec", {}).get("validationFailureAction", ""))
|
||||
check(action == "Enforce" or action.startswith("${"),
|
||||
f"{rel}: validationFailureAction is {action!r} — the no-plaintext-secrets gate must "
|
||||
"Enforce (or carry the Flux token that substitutes to Enforce)")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (no-plaintext-secrets): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: no-plaintext-secrets — {CHECKS} assertions over {len(FILES)} tracked files, "
|
||||
f"{secrets_seen} committed Secret document(s), {len(vault_files)} ansible vault file(s)")
|
||||
PY
|
||||
@@ -1,68 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — a placeholder value in cluster-vars is REFUSED by the guard that owns it
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE BUG CLASS THIS CLOSES (it has now bitten three times in one file). A guard in
|
||||
# `bootstrap/standup.sh` keys on the SHAPE of a value that lives in
|
||||
# `clusters/base/cluster-vars.yaml`, and nothing tied the two together. On 2026-09-05 the
|
||||
# S3 placeholder changed from `TBD-PROD-S3-BACKUP-ENDPOINT` to
|
||||
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`; the three guards matched the glob
|
||||
# `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is False. All three
|
||||
# guards silently stopped matching, and `standup.sh resume` would have resumed `velero` on
|
||||
# prod against a nonexistent hostname: installs clean, reports healthy, NO BACKUPS EXIST.
|
||||
# It was found by accident. The guard is fixed (`s3_is_placeholder`, substring); the
|
||||
# COUPLING was untested, which is what makes the class reintroducible.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# WHEN the REAL predicate is extracted from bootstrap/standup.sh and executed by bash
|
||||
# (never reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"),
|
||||
# THEN (1) every authored value of a guarded knob that is placeholder-shaped by an
|
||||
# INDEPENDENT detector is actually refused by that guard, in base and in every
|
||||
# environment overlay;
|
||||
# (2) no environment overlay carries a placeholder-shaped value for a knob no guard
|
||||
# refuses;
|
||||
# (3) every function in standup.sh that reads a guarded knob applies the guard to it
|
||||
# (the "three guards died at once" regression);
|
||||
# (4) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix
|
||||
# — the exact anchoring defect above — and still accepts a real endpoint;
|
||||
# AND (5) the extraction is faithful, and the check fails LOUDLY rather than silently
|
||||
# testing a copy if the guard stops being a self-contained predicate.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
|
||||
# PyYAML + bash); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test
|
||||
# wraps the SAME script.
|
||||
#
|
||||
# NOT proven: that the endpoint is REACHABLE or that backups exist (runtime facts —
|
||||
# `standup.sh s3check` probes the former from a pod, l3-fault-injection/dr-restore proves
|
||||
# the latter); that guards for knobs OTHER than the placeholder-refusing ones are correct;
|
||||
# and that the guard runs at all in a real standup (the dispatch path is shell, executed
|
||||
# only against a live cluster). R-STATE-03 is tagged for the narrow static slice only: the
|
||||
# authored backup target cannot be a placeholder that the guard fails to recognise.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: placeholder-guard-coupling
|
||||
labels:
|
||||
conformance.home.k3s/R-ENV-02: "true"
|
||||
conformance.home.k3s/R-STATE-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every placeholder-shaped value authored in cluster-vars, for any environment
|
||||
overlay, is actually recognised as a placeholder by the real bootstrap guard that
|
||||
is supposed to refuse it — so a value change in one file cannot silently disarm a
|
||||
guard in another and enable velero against an endpoint that does not exist.
|
||||
spec:
|
||||
steps:
|
||||
- name: couple-authored-placeholders-to-the-real-guard
|
||||
description: >-
|
||||
Extract the real placeholder predicate from bootstrap/standup.sh, run it against
|
||||
every authored cluster-vars value the repo actually carries, and assert the guard
|
||||
refuses each one an independent detector calls a placeholder.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,516 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — placeholder-guard-coupling
|
||||
# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
|
||||
# — spec docs/02-environments-and-provisioning.md)
|
||||
# R-STATE-03 ("Scheduled off-cluster backups + restore (DR)."
|
||||
# — spec docs/04-storage-and-state.md)
|
||||
#
|
||||
# WHY THIS EXISTS — A GUARD KEYED ON A VALUE THAT LIVES IN ANOTHER FILE
|
||||
# --------------------------------------------------------------------
|
||||
# `bootstrap/standup.sh` refuses to resume tiers whose configuration is still a
|
||||
# placeholder. Three of those guards keyed on the SHAPE of `S3_BACKUP_ENDPOINT`, which is
|
||||
# authored in `clusters/base/cluster-vars.yaml`. On 2026-09-05 the placeholder changed
|
||||
# from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`
|
||||
# (the MinIO -> Cloudflare R2 move). The guards matched the glob `TBD-*` — a PREFIX match —
|
||||
# and `fnmatch('https://TBD-R2-…','TBD-*')` is FALSE. All three guards silently stopped
|
||||
# matching. `standup.sh resume` would then have resumed `velero` on prod against a hostname
|
||||
# that does not exist: the HelmRelease installs clean, the Kustomization reports Ready, and
|
||||
# NO BACKUPS EXIST. The only reason it was not shipped is that someone noticed by accident.
|
||||
#
|
||||
# The predicate is fixed (`s3_is_placeholder`, substring match). The COUPLING is what was
|
||||
# never tested: a guard in one file keying on a value in another, with nothing tying them
|
||||
# together. This check is that tie, and it is derived from BOTH real files — no current
|
||||
# string is hardcoded here, so it does not rot when the placeholder or the guard changes.
|
||||
#
|
||||
# HOW THE REAL GUARD IS EXERCISED (and why not a copy)
|
||||
# ---------------------------------------------------
|
||||
# `bootstrap/standup.sh` is NOT sourceable: at top level it parses argv, hard-exits when
|
||||
# --kubeconfig/--expect-server/<step> are absent, `cd`s to the repo root, exports
|
||||
# KUBECONFIG, installs an EXIT trap and finally dispatches steps against a live cluster.
|
||||
# So this check EXTRACTS the predicate's own source lines from the file and sources ONLY
|
||||
# those, then calls the real function. That is the real bytes, not a reimplementation:
|
||||
# * every function whose name ends `_is_placeholder` is discovered, not named here;
|
||||
# * a duplicate definition of one is a failure (which of the two would run?);
|
||||
# * the extracted body must be a SELF-CONTAINED predicate — `case`/`return` and nothing
|
||||
# else, no command substitution, no call to another function. If the guard ever grows
|
||||
# a dependency, extraction stops being faithful and THIS CHECK FAILS LOUDLY, naming the
|
||||
# remedy (move the predicate into a small sourceable library that standup.sh sources)
|
||||
# rather than quietly falling back to testing a copy of the logic;
|
||||
# * a CANARY pins the extraction itself: the extracted function must refuse the empty
|
||||
# string and accept a realistic endpoint. An extraction that grabbed the wrong lines,
|
||||
# or a predicate degraded to always-true / always-false, fails there — this repo has
|
||||
# already been bitten by a check that passed when perturbed because it asked a question
|
||||
# that was always satisfiable.
|
||||
# There is deliberately NO second implementation of the matching logic anywhere in this file.
|
||||
#
|
||||
# THE INDEPENDENT DETECTOR (the other half of the comparison)
|
||||
# ----------------------------------------------------------
|
||||
# Asking "does the guard agree with the guard?" proves nothing, so the set of
|
||||
# placeholder-shaped values is decided by a detector written HERE, whose vocabulary is the
|
||||
# UNION of two independent sources:
|
||||
# * SEED — universal loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER, REPLACE-ME,
|
||||
# CHANGE-ME, FILL-ME, SET-ME, XXXX), plus `<angle-bracket>` templates and the empty
|
||||
# string. These are NOT derived from the guard, so DELETING a guard arm cannot make a
|
||||
# value stop being placeholder-shaped — it makes assertion (1) fail, which is the point.
|
||||
# A value using an idiom the guard does not know is a finding either way: fix it by
|
||||
# giving the knob a real value, or by teaching the guard the idiom.
|
||||
# * the guard's OWN vocabulary, read out of its case-arms with the glob anchors STRIPPED.
|
||||
# Stripping the anchors is what makes the comparison non-circular in the dimension that
|
||||
# actually broke: the detector matches a word ANYWHERE, case-insensitively, and the
|
||||
# guard must agree. A prefix-anchored guard cannot pass.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# WHEN the real predicate(s) are extracted from bootstrap/standup.sh and executed,
|
||||
# THEN (1) VALUES: for every knob a guard is responsible for, every authored value — the
|
||||
# base default AND each environment overlay's value — that the independent
|
||||
# detector calls placeholder-shaped is REFUSED by the real guard, and every value
|
||||
# it calls usable is ACCEPTED (a guard that refuses a real endpoint leaves velero
|
||||
# suspended forever, which is the same class of silent failure);
|
||||
# (2) OVERLAYS: no environment overlay's EFFECTIVE cluster-vars carries a
|
||||
# placeholder-shaped value for a knob that no guard refuses. (Base alone is
|
||||
# exempt: it is not a bootstrap target — `flux bootstrap --path=clusters/<env>` —
|
||||
# and its loud sentinels exist precisely to be overridden. A base placeholder an
|
||||
# overlay forgets to override IS caught, because it becomes that overlay's
|
||||
# effective value.)
|
||||
# (3) CONSUMERS: every function in standup.sh that reads a guarded knob (via `cvar`,
|
||||
# following one hop of variable aliasing) applies a placeholder guard to it. This
|
||||
# is the "all three guards died at once" regression stated as a property: the
|
||||
# guard being correct is worthless if a consumer stops calling it.
|
||||
# (4) ANCHORING: for each word in the guard's own vocabulary, the guard refuses a
|
||||
# value carrying that word in the MIDDLE (not at the start) — the precise defect
|
||||
# of 2026-09-05 — while still accepting a realistic endpoint.
|
||||
# (5) HARNESS: exactly one definition per predicate, the extracted body is verbatim
|
||||
# from the file and self-contained, the canary holds, no divergent second copy of
|
||||
# the matching logic exists elsewhere in standup.sh, and a floor on the number of
|
||||
# assertions run keeps a pass from degenerating into a no-op.
|
||||
# AND (6) LOAD-BEARING: ci/l0.sh runs the L0 conformance layer on every gate run and
|
||||
# ci/gate.sh sets the merge block when that stage fails.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps)
|
||||
# - Nothing about RUNTIME: not that the endpoint resolves, not that credentials work, not
|
||||
# that a backup exists. `standup.sh s3check` probes reachability from a pod (operator-
|
||||
# run, cluster-gated) and conformance/tests/l3-fault-injection/dr-restore proves restore.
|
||||
# - It does not execute standup.sh's dispatch path, so it cannot prove the guard is
|
||||
# REACHED in a real run — only that, when reached, it refuses what it must. The
|
||||
# consumer property (3) is a static approximation of reachability: it is a text-level
|
||||
# taint trace over one file, not an interpreter, and a guard whose result is computed
|
||||
# and then ignored would satisfy it.
|
||||
# - It says nothing about guards keyed on shapes OTHER than placeholders (e.g. the
|
||||
# EXTERNALDNS_TXT_OWNER_ID == 'default' test), nor about placeholder-shaped values
|
||||
# outside cluster-vars (the Ansible inventory's BLOCKED/NOT-A-DEVICE sentinels are
|
||||
# asserted by the node_baseline role itself, at play time).
|
||||
# - Stage 3b of ci/l0.sh scopes the conformance run to the TOUCHED requirements. An edit
|
||||
# to bootstrap/* attributes NO requirement in ci/touched-requirements.sh's path map, so
|
||||
# the touched set is empty and the FULL L0 layer runs (verified: ci/lib.sh
|
||||
# load_touched_requirements) — this check runs. An edit to clusters/* maps to
|
||||
# R-ENV-02, which this check carries, so a cluster-vars value change also selects it.
|
||||
# A change that touches standup.sh ALONGSIDE some unrelated mapped path (say
|
||||
# platform/cilium/) would scope the run away from this check; closing that needs a
|
||||
# `bootstrap/*` entry in the path map, which is not this check's file to edit.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
STANDUP_REL="${STANDUP_REL:-bootstrap/standup.sh}"
|
||||
|
||||
cd "${ROOT}"
|
||||
|
||||
if [ ! -f "${ROOT}/${STANDUP_REL}" ]; then
|
||||
echo "check SKIPPED (placeholder-guard-coupling): ${STANDUP_REL} is absent — nothing to couple." >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "${WORK}"' EXIT
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# PHASE 1 (python): extract the real predicate(s), derive the corpus from the real files,
|
||||
# and run every assertion that does not need a shell. Writes:
|
||||
# ${WORK}/guards.sh the predicate source lines, verbatim, and nothing else
|
||||
# ${WORK}/corpus.tsv <guard>\t<expect: placeholder|usable>\t<origin>\t<value>
|
||||
# ${WORK}/findings.txt one line per failed assertion
|
||||
# ${WORK}/counts.txt <assertions-run>
|
||||
# ---------------------------------------------------------------------------------------
|
||||
"${PYTHON}" - "${ROOT}" "${WORK}" "${STANDUP_REL}" <<'PY'
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT, WORK, STANDUP_REL = Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[3]
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def bail(msg):
|
||||
"""A failure that makes the rest of the check meaningless: report and stop."""
|
||||
print(f"CHECK FAIL (placeholder-guard-coupling): {msg}", file=sys.stderr)
|
||||
(WORK / "findings.txt").write_text(msg + "\n")
|
||||
(WORK / "counts.txt").write_text("0\n")
|
||||
(WORK / "corpus.tsv").write_text("")
|
||||
(WORK / "guards.sh").write_text("")
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
TEXT = (ROOT / STANDUP_REL).read_text()
|
||||
LINES = TEXT.splitlines()
|
||||
|
||||
# ------------------------------------------------------------------ (5) extract the guard
|
||||
DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
|
||||
defs: dict[str, list[int]] = {}
|
||||
for i, line in enumerate(LINES):
|
||||
m = DEF_RE.match(line)
|
||||
if m:
|
||||
defs.setdefault(m.group(1), []).append(i)
|
||||
|
||||
if not defs:
|
||||
bail(f"{STANDUP_REL} defines no *_is_placeholder predicate. Either the guard was renamed "
|
||||
"(update this check's discovery pattern in the SAME change) or it was DELETED, which "
|
||||
"is the 2026-09-05 failure with the guard removed instead of disarmed.")
|
||||
|
||||
# An "argument-shaped" call: <name> "<something>"
|
||||
CALL_RE = re.compile(r"\b([a-z0-9_]+_is_placeholder)\s+\"([^\"]*)\"")
|
||||
|
||||
guard_src: list[str] = []
|
||||
guard_words: dict[str, list[str]] = {}
|
||||
ARM_RE = re.compile(r"^(?P<pats>[^()]*?)\)\s*return\s+[01]\s*;;$")
|
||||
ALLOWED = (
|
||||
re.compile(r"^$"),
|
||||
re.compile(r"^#"),
|
||||
re.compile(r"^case\s+\"\$\{?[0-9A-Za-z_]+\}?\"\s+in$"),
|
||||
re.compile(r"^esac$"),
|
||||
re.compile(r"^return\s+[01]$"),
|
||||
ARM_RE,
|
||||
)
|
||||
|
||||
for name, at in sorted(defs.items()):
|
||||
check(len(at) == 1,
|
||||
f"{STANDUP_REL} defines {name}() {len(at)} times (lines "
|
||||
f"{', '.join(str(a + 1) for a in at)}) — which one runs depends on file order, and "
|
||||
"this check can no longer say which one it extracted. Keep exactly one definition.")
|
||||
if len(at) != 1:
|
||||
continue
|
||||
start = at[0]
|
||||
end = next((j for j in range(start + 1, len(LINES)) if LINES[j] == "}"), None)
|
||||
if end is None:
|
||||
bail(f"{name}() at {STANDUP_REL}:{start + 1} has no closing brace at column 0 — the "
|
||||
"extraction boundary this check relies on is gone. Restore the conventional "
|
||||
"layout, or move the predicate into a sourceable library and point this check at it.")
|
||||
body = LINES[start:end + 1]
|
||||
block = "\n".join(body)
|
||||
check(block in TEXT,
|
||||
f"extracted {name}() is not a verbatim substring of {STANDUP_REL} — extraction bug; "
|
||||
"the harness would be testing something the script does not contain")
|
||||
|
||||
words: list[str] = []
|
||||
for raw in body[1:-1]:
|
||||
s = raw.strip()
|
||||
if not any(rx.match(s) for rx in ALLOWED) or "$(" in s or "`" in s:
|
||||
bail(f"{name}() at {STANDUP_REL}:{start + 1} is no longer a SELF-CONTAINED "
|
||||
f"case/return predicate (offending line: {s!r}). This check exercises the REAL "
|
||||
"function by sourcing its own lines; a body that calls out to anything else "
|
||||
"cannot be extracted faithfully, and silently testing a COPY of the logic is "
|
||||
"exactly the failure mode this check exists to prevent. REMEDY: move the "
|
||||
"predicate into a tiny library that standup.sh sources (e.g. "
|
||||
"bootstrap/lib/placeholder.sh) and point STANDUP_REL/this extractor at it — do "
|
||||
"NOT reimplement the matching here.")
|
||||
m = ARM_RE.match(s)
|
||||
if not m:
|
||||
continue
|
||||
for pat in m.group("pats").split("|"):
|
||||
pat = pat.strip()
|
||||
if pat in ('""', "''", ""):
|
||||
continue # the empty-value arm: covered by the canary
|
||||
words.append(pat.strip("'\"").strip("*"))
|
||||
check(len(words) >= 1,
|
||||
f"{name}() has no non-empty placeholder pattern left — every arm that recognises a "
|
||||
"loud placeholder string has been removed, so the guard now only refuses the empty "
|
||||
"value and any 'TBD'-style endpoint would sail straight through it")
|
||||
guard_words[name] = words
|
||||
guard_src.append(block)
|
||||
|
||||
(WORK / "guards.sh").write_text("\n".join(guard_src) + "\n")
|
||||
|
||||
# ------------------------------------------------- no divergent second copy of the logic
|
||||
# Any of the guard's own vocabulary appearing in EXECUTABLE text outside the predicate means
|
||||
# a second, independently-drifting implementation of the same test (which is how three
|
||||
# guards came to disagree with one value in the first place).
|
||||
guard_line_nums = {j for name, at in defs.items() if len(at) == 1
|
||||
for j in range(at[0], next(k for k in range(at[0] + 1, len(LINES))
|
||||
if LINES[k] == "}") + 1)}
|
||||
all_words = sorted({w.upper() for ws in guard_words.values() for w in ws if w})
|
||||
divergent: list[str] = []
|
||||
for j, line in enumerate(LINES):
|
||||
if j in guard_line_nums or line.strip().startswith("#"):
|
||||
continue
|
||||
code = line.split("#", 1)[0]
|
||||
hit = [w for w in all_words if w in code.upper()]
|
||||
if hit:
|
||||
divergent.append(f"{STANDUP_REL}:{j + 1} ({', '.join(hit)}): {line.strip()!r}")
|
||||
check(not divergent,
|
||||
"the guard's own placeholder vocabulary appears in EXECUTABLE text outside the "
|
||||
f"predicate — a SECOND implementation of the same test, which is precisely how a value "
|
||||
f"change disarmed three guards at once. Call the predicate instead. Sites: "
|
||||
+ " | ".join(divergent))
|
||||
|
||||
# ------------------------------------------------------------- cluster-vars, as authored
|
||||
CV = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
||||
if not CV.is_file():
|
||||
bail("clusters/base/cluster-vars.yaml is absent — the authored half of the coupling is "
|
||||
"missing and this check cannot mean anything")
|
||||
|
||||
base: dict[str, str] = {}
|
||||
for d in yaml.safe_load_all(CV.read_text()):
|
||||
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
|
||||
and (d.get("metadata") or {}).get("name") == "cluster-vars":
|
||||
base = {str(k): "" if v is None else str(v) for k, v in (d.get("data") or {}).items()}
|
||||
check(len(base) > 5, f"clusters/base/cluster-vars.yaml yielded {len(base)} knob(s) — the "
|
||||
"ConfigMap parse failed, so every value assertion below would be vacuous")
|
||||
|
||||
overlays: dict[str, dict[str, str]] = {}
|
||||
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
||||
env = overlay.parent.name
|
||||
if env == "base":
|
||||
continue
|
||||
eff = dict(base)
|
||||
try:
|
||||
k = yaml.safe_load(overlay.read_text()) or {}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
check(False, f"clusters/{env}/kustomization.yaml does not parse as YAML ({exc}) — this "
|
||||
"check cannot compute that overlay's effective values")
|
||||
continue
|
||||
for p in (k.get("patches") or []):
|
||||
tgt = p.get("target") or {}
|
||||
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
|
||||
continue
|
||||
try:
|
||||
ops = yaml.safe_load(p.get("patch") or "") or []
|
||||
except Exception: # noqa: BLE001
|
||||
continue
|
||||
for op in ops if isinstance(ops, list) else []:
|
||||
if not isinstance(op, dict):
|
||||
continue
|
||||
m = re.match(r"^/data/([A-Za-z0-9_]+)$", str(op.get("path", "")))
|
||||
if m and str(op.get("op", "")) in ("replace", "add"):
|
||||
eff[m.group(1)] = "" if op.get("value") is None else str(op["value"])
|
||||
overlays[env] = eff
|
||||
check(len(overlays) >= 1,
|
||||
"no clusters/<env>/kustomization.yaml overlay was parsed — assertion (2) would be vacuous")
|
||||
|
||||
# ------------------------------------------------ which knobs is a guard responsible for?
|
||||
# Function bodies of standup.sh, then a one-hop taint trace from `x="$(cvar KEY)"` to a
|
||||
# `*_is_placeholder "${x}"` call. Both halves are read out of the file; nothing is named here.
|
||||
FUNC_RE = re.compile(r"^([a-z0-9_]+)\(\)\s*\{")
|
||||
funcs: dict[str, list[str]] = {}
|
||||
i = 0
|
||||
while i < len(LINES):
|
||||
m = FUNC_RE.match(LINES[i])
|
||||
if m:
|
||||
end = next((j for j in range(i + 1, len(LINES)) if LINES[j] == "}"), len(LINES) - 1)
|
||||
funcs[m.group(1)] = LINES[i + 1:end]
|
||||
i = end + 1
|
||||
continue
|
||||
i += 1
|
||||
check(len(funcs) > 5, f"parsed only {len(funcs)} function(s) out of {STANDUP_REL} — the "
|
||||
"consumer assertion (3) would be vacuous")
|
||||
|
||||
CVAR_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\(\s*cvar\s+([A-Z0-9_]+)\s*\)\"")
|
||||
ALIAS_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\{([A-Za-z_][A-Za-z0-9_]*)[:}]")
|
||||
|
||||
guarded_keys: set[str] = set()
|
||||
consumers: list[tuple[str, str, bool]] = [] # (func, key, guarded-in-this-func)
|
||||
for fname, body in funcs.items():
|
||||
text = "\n".join(body)
|
||||
keys_of_var: dict[str, set[str]] = {}
|
||||
for m in CVAR_RE.finditer(text):
|
||||
keys_of_var.setdefault(m.group(1), set()).add(m.group(2))
|
||||
# one hop of aliasing: ep="${live}" carries live's taint
|
||||
for _ in range(3):
|
||||
for m in ALIAS_RE.finditer(text):
|
||||
src = keys_of_var.get(m.group(2))
|
||||
if src:
|
||||
keys_of_var.setdefault(m.group(1), set()).update(src)
|
||||
called_on = {m.group(2) for m in CALL_RE.finditer(text)}
|
||||
called_vars = {re.sub(r"^\$\{?|\}?$", "", a) for a in called_on}
|
||||
# Aggregate per (function, knob): the knob is guarded HERE if ANY variable carrying its
|
||||
# value is passed to a predicate — `live="$(cvar K)"; ep="${live}"; guard "${ep}"` counts.
|
||||
guarded_here: dict[str, bool] = {}
|
||||
for var, keys in keys_of_var.items():
|
||||
for key in keys:
|
||||
guarded_here[key] = guarded_here.get(key, False) or (var in called_vars)
|
||||
for key, g in guarded_here.items():
|
||||
consumers.append((fname, key, g))
|
||||
if g:
|
||||
guarded_keys.add(key)
|
||||
|
||||
if not guarded_keys:
|
||||
bail(f"no cluster-vars knob in {STANDUP_REL} is passed to a *_is_placeholder predicate. "
|
||||
"Either the guards were removed, or the way the script reads cluster-vars changed and "
|
||||
"this check's derivation must be updated in the SAME change — it must never silently "
|
||||
"verify an empty set.")
|
||||
|
||||
# (3) CONSUMERS: every reader of a guarded knob must apply the guard to it.
|
||||
for fname, key, guarded in sorted(consumers):
|
||||
if key not in guarded_keys:
|
||||
continue # nothing claims responsibility for this knob at all
|
||||
check(guarded,
|
||||
f"{STANDUP_REL}: {fname}() reads cluster-vars knob {key} but never passes it to a "
|
||||
"*_is_placeholder guard, while other functions do. On 2026-09-05 all three consumers "
|
||||
"of S3_BACKUP_ENDPOINT were disarmed at once; a consumer that reads the knob and skips "
|
||||
"the guard is that failure in one function.")
|
||||
|
||||
# ---------------------------------------------------------------- independent detector
|
||||
# Deliberately NOT derived from the guard alone (see the header): SEED is universal
|
||||
# loud-placeholder vocabulary, and the guard's own words are added so that a guard whose
|
||||
# ANCHORING is wrong still has its vocabulary tested against real values.
|
||||
SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", "REPLACE-ME", "REPLACEME",
|
||||
"CHANGE-ME", "CHANGEME", "FILL-ME", "FILLME", "SET-ME", "XXXX"]
|
||||
VOCAB = sorted({w.upper() for w in SEED} | set(all_words))
|
||||
ANGLE = re.compile(r"<[^>\s]+>")
|
||||
|
||||
|
||||
def placeholder_shaped(value: str) -> bool:
|
||||
if value.strip() == "":
|
||||
return True # an unset endpoint is not a target
|
||||
up = value.upper()
|
||||
return any(w in up for w in VOCAB) or bool(ANGLE.search(value))
|
||||
|
||||
|
||||
rows: list[tuple[str, str, str, str]] = [] # guard, expect, origin, value
|
||||
GUARD = sorted(guard_words)[0] if len(guard_words) == 1 else None
|
||||
|
||||
# (1) VALUES: base default and every overlay's value, for every guarded knob.
|
||||
for key in sorted(guarded_keys):
|
||||
seen: list[tuple[str, str]] = []
|
||||
if key in base:
|
||||
seen.append((f"clusters/base/cluster-vars.yaml:{key}", base[key]))
|
||||
for env, eff in sorted(overlays.items()):
|
||||
seen.append((f"clusters/{env} effective:{key}", eff.get(key, "")))
|
||||
check(bool(seen), f"guarded knob {key} is not defined in cluster-vars at all — the guard "
|
||||
"protects a knob nothing authors")
|
||||
for origin, value in seen:
|
||||
expect = "placeholder" if placeholder_shaped(value) else "usable"
|
||||
for name in sorted(guard_words):
|
||||
rows.append((name, expect, origin, value))
|
||||
|
||||
# (2) OVERLAYS: a placeholder-shaped effective value for an UNGUARDED knob.
|
||||
for env, eff in sorted(overlays.items()):
|
||||
for key, value in sorted(eff.items()):
|
||||
if key in guarded_keys or not placeholder_shaped(value):
|
||||
continue
|
||||
check(False,
|
||||
f"clusters/{env}: knob {key} resolves to the placeholder-shaped value {value!r}, "
|
||||
f"and NO guard in {STANDUP_REL} refuses it. Either give it a real value before it "
|
||||
"reaches an overlay, or guard it the way the S3 endpoint is guarded — an overlay "
|
||||
"that ships a placeholder no guard recognises is the 2026-09-05 failure with a "
|
||||
"different knob's name on it.")
|
||||
|
||||
# (4) ANCHORING: the guard's own vocabulary must match ANYWHERE, not just as a prefix,
|
||||
# and a realistic endpoint must still be accepted.
|
||||
for name, words in sorted(guard_words.items()):
|
||||
for w in sorted(set(words)):
|
||||
rows.append((name, "placeholder",
|
||||
f"anchoring:{name}:mid-string({w})",
|
||||
f"https://account-{w}-id.r2.cloudflarestorage.com"))
|
||||
rows.append((name, "placeholder", f"anchoring:{name}:tail({w})",
|
||||
f"https://r2-endpoint.example.invalid/{w}"))
|
||||
rows.append((name, "placeholder", f"canary:{name}:empty", ""))
|
||||
rows.append((name, "usable", f"canary:{name}:real-r2",
|
||||
"https://0123456789abcdef0123456789abcd.r2.cloudflarestorage.com"))
|
||||
rows.append((name, "usable", f"canary:{name}:real-minio", "http://192.168.1.9:9000"))
|
||||
|
||||
# Every guarded knob must reach the oracle once per authored source (base + each overlay),
|
||||
# so a knob silently dropping out of the corpus is a failure rather than a quiet pass.
|
||||
authored = [r for r in rows if not r[2].startswith(("anchoring:", "canary:"))]
|
||||
expected_rows = len(guarded_keys) * (1 + len(overlays)) * len(guard_words)
|
||||
check(len(authored) == expected_rows,
|
||||
f"{len(authored)} authored value(s) reached the guard, expected {expected_rows} "
|
||||
f"({len(guarded_keys)} guarded knob(s) x base+{len(overlays)} overlay(s) x "
|
||||
f"{len(guard_words)} predicate(s)) — a guarded knob is missing from cluster-vars or from "
|
||||
"an overlay's effective values, and the coupling is only partly exercised")
|
||||
|
||||
# ------------------------------------------------------- (6) the check is load-bearing
|
||||
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
check("--layer l0" in L0,
|
||||
"ci/l0.sh does not run the L0 conformance layer — this check would never execute in the gate")
|
||||
check(bool(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)) and bool(re.search(r"fail\).*BLOCK=1", GATE)),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
# --------------------------------------------------------------------------- handoff
|
||||
with (WORK / "corpus.tsv").open("w") as fh:
|
||||
for name, expect, origin, value in rows:
|
||||
fh.write(f"{name}\t{expect}\t{origin}\t{value}\n")
|
||||
(WORK / "findings.txt").write_text("".join(f"{f}\n" for f in FAILURES))
|
||||
(WORK / "counts.txt").write_text(f"{CHECKS}\n")
|
||||
|
||||
print(f"guards: {', '.join(sorted(guard_words))} "
|
||||
f"vocabulary: {', '.join(VOCAB)}")
|
||||
print(f"guarded knobs: {', '.join(sorted(guarded_keys))} "
|
||||
f"consumers traced: {len(consumers)} overlays: {', '.join(sorted(overlays))}")
|
||||
for name, expect, origin, value in rows:
|
||||
if not origin.startswith(("anchoring:", "canary:")):
|
||||
print(f" authored {origin} = {value!r} -> expect {expect}")
|
||||
PY
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# PHASE 2 (bash): source the REAL predicate lines and run the corpus through them.
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# shellcheck disable=SC1090,SC1091
|
||||
source "${WORK}/guards.sh"
|
||||
|
||||
oracle_failures=0
|
||||
oracle_rows=0
|
||||
while IFS=$'\t' read -r guard expect origin value; do
|
||||
[ -n "${guard}" ] || continue
|
||||
oracle_rows=$((oracle_rows + 1))
|
||||
if "${guard}" "${value}"; then verdict=placeholder; else verdict=usable; fi
|
||||
if [ "${verdict}" != "${expect}" ]; then
|
||||
oracle_failures=$((oracle_failures + 1))
|
||||
if [ "${expect}" = placeholder ]; then
|
||||
printf '%s\n' "${guard}() in ${STANDUP_REL} ACCEPTS '${value}' (from ${origin}) as a usable value, but it is placeholder-shaped. This is the 2026-09-05 failure exactly: the guard is asked about a value it does not recognise, returns 'fine', and the tier it protects is enabled against something that does not exist. Fix the predicate (match the idiom anywhere, not as a prefix) or stop authoring that idiom." >> "${WORK}/findings.txt"
|
||||
else
|
||||
printf '%s\n' "${guard}() in ${STANDUP_REL} REFUSES '${value}' (from ${origin}), which is a usable value. An over-matching guard leaves the tier it protects suspended forever and reports it as an operator decision — a silent failure in the other direction." >> "${WORK}/findings.txt"
|
||||
fi
|
||||
fi
|
||||
done < "${WORK}/corpus.tsv"
|
||||
|
||||
# ------------------------------------------------------------------------------ verdict
|
||||
py_checks="$(cat "${WORK}/counts.txt")"
|
||||
total=$((py_checks + oracle_rows))
|
||||
findings="$(cat "${WORK}/findings.txt")"
|
||||
|
||||
# NO-OP GUARD: this repo has been bitten by a check that passed when perturbed because it
|
||||
# asked a question that was always satisfiable. The floors below are the minimum evidence
|
||||
# for the verdict to mean anything, not a style rule.
|
||||
if [ "${oracle_rows}" -lt 6 ]; then
|
||||
findings="${findings}only ${oracle_rows} value(s) were put to the real guard — the coupling is not actually being exercised
|
||||
"
|
||||
fi
|
||||
if [ "${total}" -lt 20 ]; then
|
||||
findings="${findings}only ${total} assertions ran — the check degenerated to a near-no-op and its pass would be meaningless
|
||||
"
|
||||
fi
|
||||
|
||||
if [ -n "${findings}" ]; then
|
||||
echo "CHECK FAIL (placeholder-guard-coupling): $(printf '%s' "${findings}" | grep -c . ) of ${total} assertions failed" >&2
|
||||
printf '%s' "${findings}" | sed 's/^/ - /' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "check OK: placeholder-guard-coupling — ${total} assertions; ${oracle_rows} value(s) run through the REAL extracted predicate(s) from ${STANDUP_REL} (no reimplementation), covering every authored cluster-vars value of every guarded knob in base and each overlay, the guard's own vocabulary matched mid-string, and the empty/real-endpoint canaries. Static slice only: refusal, not reachability — see the header."
|
||||
@@ -1,77 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — prod is bare metal, and is not a development surface (R-ENV-03) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
# Spec: docs/00 R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local
|
||||
# storage; it MUST NOT be an active development surface until the roadmap
|
||||
# declares it ready"); docs/02 (prod = 3 physical nodes, local SSD/NVMe to
|
||||
# Longhorn, "the *same* Ansible bootstrap runs; only the machine-creation step
|
||||
# differs", "Never develop against prod"); roadmap/phases.md Phase 7.
|
||||
#
|
||||
# HOW A "MUST NOT" IS MADE TESTABLE — the same discipline as R-TEST-06's test: a
|
||||
# governance rule is verified through the property that ENFORCES it, never by observing
|
||||
# that nobody misbehaved today. Here that property is structural: prod receives the base
|
||||
# configuration plus environment-scoped overrides and nothing else, from the same branch
|
||||
# and the same Ansible code as dev, so there is no place to put prod-only work at all.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC: the prod inventory group is DERIVED (it is the group whose
|
||||
# group_vars retargets the k3s roles away from the global default), the prod overlay is
|
||||
# derived from CLUSTER_ENV, and the storage/replica expectations are compared against the
|
||||
# node count that inventory resolves to. No hostname, address or environment name is
|
||||
# written into the test.
|
||||
#
|
||||
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
||||
# WHEN the prod inventory, overlay, IaC module and readiness record are inspected — and
|
||||
# the shared provisioning chain is actually retargeted at prod with --list-hosts,
|
||||
# THEN prod resolves to exactly three nodes reached directly on their own sshd; no IaC
|
||||
# resource creates any of them (bare metal is imaged, not instantiated); prod's
|
||||
# Longhorn path is node-local with one replica per node and the default
|
||||
# StorageClass is the local driver; prod runs the same playbooks/roles as dev,
|
||||
# retargeted only by variables the shared code already defines; the prod overlay
|
||||
# carries only the base plus environment-scoped patches on the same Git ref dev
|
||||
# tracks, with no prod-only manifest anywhere in the shared configuration; and prod
|
||||
# artifacts exist only alongside the operator's recorded opening of the prod phase.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
|
||||
# where present — ansible-playbook in --list-hosts mode, which opens no connection);
|
||||
# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
|
||||
# wraps the SAME script.
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
||||
# * THE HARDWARE. That the three machines are mini-PCs, are bare metal, and hold local
|
||||
# SSD/NVMe is operator-attested (infra/PROD-STANDUP.md, PROD-PROVISIONING.md); the
|
||||
# node-health/BIOS surveys that establish it write to gitignored artifacts, and the
|
||||
# Kubernetes API exposes no "am I virtual" fact, so nothing here asserts it.
|
||||
# * The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
|
||||
# disks at the local path) — cluster state, belonging to R-PLAT-01's still-uncovered
|
||||
# HA test; asserting it here would also make this test environment-specific, since
|
||||
# the same suite must pass against dev.
|
||||
# * Git HISTORY: this is the working tree at HEAD.
|
||||
# * Whether the roadmap SHOULD have opened prod — only that prod work is accompanied by
|
||||
# the recorded declaration.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: prod-baremetal-not-dev-surface
|
||||
labels:
|
||||
conformance.home.k3s/R-ENV-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Given the tracked tree, when prod's inventory, overlay, IaC and readiness record are
|
||||
inspected, then prod is authored as three bare-metal nodes with node-local storage,
|
||||
bootstrapped by dev's own code retargeted by variables, receiving only
|
||||
environment-scoped overrides on the same branch — so there is nowhere to develop
|
||||
against prod — and it exists only because the roadmap phase was opened.
|
||||
spec:
|
||||
steps:
|
||||
- name: then-prod-is-bare-metal-and-carries-no-development-of-its-own
|
||||
description: >-
|
||||
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
||||
conformance/run.sh --layer l0 execute).
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,528 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — prod-baremetal-not-dev-surface
|
||||
# Serves: R-ENV-03 ("prod is a 3× mini-PC bare-metal k3s cluster with local storage; it
|
||||
# MUST NOT be an active development surface until the roadmap declares it ready."
|
||||
# — spec docs/00-vision-and-scope.md. Detail in docs/02: prod substrate "3× mini-PC
|
||||
# bare metal", "3 physical nodes", storage "local SSD/NVMe presented to Longhorn",
|
||||
# "For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation
|
||||
# step differs", and "**Never develop against prod.** Prod hardware is a target, not
|
||||
# a sandbox (R-ENV-03)". The roadmap gate is roadmap/phases.md Phase 7: entry
|
||||
# "operator declares prod hardware ready"; note "Do not begin building on prod
|
||||
# hardware before this phase is explicitly opened".)
|
||||
#
|
||||
# HOW A "MUST NOT" IS MADE TESTABLE
|
||||
# --------------------------------
|
||||
# The second clause is a governance rule, and the same discipline applies as to R-TEST-06
|
||||
# (tests/l0-static/spec-immutability-enforced): "nobody developed against prod today"
|
||||
# cannot be observed, but the PROPERTY THAT MAKES DEVELOPING AGAINST PROD IMPOSSIBLE can
|
||||
# be. In this repo that property is structural — prod receives the base configuration plus
|
||||
# environment-scoped overrides and NOTHING ELSE, so there is no place to put prod-only work
|
||||
# even if someone wanted to. This check asserts that, together with the first clause's
|
||||
# authored facts (three nodes, no VM IaC behind them, node-local storage, the same
|
||||
# bootstrap code as dev).
|
||||
#
|
||||
# The first clause's HARDWARE facts (that the three machines really are mini-PCs, really
|
||||
# are bare metal, really hold local NVMe) are operator-attested, not machine-checkable —
|
||||
# see the gap list. What IS checkable is that the repository describes prod that way and
|
||||
# that no IaC creates prod nodes as virtual machines.
|
||||
#
|
||||
# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
|
||||
# WHEN the prod inventory, the prod overlay, the IaC module and the roadmap record are
|
||||
# inspected — and, where offline execution is possible, the provisioning chain is
|
||||
# actually retargeted at prod with `--list-hosts`,
|
||||
# THEN (1) prod resolves to EXACTLY THREE nodes, reached directly on their own sshd
|
||||
# (no NAT indirection, i.e. real machines on the LAN);
|
||||
# (2) NO IaC resource creates any prod node: the VM module's generated node names are
|
||||
# disjoint from the prod host set and no prod hostname appears anywhere under
|
||||
# infra/tofu — prod is imaged by the operator (docs/02 stage 1), not instantiated;
|
||||
# (3) prod's storage is NODE-LOCAL: the Longhorn data path for prod is an absolute
|
||||
# local path, and the replica count equals the node count (one replica per node's
|
||||
# own disk). A distinct bulk/media tier is explicitly permitted (R-STATE-04) and
|
||||
# is not counted against this;
|
||||
# (4) prod runs THE SAME bootstrap code as dev: the same playbooks/roles, retargeted
|
||||
# by variables only — every prod group_vars key is an existing role/global
|
||||
# variable (an invented prod-only knob would be a fork), and no prod-only role
|
||||
# or playbook exists;
|
||||
# (5) prod is NOT a development surface: clusters/prod carries only the base plus
|
||||
# sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object
|
||||
# (cluster-vars, a Flux Kustomization's sops decryption, or the Git source),
|
||||
# prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on),
|
||||
# and no prod-only manifest exists anywhere in the rendered configuration;
|
||||
# (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap
|
||||
# carries the phase that opens prod build (checked when the spec repo is mounted),
|
||||
# and the repository records that the operator opened it.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
|
||||
# - The HARDWARE. That the three nodes are mini-PCs, are bare metal, and present local
|
||||
# SSD/NVMe is operator-attested in infra/PROD-STANDUP.md / PROD-PROVISIONING.md; the
|
||||
# node-health and BIOS surveys that establish it write to gitignored artifacts, so
|
||||
# nothing tracked can be asserted against. Cluster-side, the Kubernetes API exposes no
|
||||
# "am I virtual" fact either — a bare-metal check would need a privileged host-path
|
||||
# probe of DMI data, which is a heavier and more invasive test than the claim warrants.
|
||||
# - The LIVE prod cluster's shape (three Ready nodes, all control-plane+etcd, Longhorn
|
||||
# disks at the local path). That is cluster state and belongs to R-PLAT-01's
|
||||
# (currently uncovered) HA test; asserting it here would also make this test
|
||||
# environment-specific, since the same suite must pass against dev.
|
||||
# - That prod's HISTORY contains no development. This is the working tree at HEAD.
|
||||
# - Whether the roadmap SHOULD have opened prod (the entry-gate judgement). Only that
|
||||
# prod work is accompanied by the operator's recorded declaration.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
|
||||
export SPEC_REPO
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
NOTES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def note(msg):
|
||||
NOTES.append(msg)
|
||||
|
||||
|
||||
# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
|
||||
# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
|
||||
# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
|
||||
# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
|
||||
# narrow: every other non-zero exit is a real failure. The parse assertions always run.
|
||||
ENV_ONLY_FAILURE = re.compile(
|
||||
r"vault password|vault-id|Attempting to decrypt"
|
||||
r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
|
||||
|
||||
ANS = ROOT / "infra" / "ansible"
|
||||
INV = ANS / "inventory" / "hosts.ini"
|
||||
GV = ANS / "inventory" / "group_vars"
|
||||
TOFU = ROOT / "infra" / "tofu"
|
||||
check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — prod is not described anywhere")
|
||||
if not INV.is_file():
|
||||
print("CHECK FAIL (prod-baremetal-not-dev-surface): no inventory", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
# --------------------------------------------------------------- inventory parsing
|
||||
ini_groups: dict[str, list[dict]] = {}
|
||||
children: dict[str, list[str]] = {}
|
||||
cur = None
|
||||
for line in INV.read_text().splitlines():
|
||||
s = line.strip()
|
||||
if not s or s.startswith("#"):
|
||||
continue
|
||||
m = re.match(r"^\[([^\]]+)\]$", s)
|
||||
if m:
|
||||
cur = m.group(1)
|
||||
continue
|
||||
if cur is None or cur.endswith(":vars"):
|
||||
continue
|
||||
if cur.endswith(":children"):
|
||||
children.setdefault(cur.split(":")[0], []).append(s.split()[0])
|
||||
continue
|
||||
parts = s.split()
|
||||
host = {"name": parts[0]}
|
||||
for kv in parts[1:]:
|
||||
if "=" in kv:
|
||||
k, v = kv.split("=", 1)
|
||||
host[k] = v
|
||||
ini_groups.setdefault(cur, []).append(host)
|
||||
|
||||
|
||||
def hosts_in(group: str, seen=None) -> list[dict]:
|
||||
seen = seen or set()
|
||||
if group in seen:
|
||||
return []
|
||||
seen.add(group)
|
||||
out = list(ini_groups.get(group, []))
|
||||
for child in children.get(group, []):
|
||||
out += hosts_in(child, seen)
|
||||
return out
|
||||
|
||||
|
||||
def load_gv(group: str) -> dict:
|
||||
d: dict = {}
|
||||
for p in [GV / f"{group}.yml", GV / group / "vars.yml"]:
|
||||
if p.is_file():
|
||||
try:
|
||||
d.update(yaml.safe_load(p.read_text()) or {})
|
||||
except Exception:
|
||||
pass
|
||||
return d
|
||||
|
||||
|
||||
# WHICH GROUP IS PROD? Derived, never named here: it is the group whose group_vars
|
||||
# RETARGETS the k3s roles (k3s_primary_group / k3s_servers_group) away from the global
|
||||
# dev default. That is exactly the seam docs/02 + infra/ansible/README describe for prod.
|
||||
ALL_VARS = load_gv("all")
|
||||
dev_primary_group = str(ALL_VARS.get("k3s_primary_group", ""))
|
||||
candidates = []
|
||||
for entry in sorted(GV.iterdir()) if GV.is_dir() else []:
|
||||
g = entry.name[:-4] if entry.name.endswith(".yml") else entry.name
|
||||
if g == "all":
|
||||
continue
|
||||
v = load_gv(g)
|
||||
if v.get("k3s_primary_group") and str(v["k3s_primary_group"]) != dev_primary_group:
|
||||
candidates.append((g, v))
|
||||
check(len(candidates) == 1,
|
||||
f"expected exactly ONE environment whose group_vars retargets the k3s roles (prod); found "
|
||||
f"{[c[0] for c in candidates]} — R-ENV-03 describes a single prod estate reached through "
|
||||
"the same roles as dev")
|
||||
if not candidates:
|
||||
print("CHECK FAIL (prod-baremetal-not-dev-surface): no prod group_vars found", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
PROD_GROUP, PROD_VARS = candidates[0]
|
||||
PROD_PRIMARY = str(PROD_VARS["k3s_primary_group"])
|
||||
prod_hosts = hosts_in(PROD_GROUP)
|
||||
prod_names = sorted(h["name"] for h in prod_hosts)
|
||||
|
||||
# ------------------------------------------------- (1) exactly three, directly reachable
|
||||
check(len(prod_names) == 3,
|
||||
f"prod group {PROD_GROUP!r} resolves to {len(prod_names)} host(s) {prod_names} — R-ENV-03 "
|
||||
"says prod is a 3× mini-PC cluster")
|
||||
dev_hosts = hosts_in(str(ALL_VARS.get("k3s_servers_group", "k3s_servers")))
|
||||
for h in prod_hosts:
|
||||
# Real machines on the LAN answer their own sshd. Dev's hosts are reached through a
|
||||
# NAT port-forward (ansible_port 225x -> a VM's :22), which is precisely the
|
||||
# indirection a hypervisor-hosted estate needs and bare metal does not.
|
||||
check(str(h.get("ansible_port", "22")) == "22",
|
||||
f"prod node {h['name']} is reached on port {h.get('ansible_port')} — a forwarded port "
|
||||
"implies an indirection (NAT to a virtual estate), not a machine on the LAN")
|
||||
check(h.get("k3s_node_ip") == h.get("ansible_host"),
|
||||
f"prod node {h['name']} has ansible_host={h.get('ansible_host')} but "
|
||||
f"k3s_node_ip={h.get('k3s_node_ip')} — a split address is the dev NAT pattern; a "
|
||||
"bare-metal node's SSH address IS its cluster address")
|
||||
|
||||
# ------------------------------------------------------ (2) nothing creates prod as a VM
|
||||
TF = "\n".join(p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))) \
|
||||
if TOFU.is_dir() else ""
|
||||
check(len(TF) > 500, "infra/tofu/*.tf is empty — cannot show that it does NOT create prod")
|
||||
|
||||
|
||||
def tf_default(var: str):
|
||||
m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
|
||||
if not m:
|
||||
return None
|
||||
d = re.search(r"default\s*=\s*(.+)", m.group(1))
|
||||
return d.group(1).strip() if d else None
|
||||
|
||||
|
||||
prefix = (tf_default("node_name_prefix") or "").strip('"')
|
||||
try:
|
||||
node_count = int((tf_default("node_count") or "0").strip())
|
||||
except ValueError:
|
||||
node_count = 0
|
||||
iac_names = {f"{prefix}{i + 1}" for i in range(node_count)}
|
||||
check(len(iac_names) > 0, "the IaC module generates no node names — cannot compare with prod")
|
||||
check(iac_names.isdisjoint(set(prod_names)),
|
||||
f"the IaC VM module would create {sorted(iac_names & set(prod_names))}, which are PROD nodes "
|
||||
"— prod is bare metal that the operator images (docs/02 stage 1), never instantiated by the "
|
||||
"hypervisor module")
|
||||
for name in prod_names:
|
||||
check(name not in TF,
|
||||
f"prod node {name} is named inside infra/tofu — prod hardware must not be described by "
|
||||
"the VM module at all")
|
||||
# The prod host addresses must not appear as IaC node addresses either (a renamed VM is
|
||||
# still a VM).
|
||||
tf_ips = set(re.findall(r'"(\d+\.\d+\.\d+\.\d+)"', tf_default("node_ips") or ""))
|
||||
prod_ips = {str(h.get("ansible_host")) for h in prod_hosts}
|
||||
check(tf_ips.isdisjoint(prod_ips),
|
||||
f"the IaC module addresses {sorted(tf_ips & prod_ips)}, which are prod node addresses")
|
||||
|
||||
# --------------------------------------------------------------- (3) node-local storage
|
||||
# Every environment's value for a cluster-vars knob (base default + each overlay patch).
|
||||
def env_values(key: str) -> dict[str, str]:
|
||||
vals: dict[str, str] = {}
|
||||
base = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
||||
if base.is_file():
|
||||
for d in yaml.safe_load_all(base.read_text()):
|
||||
if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
|
||||
and (d.get("metadata") or {}).get("name") == "cluster-vars":
|
||||
v = (d.get("data") or {}).get(key)
|
||||
if v is not None:
|
||||
vals["base"] = str(v)
|
||||
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
||||
env = overlay.parent.name
|
||||
if env == "base":
|
||||
continue
|
||||
try:
|
||||
k = yaml.safe_load(overlay.read_text()) or {}
|
||||
except Exception:
|
||||
continue
|
||||
for p in (k.get("patches") or []):
|
||||
tgt = p.get("target") or {}
|
||||
if tgt.get("kind") != "ConfigMap" or tgt.get("name") != "cluster-vars":
|
||||
continue
|
||||
try:
|
||||
ops = yaml.safe_load(p.get("patch") or "") or []
|
||||
except Exception:
|
||||
continue
|
||||
for op in ops if isinstance(ops, list) else []:
|
||||
if isinstance(op, dict) and str(op.get("path", "")) == f"/data/{key}":
|
||||
vals[env] = str(op.get("value"))
|
||||
return vals
|
||||
|
||||
|
||||
# The prod OVERLAY is the one whose CLUSTER_ENV is not the base default.
|
||||
cluster_envs = env_values("CLUSTER_ENV")
|
||||
prod_overlays = [e for e, v in cluster_envs.items()
|
||||
if e != "base" and v != cluster_envs.get("base")]
|
||||
check(len(prod_overlays) >= 1,
|
||||
"no clusters/ overlay sets its own CLUSTER_ENV — prod has no overlay of its own")
|
||||
prod_overlay = None
|
||||
for e in prod_overlays:
|
||||
if PROD_GROUP.replace("k3s_", "") in cluster_envs[e] or cluster_envs[e] in PROD_GROUP:
|
||||
prod_overlay = e
|
||||
check(prod_overlay is not None,
|
||||
f"no overlay's CLUSTER_ENV corresponds to the prod inventory group {PROD_GROUP!r} — the "
|
||||
"cluster config and the machine inventory disagree about which estate is prod")
|
||||
|
||||
data_paths = env_values("LONGHORN_DATA_PATH")
|
||||
replicas = env_values("LONGHORN_DEFAULT_REPLICA_COUNT")
|
||||
if prod_overlay:
|
||||
p = data_paths.get(prod_overlay, data_paths.get("base", ""))
|
||||
check(p.startswith("/"),
|
||||
f"prod's LONGHORN_DATA_PATH is {p!r} — R-ENV-03's 'local storage' requires an absolute "
|
||||
"node-local filesystem path")
|
||||
check(not re.match(r"^/(net|nfs|mnt/nfs)|:", p),
|
||||
f"prod's LONGHORN_DATA_PATH {p!r} looks like a network mount, not local SSD/NVMe")
|
||||
r = replicas.get(prod_overlay, replicas.get("base", ""))
|
||||
check(r.isdigit() and int(r) == len(prod_names),
|
||||
f"prod's LONGHORN_DEFAULT_REPLICA_COUNT is {r!r} for {len(prod_names)} node(s) — "
|
||||
"'local storage' on a 3-node estate means one replica per node's own disk")
|
||||
# The DEFAULT storage class must be the local (Longhorn) driver. A separate bulk/media tier
|
||||
# on network storage is explicitly a different requirement (R-STATE-04, docs/04) and is not
|
||||
# counted here.
|
||||
default_scs = []
|
||||
for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
|
||||
capture_output=True, text=True, check=True).stdout.split("\0"):
|
||||
if not f:
|
||||
continue
|
||||
try:
|
||||
for d in yaml.safe_load_all((ROOT / f).read_text(errors="replace")):
|
||||
if isinstance(d, dict) and d.get("kind") == "StorageClass" and \
|
||||
str(((d.get("metadata") or {}).get("annotations") or {})
|
||||
.get("storageclass.kubernetes.io/is-default-class", "")) == "true":
|
||||
default_scs.append((f, str(d.get("provisioner", ""))))
|
||||
except Exception:
|
||||
continue
|
||||
check(len(default_scs) == 1,
|
||||
f"expected exactly one default StorageClass, found {default_scs} — the tier prod's workloads "
|
||||
"land on by default must be unambiguous")
|
||||
for f, prov in default_scs:
|
||||
check(prov == "driver.longhorn.io",
|
||||
f"{f}: the default StorageClass provisioner is {prov!r}, not Longhorn — prod's default "
|
||||
"tier would not be its own local disks")
|
||||
|
||||
# ------------------------------------- (4) prod runs the SAME bootstrap code, retargeted
|
||||
PLAYBOOKS = ANS / "playbooks"
|
||||
ROLES = ANS / "roles"
|
||||
prod_token = PROD_GROUP.replace("k3s_", "")
|
||||
for d, what in ((ROLES, "role"), (PLAYBOOKS, "playbook")):
|
||||
for entry in sorted(d.iterdir()) if d.is_dir() else []:
|
||||
check(prod_token not in entry.name.lower(),
|
||||
f"infra/ansible/{d.name}/{entry.name} is a prod-specific {what} — docs/02 requires "
|
||||
"the SAME Ansible bootstrap for prod, differing only in the machine-creation step")
|
||||
# Every prod group_vars key must be a variable the shared code already knows: a role default,
|
||||
# a global, or an ansible built-in. An invented key is a fork of the role by another name.
|
||||
known: set[str] = set(ALL_VARS)
|
||||
for rd in sorted(ROLES.iterdir()) if ROLES.is_dir() else []:
|
||||
for p in (rd / "defaults").glob("*.yml") if (rd / "defaults").is_dir() else []:
|
||||
try:
|
||||
known |= set((yaml.safe_load(p.read_text()) or {}).keys())
|
||||
except Exception:
|
||||
pass
|
||||
check(len(known) > 20, f"only {len(known)} shared variables found — (4) would prove nothing")
|
||||
for k in sorted(PROD_VARS):
|
||||
check(k.startswith("ansible_") or k in known,
|
||||
f"prod group_vars sets {k!r}, which is not a role default or a global — prod-only "
|
||||
"configuration that the dev path never exercises is developing against prod")
|
||||
# ...and it must retarget, not re-implement: the group it names must exist in the inventory.
|
||||
check(PROD_PRIMARY in ini_groups or PROD_PRIMARY in children,
|
||||
f"prod group_vars names primary group {PROD_PRIMARY!r}, which the inventory does not define")
|
||||
|
||||
# ---------------------------------------- (5) prod is not a development surface (overlay)
|
||||
if prod_overlay:
|
||||
odir = ROOT / "clusters" / prod_overlay
|
||||
files = sorted(p.name for p in odir.iterdir() if p.is_file())
|
||||
check(files == ["kustomization.yaml", "sync.yaml"],
|
||||
f"clusters/{prod_overlay}/ contains {files} — a prod overlay may hold only its "
|
||||
"kustomization and its own root sync; anything else is configuration authored FOR PROD")
|
||||
k = yaml.safe_load((odir / "kustomization.yaml").read_text()) or {}
|
||||
check(sorted(str(r) for r in (k.get("resources") or [])) == ["../base", "sync.yaml"],
|
||||
f"clusters/{prod_overlay} resources are {k.get('resources')} — prod must render the base "
|
||||
"plus its own sync only (docs/02: 'apply the prod overlay (no base edits)')")
|
||||
# Every patch must be an ENVIRONMENT-SCOPED override, never a change to what runs.
|
||||
ALLOWED = {("ConfigMap", "cluster-vars"), ("GitRepository", None)}
|
||||
for p in (k.get("patches") or []):
|
||||
tgt = p.get("target") or {}
|
||||
body = p.get("patch") or ""
|
||||
kind = tgt.get("kind")
|
||||
if kind is None: # inline patch: kind comes from the body
|
||||
try:
|
||||
doc = yaml.safe_load(body) or {}
|
||||
except Exception:
|
||||
doc = {}
|
||||
kind = doc.get("kind")
|
||||
name = (doc.get("metadata") or {}).get("name")
|
||||
if kind == "Kustomization":
|
||||
# Flux Kustomization patches are allowed ONLY to wire sops decryption —
|
||||
# anything else (path, interval, images, values) changes what prod RUNS.
|
||||
keys = set((doc.get("spec") or {}).keys())
|
||||
check(keys <= {"decryption"},
|
||||
f"clusters/{prod_overlay}: patch of Kustomization/{name} touches {sorted(keys)} "
|
||||
"— only `decryption` (the per-cluster sops key) is an environment-scoped "
|
||||
"override; changing anything else is developing against prod")
|
||||
continue
|
||||
check((kind, None) in ALLOWED or (kind, name) in ALLOWED,
|
||||
f"clusters/{prod_overlay}: patch targets {kind}/{name}, which is not an "
|
||||
"environment-scoped object — prod must not carry configuration of its own")
|
||||
else:
|
||||
check((kind, tgt.get("name")) in ALLOWED or (kind, None) in ALLOWED,
|
||||
f"clusters/{prod_overlay}: patch targets {kind}/{tgt.get('name')}, which is not "
|
||||
"an environment-scoped object — prod must not carry configuration of its own")
|
||||
|
||||
# Same Git ref as the other environment(s): a prod-only branch is a development line
|
||||
# that only prod runs, which is exactly what R-ENV-03 forbids.
|
||||
def git_ref(overlay: str):
|
||||
kk = yaml.safe_load((ROOT / "clusters" / overlay / "kustomization.yaml").read_text()) or {}
|
||||
for p in (kk.get("patches") or []):
|
||||
body = p.get("patch") or ""
|
||||
try:
|
||||
doc = yaml.safe_load(body) or {}
|
||||
except Exception:
|
||||
continue
|
||||
if doc.get("kind") == "GitRepository":
|
||||
return (doc.get("spec") or {}).get("ref")
|
||||
return None
|
||||
|
||||
others = [o.name for o in (ROOT / "clusters").iterdir()
|
||||
if o.is_dir() and o.name not in ("base", prod_overlay)]
|
||||
prod_ref = git_ref(prod_overlay)
|
||||
check(prod_ref is not None,
|
||||
f"clusters/{prod_overlay} does not pin a Git ref — prod would follow whatever the base "
|
||||
"points at, with no record of what it runs")
|
||||
for o in others:
|
||||
check(git_ref(o) == prod_ref,
|
||||
f"clusters/{o} tracks {git_ref(o)} but prod tracks {prod_ref} — a prod-only ref is a "
|
||||
"development line only prod runs (R-ENV-03: prod is a target, not a sandbox)")
|
||||
|
||||
# No prod-only manifest anywhere in the rendered configuration.
|
||||
tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True
|
||||
).stdout.split("\0") if f]
|
||||
RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/")
|
||||
for f in tracked:
|
||||
if f.startswith(RENDERED_DIRS) and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)",
|
||||
f, re.I):
|
||||
check(False,
|
||||
f"{f} is a prod-specific file inside the shared configuration — every environment "
|
||||
"renders these paths, so prod-only content here is development against prod")
|
||||
check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout")
|
||||
|
||||
# --------------------------- (6) prod work exists only with a declaration of readiness
|
||||
# In-repo: the operator's recorded decision to open the prod phase.
|
||||
log = ROOT / "DECISIONS-LOG.md"
|
||||
check(log.is_file() and re.search(r"PHASE\s*7\b.*OPEN", log.read_text(errors="replace"), re.I),
|
||||
"DECISIONS-LOG.md carries no record of the operator OPENING the prod phase, yet prod "
|
||||
"inventory/overlay exist — R-ENV-03 allows prod build only once the roadmap declares it ready")
|
||||
# In the spec (read-only, may not be mounted): the roadmap phase that opens prod.
|
||||
spec = Path(os.environ.get("SPEC_REPO", ""))
|
||||
phases = spec / "roadmap" / "phases.md"
|
||||
if phases.is_file():
|
||||
txt = phases.read_text(errors="replace")
|
||||
check(re.search(r"##\s*Phase\s*7\b", txt) is not None and
|
||||
re.search(r"prod", txt, re.I) is not None,
|
||||
"the spec roadmap has no Phase 7 opening prod build — the declaration R-ENV-03 defers "
|
||||
"to does not exist")
|
||||
check(re.search(r"explicitly opened|declares prod hardware ready|operator declares", txt, re.I)
|
||||
is not None,
|
||||
"the spec roadmap's prod phase states no explicit entry declaration — the gate R-ENV-03 "
|
||||
"names would be unenforceable")
|
||||
else:
|
||||
note(f"spec repo not mounted at {spec} — the roadmap half of (6) was NOT checked "
|
||||
"(the in-repo declaration still was)")
|
||||
|
||||
# --------------------------- (4b) DRIVE the retarget offline: same chain, prod hosts
|
||||
if shutil.which("ansible-playbook") is None:
|
||||
note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
|
||||
"prod retarget was PARSED but not DRIVEN; the merge gate's container has it")
|
||||
else:
|
||||
sec_groups = [g for g in children.get(PROD_GROUP, []) if g != PROD_PRIMARY]
|
||||
args = ["--list-hosts", "playbooks/site.yml",
|
||||
"-e", f"k3s_hosts={PROD_GROUP}",
|
||||
"-e", f"k3s_primary_hosts={PROD_PRIMARY}"]
|
||||
if sec_groups:
|
||||
args += ["-e", f"k3s_secondary_hosts={sec_groups[0]}"]
|
||||
lh = subprocess.run(["ansible-playbook", *args], cwd=ANS, capture_output=True, text=True)
|
||||
out = (lh.stderr or "") + (lh.stdout or "")
|
||||
if lh.returncode != 0 and ENV_ONLY_FAILURE.search(out):
|
||||
# See the same narrow, self-policing skip in dev-nodes-provisioned-as-code.
|
||||
note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
|
||||
"passphrase and/or the gitignored Galaxy collections), so the prod retarget was not "
|
||||
"driven; every parse assertion above still ran")
|
||||
else:
|
||||
check(lh.returncode == 0,
|
||||
f"retargeting the SHARED chain at prod failed: {out.strip()[:400]} — docs/02 requires "
|
||||
"the same Ansible bootstrap to run for prod")
|
||||
seen: set[str] = set()
|
||||
in_hosts = False
|
||||
for line in lh.stdout.splitlines():
|
||||
if re.match(r"\s*hosts \(\d+\):", line):
|
||||
in_hosts = True
|
||||
continue
|
||||
if re.search(r"play #\d+", line):
|
||||
in_hosts = False
|
||||
continue
|
||||
if in_hosts and line.strip():
|
||||
seen.add(line.strip())
|
||||
check(seen == set(prod_names),
|
||||
f"the shared chain retargeted at prod resolves to {sorted(seen)}, not the three prod "
|
||||
f"nodes {prod_names} — prod would be bootstrapped by something other than dev's path")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
check(CHECKS >= 25,
|
||||
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
|
||||
"would be meaningless")
|
||||
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (prod-baremetal-not-dev-surface): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
for n in NOTES:
|
||||
print(f" note: {n}")
|
||||
print(f"check OK: prod-baremetal-not-dev-surface — {CHECKS} assertions; prod group {PROD_GROUP!r} = "
|
||||
f"{prod_names} (3 nodes, none created by IaC), overlay clusters/{prod_overlay} renders the base "
|
||||
f"plus environment-scoped patches only, local Longhorn path "
|
||||
f"{data_paths.get(prod_overlay, data_paths.get('base'))!r} with "
|
||||
f"{replicas.get(prod_overlay, replicas.get('base'))} replica(s). R-ENV-03's AUTHORED clauses "
|
||||
"only — the hardware facts (mini-PC, bare metal, local NVMe) are operator-attested and the "
|
||||
"live cluster's shape is NOT asserted here (see the header's gap list).")
|
||||
PY
|
||||
Loaded 100 of 274 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user