Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 4.8
|
7a34590b19 |
No files matched your search
+31
-48
@@ -1,27 +1,19 @@
|
||||
# .gitea/workflows/gate.yml — the ENFORCED PR merge gate (Phase 6, WS-GITOSS).
|
||||
#
|
||||
# This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
|
||||
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `make -C ci gate` — so CI
|
||||
# runs BYTE-FOR-BYTE the same command a human runs in the devcontainer (R-TEST-02): the
|
||||
# runner just installs the pinned tools from ci/versions.env first. `make -C ci gate`
|
||||
# (ci/gate.sh) runs every stage in order — spec-immutability, lint, Kyverno L0,
|
||||
# render==golden, L1 offline render — and applies the R-TEST-03 decision, printing
|
||||
# "MERGE GATE: PASS" / "MERGE GATE: BLOCKED". It is the single required status check on
|
||||
# `main` (see branch protection). A non-zero exit fails the job, so a BLOCKED gate blocks
|
||||
# the merge.
|
||||
# Docker executor + DinD). It runs the STATIC layers of the conformance gate — the ones
|
||||
# docs/14 says run "without a cluster": spec-immutability (best-effort), lint, Kyverno L0,
|
||||
# the render==golden reproducibility check, and L1 offline render checks. It is the single
|
||||
# required status check on `main`
|
||||
# (see branch protection). `make -C ci lint l0 render-golden l1` here is byte-identical to a laptop run
|
||||
# (R-TEST-02) — the runner just installs the pinned tools from ci/versions.env first.
|
||||
#
|
||||
# The cluster-gated L2/L3 conformance stage SKIPS honestly here (exit 3, surfaced not faked
|
||||
# — HARD RULE 4) because this runner declares no cluster: `make -C ci gate` does not set
|
||||
# CLUSTER_AVAILABLE, so gate.sh reports the touched requirements UNVERIFIED without blocking
|
||||
# Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
|
||||
# no broad kubeconfig) driven by the separate `conformance` job below.
|
||||
#
|
||||
# NOTE (R-TEST-06 / gate-semantics): spec-immutability is ALSO surfaced as its own explicit
|
||||
# early step (below) so its result is a distinct, tolerant status in the Gitea UI; gate.sh
|
||||
# re-runs it as its first stage, which is the authoritative one. `make -C ci gate` here is
|
||||
# the byte-identical laptop gate; the extra pre-step is a read-only git-diff check and is
|
||||
# purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
|
||||
# is retained dormant in merge-gate.yaml.
|
||||
# The L2/L3 CONFORMANCE layer (behavioural + fault-injection, needs a live cluster) is
|
||||
# NOT wired yet — it is a deliberate design decision (a CI toolchain image + a scoped
|
||||
# in-cluster kubeconfig/RBAC for the runner). It is present below as an `if: false`
|
||||
# placeholder so the shape is visible and the wiring is a one-line flip once decided.
|
||||
# The full pipeline blueprint (spec-checkout, per-layer jobs, decision job) is retained
|
||||
# dormant in merge-gate.yaml.
|
||||
---
|
||||
name: gate
|
||||
|
||||
@@ -55,27 +47,19 @@ jobs:
|
||||
./.gitea/workflows/scripts/install-tools.sh l0
|
||||
./.gitea/workflows/scripts/install-tools.sh l1
|
||||
|
||||
# R-TEST-06: the spec is read-only to agents. An EXPLICIT early surfacing of the
|
||||
# spec-immutability check, so its result is a distinct, tolerant status line in the
|
||||
# Gitea UI before the (longer) aggregate gate log. The spec repo is NOT checked out
|
||||
# into CI here (separate repo; SPEC_REPO_SLUG not wired into this runner), so the guard
|
||||
# honestly SKIPS (exit 3) rather than faking green; a real drift/edit of a co-located
|
||||
# spec exits 1 and fails. Tolerated: exit 3 only. Call the script DIRECTLY, not via
|
||||
# `make spec-immutability`: GNU make masks a recipe's exit code as its own 2, so
|
||||
# `make spec-immutability || [ $? -eq 3 ]` would see 2 and fail. The AUTHORITATIVE
|
||||
# spec-immutability run is the FIRST stage of `make -C ci gate` below (gate.sh calls
|
||||
# the script directly, so it sees exit 3 → SKIP, not a mask); this pre-step is a cheap,
|
||||
# read-only git-diff duplicate that only adds early UI signal.
|
||||
# R-TEST-06: the spec is read-only to agents. The spec repo is NOT checked out into
|
||||
# CI here (it is a separate repo; SPEC_REPO_SLUG is not wired into this runner), so
|
||||
# the guard honestly SKIPS (exit 3) rather than faking green. A real drift/edit of a
|
||||
# co-located spec would exit 1 and fail. Tolerated: exit 3 only.
|
||||
# Call the script DIRECTLY, not via `make`: GNU make masks a recipe's exit code as
|
||||
# its own 2 on failure, so `make spec-immutability || [ $? -eq 3 ]` would see 2 and
|
||||
# fail. The script itself exits 3 on an honest skip (spec repo not checked out here),
|
||||
# which we tolerate; a real drift/edit exits 1 and fails the gate.
|
||||
- name: Spec immutability (R-TEST-06, skips if spec repo absent)
|
||||
run: ./ci/spec-immutability.sh || [ "$?" -eq 3 ]
|
||||
|
||||
# Aggregate merge gate: byte-for-byte the command a human runs in the devcontainer
|
||||
# (`make -C ci gate` -> ci/gate.sh). Runs spec-immutability, lint, Kyverno L0,
|
||||
# render==golden and L1 render, then the R-TEST-03 decision. Exits non-zero on a
|
||||
# BLOCKED gate, which fails this job. The cluster-gated L2/L3 stage SKIPS here (no
|
||||
# CLUSTER_AVAILABLE) — surfaced as UNVERIFIED, never faked green (see header).
|
||||
- name: Merge gate — make -C ci gate
|
||||
run: make -C ci gate
|
||||
- name: Static gate — lint + Kyverno L0 + render==golden + L1 render
|
||||
run: make -C ci lint l0 render-golden l1
|
||||
|
||||
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
||||
# ("not supported on GHES" — Gitea does not implement the artifact-v2 backend), which
|
||||
@@ -84,16 +68,15 @@ jobs:
|
||||
# Gitea-compatible @v3 action or Gitea's own artifact backend if retention is wanted.
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────
|
||||
# PLACEHOLDER — NOT WIRED (deferred; concrete plan written, awaiting operator decision).
|
||||
# L2/L3 conformance (Chainsaw behavioural + fault-injection) needs a LIVE cluster and is
|
||||
# gated behind vars.CLUSTER_AVAILABLE + a narrow secrets.CONFORMANCE_TRIGGER_KUBECONFIG.
|
||||
# The intended model: an in-cluster Chainsaw Job (conformance/incluster/) under its OWN
|
||||
# scoped ServiceAccount — CI holds NO broad kubeconfig, only a trigger credential. The
|
||||
# blocker is a real privilege-escalation design question (a `create jobs` credential can
|
||||
# create a Job running as the privileged runner SA), plus a runner image + repo-clone +
|
||||
# results-collection that don't exist yet. Full analysis + step sequence:
|
||||
# docs/plans/l2-l3-conformance-ci-wiring.md
|
||||
# The blueprint job shape is retained dormant in merge-gate.yaml.
|
||||
# PLACEHOLDER — NOT WIRED (STOP-and-report item, WS-GITOSS Phase 6).
|
||||
# L2/L3 conformance (Chainsaw behavioural + fault-injection) needs a LIVE cluster.
|
||||
# Turning this on is a design decision with two open sub-decisions:
|
||||
# 1. Toolchain: build+push a pinned CI image to the Gitea container registry and
|
||||
# reference it as the job image, vs. install-in-job from ci/versions.env (slower).
|
||||
# 2. Cluster creds: how the runner reaches the cluster — the intended model is an
|
||||
# in-cluster Chainsaw Job driven by its OWN scoped ServiceAccount (CI holds only a
|
||||
# narrow trigger credential; the runner mounts NO SA token today). That SA + RBAC
|
||||
# + the trigger secret do not exist yet.
|
||||
# Kept `if: false` so it never runs (shows skipped, never fakes green — HARD RULE 4).
|
||||
# ───────────────────────────────────────────────────────────────────────────
|
||||
conformance:
|
||||
|
||||
@@ -41,6 +41,3 @@ infra/tofu/.r2.env
|
||||
# ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
|
||||
# stage (which scans untracked files) from tripping on .claude/worktrees/.
|
||||
.claude/
|
||||
|
||||
# Vendored ansible collections (installed by ansible-galaxy for node re-provision; not tracked or linted)
|
||||
infra/ansible/collections/
|
||||
@@ -1150,611 +1150,3 @@ gitignored (agent worktrees were tripping the lint stage's untracked-file scan).
|
||||
Plex uses `/config` → Plex writes the ephemeral layer, is unclaimed, loses config on restart
|
||||
(seerr↔plex connectivity is actually fine — 200 on `/identity`). FIX PENDING operator go (resets
|
||||
Plex, needs re-claim): mount PVC at `/config` + wire `PLEX_CLAIM`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-08-30 — Upgrade campaign landed: cluster now k3s v1.36.4 / Debian 13 / Cilium 1.20.1 / Longhorn 1.12.1
|
||||
|
||||
The platform-wide upgrade campaign completed and is live. Verified current state of the dev cluster:
|
||||
|
||||
- **k3s v1.36.4+k3s1** (was v1.31.5+k3s1), 3-server embedded-etcd HA.
|
||||
- **Debian 13 (trixie)** node OS (was Debian 12 bookworm).
|
||||
- **Cilium 1.20.1** (was 1.16.5); **Gateway API v1.6.1** (was v1.2.x).
|
||||
- **Longhorn 1.12.1** (was 1.7.3).
|
||||
- cert-manager >=1.21; Velero chart 12.x / Velero v1.18; kube-prometheus-stack 88 (operator
|
||||
v0.93.1, Grafana 13.2.0); kube-vip v1.2.3.
|
||||
- Node RAM **24 GiB / 24576 MB** (tofu default; earlier docs said 16 GiB).
|
||||
- Each node now carries a **dedicated virtio2 SSD etcd disk** (`/dev/vdc`, `local-zfs`) isolating
|
||||
etcd WAL fsync from HDD Longhorn I/O — see `docs/plans/ssd-etcd-isolation.md`.
|
||||
|
||||
**The "LIVE STATE VERIFIED 2026-08-26" line above (k3s v1.31.5) is SUPERSEDED by this entry.**
|
||||
That older line is retained as historical record — this append is the current source of truth.
|
||||
Documentation and comments across the repo were reconciled to these facts in the same change;
|
||||
toolchain kubectl bumped 1.31.4 → 1.36.4 (the 1.31 pin exceeded the ±1 version skew against a 1.36
|
||||
cluster). Revalidation items deliberately left open: external-snapshotter v8.2.0 pin vs 1.36, the
|
||||
Cilium ext_authz/forward-auth workaround relevance under 1.20.1, and the conformance ImageVolume
|
||||
feature-gate assertions under 1.36.
|
||||
|
||||
## 2026-08-31 — Post-upgrade stabilization: SSO fixed, all UIs gated, 8 vCPU, CI live. **PM HANDOFF.**
|
||||
|
||||
Everything the 08-30 campaign deferred is resolved; the platform is fully green. **This entry is the
|
||||
current source of truth and the handoff for the next PM.** (Session ran on a different workstation; a
|
||||
fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` and start at Next Steps.)
|
||||
|
||||
### LIVE STATE (verified 2026-08-31, all green)
|
||||
- 3× k3s **v1.36.4+k3s1**, Debian 13, **8 vCPU / 24 GiB** each (raised from 6 — see #5). Proxmox host 24 cores, ~idle.
|
||||
- Cilium 1.20.1 (GW-API v1.6.1), Longhorn 1.12.1, cert-manager 1.21, Velero chart 12/v1.18, kube-prometheus-stack 88
|
||||
(Grafana 13.2), kube-vip v1.2.3, **Loki chart 7.x / v3.6.11**, **Kyverno chart 3.9 / v1.19** (the last two were the
|
||||
08-30 "deferred" items — now done). etcd on the dedicated SSD virtio2 disk per node.
|
||||
- All Flux kustomizations + HelmReleases Ready; 0 not-ready pods.
|
||||
- **SSO end-to-end**: native OIDC for grafana/gitea/nextcloud/semaphore/headlamp; the 5 forward-auth UIs
|
||||
(pihole/jackett/qbittorrent/radarr/sonarr) gated by per-app oauth2-proxy sidecars (fail-closed, 302→Keycloak).
|
||||
Platform-admins require TOTP MFA.
|
||||
- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
|
||||
required status check; runner `incluster-dind` online (ns gitea-actions).
|
||||
|
||||
### What changed this session (durable)
|
||||
1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
|
||||
Gateway L7 proxy (`enforce_policy_on_l7lb`) re-check a pod's egress policy against Keycloak's REAL backend
|
||||
identity on the split-horizon hairpin; `toEntities: world` no longer covers it → all app SSO 403'd
|
||||
("Access denied", `server: envoy`). Fix: explicit `toEndpoints` egress allow to the keycloak backend —
|
||||
`catalog/render.py` emits it for `auth.mode ∈ {oidc,forward-auth}`; headlamp/crafty hand-written. (68690e5, 8a424bc)
|
||||
2. **Keycloak MFA config bug**: browser-mfa conditional-OTP used a non-existent config key → "Invalid role name: null"
|
||||
→ 403 for every platform-admin login. Fixed to `condUserRole: platform-admin` in git + live via kcadm. (45deac9)
|
||||
3. **VPN egress restored**: the Debian 12→13 re-provision wiped k3s-dev-3's node-host OpenVPN (PIA); re-ran
|
||||
`vpn.egress.gateway.yml`. Re-provision runbook (infra/ansible/README.md) updated — node-host roles (k3s, kube-vip,
|
||||
vpn.egress.gateway) must be re-run after any reprovision.
|
||||
4. **Grafana OOM** (kps 65→88 ships Grafana 13) — the hidden blocker that stalled the whole kps upgrade + platform
|
||||
tier. limits 256→512→768Mi; Prometheus 2→3Gi. (b2b1aa5 and follow-ups)
|
||||
5. **Node vCPU 6→8** — steady CPU PSI was 15-23% / load at-or-over core count (no spike headroom); now PSI ~4.5-7.7%.
|
||||
bpg reboots the VM on a cores change → rolled ONE node at a time, waiting Longhorn healthy between. (676c37a)
|
||||
+ node CPU PSI/load alerts + a "Nodes — CPU/Memory Pressure" Grafana dashboard. (a720c73)
|
||||
6. **Security: 5 forward-auth UIs gated** with per-app oauth2-proxy sidecars (Option B, crafty's proven pattern) +
|
||||
per-app Keycloak clients + SOPS creds. Closes the un-gated-admin-UI exposure. (72d4720, b2752ae)
|
||||
7. **Gitea Actions CI finished**: `ci/touched-requirements.sh` now excludes comment-only R-ID prose, so the workflow
|
||||
runs the exact aggregate `make -C ci gate`. (27bc024)
|
||||
8. **external-snapshotter vendored** — was remote github kustomize bases; the kustomize-controller's per-reconcile
|
||||
`git fetch github.com` flaked past its 27s timeout → `BuildFailed` stalled the platform tier. Now local
|
||||
`crds.yaml` + `snapshot-controller.yaml` (offline, deterministic build). (7a79cea)
|
||||
9. **08-30 revalidations closed**: ImageVolume gate is now ON in 1.36 → conformance rationale comments corrected
|
||||
(d60e553); external-snapshotter v8.2.0 pin kept (CRDs served at v1, works on 1.36); Cilium native ExternalAuth
|
||||
evaluated → **DEFER** (see plan).
|
||||
|
||||
### Operational gotchas that STILL matter (for the next PM)
|
||||
- **Access**: `./infra/devcontainer/dev.sh run <cmd>` (kubectl/flux/helm/tofu/ansible/kcadm). Do NOT run `dev.sh`/`make
|
||||
gate` concurrently — execs into the one container interfere → transient failures, just retry. Deploy = `git push
|
||||
gitea main` → Flux. Gate before push: `dev.sh run make -C ci gate` → "MERGE GATE: PASS".
|
||||
- **NAT topology** (see PHASE1-STATUS): API via `192.168.2.50:6443`→VIP .1.49; node SSH `192.168.2.102:225X`;
|
||||
Proxmox API `192.168.2.102:8006`. A single node reboot can briefly drop the kubeconfig NAT — wait ~3 min for boot.
|
||||
- **Keycloak realm import is ONE-SHOT**: the operator logs "Job already executed - not recreating" — a git change to
|
||||
`keycloak-realm-import.yaml` does NOT re-import. Edit the live realm via `kcadm.sh` in keycloak-0 (admin creds:
|
||||
secret `keycloak-admin-credentials`; kcadm present, curl/python3 are NOT), AND fix git for reproducibility.
|
||||
- **Cilium world-hairpin (CVE-2025-30162)**: any in-cluster→`*.home` call hairpins through the Gateway and gets its
|
||||
egress re-checked against the DESTINATION's real cluster identity — `toEntities: world` won't cover it. render.py
|
||||
handles the keycloak case for SSO apps; a NEW app→app-by-hostname dependency needs an explicit `toEndpoints` allow.
|
||||
- **Remote github kustomize bases flake** (27s git-fetch timeout). external-snapshotter is now vendored;
|
||||
**platform/kubevirt still uses remote bases** — vendor it the same way if it `BuildFail`s.
|
||||
- **Longhorn on the HDD**: every node reboot degrades all 18 volumes → full rebuild (slow; 30%+ Proxmox IO-delay).
|
||||
Roll reboots one node at a time; wait `robustness=healthy` before the next. Fix = `docs/plans/longhorn-ssd-tiering.md`.
|
||||
- **Gitea** can go read-only (ext4 remount-ro) during node churn → `kubectl -n gitea rollout restart deploy/gitea`.
|
||||
|
||||
### NEXT STEPS (prioritized) — where to start
|
||||
1. **Longhorn → SSD hot-tier move** — `docs/plans/longhorn-ssd-tiering.md`. The "won't fit" blocker was a measurement
|
||||
artifact (working set ~79 GB; SSD ~323 GB free). Moving the hot tier (Prometheus/CNPG DBs/Loki/Gitea ≈ 51 GB) off
|
||||
the HDD removes the reboot-rebuild tax on every future maintenance op. **Highest-leverage infra item.** Pre-flight:
|
||||
re-verify live SSD free + read HDD SMART.
|
||||
2. **L2/L3 conformance in CI** — `docs/plans/l2-l3-conformance-ci-wiring.md`. Needs the trigger-credential decision
|
||||
(scoped in-cluster SA + Kyverno-pinned Job shape; never a broad kubeconfig in CI). The gate runs no-cluster stages today.
|
||||
3. **Forward-auth long-term** — `docs/plans/cilium-native-externalauth-evaluation.md` (DEFER now: experimental CRD
|
||||
channel + fail-open history). The 5 UIs are safely sidecar-gated in the meantime.
|
||||
4. **Remaining Phase-6 onboarding** — see DESIGN-waves.md / `catalog/` (services rendered-but-not-onboarded).
|
||||
5. **Vendor platform/kubevirt** remote bases proactively (same flake class as #8).
|
||||
6. **Human action**: finish Headlamp login = complete TOTP enrollment in the browser.
|
||||
|
||||
---
|
||||
|
||||
## 2026-08-31 — **PHASE 7 OPENED (prep only)**: first prod node on the bench
|
||||
|
||||
The operator added the first **production** node and opened roadmap Phase 7. This entry records the
|
||||
opening, the state of the hardware, the operator's three decisions, and the blockers that stand
|
||||
between "a node exists" and "prod reconciles from `clusters/prod`".
|
||||
|
||||
### Governance — Phase 7 entry gate, recorded not waived
|
||||
|
||||
`roadmap/phases.md` gates Phase 7 on **two** conditions: *"Phases 1–6 proven in dev/test; operator
|
||||
declares prod hardware ready."* The operator has declared hardware ready. **The first condition is
|
||||
not met** — conformance is 37/51, and the uncovered set includes **R-ENV-01/02/03**, the exact
|
||||
requirements prod provisioning serves. We proceed with *prep only* on that basis, with no prod
|
||||
install until the §7 repo blockers close and the gate is green. This is an accepted, explicit risk,
|
||||
not an oversight.
|
||||
|
||||
### Node 1 as-found (`192.168.2.10`, verified by SSH 2026-08-31)
|
||||
|
||||
Not blank — imaged and reachable. **HP EliteDesk 800 G5 Desktop Mini** (matches the spec's `3× mini-PC
|
||||
bare metal`), i7-9700T 8c/8t, **31 GiB RAM**, Debian 13 trixie / kernel 6.12, hostname already
|
||||
`k3s-prod-1`, `k3s` user reachable with the `home_network` key, VT-x + `/dev/kvm` present
|
||||
(KubeVirt-capable), k3s not installed. Firmware HP R21 Ver. 02.26.00.
|
||||
|
||||
**One 477 GB SATA SSD** (`C400-MTFDDAC512M` — a ~2011-era Micron/Crucial consumer drive) carrying
|
||||
everything: `/` 30G, `/var` 12G, swap, and `/srv` 433G. No second device. Dev, by contrast, runs two
|
||||
dedicated data disks per node (Longhorn + a dedicated SSD for etcd, added after HDD contention
|
||||
starved etcd WAL fsync).
|
||||
|
||||
### Operator decisions (2026-08-31)
|
||||
|
||||
1. **Phase 7 opens, prep work only** — repo work and node baselining now; no k3s install until the
|
||||
parameterization blockers land and the gate is green.
|
||||
2. **Wait for all three nodes before `cluster-init`** — prod comes up as spec-conformant 3-node HA
|
||||
embedded etcd on day one; no single-server-etcd window, no later member-add.
|
||||
3. **Check drive health before choosing the disk layout** — a 15-year-old consumer SSD is not an
|
||||
obvious home for prod state.
|
||||
|
||||
### Findings — prod would build WRONG today
|
||||
|
||||
1. **R-ENV-02 is violated in the base (the headline blocker).** The overlay model promises dev and
|
||||
prod differ *only* by `cluster-vars` values. In fact dev LAN addresses are hardcoded into base
|
||||
manifests: `k8sServiceHost: 192.168.1.50` (`platform/cilium/helmrelease.yaml:50` + the
|
||||
`values.yaml` mirror), the Cilium LB pool `192.168.1.53-.60`
|
||||
(`platform-config/cilium/l2-ip-pool.yaml`), crafty `.55`, the UniFi VM `.54`, and three S3 backup
|
||||
endpoints at `192.168.1.9:9000`. **Prod is on `192.168.2.x` — none of these follow.** Fix is to
|
||||
tokenize each into a `cluster-vars` knob and set it in both overlays; the `postBuild.substituteFrom`
|
||||
seam already exists on every relevant Kustomization. Note `platform/cilium/values.yaml` is only a
|
||||
*reference mirror* — the authoritative pair is the HelmRelease (Flux) and the Ansible j2 template
|
||||
(already parameterized), so the Ansible side needs no change.
|
||||
2. **`clusters/prod` never sets `GATEWAY_VIP`** → it would inherit the base default `10.73.0.240`,
|
||||
an address on the **retired libvirt network**. `cluster-vars`' own comment flags this as owed
|
||||
"when hardware is built".
|
||||
3. **Sudo on node 1 requires a password** — every Ansible playbook fails at its first `become`.
|
||||
**Operator ruling: prod does NOT get NOPASSWD sudo** (my initial recommendation to mirror dev was
|
||||
rejected, correctly — dev's passwordless sudo is a convenience that should not propagate to
|
||||
prod). Instead prod uses Ansible `become` with an **ansible-vault-encrypted**
|
||||
`ansible_become_password` in `inventory/group_vars/k3s_prod/vault.yml`, mirroring the existing
|
||||
`hypervisor` and `proxmox_hosts` groups; the decrypting passphrase stays outside the repo at
|
||||
`~/.config/home-k3s/ansible-vault-pass`. Plays still run unattended, no plaintext secret in git,
|
||||
and prod keeps a real sudo boundary.
|
||||
4. **`node_baseline` device defaults are virtio** (`/dev/vdb`, `/dev/vdc`) and do not exist on bare
|
||||
metal; the role makes a filesystem on a *block device*, so a single-disk node needs either a
|
||||
second drive, `/dev/sda5`, or a repartition. Prod `group_vars` must override.
|
||||
5. **Single upstream resolver on node 1** (`192.168.2.1`) — the same SPOF class that broke
|
||||
containerd pulls and CoreDNS forwarding on dev. The fix is **stranded unmerged** on
|
||||
`fix/node-dns-resilient-upstream`.
|
||||
|
||||
### RESOLVED BY OPERATOR — status comes from the repository, not the task board
|
||||
|
||||
I raised an escalation here: `AGENTS.md` §11 designates `agents/task-board.md` as the one surface
|
||||
agents may edit, yet `ci/spec-immutability.sh` fails the merge gate on **any** spec-repo dirt with no
|
||||
path exception, and committing the board there moves spec `HEAD` off `ci/SPEC_PIN` — which also
|
||||
fails. Editing the board as instructed blocks every merge. That is why the board still reads
|
||||
all-`todo` after six delivered phases.
|
||||
|
||||
**Operator ruling (2026-08-31): the task board is not the status surface. The repository is.**
|
||||
Status of each requirement/spec is to be *derived from the state of the repo* — what is authored,
|
||||
what is reconciled, what has a green requirement-tagged test — not from a hand-maintained board.
|
||||
|
||||
Consequences, adopted:
|
||||
- The spec board is left alone. It is a historical seed, not a live surface. No agent edits it, and
|
||||
no `SPEC_PIN` churn is spent keeping it current.
|
||||
- `ci/spec-immutability.sh` keeps its full strictness — **no path exception was added**. Weakening
|
||||
an R-TEST-06 gate to make a bookkeeping file writable would have been a defect, not a fix.
|
||||
- The **repo-derived spec status report** becomes the artifact of record, alongside the existing
|
||||
requirement-coverage report and this log. Coverage already answers "does a tagged test exist";
|
||||
the status view must also answer "is this requirement actually *implemented and reconciled* in
|
||||
the repo as it stands".
|
||||
|
||||
### Next steps
|
||||
|
||||
1. **Operator:** configure NOPASSWD sudo on `k3s-prod-1`, then capture `smartctl -a /dev/sda`
|
||||
(commands in `infra/PROD-PROVISIONING.md` §5). The disk decision waits on that output.
|
||||
2. **Land the R-ENV-02 tokenization** — the critical path for prod reconciling at all. Dev overlay
|
||||
must set every new knob to today's exact value so live dev sees zero drift (`make -C ci render-golden`).
|
||||
3. **Complete `clusters/prod`** — `GATEWAY_VIP`, the L2 pool, and the prod backup target.
|
||||
4. **Land the two stranded fix branches** (`fix/node-dns-resilient-upstream`,
|
||||
`fix/act-runner-dind-readiness`) — the DNS one now matters for prod too.
|
||||
5. Resolve the task-board escalation above.
|
||||
6. Nodes 2 and 3 through `infra/PROD-PROVISIONING.md` §1–§5 as they arrive.
|
||||
|
||||
---
|
||||
|
||||
## 2026-08-31 — CI is the enforcement point; the LOCAL gate produced a FALSE GREEN
|
||||
|
||||
**Operator directive: "Conformance and gates are enforced at the gitea runner and CI/CD level.
|
||||
Use it."** Adopted. The authority on whether a change is mergeable is the Gitea Actions run, not a
|
||||
local `make -C ci gate`. Local runs are a fast pre-check only.
|
||||
|
||||
This was proven immediately and expensively. Branch `docs/phase-7-open-prod-node-1`:
|
||||
|
||||
- **Local** `make -C ci gate` → `no changed files vs base; empty touched-requirement set` →
|
||||
**MERGE GATE: PASS**.
|
||||
- **CI**, same commit (`a1fb31a`, task 200) → 17 touched requirements correctly computed →
|
||||
`REQUIRE: FAILED — required requirement(s) uncovered: ['R-SEC-01','R-SEC-02','R-TEST-06']` →
|
||||
**MERGE GATE: BLOCKED**.
|
||||
|
||||
The local run resolved an empty diff base and therefore checked nothing, reporting green. **A local
|
||||
false green is the most dangerous failure mode this project has** — it hid real coverage debt from
|
||||
the PM and would have hidden it from any future agent. `ci/touched-requirements.sh` base-ref
|
||||
resolution is being fixed so local agrees with CI, or announces loudly that it cannot determine a
|
||||
base. An honest noisy skip beats a quiet false pass (`ci/README.md` exit-code contract).
|
||||
|
||||
The gate itself was RIGHT: we authored a vaulted-secret flow (R-SEC-01) and status tooling that
|
||||
reasons about spec immutability (R-TEST-06) while those requirements had **no covering test at
|
||||
all**. Tests are being authored rather than the rule being narrowed.
|
||||
|
||||
**How to read CI from a workstation** (the API paths are non-obvious, so recording them):
|
||||
```bash
|
||||
CRED=$(printf 'protocol=https\nhost=gitea.home.keithjackson.dev\n\n' | git credential fill)
|
||||
# list recent runs (id | workflow | branch | status | sha):
|
||||
curl -sk -u "$U:$P" "$B/api/v1/repos/Home-Network/home-network-k3s-impl/actions/tasks?limit=10"
|
||||
# fetch a task's log — NOTE the job-id/task-id OFFSET: task N is at jobs/(N+5)/logs on this instance
|
||||
curl -sk -u "$U:$P" "$B/api/v1/repos/Home-Network/home-network-k3s-impl/actions/jobs/205/logs"
|
||||
```
|
||||
`.../actions/runs/<run>/jobs` is 404 on this Gitea version; use `actions/tasks` + `actions/jobs/<id>/logs`.
|
||||
|
||||
### OPERATOR ATTESTATIONS (2026-08-31) — capability vs. automated proof
|
||||
|
||||
The operator states, and this log records as fact:
|
||||
|
||||
1. **DR is PROVEN** — Velero backups to the Proxmox server, restore demonstrated. R-STATE-03's
|
||||
capability is delivered and in operational use.
|
||||
2. **Cluster resilience is PROVEN at the dev-cluster level** — node loss / rebuild demonstrated.
|
||||
It is slow because Longhorn storage is on HDD (etcd is the exception, on its dedicated SSD),
|
||||
which is a performance property, not a correctness gap.
|
||||
|
||||
**Distinguish capability from automated proof.** The new repo-derived status report flags
|
||||
`R-STATE-03` as a safety-critical requirement whose L3 test body asserts nothing. That finding
|
||||
stands and is NOT contradicted by the attestation above — it is **test debt, not capability debt**.
|
||||
The system does the thing; the suite does not yet assert that it does. Per the project's own
|
||||
definition of done, only a green requirement-tagged test is acceptance — so the remaining work is
|
||||
to make `dr-restore/chainsaw-test.yaml` and the node-loss test actually assert the behaviour the
|
||||
operator has already demonstrated, and let the runner enforce it. Recording the attestation here
|
||||
does not close the requirement, and the gate must not be softened to make it appear closed.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-01 — Phase-7 prep session complete. **PM HANDOFF.**
|
||||
|
||||
Everything below is on branch `docs/phase-7-open-prod-node-1` / **PR #4**, 13 commits, the
|
||||
enforced Gitea gate **green**. Nothing has been installed on prod hardware. `main` is untouched
|
||||
until the operator merges.
|
||||
|
||||
### Delivered
|
||||
1. **Phase 7 opened (prep only)** with the entry-gate shortfall recorded, not waived.
|
||||
2. **T-7.1 checklist** (`infra/PROD-PROVISIONING.md`) written against the real hardware.
|
||||
3. **R-ENV-02 tokenization** — 6 `cluster-vars` knobs; prod would otherwise have come up
|
||||
announcing DEV addresses. Dev proven byte-identical.
|
||||
4. **Prod Ansible**: inventory, vaulted `become` (prod gets NO passwordless sudo), and
|
||||
**target-group parameterisation** so prod reuses the roles unchanged.
|
||||
5. **`node_health` role** — read-only, portable dev+prod, run against both.
|
||||
6. **Repo-derived spec status report** (`run.sh --status`).
|
||||
7. **CI false-green killed** + L0 static tests for R-SEC-01/02 and R-TEST-06.
|
||||
8. **dind readiness fix**; **DNS resilience re-authored** for the current stack.
|
||||
9. **`dr-restore` now asserts** — safety-critical scaffolds 0.
|
||||
|
||||
### Numbers
|
||||
Coverage 37/51 → **40/51**. Executable tests 32 → **36**. Scaffold-only 5 → **4**.
|
||||
Safety-critical without an executable L3 1 → **0**. **PROVEN: 0 — still not derivable**, by
|
||||
design; proof-of-pass is not recoverable from repo state, so a green-run ledger is the only
|
||||
thing that ever closes a requirement.
|
||||
|
||||
### Corrections made during the session (recorded so they are not re-litigated)
|
||||
- **NOPASSWD sudo on prod: REJECTED** by the operator. Prod uses vaulted `become`. My original
|
||||
checklist copied dev's convenience onto prod; that was wrong.
|
||||
- **The "`$i` envsubst bug" was FALSE.** An assessment claimed Flux envsubst ate bare `$i`,
|
||||
making the dind cert-wait bound a no-op. Running the real `flux envsubst` disproved it: bare
|
||||
`$VAR` passes through; only `${VAR}` expands — matching this repo's own contract in
|
||||
`clusters/base/apps-vms.yaml`, itself written after empirical probing. **No escaping was
|
||||
applied**; `$${i}` would have been an active regression (correct only on the envsubst path —
|
||||
on a plain `kustomize build`, `sh` reads `$$` as its PID). Agent reports are not evidence;
|
||||
the tool is.
|
||||
- **The stranded DNS branch was NOT landed.** Its netplan mechanism no longer exists on Debian
|
||||
13; landing it would have broken `node-baseline.yml` on every node.
|
||||
|
||||
### Near-misses worth remembering
|
||||
- A YAML **comment** nearly broke every merge in the repo: Flux envsubst runs over comments too,
|
||||
so a braced name with punctuation is a fatal parse error that fails the whole `apps-wave-1`
|
||||
Kustomization.
|
||||
- `roles/cilium`, `cluster_verify`, `flux_bootstrap` all hardcoded `.cache/dev.kubeconfig` — the
|
||||
file `dev.sh` exports as `$KUBECONFIG`. The moment prod became targetable, a prod run would
|
||||
have silently overwritten it and left the operator's `kubectl` answering from **prod**.
|
||||
- Prod's `GATEWAY_VIP` outside the LB pool would have brought the Gateway up on `.21` while
|
||||
split-horizon DNS answered `.20` — blackholing in-cluster SSO while reporting healthy.
|
||||
|
||||
### BLOCKED ON THE OPERATOR — nothing else moves without these
|
||||
1. **Create `inventory/group_vars/k3s_prod/vault.yml`** (`ansible_become_password`). The
|
||||
passphrase file already exists. This gates SMART, the packages-only baseline, and every
|
||||
future prod play.
|
||||
2. **Drive health decision** — run `smartctl -a /dev/sda` once `smartmontools` lands via
|
||||
`node-baseline.yml -e k3s_hosts=k3s_prod --tags packages`. Prod state on an unmeasured
|
||||
~2011 consumer SSD is the open risk.
|
||||
3. **Prod's second DNS upstream** — `192.168.2.1` is both gateway and only IPv4 resolver.
|
||||
4. **Prod backup target** — `S3_BACKUP_ENDPOINT` is still `TBD-PROD-*`; dev's versitygw is on
|
||||
the far side of the pfSense boundary.
|
||||
5. **`UNIFI_L4_VIP` / `CRAFTY_L4_VIP`** — allocatable from `.20-.30`, not invented here.
|
||||
6. **Merge PR #4.**
|
||||
|
||||
### Known, documented, NOT fixed
|
||||
- `roles/flux_bootstrap` shells to `bootstrap/flux/bootstrap.sh` with a hardcoded
|
||||
`--path=clusters/dev` — retargeting prod needs a `bootstrap/` change.
|
||||
- Prod has no VPN egress group (needs its own vaulted PIA credentials).
|
||||
- **`dr-restore` is AUTHORED, NEVER EXECUTED** (`conformance.home.k3s/never-executed`). It is
|
||||
destructive; converting it to proven needs an operator maintenance-window run. The log must
|
||||
show `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`, **not** `DR_RESTORE_SKIPPED_OK` — green but
|
||||
proving nothing.
|
||||
- `dind` → native sidecar `initContainer` with a `docker info` startupProbe (NOT a bare
|
||||
tcpSocket probe) is the right shape on k3s 1.36; deferred so a merge-gate failure is
|
||||
unambiguous.
|
||||
- **Non-ECC RAM — RISK ACCEPTED by the operator (2026-09-01), not an open item.** `node_health`
|
||||
correctly reports ECC status as unknowable (no EDAC controller bound). Operator rationale:
|
||||
this is a 3-node k3s homelab, not enterprise k8s; corruption on one node is outweighed by the
|
||||
other two and results in node ejection. That holds for the dominant failure mode — etcd raft
|
||||
(per-entry CRC + boltdb page checksums, quorum of 2 survives) and Longhorn (3 replicas, hard
|
||||
anti-affinity, rebuild from healthy peers) both detect a *divergent* node and eject/rebuild it.
|
||||
The residual it does not cover is corruption introduced *upstream* of replication: bytes
|
||||
corrupted before the checksum is computed replicate faithfully to all three replicas, so
|
||||
quorum sees no disagreement. Low probability, and the mitigation (ECC hardware) is out of
|
||||
scope for this platform. **Do not re-litigate.** The one thing that operationalises the
|
||||
operator's own argument is Longhorn's `snapshotDataIntegrity`, which is currently left at the
|
||||
chart default rather than set explicitly — see the follow-up in `platform/longhorn/`.
|
||||
- L2/L3 conformance still SKIPs in CI (`docs/plans/l2-l3-conformance-ci-wiring.md`); it needs
|
||||
the trigger-credential decision — a scoped in-cluster SA, never a broad kubeconfig in CI.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-01 — INCIDENT: merging PR #4 briefly broke Cilium on 2/3 dev nodes. **Root cause: a substitution race that fails OPEN.**
|
||||
|
||||
Merging the R-ENV-02 tokenization (PR #4, `dce0e8f`) triggered a Cilium Helm upgrade that put
|
||||
`cilium` agents into `Init:Error` on `k3s-dev-1` and `k3s-dev-2`. It **self-healed in ~6 minutes**
|
||||
and the cluster is fully green (12/12 Kustomizations Ready, 0 HelmReleases unready, 0 pods down).
|
||||
Recording it because the mechanism is systemic, will recur, and is far more dangerous on a cold
|
||||
prod bootstrap than it was here.
|
||||
|
||||
### What happened
|
||||
|
||||
The HelmRelease config digest tells the whole story:
|
||||
|
||||
```
|
||||
v8 05:49:39 digest e83bf695… <- before
|
||||
v9 05:50:11 digest eebd6c88… <- the bad render
|
||||
v10 05:55:23 digest e83bf695… <- identical to v8 again
|
||||
```
|
||||
|
||||
The config changed and then changed **back to byte-identical**. Nothing in git differed
|
||||
semantically — `git diff` over `platform/cilium/helmrelease.yaml` shows only comments plus
|
||||
`k8sServiceHost: "192.168.1.50"` → `k8sServiceHost: "${K3S_API_HOST}"`, and the dev overlay sets
|
||||
`K3S_API_HOST=192.168.1.50`. The pre-merge byte-identity verification was **correct**; the steady
|
||||
state is identical. The damage happened in a transient intermediate state.
|
||||
|
||||
### Root cause — the var and its consumer are reconciled by DIFFERENT Kustomizations
|
||||
|
||||
- `cluster-vars` (which *defines* `K3S_API_HOST`) is owned by the **`flux-system`** Kustomization.
|
||||
- `platform/cilium/helmrelease.yaml` (which *references* it) is owned by the **`platform`**
|
||||
Kustomization, via `postBuild.substituteFrom: cluster-vars`.
|
||||
- There is **no dependency ordering between defining a variable and using it.**
|
||||
|
||||
So there is a window where `platform` renders the NEW manifest against the OLD `cluster-vars`,
|
||||
which has no `K3S_API_HOST` key yet. **Flux's post-build substitution is non-strict by default: an
|
||||
undefined variable becomes the EMPTY STRING, silently.** That produced `k8sServiceHost: ""`, Helm
|
||||
upgraded Cilium with it, and the agents fell back to the in-cluster ClusterIP `10.43.0.1:443` —
|
||||
which only the Cilium agent itself can translate. The agent was the thing restarting, so its own
|
||||
`config` init container could not reach the API server:
|
||||
|
||||
```
|
||||
level=error msg="Unable to contact k8s api-server" ipAddr=https://10.43.0.1:443
|
||||
error="dial tcp 10.43.0.1:443: i/o timeout"
|
||||
level=fatal msg="Build config failed"
|
||||
```
|
||||
|
||||
Once `flux-system` applied the updated `cluster-vars`, the next `platform` reconcile rendered the
|
||||
correct value, digest returned to `e83bf695…`, and all three agents came up.
|
||||
|
||||
`optional: false` on the `substituteFrom` entry does **not** protect against this — it only requires
|
||||
the ConfigMap to *exist*, not that every referenced key is defined.
|
||||
|
||||
### Why this matters much more for prod
|
||||
|
||||
On dev, one Cilium agent (`k3s-dev-3`) stayed up throughout, the API server was reachable, and the
|
||||
race closed on its own. **On a cold prod bootstrap every one of these knobs is new simultaneously**,
|
||||
there is no already-running CNI to fall back on, and an empty `k8sServiceHost` means Cilium may
|
||||
never come up at all — with no healthy peer to recover from. This is a plausible
|
||||
bring-up-blocker for `clusters/prod`.
|
||||
|
||||
### The fix — make substitution FAIL CLOSED
|
||||
|
||||
`kustomize-controller` currently runs with **no** `--feature-gates` argument, i.e. non-strict
|
||||
substitution. Flux supports `StrictPostBuildSubstitutions`, which makes an **undefined variable fail
|
||||
the Kustomization** instead of silently emptying it. That is exactly this project's stated doctrine
|
||||
("if unsure, deny/close"; "never fake green") applied to the substitution layer, and it converts
|
||||
this entire bug class from *silently corrupt a critical value* into *a loud, blocking reconcile
|
||||
failure*.
|
||||
|
||||
**Not enabled unilaterally** — turning it on cluster-wide would immediately fail any Kustomization
|
||||
that references a variable not defined in `cluster-vars`, including legitimate shell `${…}` in
|
||||
embedded scripts. The safe sequence, before flipping it:
|
||||
1. Enumerate every `${VAR}` in every substituted path and prove each is either defined in
|
||||
`cluster-vars` or correctly escaped as `$${VAR}`.
|
||||
2. Then enable the gate, and let CI + a dev reconcile prove it.
|
||||
|
||||
Interim discipline, independent of the gate: **add a `cluster-vars` knob in one commit and
|
||||
reference it in a later one**, so the definition is always reconciled before its first use.
|
||||
|
||||
### Operational note worth keeping
|
||||
|
||||
A Cilium rolling upgrade on this cluster is not risk-free even when the values are right: the
|
||||
`config` init container resolves the API server through the ClusterIP unless `k8sServiceHost` is
|
||||
set, and the agent that translates that ClusterIP is the one being replaced. Keep
|
||||
`k8sServiceHost`/`k8sServicePort` correct at all times, and roll Cilium one node at a time.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-01 — RESEARCH: Flux substitution + HelmRelease best practice. **Measured semantics + ranked plan.**
|
||||
|
||||
Three parallel research passes (substitution/ordering, HelmRelease/CNI, and a full repo audit),
|
||||
prompted by the incident above. Two of my own earlier conclusions were WRONG and are corrected
|
||||
here. Everything below marked *measured* was run against flux 2.4.0 / the live cluster; everything
|
||||
else is cited inference.
|
||||
|
||||
### THE SUBSTITUTION SEMANTICS TABLE (measured — trust this over prose elsewhere)
|
||||
|
||||
| form | behaviour |
|
||||
|---|---|
|
||||
| `${DEFINED}` | substituted |
|
||||
| `${UNDEFINED}` | **silently empty** (non-strict) · fails loudly (strict) |
|
||||
| `${UNDEFINED:-d}` | **default applied at reconcile time — `--strict` does NOT catch it** |
|
||||
| `${VAR:?msg}` | renders **the message as the value** — a non-empty WRONG value. ANTI-PATTERN. |
|
||||
| `${VAR:=192.168.1.50}` | silently ships the dev value to prod. ANTI-PATTERN. |
|
||||
| `$${VAR}` / `$${VAR:-d}` / `$${ARR[0]}` | literal — **escaping is TOTAL** |
|
||||
| `$VAR` / `$(cmd)` | untouched, no escaping needed |
|
||||
| `${...}` / `${a b}` / `${ARR[0]}` unescaped | **hard parse error — fails the whole Kustomization** |
|
||||
|
||||
Flux exempts default-providing operators (`-`, `:-`, `+`, `:+`, `=`, `:=`, `?`, `:?`) from strict
|
||||
mode by design ("they intentionally handle the unset case"). **That is why strict mode would never
|
||||
have caught the gitea/nextcloud dead-guard bug**, and why a targeted check was required.
|
||||
|
||||
**Doctrine corrected:** `clusters/base/apps-vms.yaml` claimed an indexed expansion is fatal
|
||||
"escaped or not". Measured: escaping is total; only the UNESCAPED form is fatal. Fixed in place.
|
||||
|
||||
**Comment nuance:** `kustomize build` strips *structural* YAML comments before envsubst sees them,
|
||||
so those are usually harmless. But `#` lines INSIDE a block scalar are data and DO reach envsubst,
|
||||
and a `$`-carrying body comment was observed surviving into built `apps/wave-1`. Do not rely on
|
||||
stripping — escape in comments too. (An earlier claim in this log that a YAML comment "nearly broke
|
||||
every merge" was overstated for the structural case; the hazard is real for block-scalar bodies.)
|
||||
|
||||
### CORRECTION 1 (mine) — `StrictPostBuildSubstitutions` alone is INSUFFICIENT
|
||||
|
||||
It fixes nothing about ordering; it converts "render a corrupt value and ship it" into "fail loudly
|
||||
and keep last-good". **The actual ordering fix is one line**, verified against
|
||||
`kustomization_controller.go :: checkDependencies` at our exact version (v1.4.0): `dependsOn` is
|
||||
**revision-gated**, not merely Ready-gated — when the dependency shares a GitRepository, the
|
||||
dependent cannot reconcile revision N until the dependency has APPLIED revision N.
|
||||
|
||||
`platform` is the only Kustomization in the graph with no `dependsOn`; every other consumer chains
|
||||
transitively through it. So:
|
||||
|
||||
```yaml
|
||||
# clusters/base/platform.yaml
|
||||
spec:
|
||||
dependsOn:
|
||||
- name: flux-system # cluster-vars must be applied at THIS revision first
|
||||
```
|
||||
|
||||
**Cost, recorded honestly:** the root becomes a gate — a malformed manifest anywhere under
|
||||
`clusters/dev` now stops the whole graph where today it would not. Fail-closed, consistent with
|
||||
doctrine, but a real blast-radius change. No deadlock risk: the root runs `wait: false`.
|
||||
|
||||
Rejected alternative: moving `cluster-vars` into the consuming Kustomization is **strictly worse** —
|
||||
`SubstituteVariables` reads the ConfigMap from the API at build time, *before* the Kustomization
|
||||
applies its own resources, so it would always read the PREVIOUS value. That converts an occasional
|
||||
race into a permanent one-reconcile lag.
|
||||
|
||||
### CORRECTION 2 (an agent's) — the safety machinery DID fire
|
||||
|
||||
A research pass concluded that Helm considered the bad release healthy (because
|
||||
`updateStrategy.rollingUpdate.maxUnavailable: 2` on 3 nodes means `expectedReady = 1`), and
|
||||
therefore remediation never engaged. **Measured on the live cluster: that is wrong.** The Helm
|
||||
release secrets show **v9 = `failed`**; Helm's wait DID fail and the release was remediated, which
|
||||
is why it self-healed. `maxUnavailable: 2 of desired 3` IS confirmed live — so the real harm was
|
||||
blast radius (2 of 3 agents down at once), not a missed detection.
|
||||
|
||||
### Verified, and worth knowing
|
||||
|
||||
- **Flux 2.4.0 is OUT OF SUPPORT** — the project supports the last three minors; 2.9.x is current.
|
||||
- **Strict substitution became DEFAULT-ON** in kustomize-controller v1.9.0. So upgrading Flux flips
|
||||
this behaviour whether or not we opt in — the variable audit is a prerequisite for the upgrade
|
||||
regardless of today's decision.
|
||||
- **Cilium's `values.schema.json` permits `k8sServiceHost: ""`** — no `minLength`, no `pattern`.
|
||||
Helm was never going to reject the bad render.
|
||||
- **Cilium docs make `k8sServiceHost` REQUIRED under kube-proxy replacement**, precisely because the
|
||||
environment exports the API ClusterIP and there is no kube-proxy to provision it.
|
||||
- **The Cilium chart ships zero test templates**, so `spec.test` is useless for it. A post-upgrade
|
||||
check must live outside Helm.
|
||||
- `valuesFrom` fails HARD on a missing key (verified in `chartutil/values.go`) unlike postBuild's
|
||||
silent empty — but only on a MISSING key, not an empty value, and it cannot cross namespaces
|
||||
(`cluster-vars` is in `flux-system`; the Cilium HelmRelease is in `kube-system`).
|
||||
- **Do NOT enable `driftDetection.mode: enabled` on a CNI.** It compares the cluster against the
|
||||
Helm storage manifest — during the incident that manifest WAS the bad render, so drift correction
|
||||
would have actively re-asserted the broken value.
|
||||
|
||||
### RANKED PLAN
|
||||
|
||||
| P | Action | State |
|
||||
|---|---|---|
|
||||
| **P0** | `platform.dependsOn: [flux-system]` — closes the race completely | **PENDING — operator call** (blast-radius change) |
|
||||
| **P0** | Escape the dead OIDC guards; renderer now refuses the class | ✅ DONE |
|
||||
| **P1** | Substitution-safety L0 gate, unconditional in `ci/l0.sh` | ✅ DONE — 102 assertions |
|
||||
| **P1** | Enable `StrictPostBuildSubstitutions=true` | **READY — proven safe**, all 11 paths exit 0 under `flux envsubst --strict`. Sequence AFTER P0. |
|
||||
| **P2** | `updateStrategy.rollingUpdate.maxUnavailable: 1` on Cilium, as a new **[parity]** field in BOTH the HelmRelease and the Ansible j2 | PENDING — bounds blast radius to one node |
|
||||
| **P3** | Split Cilium into its own Kustomization (`wait: true`, `interval: 1h`, `timeout: 10m`); `platform` dependsOn it | PENDING — gives `flux suspend kustomization cilium` as an emergency valve |
|
||||
| **P4** | Upgrade Flux 2.4.0 → 2.9.x | PENDING — out of support; brings strict-by-default |
|
||||
| **P5** | Cilium MINOR bumps become manual (Renovate PR → human runs the documented pre-flight → merge). Never automerge a CNI minor. | PENDING — process |
|
||||
| — | Record `${VAR:?}` / `${VAR:=}` as anti-patterns | ✅ DONE (table above) |
|
||||
| — | *Not recommended*: drift detection on Cilium; `spec.test` on Cilium; `k8sServiceHost: auto` | — |
|
||||
|
||||
### Follow-ups surfaced, not yet actioned
|
||||
- 6 dead `cluster-vars` knobs; `BASE_DOMAIN` and `CLUSTER_ENV` look like genuine rot.
|
||||
- `platform/{cilium,longhorn,velero,oauth2-proxy}/values.yaml` reference mirrors carry decorative
|
||||
`${...}` that can silently drift from the real HelmRelease.
|
||||
- `platform` Kustomization `timeout: 5m` vs Longhorn HelmRelease `timeout: 15m` — the tier can time
|
||||
out while a legitimate Longhorn upgrade is still running.
|
||||
|
||||
---
|
||||
|
||||
## 2026-09-01 — P0/P1/P2 LANDED + a second incident: two bugs stacked, the outer one hid the inner
|
||||
|
||||
All three approved changes are merged (PR #5) and reconciled on dev, plus a hotfix (PR #6).
|
||||
|
||||
### Landed and verified on the live cluster
|
||||
|
||||
| Change | Verified |
|
||||
|---|---|
|
||||
| **P0** `platform.dependsOn: [flux-system]` | Flux on `63002d1`; all 12 Kustomizations Ready; steady-state graph unchanged |
|
||||
| **P1** strict substitution | Committed to `bootstrap/flux/components/`. **NOT YET ROLLED OUT** — see below |
|
||||
| **P2** Cilium `maxUnavailable: 1` | Applied. **Pod ages unchanged (15h/39h), restart counts unchanged** — the clean proof nothing rolled, exactly as predicted |
|
||||
| OIDC guard fix | Deployed guards now read `${..._OIDC_CLIENT_SECRET:-}` instead of `""`; both providers registered |
|
||||
| Prod overlay | SOPS decryption (same age key as dev) + source on `main` |
|
||||
|
||||
### THE SECOND INCIDENT — nextcloud CrashLoopBackOff
|
||||
|
||||
Immediately after the PR #5 reconcile, `nextcloud` went `CrashLoopBackOff` (7 restarts,
|
||||
`FailedPostStartHook`) and `apps-wave-1`/`apps-wave-2` stalled. Cause:
|
||||
|
||||
```sh
|
||||
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries" # LAST line of the hook
|
||||
```
|
||||
|
||||
On the **success** path (`p < 15`) the test returns 1, and as the script's last command that became
|
||||
the hook's exit status. **A non-zero postStart exit makes the kubelet kill the container.**
|
||||
|
||||
**The previous commit did not introduce this — it UNCOVERED it.** The bug was unreachable for as
|
||||
long as the OIDC guard above it was dead code that `exit 0`'d first. Fixing the guard made the block
|
||||
execute for the first time in these apps' deployed lifetime, and the latent defect fired instantly.
|
||||
**Two bugs stacked, and the outer one hid the inner one.** Worth remembering as a pattern: fixing
|
||||
dead code is not a safe no-op — it is the first execution of never-tested code, and should be
|
||||
treated with the caution of new code, not of a one-character escape.
|
||||
|
||||
Fixed by converting to a plain `if` and adding an explicit `exit 0`. Gitea's hook already returned 0
|
||||
on both paths, but only by accident of which statement happened to be last — made explicit there too
|
||||
so the class cannot return. Recovery confirmed: both pods `1/1 Running`, 0 restarts, all
|
||||
Kustomizations Ready, both OIDC providers still registered.
|
||||
|
||||
### STILL TO DO — strict substitution is committed but NOT ACTIVE
|
||||
|
||||
`StrictPostBuildSubstitutions` does **not** arrive on a git push. This repo does not use
|
||||
`flux bootstrap`; the controllers were installed imperatively by `flux install` and carry no Flux
|
||||
ownership labels, so they are outside Flux's inventory and nothing under `clusters/**` can reach a
|
||||
controller flag. An operator must run `bootstrap/flux/bootstrap.sh` step 1 (idempotent; the
|
||||
standalone command is in `bootstrap/flux/README.md`), which restarts kustomize-controller for
|
||||
~10-20s. Flux is level-triggered, so an interrupted reconcile is simply retried.
|
||||
|
||||
**Ordering is satisfied:** P0 is merged and reconciled, so the gate can now be rolled out safely.
|
||||
With the gate on but the ordering edge absent, the same race would turn a transient empty render
|
||||
into a transient hard Kustomization failure.
|
||||
|
||||
### Prod consequences recorded this session
|
||||
|
||||
- **Every merge to `main` will reconcile BOTH clusters** once prod is live — no promotion step, no
|
||||
soak window. A deliberate consequence of dev being temporary, but it removes the "prove it on dev
|
||||
first" safety net that the last two days have leaned on heavily, twice.
|
||||
- **Gitea migration is the FIRST task of the dev decommission, not the last.** Prod's GitOps source
|
||||
currently points at Gitea running on the dev cluster.
|
||||
- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
|
||||
with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
|
||||
Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
|
||||
@@ -89,16 +89,7 @@ spec:
|
||||
# userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
|
||||
# signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
|
||||
# Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
|
||||
# NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
|
||||
# ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
|
||||
# Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
|
||||
# as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
|
||||
# by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
|
||||
# an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
|
||||
# and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
|
||||
# flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
|
||||
# text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
|
||||
# wording in clusters/base/apps-vms.yaml overstated it.
|
||||
# NOTE: apps-vms has NO postBuild.substituteFrom, so shell ${...} below is NOT envsubst'd.
|
||||
userData: |-
|
||||
#cloud-config
|
||||
hostname: unifi
|
||||
@@ -112,11 +103,11 @@ spec:
|
||||
content: |
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
if [ -z "$${UOS_URL:-}" ]; then
|
||||
if [ -z "${UOS_URL:-}" ]; then
|
||||
UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
|
||||
fi
|
||||
[ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
|
||||
t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
|
||||
[ -n "${UOS_URL}" ] && [ "${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
|
||||
t="$(mktemp)"; echo "downloading ${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
|
||||
runcmd:
|
||||
- [ systemctl, enable, --now, qemu-guest-agent ]
|
||||
- [ systemctl, enable, --now, podman.socket ]
|
||||
|
||||
@@ -8,9 +8,8 @@
|
||||
# CANNOT express: raw TCP on non-HTTP ports (8080 inform, 6789), UDP at all (3478 STUN, 10001
|
||||
# discovery), or the device-facing protocols that never speak HTTP. UniFi device ADOPTION rides
|
||||
# exactly those L4 ports. So the controller is exposed as a type=LoadBalancer Service whose VIP
|
||||
# is pulled from the CiliumLoadBalancerIPPool `default-pool` (dev 192.168.1.53-.60 — the
|
||||
# CILIUM_LB_POOL_START/STOP cluster-vars knobs) and ARP-announced on the flat LAN by the existing
|
||||
# CiliumL2AnnouncementPolicy `default-l2` — the SAME L2/LB-IPAM
|
||||
# is pulled from the CiliumLoadBalancerIPPool `default-pool` (192.168.1.53-.60) and ARP-announced
|
||||
# on the flat LAN by the existing CiliumL2AnnouncementPolicy `default-l2` — the SAME L2/LB-IPAM
|
||||
# machinery the Gateway ingress VIP already uses (platform-config/cilium/*). No new Cilium config
|
||||
# is required; these Services just consume the pool.
|
||||
#
|
||||
@@ -24,11 +23,8 @@
|
||||
# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
|
||||
# VIP pinned to the node running the VM (also what L2 announcement does).
|
||||
#
|
||||
# VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
|
||||
# default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
|
||||
# Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for
|
||||
# exactly this token; every shell $${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
|
||||
# accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool.
|
||||
# FLAG: confirm 192.168.1.54 is free, inside default-pool, and reserved in tofu (the Gateway
|
||||
# already holds one pool IP, typically .53). Adjust if it collides.
|
||||
#
|
||||
# REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an
|
||||
# L4-only workload as a Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing
|
||||
@@ -43,8 +39,8 @@ metadata:
|
||||
app.kubernetes.io/name: unifi
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "unifi"
|
||||
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
||||
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
|
||||
lbipam.cilium.io/ips: "192.168.1.54"
|
||||
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> 192.168.1.54.
|
||||
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
|
||||
external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
|
||||
spec:
|
||||
@@ -69,7 +65,7 @@ metadata:
|
||||
app.kubernetes.io/name: unifi
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "unifi" # same key + same IP => shares the VIP with -tcp
|
||||
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
||||
lbipam.cilium.io/ips: "192.168.1.54"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
@@ -127,8 +127,8 @@ spec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1;\
|
||||
\ break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
|
||||
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name\
|
||||
\ groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n"
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
@@ -185,10 +185,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute gitea-3000 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: jackett-oauth2-proxy
|
||||
namespace: jackett
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:n5h6NG0emA==,iv:d6lvZfgRq6+IKNpsSoRa5a38R2f2+tWBO+j9QKDThMY=,tag:P+cvQXlWR4aZs7a9/lVuMQ==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:WL5G7PuVTdHQoe7Zp2eiE9ppxSGE0+W2RMfISuwqD6JA9auG97ZYnPUjG8HNx7sosOvsmqeE9kONK23IqIeaQw==,iv:kSP10UKeDv/8BsgX36+uKwEY+LBovcicce1RkD14M4g=,tag:fEUsVuajWPxAmAo8Nnrrrg==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:WChW9YLhzG9Nwp8dCWTFtirufXxGk3Ep9HXpfqcNz69PXQW3RaMC4EJpw2U=,iv:f6obIerkog8Q2gfj1tVL259LOl5jj59VE29KnJwmCXQ=,tag:s0FwNtU6vffKgDcGUNUhCg==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLWVptVHc4aFl0MXRWTHVP
|
||||
cC9qcVoyZUV2TGZnTEpHY3RLOU8vQnNNNm00ClZRejE2bzFaWXNSUmQ5MEZtV3VK
|
||||
djB2TXJqQjVUOE93QjN6U0lMVDBEbm8KLS0tIFFTZnB0c2pCZWZMZ084WEIxT2tW
|
||||
WUxxemJHdWRZbnBpL280dzlpMmhwSW8Kra0bkx3iJYQi+3rnpU/89fJbJHzQjKi2
|
||||
ZWNwYhLpULw1xNo3xbCm+PUp5f30kpyMKQld2qaLOZAqWI5vNl9Byw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:41:37Z"
|
||||
mac: ENC[AES256_GCM,data:zU9nWCrSomAXAZ/Ve8U3yLHnoxhu8nkQ7523neJTGv810ONzAAZ7Gr8nFHR7bTqz2QwuGYD3P1ypdqYnKLM8ri2Zf1LDo/Su9ptnvQhUbfXKlRorcW6uAWsViCwvTM05wZNkdzZfza7tGwJQVTD6hc0cP+/7Y7JnpT5FEstK254=,iv:oggTRjcWqAJizA/a22+mtslkEWka8hUG0wudM8obO4g=,tag:i0TtNmqGe+b5t4iq8LpsqA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --upstream=http://127.0.0.1:9117
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://jackett.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: jackett-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: jackett-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: jackett-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-9117
|
||||
port: 9117
|
||||
targetPort: 9117
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): jackett-allow-radarr-9117
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -233,7 +166,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '9117'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
|
||||
@@ -251,10 +184,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute jackett-9117 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -281,7 +210,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: jackett
|
||||
port: 4180
|
||||
port: 9117
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
|
||||
@@ -8,15 +8,12 @@ kind: Kustomization
|
||||
resources:
|
||||
- gitea-bootstrap-secret.sops.yaml
|
||||
- gitea.yaml
|
||||
- jackett-oauth2-proxy-credentials.sops.yaml
|
||||
- jackett.yaml
|
||||
- nextcloud-db-app.sops.yaml
|
||||
- nextcloud-db-backup-credentials.sops.yaml
|
||||
- nextcloud-db.yaml
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
- pihole.yaml
|
||||
- plex.yaml
|
||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||
- qbittorrent.yaml
|
||||
- semaphore-db-app.sops.yaml
|
||||
- semaphore-db-backup-credentials.sops.yaml
|
||||
|
||||
@@ -74,13 +74,11 @@ spec:
|
||||
# (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP.
|
||||
# Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials
|
||||
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
|
||||
# endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
|
||||
# substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
|
||||
backup:
|
||||
retentionPolicy: "30d"
|
||||
barmanObjectStore:
|
||||
destinationPath: s3://cnpg
|
||||
endpointURL: ${S3_BACKUP_ENDPOINT}
|
||||
endpointURL: http://192.168.1.9:9000
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: nextcloud-db-backup-credentials
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: pihole-oauth2-proxy
|
||||
namespace: pihole
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:C6IXgQxi,iv:A9bWygwabOKIRi55jV9+TRU7eiZels3vqqLDTZ469aA=,tag:hC7sn/Xr4vQFkRjDLMgEmQ==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:50HEfMG+CoWXokS+IxQgCPQEt+EBCUontOc0GrTyauHuV/U7w9IPo83iJECO0QhTPiQIgD3j+F+YXfubCbTnuw==,iv:ejQxin0NMs0Ye0fhqfPuh8JmRrEimDrSAcZj28uS+GM=,tag:FVassLZPZMY9ry48ZjBgmw==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:nfTEg5DlIXCLI/hhHngyqLwsQh2SgXU21LaSkVnr91sTi4wODrEH4C7wvPY=,iv:WWXnHnFP2IqLrw1yXAlD0Sw+Vc1m4rxXIHT1u2UWH3o=,tag:pYZEuwPYH+b8S4kGtaWqEA==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuTnJiWTVPR01aY2h3dkJI
|
||||
OTV0WTFQeHoyUGluVzV4NFpSZjd6bjlHZkE0ClZmTGgzQWo4VTROaUg5Wkd1TTVV
|
||||
cGRjZjJuMitPZC9OaVZUZzRGeEJmemcKLS0tIGRjUFd3SzFzeFBBQVRacjZWQmNH
|
||||
ZjllZTQxbzRWQXNiVllkc1hESGN4alEK/hFOtchsYMx09/WE5uaCe6JetUurzhhQ
|
||||
7imb8RIjh0cfP4PDT3dZhhPF+ehcV7ttslFkcyaqlG1F8Gk2jtN0pw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:33:15Z"
|
||||
mac: ENC[AES256_GCM,data:Tse6fLhc08DxCAuZW5KSZ3M5OQUTlbeHnWiO3jIROUVLrBfbXCuDNWnQyOPjwVPqyzeAzC3APEKFzQ2S8rYjRyO08SJ4ZFPPUeRblcXgXHZ4Y/E4Renq7axk61VvFGtZteHWuQz9ss//Qt+KwPMDBGTivLCoVTOh2K01AXPFXdA=,iv:fyhkDLVN28umXDR4/f4ZzQuu3OG6QBkpUgf3E9rknKk=,tag:iNqor6cD9tJ6MsqAmmRP0Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+2
-73
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --upstream=http://127.0.0.1:80
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://pihole.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: pihole-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: pihole-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: pihole-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-80
|
||||
port: 80
|
||||
targetPort: 80
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): pihole-allow-gateway-ingress
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -189,7 +122,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '80'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
|
||||
@@ -207,10 +140,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute pihole-80 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -237,7 +166,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: pihole
|
||||
port: 4180
|
||||
port: 80
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
namespace: qbittorrent
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:ytfVRDabEfRM8NI=,iv:ERtFy5iss10id/cGLMZDmIAazkYtFG/spkxRYkoT2pw=,tag:FzOepKeYT05BBrz6RlnACQ==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:klZzTdyAqQgG+1s074/0Bh5ybuHehiEo/RYceRGAGwuUrgCMb0AmYHCrE9gvtnPkSOBqWvFnPnorWkERc+V73g==,iv:nOUO/p+WnI5657I3Lm6zQAsumkscjF7J0fK0EVFHGo0=,tag:dF3AHYP4sxDChroRer9kMQ==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:jVpsZjYHRrPvblfsb2f9OuneqoZ0cwcTLKrPb8XJ5oves1PmHtN5njrkk4o=,iv:bNjUK/ar4nniR7AjJ9fsGVkKd8nxwphkptmhFaYh9Ek=,tag:2axf+AshSdIvvqwyIfhq6g==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAreWdPdUxiYXBxYWxZNlhy
|
||||
WHFTZmo1RW1McXJJTjlWeTBCd2Y4WVU2TVRNCnExbEQrQXZCS2RRaGxJQlQ5eHpY
|
||||
Z1UrZWlubUJpRC9YeEhJZW8zNVdLUVUKLS0tIGtmblFDRzk3ZC8wWVo2ZHJHWmEx
|
||||
OW9DRUpaSlFrbHNDTGYvQUdyamh2bWsK5VvV4cl1iNXYECnSk3fi1ZuAPKGN+n8B
|
||||
g51lT4gi03KA67ARzwVGVBfP5E1Cdmx4rJhi305tG+Ug99VyLBCUfQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:42:02Z"
|
||||
mac: ENC[AES256_GCM,data:7GdlUKnZ/ST7EOhfPB6I2e2wyVHWmZ2G554yBcZQzqoZZWZ1hn+lkosNJxWD7E0hRId6P320EKj+Sf0g2ur3Mna5zii1GKFJHGtDEoccKvJQRx2JMkZlEC/x6xwc4BeJNtvidECVRvKC2WdrByRMr9rrI8NfXpmWH4e7iz1Xac0=,iv:UqsNOPRwzT+Dt043ldxIubkAAkPr0/nvjxlgVPiXIbw=,tag:7Wex5Rng3Nwtr8wxkFJD9Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --upstream=http://127.0.0.1:8080
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://qbittorrent.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-8080
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): qbittorrent-allow-radarr-8080
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -233,7 +166,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '8080'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
|
||||
@@ -251,10 +184,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute qbittorrent-8080 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -281,7 +210,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: qbittorrent
|
||||
port: 4180
|
||||
port: 8080
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
|
||||
@@ -75,13 +75,11 @@ spec:
|
||||
# (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
|
||||
# Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
|
||||
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
|
||||
# endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
|
||||
# substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
|
||||
backup:
|
||||
retentionPolicy: "30d"
|
||||
barmanObjectStore:
|
||||
destinationPath: s3://cnpg
|
||||
endpointURL: ${S3_BACKUP_ENDPOINT}
|
||||
endpointURL: http://192.168.1.9:9000
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: semaphore-db-backup-credentials
|
||||
|
||||
@@ -103,65 +103,21 @@ spec:
|
||||
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
|
||||
- name: runner
|
||||
image: gitea/act_runner:0.2.11
|
||||
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
|
||||
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
|
||||
# boot and runs `act_runner daemon`.
|
||||
#
|
||||
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
|
||||
# and hard-exits if it is missing (a startup race against the sidecar).
|
||||
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
|
||||
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
|
||||
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
|
||||
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
|
||||
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
|
||||
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
|
||||
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
|
||||
#
|
||||
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
|
||||
# braced form or the double-dollar form; both are wrong here, for different reasons.
|
||||
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
|
||||
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
|
||||
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
|
||||
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
|
||||
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
|
||||
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
|
||||
# cluster verbatim and both 60s bounds below are real, enforced bounds.
|
||||
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
|
||||
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
|
||||
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
|
||||
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
|
||||
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
|
||||
# form — still wrong for us, because on any path where envsubst does NOT run
|
||||
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
|
||||
# reads that leading double-dollar as its own PID, so a bound would compare a
|
||||
# string like 1234{i} against 60. The bare form is the only spelling that is
|
||||
# correct on BOTH paths.
|
||||
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
|
||||
# name") that fails this whole Kustomization — and this Kustomization gates every
|
||||
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
|
||||
# comment. That is why this note names the three forms in words rather than
|
||||
# showing them: an earlier draft of it spelled them out literally and broke
|
||||
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
|
||||
# clusters/base/apps-vms.yaml describes.)
|
||||
# Wait for the DinD sidecar to publish its TLS client certs into the shared
|
||||
# emptyDir before starting the daemon — act_runner reads /certs/client/ca.pem at
|
||||
# startup and hard-exits if it is missing (a startup race against the sidecar).
|
||||
# Then exec the image's normal entrypoint (/sbin/tini -- /opt/act/run.sh), which
|
||||
# registers-from-env on first boot and runs `act_runner daemon`.
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
|
||||
i=0
|
||||
until [ -f /certs/client/ca.pem ]; do
|
||||
i=$((i + 1))
|
||||
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
|
||||
echo "waiting for dind TLS certs ($i)"; sleep 1
|
||||
done
|
||||
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
|
||||
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
|
||||
j=0
|
||||
until nc -z -w1 127.0.0.1 2376; do
|
||||
j=$((j + 1))
|
||||
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
|
||||
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
|
||||
done
|
||||
exec /sbin/tini -- /opt/act/run.sh
|
||||
env:
|
||||
- name: GITEA_INSTANCE_URL
|
||||
@@ -199,31 +155,6 @@ spec:
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
|
||||
#
|
||||
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
|
||||
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
|
||||
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
|
||||
# ordering that the loops above hand-roll — which also removes the envsubst hazard
|
||||
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
|
||||
# Shape to build:
|
||||
# 1. Move this whole container into `initContainers:` as the FIRST entry.
|
||||
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
|
||||
# initContainer a sidecar: it starts before the regular containers, keeps
|
||||
# running alongside them, and is terminated after them.
|
||||
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
|
||||
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
|
||||
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
|
||||
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
|
||||
# it would re-open the same "port accepting but TLS/certs not ready" gap that
|
||||
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
|
||||
# preserves today's 60s cap.
|
||||
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
|
||||
# `runner` can drop its `command:` override completely and run the image's
|
||||
# default entrypoint.
|
||||
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
|
||||
# entire repository. If the conversion is wrong, nothing can merge — so it must land
|
||||
# alone, where the failure is unambiguous and not entangled with anything else, and
|
||||
# only after a real workflow run has been observed green on the converted pod.
|
||||
- name: dind
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
|
||||
@@ -10,9 +10,7 @@ resources:
|
||||
- gitea-actions.yaml
|
||||
- nextcloud-bootstrap-secret.sops.yaml
|
||||
- nextcloud.yaml
|
||||
- radarr-oauth2-proxy-credentials.sops.yaml
|
||||
- radarr.yaml
|
||||
- semaphore-bootstrap-secret.sops.yaml
|
||||
- semaphore.yaml
|
||||
- sonarr-oauth2-proxy-credentials.sops.yaml
|
||||
- sonarr.yaml
|
||||
@@ -125,9 +125,8 @@ spec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"\
|
||||
[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"\
|
||||
$ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
|
||||
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra\
|
||||
\ positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n"
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
@@ -206,10 +205,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute nextcloud-80 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: radarr-oauth2-proxy
|
||||
namespace: radarr
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:vFLxqzxt,iv:6k0ZjKP8rvg56ZFPWEGdyj+FLgK+LBOE0pLBWfC0bjA=,tag:LiZb5BasJAEWM1c3I8vFTA==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:JzfHF8JF4YUxBxiC738N4UsMlcCOvxn4/AE2B4meZNGAZb2uhG4n51uEY8OQsbsM0Ni4qM2Wi17KuLDwAicfUA==,iv:A3AnUhYxw/IWfvfcZf6MgC5u37dMoNToIWxRG9AxEJQ=,tag:1OY2pWGTbCxPYYdaiBa/oA==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:LPt5nghLfoNvr7yBes8rLeEK3M6UVIO1h9POHMOAj8iZX3l2RJJa4w8GcSg=,iv:Q5+7zH8Tb1SWe92B0pxekfMgOXlBG8nixWn/ho4oJw4=,tag:D+41dBFPVTGj86xv5/GEng==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBHVG5MWVRyOVl1RWFpOExC
|
||||
SjZKcEd1ZlFnZVZ5NmtyUlQzRGI1b1o3dG04CkwrTUNjd3crM1VzaUdPczliSDY1
|
||||
a3dONnBFWk8yOW5kL2xES1RZbUN6L3cKLS0tIG9hWG9Cbm01RlMrcC9zMWkxeFgr
|
||||
ZnRGRUVTTS81SFRyenNsZUZEcG1TUm8K5XUGP/1NVsoaFcBZUER3/zMPTtCuVisT
|
||||
LrbSmTkIyVfzjN/vRpRH92b1VzENJvhGhRghhQaKY0rbG34i6yh3Vw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:42:25Z"
|
||||
mac: ENC[AES256_GCM,data:TxdPz2zE6+ypgQG2b0Rp3VFiPL7LLls4NNkC7YQ/bVCLvyvaXdGsEWkiHJE+Po7hcESmZRt/NMS25mEXPMfvlSCltwFUTcrxoZDS0NI27kIzbtceTbxmznt/dMS/iZm21lCgA6Aag9ByLjWusPLkRKWhep+G+0G54qtfQzv3s80=,iv:44f0KikrpKpnSaLLyzTvnCfrwEXylRgVIlKMtSpMyNc=,tag:jDx43n0usDtbwzo8KzDBqw==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+2
-73
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --upstream=http://127.0.0.1:7878
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://radarr.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: radarr-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: radarr-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: radarr-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-7878
|
||||
port: 7878
|
||||
targetPort: 7878
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy egress allow (R-NET-02): radarr-allow-qbittorrent-8080
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -255,7 +188,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '7878'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
|
||||
@@ -273,10 +206,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute radarr-7878 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -303,7 +232,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: radarr
|
||||
port: 4180
|
||||
port: 7878
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
|
||||
@@ -209,10 +209,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute semaphore-3000 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: sonarr-oauth2-proxy
|
||||
namespace: sonarr
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:JMYoJymH,iv:CM9HF71QZCeETzrOtkeTUs+BLTp/gGSExulXUP8yS/M=,tag:l73RaFQolO7+wZBo8nYCvw==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:Fq271KdbtKWKWOe1Sun6+SLv/kSBsvAeFUhZIq6J2DNPGM4E8dgc2YANITcpezcJR07PAbSlUcS0L/qF6K+YhQ==,iv:/mVD1uOkweyO26RMV3qRQmLdyxXJR4I7p+OjYiiuSMU=,tag:9ZOKye5Sp2NHU2RXH1Y+wQ==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:jMZZEi8fdh20BbPzCoVfwdS1g0/ma5n/QISU5A/gQCu/2Rkylt7mp798CUg=,iv:6IPqDNQyJHze2DUOQ93IwruWSge5nXjkVcQIOMfJSwY=,tag:cOKRcDKywV3PBOBrhoddRQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoNU5NbnJnV0dKRnpIN2dw
|
||||
QmhBRWI4aEpWajRHVkxZcGxCRk1hMDBjSlRVCm91b0U2N08vNi8rSEFaVlRJazFL
|
||||
bllPcm9EMUgrQ3ZlUDlrdEZpZE8zNmMKLS0tIDU0ajdycXZrazlQb3RVRGQyY25h
|
||||
R3RDajltcmFvQWdLYWQ2Mjh0VkplaWsKJprnLNJYzMd0yHbxjsD8P66Wu+sefmaV
|
||||
nduEgVQY0t4EsYN6iKVDLMUAp+o3gXU8QuMuBnFZznrmBghfG+ZruQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:42:49Z"
|
||||
mac: ENC[AES256_GCM,data:ckVR9OyFvle0fLhBjHtJ6hH2St2CRJeu6abTCnqtxAfMDVXdbIkgHEesmN2kAgA7s3LkKJC2m7EFIpt3pO94VQNnZvC1KKD7qUSIfJN1DLVv78m8JbEQfVTJk/WgWy1ObxLriw5hDuMiEuOvKND+s6eqS5j6er4Q72lBsBpoEeY=,iv:uguThOQbR9Ks1qs1U0HO7EhUj1BT60fvM+3xVEC9hUU=,tag:m4P5nhlspCZyJRuOqffMog==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+2
-73
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --upstream=http://127.0.0.1:8989
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-8989
|
||||
port: 8989
|
||||
targetPort: 8989
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy egress allow (R-NET-02): sonarr-allow-qbittorrent-8080
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -255,7 +188,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '8989'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
|
||||
@@ -273,10 +206,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute sonarr-8989 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -303,7 +232,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: sonarr
|
||||
port: 4180
|
||||
port: 8989
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
|
||||
@@ -11,76 +11,16 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux
|
||||
## Files
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
|
||||
| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
|
||||
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
|
||||
| `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
|
||||
| `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
|
||||
| `patches/apps-sops-decryption.yaml` | Same, for the `apps` Kustomization. |
|
||||
| `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |
|
||||
|
||||
## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster
|
||||
|
||||
Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
|
||||
and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):
|
||||
|
||||
```
|
||||
$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
|
||||
{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
|
||||
"app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
|
||||
# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1
|
||||
```
|
||||
|
||||
No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under
|
||||
`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux
|
||||
documents for hand-installed clusters ([Customize Flux
|
||||
manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)):
|
||||
|
||||
```
|
||||
flux install --export … -> <tmp>/gotk-components.yaml # regenerated from the pinned flux CLI
|
||||
components/kustomization.yaml -> <tmp>/kustomization.yaml # the only hand-authored part (in git)
|
||||
kubectl kustomize <tmp> | kubectl apply -f -
|
||||
```
|
||||
|
||||
Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION`
|
||||
(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations,
|
||||
more feature gates) is another entry in `components/kustomization.yaml`.
|
||||
|
||||
### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)
|
||||
|
||||
> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the
|
||||
> controller Deployment is outside Flux's inventory. Someone must run step 1.
|
||||
|
||||
Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1:
|
||||
|
||||
```bash
|
||||
# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
|
||||
d=$(mktemp -d)
|
||||
flux install --export --namespace=flux-system \
|
||||
--components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
|
||||
cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
|
||||
kubectl kustomize "$d" | kubectl apply -f - # add --server-side --force-conflicts on a field-manager conflict
|
||||
kubectl -n flux-system rollout status deploy/kustomize-controller
|
||||
```
|
||||
|
||||
**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any
|
||||
reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing
|
||||
is lost. Confirm afterwards:
|
||||
|
||||
```bash
|
||||
kubectl -n flux-system get deploy kustomize-controller \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].args}' # must contain --feature-gates=StrictPostBuildSubstitutions=true
|
||||
flux get kustomizations -A # all 11 must return to Ready
|
||||
```
|
||||
|
||||
**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
|
||||
simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
|
||||
|
||||
## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
|
||||
|
||||
The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):
|
||||
`flux-system.yaml` is the `flux-system` **GitRepository**; `platform.yaml`, `platform-config.yaml`
|
||||
and the app-wave Kustomizations (`apps-wave-0/1/2.yaml` + `apps-vms.yaml` — the single `apps.yaml`
|
||||
was retired) are the Flux **Kustomizations**; `clusters/dev` is the overlay that patches
|
||||
`cluster-vars` (R-ENV-02) and layers SOPS decryption onto each Kustomization by name.
|
||||
`flux-system.yaml` is the `flux-system` **GitRepository**; `platform.yaml`/`apps.yaml` are the Flux
|
||||
**Kustomizations**; `clusters/dev` is the overlay that patches `cluster-vars` (R-ENV-02).
|
||||
|
||||
`flux bootstrap git` **generates and owns its own** `gotk-sync` GitRepository + Kustomization named
|
||||
`flux-system` and commits them into the repo — which would **duplicate/conflict** with the authored
|
||||
@@ -101,9 +41,8 @@ tofu apply → cloud-init → Ansible (k3s + Cilium) [Infra worker, T-1
|
||||
3. bootstrap/flux/bootstrap.sh # flux install → verify sops-age → apply -k clusters/dev
|
||||
│
|
||||
▼
|
||||
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
|
||||
→ Kustomization(apps-wave-0 → apps-wave-1 → apps-wave-2) + Kustomization(apps-vms),
|
||||
with per-Kustomization SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).
|
||||
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(apps),
|
||||
with SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).
|
||||
```
|
||||
|
||||
## Interface note — for the Infra worker (T-1.3 Ansible)
|
||||
@@ -133,8 +72,7 @@ patches:
|
||||
- path: ../../bootstrap/flux/patches/apps-sops-decryption.yaml # SOPS on apps
|
||||
```
|
||||
|
||||
Alternatively the base TODOs in `clusters/base/platform.yaml` and each app-wave Kustomization
|
||||
(`apps-wave-0/1/2.yaml`, `apps-vms.yaml`) and the `url` in
|
||||
Alternatively the base TODOs in `clusters/base/{platform,apps}.yaml` and the `url` in
|
||||
`clusters/base/flux-system.yaml` can be filled in directly (same effect) — the clusters owner's call.
|
||||
Either way the `sops-age` `secretRef` name and `flux-system` source name are kept in lockstep with
|
||||
the authored base.
|
||||
|
||||
@@ -25,9 +25,7 @@
|
||||
# We do NOT use `flux bootstrap git` in Phase 1 because that command commits
|
||||
# and OWNS its own gotk-sync GitRepository+Kustomization named `flux-system`.
|
||||
# Our entrypoints are ALREADY authored in clusters/base (flux-system.yaml =
|
||||
# the GitRepository; platform.yaml + platform-config.yaml + the app-wave
|
||||
# Kustomizations apps-wave-0/1/2.yaml + apps-vms.yaml = the Kustomizations;
|
||||
# the single apps.yaml was retired). Letting
|
||||
# the GitRepository; platform.yaml/apps.yaml = the Kustomizations). Letting
|
||||
# bootstrap regenerate them would DUPLICATE/conflict. `flux install` +
|
||||
# applying the authored clusters/dev overlay is the reconciling path.
|
||||
# 2. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
|
||||
@@ -35,9 +33,8 @@
|
||||
# only VERIFIES it exists; it never handles key material.
|
||||
# 3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
|
||||
# GitRepository `flux-system` (source) -> Kustomization `platform` ->
|
||||
# `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
|
||||
# (chained dependsOn) + `apps-vms`, with cluster-vars overlay
|
||||
# (R-ENV-02) and per-Kustomization SOPS decryption enabled via ./patches/.
|
||||
# Kustomization `apps` (dependsOn platform), with cluster-vars overlay
|
||||
# (R-ENV-02) and SOPS decryption enabled via the patches in ./patches/.
|
||||
#
|
||||
# PREREQUISITES the operator/PM resolves before this runs (see ../README.md "Handoff"):
|
||||
# - clusters/base/flux-system.yaml url placeholder is overridden — wire ./patches/flux-system-
|
||||
@@ -69,59 +66,16 @@ for bin in flux kubectl; do
|
||||
command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
|
||||
done
|
||||
|
||||
# --- 1. install Flux controllers (rendered through ./components) --------------
|
||||
# --- 1. install Flux controllers ---------------------------------------------
|
||||
# Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates,
|
||||
# ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the
|
||||
# image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly.
|
||||
#
|
||||
# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently:
|
||||
# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined
|
||||
# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see
|
||||
# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md).
|
||||
#
|
||||
# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the
|
||||
# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux
|
||||
# documents for hand-installed clusters ("Customize Flux manifests"):
|
||||
#
|
||||
# flux install --export > gotk-components.yaml # exact manifests `flux install` would apply
|
||||
# kustomize overlay = ./components/kustomization.yaml (checked in — the ONLY hand-authored part)
|
||||
# kubectl apply -k
|
||||
#
|
||||
# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION,
|
||||
# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is
|
||||
# a temp dir — the repo tree is never written to.
|
||||
#
|
||||
# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
|
||||
# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
|
||||
echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
|
||||
flux_install_args=(--namespace="${FLUX_NAMESPACE}")
|
||||
[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
|
||||
|
||||
if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then
|
||||
echo " WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED."
|
||||
echo " StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN."
|
||||
flux install "${flux_install_args[@]}"
|
||||
if [ -n "${FLUX_COMPONENTS_EXTRA}" ]; then
|
||||
echo "==> [1/3] flux install (default controllers + extra: ${FLUX_COMPONENTS_EXTRA})"
|
||||
flux install --namespace="${FLUX_NAMESPACE}" --components-extra="${FLUX_COMPONENTS_EXTRA}"
|
||||
else
|
||||
components_dir="${script_dir}/components"
|
||||
[ -f "${components_dir}/kustomization.yaml" ] || {
|
||||
echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2
|
||||
echo " (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2
|
||||
exit 1
|
||||
}
|
||||
render_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "${render_dir}"' EXIT
|
||||
echo " rendering flux install --export through ${components_dir}"
|
||||
flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml"
|
||||
cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml"
|
||||
# Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux
|
||||
# version bump) — better a hard stop than silently installing without the feature gate.
|
||||
kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml"
|
||||
grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || {
|
||||
echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2
|
||||
echo " Check ./components/kustomization.yaml against this flux CLI version." >&2
|
||||
exit 1
|
||||
}
|
||||
kubectl apply -f "${render_dir}/rendered.yaml"
|
||||
echo "==> [1/3] flux install (default controllers only)"
|
||||
flux install --namespace="${FLUX_NAMESPACE}"
|
||||
fi
|
||||
|
||||
# --- 2. verify the out-of-band age key Secret exists -------------------------
|
||||
|
||||
@@ -1,64 +0,0 @@
|
||||
---
|
||||
# Controller-level customization of the Flux CONTROLLERS themselves (the `flux-system` Deployments).
|
||||
#
|
||||
# WHY THIS DIRECTORY EXISTS
|
||||
# =========================
|
||||
# This repo does NOT use `flux bootstrap` (see ../README.md "Why `flux install` + apply"): the
|
||||
# reconciliation graph is authored by hand in clusters/base, and the controllers are installed
|
||||
# imperatively by `flux install`. Consequence, verified on the live dev cluster 2026-09-01:
|
||||
#
|
||||
# kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
|
||||
# -> NO kustomize.toolkit.fluxcd.io/name label, deployment.kubernetes.io/revision: 1
|
||||
#
|
||||
# i.e. **the controllers are not managed by Flux and are not in git.** There was therefore no
|
||||
# existing mechanism to customize them, and no amount of editing under clusters/** can reach them.
|
||||
#
|
||||
# This is the mechanism Flux documents for exactly that situation ("Customize Flux manifests" —
|
||||
# https://fluxcd.io/flux/installation/#customize-flux-manifests): export the component manifests
|
||||
# with `flux install --export`, overlay them with kustomize `patches:`, and apply the result.
|
||||
# `../bootstrap.sh` step 1 does that render — it copies THIS file next to a freshly exported
|
||||
# `gotk-components.yaml` in a temp dir and runs `kubectl apply -k`. Nothing is generated into the
|
||||
# repo, so the manifests can never drift from the pinned FLUX_VERSION
|
||||
# (infra/devcontainer/versions.env) the way a vendored copy would.
|
||||
#
|
||||
# Every future controller customization (vertical scaling, tolerations, extra feature gates,
|
||||
# --concurrent) belongs HERE, as another entry under `patches:`.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
# Written by `bootstrap.sh` from `flux install --export ...` at render time. NOT checked in.
|
||||
- gotk-components.yaml
|
||||
patches:
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# StrictPostBuildSubstitutions — make postBuild substitution FAIL CLOSED.
|
||||
#
|
||||
# Default behaviour is fail-OPEN: a `${VAR}` that is not defined in `cluster-vars` renders as the
|
||||
# EMPTY STRING, silently. On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium
|
||||
# HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry). With this
|
||||
# gate the undefined variable FAILS the Kustomization instead, and Flux keeps the last-good state.
|
||||
#
|
||||
# PROVEN SAFE before enabling: every one of the 11 substituted paths was rendered with
|
||||
# `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst --strict` and
|
||||
# exited 0, with a negative control reproducing the incident. The permanent guard that keeps it
|
||||
# true is the L0 gate conformance/tests/l0-static/flux-postbuild-substitution/.
|
||||
#
|
||||
# NOTE the gate does NOT catch default-providing forms (`${VAR:-d}`, `${VAR:?msg}`, `${VAR:=v}`) —
|
||||
# Flux exempts them by design. That class is what the L0 gate covers; the two are complementary.
|
||||
#
|
||||
# This also pre-pays for the Flux 2.4.0 -> 2.9.x upgrade: strict substitution became DEFAULT-ON in
|
||||
# kustomize-controller v1.9.0, so the upgrade flips this behaviour whether or not we opt in.
|
||||
#
|
||||
# JSON-6902 `add ... /args/-` (append), NOT a strategic merge: a container's `args` is an atomic
|
||||
# list, so a strategic-merge patch would REPLACE every arg `flux install` generates
|
||||
# (--events-addr, --watch-all-namespaces, --log-level, --log-encoding, --enable-leader-election)
|
||||
# and silently drop any the CLI adds in a future version.
|
||||
- target:
|
||||
group: apps
|
||||
version: v1
|
||||
kind: Deployment
|
||||
name: kustomize-controller
|
||||
namespace: flux-system
|
||||
patch: |-
|
||||
- op: add
|
||||
path: /spec/template/spec/containers/0/args/-
|
||||
value: --feature-gates=StrictPostBuildSubstitutions=true
|
||||
+1
-11
@@ -10,7 +10,7 @@ policy, egress, TLS, DNS, monitoring, and update policy from it (see
|
||||
[`docs/13-service-catalog.md`](../../home-network-k3s/docs/13-service-catalog.md) (14-service table +
|
||||
SSO/egress/dependency intent).
|
||||
|
||||
## Current examples
|
||||
## Current examples (Phase 0)
|
||||
| File | docs/13 row | exposure | egress | auth | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| `pihole.yaml` | 6 | internal | internet | forward-auth | admin UI behind oauth2-proxy; config on Longhorn (DR-critical); also platform DNS |
|
||||
@@ -28,16 +28,6 @@ inherit every guarantee automatically.
|
||||
`auth.groups`; `native`/`none` are documented exceptions only.
|
||||
- Irreplaceable state (`class: longhorn` configs/DBs) must set `backup.policy` to an off-cluster
|
||||
schedule — never `reacquirable` (R-STATE-03). Only bulk media may be `reacquirable`.
|
||||
- **Shell braces must be escaped `$${...}`.** Anything rendered verbatim onto the workload — a
|
||||
`lifecycle` postStart script, an `env` value — lands under `apps/wave-<N>/`, and every
|
||||
`apps-wave-<N>` Flux Kustomization declares `postBuild.substituteFrom: cluster-vars`. Flux runs
|
||||
envsubst over the WHOLE manifest at reconcile time, so an unescaped `${VAR:-}` meant for the
|
||||
*shell* is replaced with the empty string before the container ever starts (this silently killed
|
||||
gitea's and nextcloud's OIDC registration for days). Bare `$VAR` and `$(cmd)` are left alone and
|
||||
need no escaping; a genuine environment knob is declared in `clusters/base/cluster-vars.yaml` and
|
||||
written bare `${KNOB}`. Full contract: [`../clusters/base/apps-vms.yaml`](../clusters/base/apps-vms.yaml).
|
||||
`render.py` REFUSES to emit a manifest that breaks this rule (Flux's *strict* substitution mode
|
||||
cannot: it deliberately exempts the `${V:-}` / `${V:=x}` / `${V:?}` default-operator forms).
|
||||
|
||||
## Validate before commit — single command, no cluster (R-TEST-02)
|
||||
```bash
|
||||
|
||||
+1
-19
@@ -109,19 +109,7 @@ lifecycle:
|
||||
echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
|
||||
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
|
||||
}
|
||||
# NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild.
|
||||
# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
|
||||
# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not
|
||||
# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE
|
||||
# — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on
|
||||
# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was
|
||||
# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.
|
||||
# Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design.
|
||||
# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.
|
||||
# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the
|
||||
# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.
|
||||
# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
|
||||
if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
|
||||
else
|
||||
# (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
|
||||
@@ -162,12 +150,6 @@ lifecycle:
|
||||
gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
|
||||
fi
|
||||
fi
|
||||
# A postStart hook MUST exit 0 — a non-zero exit makes the kubelet KILL the container
|
||||
# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but
|
||||
# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to
|
||||
# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
|
||||
# the moment its OIDC guard was fixed and the block actually ran for the first time.
|
||||
exit 0
|
||||
exposure: public # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
||||
storage:
|
||||
|
||||
+2
-24
@@ -111,19 +111,7 @@ lifecycle:
|
||||
# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.
|
||||
OCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \
|
||||
|| echo "[nextcloud-oidc] WARN: failed to set allow_local_remote_servers" >&2
|
||||
# NOTE the `$$` escape below — it is load-bearing. apps-wave-1 declares postBuild.
|
||||
# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
|
||||
# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is
|
||||
# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS
|
||||
# TRUE — so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never
|
||||
# installing/enabling user_oidc nor registering the provider. SSO kept working only because
|
||||
# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with
|
||||
# NO OIDC at all. Flux STRICT substitution cannot catch this — it EXEMPTS default-operator
|
||||
# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion
|
||||
# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why
|
||||
# `--clientsecret-env` and the bare uses below always worked and only the guard broke.
|
||||
# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
|
||||
if [ -z "$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
echo "[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)"
|
||||
exit 0
|
||||
fi
|
||||
@@ -203,17 +191,7 @@ lifecycle:
|
||||
[ "$missing" = 0 ] && break
|
||||
p=$((p + 1)); sleep 2
|
||||
done
|
||||
if [ "$p" -ge 15 ]; then
|
||||
nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
|
||||
fi
|
||||
# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container
|
||||
# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:
|
||||
# the final statement used to be `[ "$p" -ge 15 ] && nextcloud_oidc_fail ...`, so on the
|
||||
# SUCCESS path the test returned 1 and became the script's exit status. That was latent for
|
||||
# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard
|
||||
# exposed it immediately. Every failure path here logs loudly and writes a marker file and
|
||||
# deliberately does NOT crash the pod — the break-glass local admin must stay usable.
|
||||
exit 0
|
||||
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
|
||||
exposure: public # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks
|
||||
storage:
|
||||
|
||||
+19
-469
@@ -29,7 +29,6 @@ conformance harness stays stdlib-only and only shells out to this script + diffs
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
@@ -53,20 +52,6 @@ KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery
|
||||
EGRESS_CLASSES = ("internet", "vpn")
|
||||
AUTH_MODES = ("oidc", "forward-auth", "native", "none")
|
||||
|
||||
# --- forward-auth (Option B) per-app oauth2-proxy sidecar ------------------------------
|
||||
# A `forward-auth` app has a bare web UI with no usable native OIDC (PiHole, *arr, Jackett,
|
||||
# qBittorrent). Cilium 1.16.x cannot inject ext_authz into the shared Gateway's Envoy config
|
||||
# (docs/plans/cilium-native-externalauth-evaluation.md; platform/oauth2-proxy/README.md), so
|
||||
# the platform's forward-auth CEC is an UNWIRED template. Instead each forward-auth app is
|
||||
# gated by a PER-POD oauth2-proxy REVERSE-PROXY sidecar (Option B) — EXACTLY the pattern
|
||||
# platform/crafty proves working: the shared Gateway routes <app>.home -> the sidecar (:4180),
|
||||
# the sidecar authenticates the browser against Keycloak (homelab realm, the app's auth.groups)
|
||||
# and only THEN proxies to the co-located app UI at http://127.0.0.1:<uiport>. The gate is IN
|
||||
# the data path (fail-closed: no valid session in an authorized group => no upstream; R-IDP-02).
|
||||
OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
|
||||
OAUTH2_PROXY_PORT = 4180
|
||||
KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
|
||||
@@ -267,13 +252,6 @@ def r_deployment(s: dict) -> dict:
|
||||
|
||||
pod_spec = {"containers": [container]}
|
||||
|
||||
# forward-auth (Option B): append the per-pod oauth2-proxy reverse-proxy sidecar AFTER the app
|
||||
# container (app stays containers[0]). The shared Gateway routes to the sidecar (:4180); it gates
|
||||
# the co-located UI against Keycloak and only then proxies to http://127.0.0.1:<uiport>. `oidc`
|
||||
# apps (native client) and `native`/`none` render byte-identical (no sidecar).
|
||||
if _needs_oauth2_sidecar(s):
|
||||
pod_spec["containers"].append(_oauth2_sidecar_container(s))
|
||||
|
||||
# podSecurityContext (schema-optional): the k8s PodSecurityContext is emitted VERBATIM onto the
|
||||
# pod spec. Its primary use is `fsGroup`, which makes a mounted Longhorn PVC group-writable by a
|
||||
# non-root image (e.g. semaphore runs as UID 1001 and cannot otherwise write its config PVC).
|
||||
@@ -316,14 +294,6 @@ def r_service(s: dict):
|
||||
ports = _service_ports(s)
|
||||
if not ports:
|
||||
return None
|
||||
# forward-auth (Option B): the shared Gateway must reach the oauth2-proxy sidecar, so front it
|
||||
# with a `authproxy` :4180 Service port (the HTTPRoute backendRef targets this). The app UI/API
|
||||
# port(s) stay in the Service for EAST-WEST callers (e.g. arr -> qbittorrent:8080 / jackett:9117);
|
||||
# north-south to those app ports is denied at the Gateway (see r_gateway_ingress_policy).
|
||||
if _needs_oauth2_sidecar(s):
|
||||
ports = ports + [
|
||||
{"name": "authproxy", "port": OAUTH2_PROXY_PORT, "targetPort": OAUTH2_PROXY_PORT}
|
||||
]
|
||||
return {
|
||||
"apiVersion": "v1",
|
||||
"kind": "Service",
|
||||
@@ -494,7 +464,7 @@ def r_gateway_ingress_policy(s: dict):
|
||||
north-south). This CiliumNetworkPolicy is the ADDITIVE, least-privilege allow that opens
|
||||
ONLY the north-south path (docs/09 "Ingress opens north-south only — it does not relax
|
||||
east-west default-deny"): ingress from Cilium's reserved `ingress` identity (the Gateway/
|
||||
Ingress Envoy proxy, `reserved:ingress`, numeric identity 8 in Cilium 1.20.1) to ONLY the
|
||||
Ingress Envoy proxy, `reserved:ingress`, numeric identity 8 in Cilium 1.16.5) to ONLY the
|
||||
exposed service port(s). Everything else stays denied; undeclared ports stay denied.
|
||||
|
||||
Verified empirically against the cluster: with default-deny in place, Hubble shows the
|
||||
@@ -504,15 +474,7 @@ def r_gateway_ingress_policy(s: dict):
|
||||
"""
|
||||
if not s["ingressRoutes"]:
|
||||
return None
|
||||
# forward-auth (Option B): the ONLY north-south path is the oauth2-proxy sidecar. Open the
|
||||
# Gateway to :4180 ONLY — never the app UI port — so the un-gated UI is unreachable from the
|
||||
# Gateway (fail-closed; R-IDP-02). Every other app opens its exposed service port(s) as before.
|
||||
if _needs_oauth2_sidecar(s):
|
||||
allow_ports = [{"port": str(OAUTH2_PROXY_PORT), "protocol": "TCP"}]
|
||||
else:
|
||||
allow_ports = [
|
||||
{"port": str(p["port"]), "protocol": "TCP"} for p in _service_ports(s)
|
||||
]
|
||||
ports = _service_ports(s)
|
||||
return {
|
||||
"apiVersion": "cilium.io/v2",
|
||||
"kind": "CiliumNetworkPolicy",
|
||||
@@ -528,7 +490,14 @@ def r_gateway_ingress_policy(s: dict):
|
||||
"ingress": [
|
||||
{
|
||||
"fromEntities": ["ingress"],
|
||||
"toPorts": [{"ports": allow_ports}],
|
||||
"toPorts": [
|
||||
{
|
||||
"ports": [
|
||||
{"port": str(p["port"]), "protocol": "TCP"}
|
||||
for p in ports
|
||||
]
|
||||
}
|
||||
],
|
||||
}
|
||||
],
|
||||
},
|
||||
@@ -558,28 +527,6 @@ def r_internet_egress(s: dict) -> dict:
|
||||
switch (a cluster-wide singleton keyed on the egress-class pod label) decide the PATH; this
|
||||
policy decides that north-south egress is ALLOWED at all.
|
||||
"""
|
||||
egress = [{"toEntities": ["world"]}]
|
||||
# SSO apps (oidc/forward-auth) reach Keycloak at keycloak.home, which coredns split-horizon
|
||||
# resolves to the shared Gateway VIP (an L2-announced LoadBalancer IP Cilium classifies as
|
||||
# `world`), so the pod hairpins THROUGH the Gateway. Under Cilium >=1.16.8/1.17.2 (the
|
||||
# CVE-2025-30162 fix, present in 1.20), the Gateway's L7 proxy re-evaluates THIS pod's egress
|
||||
# policy against Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) — `world`
|
||||
# matches the VIP at L3/L4 but NOT the resolved keycloak identity at L7, so OIDC discovery/
|
||||
# token 403s ("Access denied") without an explicit allow to the keycloak backend. Scope it to
|
||||
# the apps that actually talk to Keycloak (auth mode oidc or forward-auth).
|
||||
if s.get("auth", {}).get("mode") in ("oidc", "forward-auth"):
|
||||
egress.append(
|
||||
{
|
||||
"toEndpoints": [
|
||||
{
|
||||
"matchLabels": {
|
||||
"io.kubernetes.pod.namespace": "keycloak",
|
||||
"app": "keycloak",
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
return {
|
||||
"apiVersion": "cilium.io/v2",
|
||||
"kind": "CiliumNetworkPolicy",
|
||||
@@ -590,7 +537,7 @@ def r_internet_egress(s: dict) -> dict:
|
||||
},
|
||||
"spec": {
|
||||
"endpointSelector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
|
||||
"egress": egress,
|
||||
"egress": [{"toEntities": ["world"]}],
|
||||
},
|
||||
}
|
||||
|
||||
@@ -603,10 +550,6 @@ def r_httproutes(s: dict) -> list:
|
||||
"""
|
||||
out = []
|
||||
for route in s["ingressRoutes"]:
|
||||
# forward-auth (Option B): the Gateway backendRef targets the oauth2-proxy sidecar (:4180),
|
||||
# NOT the app UI port — so the gate is in the data path and the bare UI is never served
|
||||
# un-authenticated. Other apps route straight to their exposed app port.
|
||||
backend_port = OAUTH2_PROXY_PORT if _needs_oauth2_sidecar(s) else route["port"]
|
||||
out.append(
|
||||
{
|
||||
"apiVersion": "gateway.networking.k8s.io/v1",
|
||||
@@ -627,7 +570,7 @@ def r_httproutes(s: dict) -> list:
|
||||
{"path": {"type": "PathPrefix", "value": "/"}}
|
||||
],
|
||||
"backendRefs": [
|
||||
{"name": s["name"], "port": backend_port}
|
||||
{"name": s["name"], "port": route["port"]}
|
||||
],
|
||||
}
|
||||
],
|
||||
@@ -637,94 +580,6 @@ def r_httproutes(s: dict) -> list:
|
||||
return out
|
||||
|
||||
|
||||
def _needs_oauth2_sidecar(s: dict) -> bool:
|
||||
"""True iff this service gets a per-pod oauth2-proxy reverse-proxy sidecar (Option B).
|
||||
|
||||
A `forward-auth` app with an exposed UI has no usable native OIDC, so its Gateway route is
|
||||
gated IN the data path by a co-located oauth2-proxy sidecar (crafty's proven pattern). `oidc`
|
||||
apps authenticate themselves (native Keycloak client) and `native`/`none` are unchanged."""
|
||||
return s["auth"]["mode"] == "forward-auth" and bool(s["ingressRoutes"])
|
||||
|
||||
|
||||
def _oauth2_sidecar_container(s: dict) -> dict:
|
||||
"""The oauth2-proxy REVERSE-PROXY sidecar (Option B) that gates a forward-auth app's UI.
|
||||
|
||||
Mirrors platform/crafty/deployment.yaml's `oauth2-proxy` container EXACTLY, adapted per app:
|
||||
the upstream is the co-located app UI over plain HTTP (http://127.0.0.1:<uiport>) — the *arr /
|
||||
PiHole / Jackett / qBittorrent UIs are HTTP, so (unlike crafty's self-signed HTTPS :8443) there
|
||||
is NO --ssl-upstream-insecure-skip-verify. --allowed-group is emitted once per declared
|
||||
auth.group (R-IDP-06: a valid session is not enough — the user must be in an authorized group,
|
||||
else /oauth2 returns 403). client-id/secret + cookie-secret come by name from the SOPS-encrypted
|
||||
`<app>-oauth2-proxy` Secret (R-IDP-04). Single provider => --skip-provider-button 302s straight
|
||||
to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy."""
|
||||
route = s["ingressRoutes"][0]
|
||||
host = route["host"]
|
||||
upstream_port = route["port"]
|
||||
args = [
|
||||
f"--http-address=0.0.0.0:{OAUTH2_PROXY_PORT}",
|
||||
"--reverse-proxy=true",
|
||||
"--provider=oidc",
|
||||
f"--oidc-issuer-url={KEYCLOAK_ISSUER}",
|
||||
f"--upstream=http://127.0.0.1:{upstream_port}",
|
||||
"--email-domain=*",
|
||||
"--scope=openid email profile",
|
||||
"--oidc-groups-claim=groups",
|
||||
]
|
||||
# R-IDP-06: authorize by group. One flag per declared group (fail-closed to 403 otherwise).
|
||||
for group in s["auth"].get("groups", []):
|
||||
args.append(f"--allowed-group={group}")
|
||||
args += [
|
||||
"--skip-provider-button=true",
|
||||
f"--redirect-url=https://{host}/oauth2/callback",
|
||||
f"--cookie-domain=.{ZONE}",
|
||||
f"--whitelist-domain=.{ZONE}",
|
||||
"--cookie-secure=true",
|
||||
"--pass-access-token=true",
|
||||
"--set-xauthrequest=true",
|
||||
]
|
||||
secret = f"{s['name']}-oauth2-proxy"
|
||||
return {
|
||||
"name": "oauth2-proxy",
|
||||
"image": OAUTH2_PROXY_IMAGE,
|
||||
"imagePullPolicy": "IfNotPresent",
|
||||
"args": args,
|
||||
"env": [
|
||||
{
|
||||
"name": "OAUTH2_PROXY_CLIENT_ID",
|
||||
"valueFrom": {"secretKeyRef": {"name": secret, "key": "client-id"}},
|
||||
},
|
||||
{
|
||||
"name": "OAUTH2_PROXY_CLIENT_SECRET",
|
||||
"valueFrom": {"secretKeyRef": {"name": secret, "key": "client-secret"}},
|
||||
},
|
||||
{
|
||||
"name": "OAUTH2_PROXY_COOKIE_SECRET",
|
||||
"valueFrom": {"secretKeyRef": {"name": secret, "key": "cookie-secret"}},
|
||||
},
|
||||
],
|
||||
"ports": [{"name": "http", "containerPort": OAUTH2_PROXY_PORT, "protocol": "TCP"}],
|
||||
"resources": {
|
||||
"requests": {"cpu": "25m", "memory": "64Mi"},
|
||||
"limits": {"memory": "128Mi"},
|
||||
},
|
||||
"securityContext": {
|
||||
"allowPrivilegeEscalation": False,
|
||||
"readOnlyRootFilesystem": True,
|
||||
"capabilities": {"drop": ["ALL"]},
|
||||
},
|
||||
"livenessProbe": {
|
||||
"httpGet": {"path": "/ping", "port": "http"},
|
||||
"initialDelaySeconds": 5,
|
||||
"periodSeconds": 15,
|
||||
},
|
||||
"readinessProbe": {
|
||||
"httpGet": {"path": "/ping", "port": "http"},
|
||||
"initialDelaySeconds": 5,
|
||||
"periodSeconds": 15,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _auth_annotations(s: dict) -> dict:
|
||||
"""Auth wiring hints (renderer-contract §auth): the platform (oauth2-proxy/Keycloak)
|
||||
consumes these to protect ingressRoutes. oidc/forward-auth carry the authorized groups
|
||||
@@ -812,7 +667,7 @@ def r_image_automation(s: dict) -> list:
|
||||
upd = s["update"]
|
||||
interval = "1h0m0s"
|
||||
imagerepo = {
|
||||
"apiVersion": "image.toolkit.fluxcd.io/v1",
|
||||
"apiVersion": "image.toolkit.fluxcd.io/v1beta2",
|
||||
"kind": "ImageRepository",
|
||||
"metadata": {"name": s["name"], "namespace": s["name"]},
|
||||
"spec": {"image": image, "interval": interval},
|
||||
@@ -827,7 +682,7 @@ def r_image_automation(s: dict) -> list:
|
||||
else:
|
||||
policy_spec["policy"] = {"semver": {"range": ">=0.0.0"}}
|
||||
imagepolicy = {
|
||||
"apiVersion": "image.toolkit.fluxcd.io/v1",
|
||||
"apiVersion": "image.toolkit.fluxcd.io/v1beta2",
|
||||
"kind": "ImagePolicy",
|
||||
"metadata": {
|
||||
"name": s["name"],
|
||||
@@ -937,124 +792,6 @@ def r_virtualmachine(s: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Substitution safety (R-ENV-02 / R-GITOPS-02) — refuse to emit a manifest that
|
||||
# Flux's envsubst would silently eat before the workload ever runs.
|
||||
# =============================================================================
|
||||
# Every manifest this renderer writes lands under apps/wave-<N>/, and EVERY apps-wave-<N>
|
||||
# Flux Kustomization declares `postBuild.substituteFrom: cluster-vars` (see
|
||||
# flux_wave_kustomization_text below). Flux therefore runs envsubst over the WHOLE rendered
|
||||
# document — including any shell script carried verbatim in a `lifecycle` hook or an env
|
||||
# value — at reconcile time, long before a container sees it. The escape contract is stated
|
||||
# in clusters/base/apps-vms.yaml and proven with `flux envsubst`:
|
||||
#
|
||||
# `${VAR}` -> SUBSTITUTED (empty when unset) => a shell brace must be written `$${VAR}`,
|
||||
# which envsubst emits back as `${VAR}`.
|
||||
# `$VAR` -> left alone => bare shell vars need NO escaping.
|
||||
# `$(cmd)` -> left alone => command substitution needs NO escaping.
|
||||
# `${A[0]}` -> HARD ERROR ("missing closing brace"), failing the whole Kustomization.
|
||||
#
|
||||
# This bit us for real (fixed 2026-09-01): gitea/nextcloud guarded their whole Keycloak OIDC
|
||||
# registration with `if [ -z "${..._OIDC_CLIENT_SECRET:-}" ]`. Neither name is a cluster-var,
|
||||
# so envsubst rendered the guard as `if [ -z "" ]` — ALWAYS TRUE — and every boot skipped the
|
||||
# registration entirely. Flux's STRICT substitution mode cannot catch this class: it
|
||||
# deliberately EXEMPTS the default-operator forms (`${V:-}`, `${V:=x}`, `${V:?}`), which is
|
||||
# exactly the form a defensive shell script uses. So the renderer checks it itself.
|
||||
CLUSTER_VARS_PATH = (
|
||||
Path(__file__).resolve().parent.parent / "clusters" / "base" / "cluster-vars.yaml"
|
||||
)
|
||||
|
||||
# A run of `$` followed by `{...}`. envsubst consumes `$$` as a literal `$`, so a run of N
|
||||
# dollars leaves the brace SUBSTITUTED when N is odd (`${X}`, `$$${X}`) and LITERAL when N is
|
||||
# even (`$${X}` — the escape). Deliberately greedy about the body so malformed/indexed forms
|
||||
# (`${A[0]}`) are caught too rather than skipped.
|
||||
_BRACED_EXPANSION_RE = re.compile(r"(\$+)\{([^{}]*)\}")
|
||||
_PLAIN_IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
|
||||
|
||||
_CLUSTER_VARS_CACHE: dict = {}
|
||||
|
||||
|
||||
def cluster_vars_keys(path: Path | None = None) -> set:
|
||||
"""The declared cluster-vars keys — the ONLY names envsubst may legitimately expand.
|
||||
|
||||
Read from clusters/base/cluster-vars.yaml (the single knob surface, R-ENV-02) so the
|
||||
renderer never carries a stale hardcoded copy. Unreadable/absent => empty set, i.e.
|
||||
FAIL-CLOSED: with no known knobs, every braced expansion must be escaped for the shell.
|
||||
"""
|
||||
key = str(path or CLUSTER_VARS_PATH)
|
||||
if key not in _CLUSTER_VARS_CACHE:
|
||||
try:
|
||||
doc = yaml.safe_load(Path(key).read_text())
|
||||
data = doc.get("data") if isinstance(doc, dict) else None
|
||||
_CLUSTER_VARS_CACHE[key] = set(data) if isinstance(data, dict) else set()
|
||||
except Exception: # pragma: no cover - absent/malformed => fail closed
|
||||
_CLUSTER_VARS_CACHE[key] = set()
|
||||
return _CLUSTER_VARS_CACHE[key]
|
||||
|
||||
|
||||
def substitution_violations(text: str, known: set | None = None) -> list:
|
||||
"""Every unescaped `${...}` in `text` that is NOT a bare, declared cluster-vars key.
|
||||
|
||||
Returns `(line_number, token, reason)` triples in line order. An expansion is ALLOWED only
|
||||
when it is exactly `${KNOWN_KEY}` — a deliberate R-ENV-02 knob. Anything else is a bug:
|
||||
a shell expansion that envsubst will eat (`${V:-}`), a typo'd/undeclared knob that
|
||||
envsubst will blank, or an indexed form that hard-fails the whole Kustomization.
|
||||
"""
|
||||
keys = cluster_vars_keys() if known is None else known
|
||||
out = []
|
||||
for lineno, line in enumerate(text.splitlines(), start=1):
|
||||
for m in _BRACED_EXPANSION_RE.finditer(line):
|
||||
if len(m.group(1)) % 2 == 0:
|
||||
continue # even run of `$` => escaped; envsubst emits it back literally
|
||||
expr = m.group(2)
|
||||
if _PLAIN_IDENT_RE.match(expr):
|
||||
if expr in keys:
|
||||
continue # a declared cluster-vars knob, intentionally substituted
|
||||
reason = (
|
||||
f"'{expr}' is not a declared cluster-vars key — envsubst will replace "
|
||||
"it with the EMPTY STRING"
|
||||
)
|
||||
elif "[" in expr:
|
||||
reason = (
|
||||
"indexed expansion — envsubst hard-fails with 'missing closing brace' "
|
||||
"and the whole Kustomization stops reconciling"
|
||||
)
|
||||
else:
|
||||
reason = (
|
||||
"shell expansion (default-operator/parameter form) — envsubst eats it "
|
||||
"at reconcile time and STRICT mode exempts this form, so nothing else "
|
||||
"will catch it"
|
||||
)
|
||||
out.append((lineno, m.group(0), reason))
|
||||
return out
|
||||
|
||||
|
||||
def assert_substitution_safe(text: str, origin: str = "<render>") -> None:
|
||||
"""Refuse to emit a manifest carrying an unescaped braced expansion (see above).
|
||||
|
||||
Raises ValueError listing every offending token. The fix is almost always to escape the
|
||||
shell brace as `$${...}` (envsubst emits a literal `${...}` for the shell), or — if the
|
||||
value really is an environment knob — to declare it in clusters/base/cluster-vars.yaml
|
||||
and write it as a bare `${KNOB}`.
|
||||
"""
|
||||
bad = substitution_violations(text)
|
||||
if not bad:
|
||||
return
|
||||
lines = [
|
||||
f"{origin}: refusing to emit — unescaped `${{...}}` in a manifest that Flux's "
|
||||
"postBuild.substituteFrom (cluster-vars) will substitute at reconcile time:"
|
||||
]
|
||||
for lineno, token, reason in bad:
|
||||
lines.append(f" line {lineno}: {token} -> {reason}")
|
||||
lines.append(
|
||||
" FIX: a SHELL expansion must be escaped `$${...}` (envsubst emits a literal "
|
||||
"`${...}`); bare `$VAR` / `$(cmd)` need no escaping. A genuine environment knob "
|
||||
"must be declared in clusters/base/cluster-vars.yaml and written bare `${KNOB}`. "
|
||||
"Contract: clusters/base/apps-vms.yaml."
|
||||
)
|
||||
raise ValueError("\n".join(lines))
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Top-level render + emit
|
||||
# =============================================================================
|
||||
@@ -1145,14 +882,8 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
|
||||
return docs
|
||||
|
||||
|
||||
def emit(docs: list, origin: str = "<render>") -> str:
|
||||
"""Deterministic multi-document YAML text (golden-diffable).
|
||||
|
||||
Every emitted document is checked against the substitution contract before it is handed
|
||||
back (assert_substitution_safe): the rendered manifest lands on a Flux path that runs
|
||||
envsubst over it, so an unescaped `${...}` meant for a SHELL is silently blanked. Raises
|
||||
ValueError rather than emitting such a manifest.
|
||||
"""
|
||||
def emit(docs: list) -> str:
|
||||
"""Deterministic multi-document YAML text (golden-diffable)."""
|
||||
parts = []
|
||||
for comment, resource in docs:
|
||||
body = yaml.dump(
|
||||
@@ -1162,9 +893,7 @@ def emit(docs: list, origin: str = "<render>") -> str:
|
||||
width=4096,
|
||||
)
|
||||
parts.append(f"---\n# {comment}\n{body}".rstrip() + "\n")
|
||||
text = "".join(parts)
|
||||
assert_substitution_safe(text, origin)
|
||||
return text
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def load_sd(path: Path) -> dict:
|
||||
@@ -1203,9 +932,7 @@ def render_file(path: Path, image_automation: bool = True, catalog_dir: Path | N
|
||||
src_dir = catalog_dir if catalog_dir is not None else path.parent
|
||||
rev = build_reverse_map(_load_catalog(src_dir))
|
||||
inbound = rev.get(sd["name"], [])
|
||||
return emit(
|
||||
render(sd, image_automation=image_automation, inbound=inbound), origin=str(path)
|
||||
)
|
||||
return emit(render(sd, image_automation=image_automation, inbound=inbound))
|
||||
|
||||
|
||||
# =============================================================================
|
||||
@@ -1309,7 +1036,6 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
convention:
|
||||
- `<svc>.yaml` -> wave(svc)
|
||||
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
|
||||
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
|
||||
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
|
||||
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
|
||||
- `<svc>-db-backup-credentials.sops.yaml` -> wave(svc-db) (CNPG Barman S3 backup secret)
|
||||
@@ -1330,10 +1056,6 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
|
||||
w = waves[name]
|
||||
add(w, f"{name}.yaml")
|
||||
add(w, f"{name}-bootstrap-secret.sops.yaml")
|
||||
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
|
||||
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
|
||||
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
|
||||
add(w, f"{name}-oauth2-proxy-credentials.sops.yaml")
|
||||
db = f"{name}{CO_DB_SUFFIX}"
|
||||
if any(dep["to"] == db for dep in by_name[name]["dependencies"]):
|
||||
wdb = waves[db]
|
||||
@@ -1581,15 +1303,12 @@ def _selftest() -> int:
|
||||
|
||||
# 4b) exposure (ingressRoutes) -> an ADDITIVE gateway-ingress CiliumNetworkPolicy that
|
||||
# allows ONLY the reserved `ingress` (Gateway proxy) identity to the exposed port,
|
||||
# while default-deny still denies all other ingress (R-NET-01/02). Uses an explicit
|
||||
# non-forward-auth mode so this exercises the direct app-port gateway path (a
|
||||
# forward-auth app instead routes the Gateway to the :4180 sidecar — see 6c).
|
||||
# while default-deny still denies all other ingress (R-NET-01/02).
|
||||
exp_docs = render(
|
||||
{
|
||||
"name": "z",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "z.home.keithjackson.dev", "port": 80}],
|
||||
"auth": {"mode": "none"},
|
||||
}
|
||||
)
|
||||
gw = [
|
||||
@@ -1703,91 +1422,6 @@ def _selftest() -> int:
|
||||
"an SD without podSecurityContext must not gain a pod securityContext",
|
||||
)
|
||||
|
||||
# 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
|
||||
# fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.
|
||||
fa_docs = render(
|
||||
{
|
||||
"name": "fa",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "fa.home.keithjackson.dev", "port": 9117}],
|
||||
"auth": {"mode": "forward-auth", "groups": ["media-users"]},
|
||||
}
|
||||
)
|
||||
fa_dep = next(d[1] for d in fa_docs if d[1]["kind"] == "Deployment")
|
||||
fa_containers = fa_dep["spec"]["template"]["spec"]["containers"]
|
||||
check(
|
||||
len(fa_containers) == 2
|
||||
and fa_containers[0]["name"] == "fa"
|
||||
and fa_containers[1]["name"] == "oauth2-proxy",
|
||||
"forward-auth app must render the app container FIRST + an oauth2-proxy sidecar SECOND",
|
||||
)
|
||||
side = fa_containers[1]
|
||||
check(
|
||||
"--upstream=http://127.0.0.1:9117" in side["args"],
|
||||
"sidecar upstream must be the co-located app UI over http on 127.0.0.1:<uiport>",
|
||||
)
|
||||
check(
|
||||
"--allowed-group=media-users" in side["args"],
|
||||
"sidecar must enforce the declared auth.group (R-IDP-06)",
|
||||
)
|
||||
check(
|
||||
"--redirect-url=https://fa.home.keithjackson.dev/oauth2/callback" in side["args"],
|
||||
"sidecar redirect-url must be the app host's /oauth2/callback",
|
||||
)
|
||||
check(
|
||||
not any(a.startswith("--ssl-upstream-insecure-skip-verify") for a in side["args"]),
|
||||
"http upstream must NOT skip TLS verify (that flag is only for crafty's self-signed https)",
|
||||
)
|
||||
check(
|
||||
side["env"][0]["valueFrom"]["secretKeyRef"]["name"] == "fa-oauth2-proxy",
|
||||
"sidecar creds must come by name from the per-app <app>-oauth2-proxy Secret (R-IDP-04)",
|
||||
)
|
||||
fa_svc = next(d[1] for d in fa_docs if d[1]["kind"] == "Service")
|
||||
fa_svc_ports = {p["port"] for p in fa_svc["spec"]["ports"]}
|
||||
check(
|
||||
fa_svc_ports == {9117, 4180},
|
||||
"forward-auth Service must expose the app port (east-west) AND :4180 (gateway->sidecar)",
|
||||
)
|
||||
fa_route = next(d[1] for d in fa_docs if d[1]["kind"] == "HTTPRoute")
|
||||
check(
|
||||
fa_route["spec"]["rules"][0]["backendRefs"][0]["port"] == 4180,
|
||||
"forward-auth HTTPRoute backendRef must target the sidecar :4180, not the app UI port",
|
||||
)
|
||||
fa_gw = next(
|
||||
d[1] for d in fa_docs
|
||||
if d[1]["kind"] == "CiliumNetworkPolicy"
|
||||
and d[1]["metadata"]["name"] == "fa-allow-gateway-ingress"
|
||||
)
|
||||
fa_gw_ports = {
|
||||
p["port"] for r in fa_gw["spec"]["ingress"] for tp in r["toPorts"] for p in tp["ports"]
|
||||
}
|
||||
check(
|
||||
fa_gw_ports == {"4180"},
|
||||
"fail-closed: the Gateway may reach ONLY the sidecar :4180, never the un-gated app port",
|
||||
)
|
||||
# an oidc app (native client) must NOT gain a sidecar or a :4180 path.
|
||||
oidc_docs = render(
|
||||
{
|
||||
"name": "oi",
|
||||
"source": {"image": "a:1"},
|
||||
"ingressRoutes": [{"host": "oi.home.keithjackson.dev", "port": 3000}],
|
||||
"auth": {"mode": "oidc", "groups": ["platform-admins"]},
|
||||
}
|
||||
)
|
||||
oi_containers = next(
|
||||
d[1] for d in oidc_docs if d[1]["kind"] == "Deployment"
|
||||
)["spec"]["template"]["spec"]["containers"]
|
||||
check(
|
||||
len(oi_containers) == 1
|
||||
and not any("oauth2-proxy" in c["name"] for c in oi_containers),
|
||||
"an oidc app must NOT gain an oauth2-proxy sidecar (native client authenticates itself)",
|
||||
)
|
||||
oi_route = next(d[1] for d in oidc_docs if d[1]["kind"] == "HTTPRoute")
|
||||
check(
|
||||
oi_route["spec"]["rules"][0]["backendRefs"][0]["port"] == 3000,
|
||||
"an oidc app's HTTPRoute must still target its own app port (unchanged)",
|
||||
)
|
||||
|
||||
# 7) wave partitioning: longest-path over the dependency DAG, CNPG-DB strictly below.
|
||||
wsds = [
|
||||
{"name": "a", "source": {"image": "x:1"}}, # leaf
|
||||
@@ -1858,90 +1492,6 @@ def _selftest() -> int:
|
||||
check(kt["kind"] == "Kustomization" and kt["resources"] == ["svc.yaml", "sonarr.yaml"],
|
||||
"wave kustomization must list exactly the wave's resources")
|
||||
|
||||
# 8) substitution safety (R-ENV-02): the renderer must REFUSE to emit a manifest whose
|
||||
# shell script carries an unescaped `${...}`. Every rendered manifest lands under a
|
||||
# Flux path with postBuild.substituteFrom cluster-vars, so envsubst rewrites it before
|
||||
# the container runs — this is the gitea/nextcloud OIDC-guard bug (`[ -z "${X:-}" ]`
|
||||
# -> `[ -z "" ]`, always true, whole registration block skipped on every boot).
|
||||
known = {"BASE_DOMAIN", "GATEWAY_VIP"}
|
||||
# allowed: a bare, DECLARED cluster-vars knob is a deliberate R-ENV-02 substitution.
|
||||
check(
|
||||
substitution_violations("host: app.${BASE_DOMAIN}", known) == [],
|
||||
"a bare declared cluster-vars key must be allowed (that is what substitution is FOR)",
|
||||
)
|
||||
# allowed: escaped shell braces, and the forms envsubst leaves alone.
|
||||
check(
|
||||
substitution_violations('if [ -z "$${X:-}" ]; then', known) == [],
|
||||
"an ESCAPED `$${...}` must be allowed (envsubst emits it back as a literal `${...}`)",
|
||||
)
|
||||
check(
|
||||
substitution_violations('t="$(mktemp)"; echo "$HOME $1"', known) == [],
|
||||
"bare `$VAR` / `$(cmd)` are left alone by envsubst and must NOT be flagged",
|
||||
)
|
||||
# `$$` is a literal `$` to envsubst, so an ODD run of dollars still leaves the brace live.
|
||||
check(
|
||||
len(substitution_violations("$$${X:-}", known)) == 1,
|
||||
"an ODD run of `$` still substitutes ($$ -> literal $, then `${X:-}` is eaten)",
|
||||
)
|
||||
# flagged: the exact regression — a default-operator form STRICT mode deliberately exempts.
|
||||
guard_bug = substitution_violations('if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then', known)
|
||||
check(
|
||||
len(guard_bug) == 1 and guard_bug[0][1] == "${GITEA_OIDC_CLIENT_SECRET:-}",
|
||||
"an unescaped `${VAR:-}` shell guard must be flagged (strict mode exempts this form)",
|
||||
)
|
||||
# flagged: an undeclared bare knob (envsubst blanks it), and an indexed form (hard error).
|
||||
check(
|
||||
len(substitution_violations("value: ${NOT_A_KNOB}", known)) == 1,
|
||||
"an undeclared bare `${VAR}` must be flagged (envsubst blanks it silently)",
|
||||
)
|
||||
idx = substitution_violations("echo ${PIPESTATUS[0]}", known)
|
||||
check(
|
||||
len(idx) == 1 and "missing closing brace" in idx[0][2],
|
||||
"an indexed `${A[0]}` must be flagged (envsubst hard-fails the whole Kustomization)",
|
||||
)
|
||||
# the checker must be wired into emit(): rendering an SD whose lifecycle hook carries the
|
||||
# bug RAISES rather than writing a manifest whose guard is dead on arrival.
|
||||
bad_sd = {
|
||||
"name": "subbad",
|
||||
"source": {"image": "a:1"},
|
||||
"lifecycle": {
|
||||
"postStart": {
|
||||
"exec": {
|
||||
"command": [
|
||||
"/bin/sh",
|
||||
"-c",
|
||||
'if [ -z "${SUBBAD_SECRET:-}" ]; then exit 0; fi',
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
}
|
||||
try:
|
||||
emit(render(bad_sd), origin="<selftest>")
|
||||
check(False, "emit must REFUSE a manifest with an unescaped `${...}` shell expansion")
|
||||
except ValueError as exc:
|
||||
check(
|
||||
"SUBBAD_SECRET" in str(exc) and "$${" in str(exc),
|
||||
"the refusal must name the offending token AND state the `$${...}` fix",
|
||||
)
|
||||
# the ESCAPED form of the same hook renders normally (the fix must be emittable).
|
||||
good_sd = dict(bad_sd, name="subgood")
|
||||
good_sd["lifecycle"] = {
|
||||
"postStart": {
|
||||
"exec": {
|
||||
"command": [
|
||||
"/bin/sh",
|
||||
"-c",
|
||||
'if [ -z "$${SUBBAD_SECRET:-}" ]; then exit 0; fi',
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
check(
|
||||
"$${SUBBAD_SECRET:-}" in emit(render(good_sd), origin="<selftest>"),
|
||||
"the escaped `$${...}` form must render (and reach the shell as `${...}`)",
|
||||
)
|
||||
|
||||
if failures:
|
||||
for f in failures:
|
||||
sys.stderr.write(f"SELFTEST FAIL: {f}\n")
|
||||
|
||||
+3
-11
@@ -1,9 +1,8 @@
|
||||
# ci/Makefile — portable, single-command local equivalent of the Gitea Actions
|
||||
# merge gate (R-TEST-02: the gates are runnable TODAY, without Gitea).
|
||||
#
|
||||
# The LIVE Gitea Actions workflow (.gitea/workflows/gate.yml) invokes `make -C ci gate`
|
||||
# directly, so `make -C ci gate` locally IS the CI merge gate — byte-for-byte. Run from
|
||||
# anywhere:
|
||||
# The Gitea Actions workflow (.gitea/workflows/merge-gate.yaml) calls these exact
|
||||
# scripts, so `make -C ci gate` locally == the CI merge gate. Run from anywhere:
|
||||
# make -C ci gate
|
||||
#
|
||||
# Every stage is a thin wrapper over a ci/*.sh script; see ci/README.md.
|
||||
@@ -12,7 +11,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
|
||||
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
|
||||
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools
|
||||
|
||||
help: ## Show this help
|
||||
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf " \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
|
||||
@@ -43,13 +42,6 @@ all: gate ## Alias for `gate`
|
||||
touched: ## Print the touched-requirement subset for the current branch vs main
|
||||
@"$(CI_DIR)touched-requirements.sh"
|
||||
|
||||
test: ## Run ci/ script unit tests (no cluster; e.g. touched-requirements content scan)
|
||||
@rc=0; for t in "$(CI_DIR)"tests/*.test.sh; do \
|
||||
echo "== $$t"; bash "$$t" || rc=1; \
|
||||
done; \
|
||||
if [ $$rc -ne 0 ]; then echo "ci/tests: FAILED" >&2; else echo "ci/tests: all suites passed"; fi; \
|
||||
exit $$rc
|
||||
|
||||
versions: ## Print pinned tool versions (R-TEST-02)
|
||||
@cat "$(CI_DIR)versions.env"
|
||||
|
||||
|
||||
+5
-45
@@ -20,54 +20,17 @@ Individual stages:
|
||||
|
||||
```bash
|
||||
make -C ci lint # yaml/lint (no cluster; live now)
|
||||
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
|
||||
make -C ci l0 # ServiceDefinition schema + Kyverno (no cluster; live now)
|
||||
make -C ci l1 # L1 rendering checks (no cluster; live now)
|
||||
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
|
||||
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
|
||||
```
|
||||
|
||||
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
|
||||
(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
|
||||
per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
|
||||
spec-immutability enforcement). See `conformance/tests/README.md`.
|
||||
|
||||
## Exit-code contract
|
||||
|
||||
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
|
||||
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
||||
the gate stays honest in Phase 0 (no fake green).
|
||||
|
||||
### The touched-requirement subset can also be *undeterminable* (exit 3)
|
||||
|
||||
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
|
||||
order, **verifying each candidate exists** before using it:
|
||||
|
||||
```
|
||||
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
|
||||
```
|
||||
|
||||
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
|
||||
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
|
||||
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
|
||||
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
|
||||
|
||||
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
|
||||
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
|
||||
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
|
||||
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
|
||||
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
|
||||
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
|
||||
|
||||
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
|
||||
|
||||
| reason | headline | exit |
|
||||
|---|---|---|
|
||||
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
|
||||
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
|
||||
|
||||
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
|
||||
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
|
||||
|
||||
## Reproducibility (R-TEST-02)
|
||||
|
||||
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
|
||||
@@ -80,19 +43,16 @@ in that case — nothing goes untested — and says plainly that the *scoped* ch
|
||||
|
||||
```
|
||||
ci/
|
||||
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
|
||||
Makefile targets: lint · l0 · l1 · conformance · gate · touched · tools · versions
|
||||
versions.env PINNED tool versions (R-TEST-02) — single source of truth
|
||||
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
|
||||
yamllint.yaml lint config for the yaml/lint stage
|
||||
lint.sh Stage: yaml/lint
|
||||
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
|
||||
l0.sh Stage: ServiceDefinition schema validation + Kyverno L0 policy gate
|
||||
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
||||
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
|
||||
exits 3 LOUDLY rather than guessing when the diff base is unreliable
|
||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter)
|
||||
gate.sh run all stages + the merge decision (the single command)
|
||||
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
|
||||
throwaway git repos via the REPO_ROOT override — no network, no cluster
|
||||
README.md this file
|
||||
```
|
||||
|
||||
@@ -107,7 +67,7 @@ below are this workstream's **assumptions** — flagged for the PM to verify at
|
||||
|---|---|---|---|---|
|
||||
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
|
||||
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
|
||||
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` |
|
||||
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l1\|l2,l3> [--requirements R-A,R-B] [--trigger in-cluster-job] --junit <file> --coverage <dir>` |
|
||||
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
|
||||
|
||||
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
|
||||
|
||||
+8
-56
@@ -48,32 +48,6 @@ echo >&2
|
||||
log "==== gate decision ===="
|
||||
BLOCK=0
|
||||
|
||||
# --- R-TEST-03 input: the touched-requirement subset --------------------------
|
||||
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
|
||||
# reliable diff base; that used to degrade to an empty set, which silently disabled the
|
||||
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
|
||||
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
|
||||
# as an UNDECIDED merge decision, never as a pass.
|
||||
_reason_file="$(mktemp)"
|
||||
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
|
||||
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
|
||||
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
|
||||
TOUCHED_RC=$?
|
||||
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
|
||||
rm -f "$_reason_file"
|
||||
TOUCHED=()
|
||||
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
|
||||
UNDECIDED=0
|
||||
if [ "$TOUCHED_RC" -eq 3 ]; then
|
||||
UNDECIDED=1
|
||||
echo >&2
|
||||
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
|
||||
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
|
||||
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
|
||||
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED[*]}"
|
||||
fi
|
||||
|
||||
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
|
||||
# A skip means the spec repo wasn't available to verify — surfaced, not green.
|
||||
case "${OUTCOME[spec-immutability]}" in
|
||||
@@ -103,11 +77,9 @@ case "${OUTCOME[conformance]}" in
|
||||
skip)
|
||||
if cluster_available; then
|
||||
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
|
||||
else
|
||||
if [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
reqs=("${TOUCHED[@]}")
|
||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
||||
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
|
||||
warn " (no cluster — Phase 0). These are NOT certified green."
|
||||
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
|
||||
@@ -121,35 +93,15 @@ case "${OUTCOME[conformance]}" in
|
||||
fi ;;
|
||||
esac
|
||||
|
||||
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
|
||||
# on-base -> this checkout has no branch commits to gate (you are on main, or the
|
||||
# branch is already merged; a push-to-main CI run lands here). Legitimate,
|
||||
# so the job is not failed — but the headline says NOT APPLICABLE, never
|
||||
# PASS, so nobody reads it as "my branch is green".
|
||||
# anything -> the environment cannot support a merge decision (no base ref, broken
|
||||
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
|
||||
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
|
||||
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
||||
BLOCK=1
|
||||
fi
|
||||
|
||||
echo >&2
|
||||
if [ "$BLOCK" -ne 0 ]; then
|
||||
err "MERGE GATE: BLOCKED"
|
||||
summary ""
|
||||
summary "**Result: BLOCKED** — see failures above."
|
||||
exit 1
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
|
||||
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
|
||||
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
|
||||
warn " (or set BASE_REF) and run it again."
|
||||
summary ""
|
||||
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
|
||||
exit 0
|
||||
else
|
||||
if [ "$BLOCK" -eq 0 ]; then
|
||||
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
|
||||
summary ""
|
||||
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
|
||||
exit 0
|
||||
else
|
||||
err "MERGE GATE: BLOCKED"
|
||||
summary ""
|
||||
summary "**Result: BLOCKED** — see failures above."
|
||||
exit 1
|
||||
fi
|
||||
@@ -7,9 +7,6 @@
|
||||
# Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline
|
||||
# present, no plaintext secrets, required labels/egress-class,
|
||||
# ServiceDefinition schema conformance (owner: T-0.5, ADR-0011).
|
||||
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
|
||||
# a per-resource policy cannot express (secret hygiene across every
|
||||
# tracked file, least-privilege RBAC, spec-immutability enforcement).
|
||||
#
|
||||
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
|
||||
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
|
||||
@@ -76,67 +73,6 @@ else
|
||||
rm -rf "$l0_render"
|
||||
fi
|
||||
|
||||
# ---- Stage 3b: L0 STATIC conformance checks ---------------------------------
|
||||
# docs/14's L0 row is "manifests/ServiceDefinitions obey invariants", which is wider
|
||||
# than what Kyverno alone can express over single resources: whole-repo invariants
|
||||
# (no plaintext secrets anywhere in git, least-privilege RBAC across every tracked
|
||||
# binding, the spec-immutability gate's own behaviour) live in the conformance
|
||||
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
|
||||
# through the same single-command runner CI and a laptop both use (R-TEST-02).
|
||||
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
|
||||
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
|
||||
summary "- l0-static: **SKIPPED** (conformance runner missing)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
else
|
||||
load_touched_requirements
|
||||
if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
|
||||
ok "L0 static conformance checks passed"
|
||||
summary "- l0-static: **passed**"
|
||||
else
|
||||
err "L0 static conformance checks failed"
|
||||
summary "- l0-static: **FAILED**"
|
||||
rc=1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
|
||||
# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
|
||||
# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
|
||||
# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
|
||||
# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
|
||||
# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
|
||||
# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
|
||||
# a legitimate manifest, and that no expansion silently empties today.
|
||||
#
|
||||
# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
|
||||
# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
|
||||
# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
|
||||
# attributes no requirement, so the requirement-scoped selection would filter this check
|
||||
# out of exactly the change that introduces the bug. The invariant is whole-repo (same
|
||||
# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
|
||||
# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
|
||||
# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
|
||||
SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
|
||||
log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
|
||||
if [ ! -f "$SUBST_CHECK" ]; then
|
||||
skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
|
||||
summary "- subst-safety: **SKIPPED** (check missing)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
else
|
||||
subst_rc=0
|
||||
bash "$SUBST_CHECK" || subst_rc=$?
|
||||
case "$subst_rc" in
|
||||
0) ok "Flux postBuild substitution safety passed"
|
||||
summary "- subst-safety: **passed**" ;;
|
||||
3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
|
||||
summary "- subst-safety: **SKIPPED** (nothing to audit)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
|
||||
*) err "Flux postBuild substitution safety FAILED"
|
||||
summary "- subst-safety: **FAILED**"; rc=1 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
case "$rc" in
|
||||
0) ok "L0 stage complete" ;;
|
||||
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
|
||||
|
||||
@@ -20,16 +20,19 @@ if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# Touched-requirement subset (R-TEST-03). An empty OR undeterminable subset => run all
|
||||
# L1 as a superset sanity net, and SAY SO — see load_touched_requirements in ci/lib.sh.
|
||||
load_touched_requirements
|
||||
req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
||||
# Touched-requirement subset (R-TEST-03). Empty subset => run all L1 as a sanity net.
|
||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
||||
req_args=()
|
||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${reqs[*]}"
|
||||
req_args=(--requirements "$(IFS=,; echo "${reqs[*]}")")
|
||||
fi
|
||||
|
||||
# RECONCILED (PM 2026-08-16): conformance/run.sh's real interface is `--layer <l1|l2|l3>`
|
||||
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
|
||||
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
||||
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
||||
"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
|
||||
"$CONFORMANCE_CMD" --layer l1 "${req_args[@]}"
|
||||
|
||||
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
||||
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
||||
|
||||
@@ -10,12 +10,9 @@
|
||||
set -euo pipefail
|
||||
|
||||
# --- Paths -------------------------------------------------------------------
|
||||
# CI_DIR = this ci/ directory (always the real script location). REPO_ROOT = its
|
||||
# parent (impl repo root) by default, but OVERRIDABLE via the environment so the
|
||||
# scripts can be pointed at a throwaway git repo under test (see ci/tests/) without
|
||||
# copying them out of the tree. CI and normal runs leave it unset -> unchanged.
|
||||
# CI_DIR = this ci/ directory; REPO_ROOT = its parent (impl repo root).
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
: "${REPO_ROOT:=$(cd "${CI_DIR}/.." && pwd)}"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/versions.env"
|
||||
|
||||
@@ -85,38 +82,3 @@ cluster_available() {
|
||||
}
|
||||
|
||||
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
|
||||
|
||||
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
|
||||
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
|
||||
# (ready to splice into a conformance/run.sh invocation).
|
||||
#
|
||||
# ci/touched-requirements.sh exits 3 — LOUDLY — when it cannot establish a reliable
|
||||
# diff base, instead of the old silent degradation to an empty set. An empty set
|
||||
# means run.sh gets no `--requirements`, which means its coverage `--require` gate
|
||||
# never runs; treating "could not determine" as "nothing touched" is precisely the
|
||||
# false green this helper exists to prevent. A stage still runs its FULL layer in
|
||||
# that case (a superset, so nothing goes untested), but says plainly that the
|
||||
# SCOPED check did not happen; ci/gate.sh turns that into an UNDECIDED decision.
|
||||
load_touched_requirements() {
|
||||
local out rc=0
|
||||
out="$("${CI_DIR}/touched-requirements.sh")" || rc=$?
|
||||
TOUCHED_RC=$rc
|
||||
TOUCHED_REQS=()
|
||||
# `if`, not `[ ... ] && ...`: under `set -e` a false AND-list is the caller's last
|
||||
# command status and would abort the stage script on an EMPTY touched set.
|
||||
if [ -n "$out" ]; then mapfile -t TOUCHED_REQS <<< "$out"; fi
|
||||
REQ_ARGS=()
|
||||
if [ "$rc" -eq 3 ]; then
|
||||
warn "touched-requirement subset UNDETERMINED (reason logged above)."
|
||||
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
|
||||
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
|
||||
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED_REQS[*]}"
|
||||
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
|
||||
else
|
||||
log "empty touched-requirement set (base WAS established) — running the full layer."
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
@@ -39,7 +39,6 @@ PYTHON="${PYTHON:-python3}"
|
||||
# DESIGN-waves.md §"Catalog services not yet onboarded"). They render as a
|
||||
# by-product of `--all catalog` but must NOT have a golden; the gate ignores
|
||||
# their rendered output. Onboarding one = add its golden and drop it here.
|
||||
#
|
||||
: "${RENDER_GOLDEN_NOT_ONBOARDED:=keycloak vm-demo}"
|
||||
|
||||
log "Stage: render==golden reproducibility (canonical render vs committed apps/)"
|
||||
|
||||
@@ -1,169 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/touched-requirements-base.test.sh — regression tests for BASE-REF RESOLUTION
|
||||
# in ci/touched-requirements.sh.
|
||||
#
|
||||
# THE BUG THESE LOCK DOWN (2026-08-31, local/CI divergence):
|
||||
# `make -C ci gate` on a feature branch printed
|
||||
# "no changed files vs base; empty touched-requirement set"
|
||||
# and then "MERGE GATE: PASS" — while CI, with a correct base, computed 17 touched
|
||||
# requirements and BLOCKED on real coverage debt. Root cause: EVERY way of failing to
|
||||
# establish a diff base degraded silently to an EMPTY touched set and exit 0. An empty
|
||||
# set means conformance/run.sh is invoked without `--requirements`, which means its
|
||||
# coverage `--require` gate never runs — so the gate reported PASS having checked
|
||||
# nothing. A false green is worse than a red.
|
||||
#
|
||||
# The fix: an undeterminable base is a LOUD exit 3 (ci/README.md's SKIPPED code) with
|
||||
# EMPTY stdout and a machine-readable reason code, never exit 0. These tests assert
|
||||
# that for each way the base can be undeterminable, and assert the good paths still work.
|
||||
#
|
||||
# The tests point the real script at THROWAWAY git repos via the REPO_ROOT override
|
||||
# (ci/lib.sh honours a pre-set REPO_ROOT); the script runs from its true location so it
|
||||
# still sources the real lib.sh. No network, no cluster, and the working repo is never
|
||||
# touched.
|
||||
#
|
||||
# Run: ci/tests/touched-requirements-base.test.sh (or `make -C ci test`)
|
||||
set -uo pipefail # NOT -e: these tests deliberately run a script that exits non-zero.
|
||||
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
# newrepo <dir> — a throwaway repo with one baseline commit on `main`.
|
||||
newrepo() {
|
||||
local d="$1"
|
||||
mkdir -p "$d/notes"
|
||||
git -C "$d" init -q -b main
|
||||
git -C "$d" config user.email [email protected]
|
||||
git -C "$d" config user.name t
|
||||
git -C "$d" config commit.gpgsign false # throwaway: never sign (ignore host gpg config)
|
||||
echo baseline > "$d/notes/base.txt"
|
||||
git -C "$d" add -A && git -C "$d" commit -qm baseline
|
||||
}
|
||||
|
||||
# commit_touching <dir> <text> — a commit whose content declares an R-ID.
|
||||
commit_touching() {
|
||||
local d="$1" text="$2"
|
||||
printf 'name: %s\n' "$text" > "$d/notes/decl.yaml"
|
||||
git -C "$d" add -A && git -C "$d" commit -qm change
|
||||
}
|
||||
|
||||
# Runs the script in <dir>; sets RC, OUT (stdout), ERRTXT (stderr), REASON.
|
||||
run_in() {
|
||||
local d="$1"; shift
|
||||
local reason_file="${TMP}/reason.$$"
|
||||
: > "$reason_file"
|
||||
OUT="$(REPO_ROOT="$d" TOUCHED_REASON_FILE="$reason_file" "$@" "$SCRIPT" 2>"${TMP}/err.$$")"
|
||||
RC=$?
|
||||
ERRTXT="$(cat "${TMP}/err.$$")"
|
||||
REASON="$(cat "$reason_file")"
|
||||
}
|
||||
|
||||
# assert_refusal <label> <expected-reason>
|
||||
assert_refusal() {
|
||||
local label="$1" want="$2"
|
||||
[ "$RC" = 3 ] && ok "${label}: exits 3 (SKIPPED), not 0" || bad "${label}: rc=${RC}, expected 3"
|
||||
[ -z "$OUT" ] && ok "${label}: stdout is EMPTY (no silent requirement list)" \
|
||||
|| bad "${label}: stdout was '${OUT}'"
|
||||
[ "$REASON" = "$want" ] && ok "${label}: reason code '${want}'" \
|
||||
|| bad "${label}: reason '${REASON}', expected '${want}'"
|
||||
grep -q "NOT DETERMINABLE" <<<"$ERRTXT" && ok "${label}: announces the refusal LOUDLY on stderr" \
|
||||
|| bad "${label}: stderr lacked the loud banner"
|
||||
}
|
||||
|
||||
echo "== case 1: a branch ahead of local main (no remote) resolves and reports"
|
||||
R1="${TMP}/r1"; newrepo "$R1"
|
||||
git -C "$R1" checkout -q -b feature
|
||||
commit_touching "$R1" implements-R-ALPHA-01
|
||||
run_in "$R1"
|
||||
[ "$RC" = 0 ] && ok "ahead-of-main: exits 0" || bad "ahead-of-main: rc=${RC}"
|
||||
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "ahead-of-main: emits the touched R-ID" \
|
||||
|| bad "ahead-of-main: output was '${OUT}'"
|
||||
grep -q "diff base: main" <<<"$ERRTXT" && ok "ahead-of-main: names the base it used" \
|
||||
|| bad "ahead-of-main: base not reported: ${ERRTXT}"
|
||||
|
||||
echo "== case 2: THE FALSE GREEN — HEAD is the base branch itself"
|
||||
# This is the state that produced 'no changed files vs base' + 'MERGE GATE: PASS'
|
||||
# locally while CI blocked. It must now refuse, not report an empty set.
|
||||
R2="${TMP}/r2"; newrepo "$R2"
|
||||
git -C "$R2" checkout -q -b feature
|
||||
commit_touching "$R2" implements-R-ALPHA-01
|
||||
git -C "$R2" checkout -q main # <- sitting on the base branch
|
||||
run_in "$R2"
|
||||
assert_refusal "on-base" "on-base"
|
||||
|
||||
echo "== case 3: a branch already merged into the base (HEAD is an ancestor)"
|
||||
R3="${TMP}/r3"; newrepo "$R3"
|
||||
git -C "$R3" checkout -q -b feature
|
||||
commit_touching "$R3" implements-R-ALPHA-01
|
||||
git -C "$R3" checkout -q main
|
||||
git -C "$R3" merge -q --no-edit feature
|
||||
git -C "$R3" checkout -q feature # feature is now behind main
|
||||
run_in "$R3"
|
||||
assert_refusal "already-merged" "on-base"
|
||||
|
||||
echo "== case 4: no base ref resolves at all (no main, no origin/main)"
|
||||
R4="${TMP}/r4"; newrepo "$R4"
|
||||
git -C "$R4" checkout -q -b work
|
||||
commit_touching "$R4" implements-R-ALPHA-01
|
||||
git -C "$R4" branch -q -D main
|
||||
run_in "$R4"
|
||||
assert_refusal "no-base-ref" "no-base"
|
||||
|
||||
echo "== case 5: a single-commit repo with no main (the removed HEAD~1 last resort)"
|
||||
# HEAD~1 does not exist here. The old code fell back to it, the diff failed, and the
|
||||
# empty result was reported as 'no changed files' + exit 0 — a silent false green.
|
||||
R5="${TMP}/r5"; mkdir -p "$R5/notes"
|
||||
git -C "$R5" init -q -b work
|
||||
git -C "$R5" config user.email [email protected]; git -C "$R5" config user.name t
|
||||
git -C "$R5" config commit.gpgsign false
|
||||
echo only > "$R5/notes/only.txt"; git -C "$R5" add -A; git -C "$R5" commit -qm only
|
||||
run_in "$R5"
|
||||
assert_refusal "single-commit-no-main" "no-base"
|
||||
|
||||
echo "== case 6: BASE_REF names a ref that does not exist"
|
||||
R6="${TMP}/r6"; newrepo "$R6"
|
||||
git -C "$R6" checkout -q -b feature
|
||||
commit_touching "$R6" implements-R-ALPHA-01
|
||||
run_in "$R6" env BASE_REF=refs/heads/nope
|
||||
assert_refusal "bogus-BASE_REF" "no-base"
|
||||
|
||||
echo "== case 7: GITHUB_BASE_REF set but absent from the checkout (shallow clone)"
|
||||
R7="${TMP}/r7"; newrepo "$R7"
|
||||
git -C "$R7" checkout -q -b feature
|
||||
commit_touching "$R7" implements-R-ALPHA-01
|
||||
run_in "$R7" env GITHUB_BASE_REF=release-9
|
||||
assert_refusal "missing-GITHUB_BASE_REF" "no-base"
|
||||
|
||||
echo "== case 8: origin/main is preferred over a stale local main, and staleness warns"
|
||||
R8="${TMP}/r8"; newrepo "$R8"
|
||||
UP="${TMP}/r8-origin.git"; git init -q --bare -b main "$UP"
|
||||
git -C "$R8" remote add origin "$UP"
|
||||
git -C "$R8" push -q origin main
|
||||
# advance origin/main beyond the local main ref, then branch off it
|
||||
git -C "$R8" checkout -q -b ahead
|
||||
echo more > "$R8/notes/more.txt"; git -C "$R8" add -A; git -C "$R8" commit -qm more
|
||||
git -C "$R8" push -q origin ahead:main
|
||||
git -C "$R8" fetch -q origin
|
||||
git -C "$R8" checkout -q -b feature origin/main
|
||||
commit_touching "$R8" implements-R-ALPHA-01
|
||||
run_in "$R8"
|
||||
[ "$RC" = 0 ] && ok "origin-preferred: exits 0" || bad "origin-preferred: rc=${RC}"
|
||||
grep -q "diff base: origin/main" <<<"$ERRTXT" && ok "origin-preferred: uses origin/main, not local main" \
|
||||
|| bad "origin-preferred: base line was: ${ERRTXT}"
|
||||
grep -q "may be stale" <<<"$ERRTXT" && ok "origin-preferred: warns that local main differs" \
|
||||
|| bad "origin-preferred: no staleness warning"
|
||||
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "origin-preferred: emits the touched R-ID" \
|
||||
|| bad "origin-preferred: output was '${OUT}'"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then
|
||||
echo "touched-requirements base-resolution tests: PASS (${PASS} assertions)"; exit 0
|
||||
fi
|
||||
echo "touched-requirements base-resolution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
|
||||
exit 1
|
||||
@@ -1,98 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/touched-requirements.test.sh — regression tests for the content scan in
|
||||
# ci/touched-requirements.sh, focused on rule 3: R-IDs cited only in a whole-line
|
||||
# YAML/shell `#` comment are PROSE (a reference), not a declaration, and must NOT be
|
||||
# attributed — while R-IDs on real code/spec lines still ARE.
|
||||
#
|
||||
# The tests point the real script at a THROWAWAY git repo via the REPO_ROOT override
|
||||
# (ci/lib.sh honours a pre-set REPO_ROOT); the script itself is run from its true
|
||||
# location so it still sources the real lib.sh. No network, no cluster.
|
||||
#
|
||||
# Run: ci/tests/touched-requirements.test.sh (or `make -C ci test`)
|
||||
set -euo pipefail
|
||||
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
# assert_absent <needle> <haystack> <msg>
|
||||
assert_absent() {
|
||||
if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))";
|
||||
else ok "$3"; fi
|
||||
}
|
||||
# assert_present <needle> <haystack> <msg>
|
||||
assert_present() {
|
||||
if grep -qxF "$1" <<<"$2"; then ok "$3";
|
||||
else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
|
||||
}
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
git -C "$TMP" init -q -b main
|
||||
git -C "$TMP" config user.email [email protected]
|
||||
git -C "$TMP" config user.name t
|
||||
git -C "$TMP" config commit.gpgsign false # throwaway repo: never sign (ignore host gpg config)
|
||||
mkdir -p "$TMP/notes" # a path OUTSIDE path_requirements() so we isolate the content scan
|
||||
echo "baseline" > "$TMP/notes/base.txt"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
|
||||
|
||||
git -C "$TMP" checkout -q -b feature
|
||||
|
||||
# A YAML file: an R-ID in a whole-line comment (must be IGNORED) and an R-ID on a real
|
||||
# value line (must be DETECTED). Also an indented comment, to prove leading space is fine.
|
||||
cat > "$TMP/notes/example.yaml" <<'YAML'
|
||||
# This header explains R-DOC-01 and must NOT be attributed.
|
||||
kind: Thing
|
||||
metadata:
|
||||
name: implements-R-REAL-01
|
||||
spec:
|
||||
note: value # trailing inline comment mentioning R-INLINE-01 (conservatively still seen)
|
||||
# deeply indented comment about R-DOC-02 — also NOT attributed
|
||||
YAML
|
||||
|
||||
# A shell file: a `#` comment R-ID (IGNORED) plus a real command line R-ID (DETECTED).
|
||||
cat > "$TMP/notes/example.sh" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
# doc: this script relates to R-SHDOC-01 (comment only — ignored)
|
||||
echo "runs check for R-SHREAL-01"
|
||||
SH
|
||||
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm feature
|
||||
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
|
||||
# Comment-only citations are dropped (rule 3):
|
||||
assert_absent "R-DOC-01" "$OUT" "whole-line YAML comment R-ID is NOT attributed"
|
||||
assert_absent "R-DOC-02" "$OUT" "indented YAML comment R-ID is NOT attributed"
|
||||
assert_absent "R-SHDOC-01" "$OUT" "shell '#' comment R-ID is NOT attributed"
|
||||
# Real code/spec lines still count:
|
||||
assert_present "R-REAL-01" "$OUT" "R-ID on a real YAML value line IS attributed"
|
||||
assert_present "R-SHREAL-01" "$OUT" "R-ID on a real shell command line IS attributed"
|
||||
# Conservative: a trailing inline comment on a REAL line is still seen (only whole-line
|
||||
# comments are dropped) — documents the intentional boundary of the fix.
|
||||
assert_present "R-INLINE-01" "$OUT" "inline comment on a real line is (conservatively) still seen"
|
||||
|
||||
# Removed comment lines are also excluded, while removed real lines still count. The file
|
||||
# must EXIST at the base, so we diff the removal against the commit that added it.
|
||||
git -C "$TMP" checkout -q -b feature2 main
|
||||
cat > "$TMP/notes/todelete.yaml" <<'YAML'
|
||||
# will delete a comment citing R-DELDOC-01 and a real line citing R-DELREAL-01
|
||||
kind: Thing
|
||||
name: implements-R-DELREAL-01
|
||||
YAML
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm add
|
||||
ADD_SHA="$(git -C "$TMP" rev-parse HEAD)"
|
||||
git -C "$TMP" rm -q "$TMP/notes/todelete.yaml" && git -C "$TMP" commit -qm remove
|
||||
OUT2="$(REPO_ROOT="$TMP" BASE_REF="$ADD_SHA" "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (removal):\n%s\n---\n' "$OUT2"
|
||||
assert_absent "R-DELDOC-01" "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
|
||||
assert_present "R-DELREAL-01" "$OUT2" "REMOVED real-line R-ID IS attributed"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
|
||||
else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
+20
-145
@@ -28,99 +28,28 @@ CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
# ---------------------------------------------------------------------------------
|
||||
# BASE-REF RESOLUTION — and why it now REFUSES rather than guesses.
|
||||
#
|
||||
# BUG THIS FIXES (local/CI divergence, 2026-08-31): every failure to establish a
|
||||
# reliable diff base used to degrade SILENTLY to an EMPTY touched set, which main()
|
||||
# reported as "no changed files vs base" and exit 0. An empty set means the coverage
|
||||
# `--require` gate is never applied, so the merge gate printed "MERGE GATE: PASS"
|
||||
# having checked NOTHING — while CI, with a correct base, computed 17 touched
|
||||
# requirements and blocked. A false green is the worst outcome this project can
|
||||
# produce, so every one of those degradations is now a LOUD exit 3:
|
||||
#
|
||||
# * HEAD~1 last resort REMOVED. It silently narrowed the diff to the last commit
|
||||
# (and on a one-commit repo failed into an empty set).
|
||||
# * two-dot fallback REMOVED. It masked a failing three-dot diff as "no changes".
|
||||
# * unresolvable base exit 3 (was: HEAD~1 or empty).
|
||||
# * diff command fails exit 3 (was: empty set).
|
||||
# * HEAD not ahead of exit 3 (was: exit 0 with an empty set — this is the case
|
||||
# the base that produced the false green: running the gate while
|
||||
# sitting on `main`, or on a branch already merged).
|
||||
#
|
||||
# Exit 3 is the ci/README.md "SKIPPED / not-run" code: NOT a pass, NOT a fail. Callers
|
||||
# (ci/gate.sh) must surface it as an UNDECIDED merge decision. An honest, noisy skip
|
||||
# beats a quiet false pass.
|
||||
#
|
||||
# Order of preference: BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs, target
|
||||
# branch) > origin/main > main. Each candidate is VERIFIED to resolve to a commit
|
||||
# before it is used, so a shallow checkout that lacks origin/main is reported, not
|
||||
# papered over.
|
||||
# Base ref to diff against. In Gitea/GitHub Actions PR runs this is the target
|
||||
# branch (main). Locally, default to main; override with BASE_REF.
|
||||
: "${BASE_REF:=}"
|
||||
BASE="" # resolved base ref
|
||||
BASE_SOURCE="" # where it came from (for the log line)
|
||||
BASE_PROBLEM="" # why resolution failed, when it did
|
||||
|
||||
rev() { git -C "$REPO_ROOT" rev-parse --verify -q "$1^{commit}" 2>/dev/null; }
|
||||
|
||||
# Loud, structured refusal. Prints to stderr only; stdout stays empty so a caller that
|
||||
# captures stdout gets NO requirement list rather than a silently-empty one.
|
||||
#
|
||||
# $1 is a machine-readable REASON CODE, also written to $TOUCHED_REASON_FILE when that
|
||||
# env var names a path. ci/gate.sh uses it to tell the two cases apart:
|
||||
# on-base HEAD is the base branch (or already merged) — nothing to gate. Legitimate
|
||||
# (a push-to-main CI run lands here), so the gate reports NOT APPLICABLE
|
||||
# rather than PASS, and does not fail the job.
|
||||
# no-base no reliable base ref exists in this checkout — the ENVIRONMENT is broken,
|
||||
# diff-failed or git could not produce the diff. A merge decision is impossible, so the
|
||||
# gate BLOCKS rather than guessing.
|
||||
undecidable() {
|
||||
local code="$1"; shift
|
||||
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "$code" > "$TOUCHED_REASON_FILE"; fi
|
||||
err "TOUCHED-REQUIREMENT SET NOT DETERMINABLE (${code})"
|
||||
err " reason: $*"
|
||||
err " consequence: the R-TEST-03 coverage gate CANNOT be applied to this checkout."
|
||||
err " This is exit 3 (SKIPPED / not-run), never a pass — see ci/README.md."
|
||||
err " Fix one of:"
|
||||
err " * check out the feature branch you mean to gate (not the base branch), or"
|
||||
err " * git fetch origin main (so origin/main exists and is current), or"
|
||||
err " * BASE_REF=<ref> make -C ci gate to name the base explicitly."
|
||||
exit 3
|
||||
}
|
||||
|
||||
resolve_base() {
|
||||
local cand
|
||||
if [ -n "$BASE_REF" ]; then
|
||||
if rev "$BASE_REF" >/dev/null; then BASE="$BASE_REF"; BASE_SOURCE="BASE_REF"; return 0; fi
|
||||
BASE_PROBLEM="BASE_REF='${BASE_REF}' does not resolve to a commit in ${REPO_ROOT}"
|
||||
return 1
|
||||
fi
|
||||
if [ -n "${GITHUB_BASE_REF:-}" ]; then
|
||||
for cand in "origin/${GITHUB_BASE_REF}" "${GITHUB_BASE_REF}"; do
|
||||
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="GITHUB_BASE_REF"; return 0; fi
|
||||
done
|
||||
BASE_PROBLEM="GITHUB_BASE_REF='${GITHUB_BASE_REF}' is set, but neither 'origin/${GITHUB_BASE_REF}' nor '${GITHUB_BASE_REF}' exists here (shallow checkout? the workflow needs fetch-depth: 0)"
|
||||
return 1
|
||||
fi
|
||||
for cand in origin/main main; do
|
||||
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="default"; return 0; fi
|
||||
done
|
||||
BASE_PROBLEM="no base ref resolves (tried origin/main, main) in ${REPO_ROOT}"
|
||||
return 1
|
||||
if [ -n "$BASE_REF" ]; then echo "$BASE_REF"; return; fi
|
||||
if [ -n "${GITHUB_BASE_REF:-}" ]; then echo "origin/${GITHUB_BASE_REF}"; return; fi
|
||||
if git -C "$REPO_ROOT" rev-parse --verify -q origin/main >/dev/null; then echo "origin/main"; return; fi
|
||||
if git -C "$REPO_ROOT" rev-parse --verify -q main >/dev/null; then echo "main"; return; fi
|
||||
echo "HEAD~1" # last resort
|
||||
}
|
||||
|
||||
# Both diff helpers assume BASE is already resolved and validated by main(). They return
|
||||
# git's exit status so the CALLER can distinguish "no changes" from "the diff failed" —
|
||||
# conflating those two is exactly what produced the false green.
|
||||
changed_files() {
|
||||
local base; base="$(resolve_base)"
|
||||
# Three-dot: changes on this branch since it diverged from base.
|
||||
git -C "$REPO_ROOT" diff --name-only "${BASE}...HEAD"
|
||||
if git -C "$REPO_ROOT" diff --name-only "${base}...HEAD" 2>/dev/null; then return; fi
|
||||
git -C "$REPO_ROOT" diff --name-only "${base}" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# The ADDED/REMOVED lines of the change, for the R-ID content scan — excluding
|
||||
# documentation.
|
||||
#
|
||||
# THREE scoping rules, all learned from the gate blocking merges it should not have:
|
||||
# TWO scoping rules, both learned from the gate blocking merges it should not have:
|
||||
#
|
||||
# 1. Scan the DIFF, not whole file contents. Grepping whole files meant a change
|
||||
# inherited every requirement those files had EVER mentioned, on lines the author
|
||||
@@ -137,34 +66,14 @@ changed_files() {
|
||||
# that matter (conformance/, schemas/, catalog/, clusters/, ci/) are covered by
|
||||
# the path->requirement fallback regardless of file type.
|
||||
#
|
||||
# 3. Skip COMMENT-ONLY lines in non-Markdown files — rule 2's logic, applied to the
|
||||
# prose that lives INSIDE code files. A requirement ID cited in a YAML/shell `#`
|
||||
# comment is documentation, not a declaration, exactly like an ID in a `*.md` file.
|
||||
# Rule 2 drops whole Markdown files, but a `.gitea/workflows/*.yml` or `ci/*.sh`
|
||||
# HEADER that explains a requirement would otherwise re-attribute it. Concretely:
|
||||
# gate.yml's header cites R-TEST-06 (enforced by the spec-immutability STAGE, not by
|
||||
# any Chainsaw test, so a known-uncovered ID); before this rule, editing that comment
|
||||
# re-attributed R-TEST-06 and the coverage `--require` check false-blocked the merge —
|
||||
# the same self-referential trap rule 2 fixed for `*.md`. Applied in main()'s content
|
||||
# scan (see there): after the diff marker is stripped, a line whose first non-blank
|
||||
# character is `#` is dropped BEFORE the R-ID grep. Deliberately conservative — only
|
||||
# WHOLE-LINE comments are dropped; an R-ID on a real code/spec line is still detected
|
||||
# (authoritative declarations — test tags, policies, ServiceDefinitions — are never a
|
||||
# bare comment), so this only narrows PROSE, never a declaration.
|
||||
#
|
||||
# Removed lines count: deleting a test's requirement tag, or a policy's self-declared
|
||||
# ID, touches that requirement just as much as adding one.
|
||||
# `:(exclude)*.md` drops Markdown from the content scan (rule 2 above).
|
||||
# `:(exclude)ci/tests/` drops this dir's OWN unit-test fixtures: those files embed example
|
||||
# R-IDs as DATA (asserting the scan's behaviour), never as declarations — the same
|
||||
# "reference, not declaration" logic as rule 2. Their gate coverage is unaffected: any
|
||||
# ci/ change (ci/tests/ included) still attributes R-GITOPS-01/R-TEST-02/R-TEST-03 via the
|
||||
# path->requirement map below. NOTE: this excludes ci/tests/ only — conformance/tests/
|
||||
# carry AUTHORITATIVE `conformance.home.k3s/R-*` labels and are deliberately still scanned.
|
||||
# Three-dot to match changed_files(): changes on this branch since it diverged from base.
|
||||
# `:(exclude)*.md` drops Markdown from the content scan (rule 2 above). Three-dot to
|
||||
# match changed_files(): changes on this branch since it diverged from base.
|
||||
changed_diff() {
|
||||
local -a excl=(':(exclude)*.md' ':(exclude)ci/tests/')
|
||||
git -C "$REPO_ROOT" diff -U0 "${BASE}...HEAD" -- . "${excl[@]}"
|
||||
local base; base="$(resolve_base)"
|
||||
if git -C "$REPO_ROOT" diff -U0 "${base}...HEAD" -- . ':(exclude)*.md' 2>/dev/null; then return; fi
|
||||
git -C "$REPO_ROOT" diff -U0 "${base}" -- . ':(exclude)*.md' 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Coarse path -> requirement map. Deliberately conservative; extend as subtrees
|
||||
@@ -189,34 +98,9 @@ path_requirements() {
|
||||
}
|
||||
|
||||
main() {
|
||||
local files reqs="" rc=0 base_sha head_sha local_main merge_base
|
||||
|
||||
resolve_base || undecidable no-base "$BASE_PROBLEM"
|
||||
base_sha="$(rev "$BASE")" || undecidable no-base "resolved base '${BASE}' vanished"
|
||||
head_sha="$(rev HEAD)" || undecidable no-base "HEAD does not resolve to a commit (empty repo?)"
|
||||
log "diff base: ${BASE} (${BASE_SOURCE}) @ ${base_sha:0:12} ... HEAD @ ${head_sha:0:12}"
|
||||
|
||||
# Advisory: a never-fetched origin/main silently WIDENS the set (over-broad, never a
|
||||
# false green) — surfaced so an unexpectedly large touched set has an explanation.
|
||||
if [ "$BASE" = "origin/main" ] && local_main="$(rev main)" && [ -n "$local_main" ] \
|
||||
&& [ "$local_main" != "$base_sha" ]; then
|
||||
warn "local 'main' (${local_main:0:12}) != origin/main (${base_sha:0:12}) — if origin/main"
|
||||
warn " was not fetched recently the touched set may be stale: git fetch origin main"
|
||||
fi
|
||||
|
||||
merge_base="$(git -C "$REPO_ROOT" merge-base "$BASE" HEAD 2>/dev/null)" \
|
||||
|| undecidable no-base "no merge base between '${BASE}' and HEAD (unrelated histories, or a shallow clone)"
|
||||
if [ "$merge_base" = "$head_sha" ]; then
|
||||
undecidable on-base "HEAD is not ahead of '${BASE}' (merge-base == HEAD): this checkout contains no branch commits to gate. The empty touched set would be empty BY DEFAULT, not by verification."
|
||||
fi
|
||||
|
||||
files="$(changed_files)" || rc=$?
|
||||
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff --name-only ${BASE}...HEAD failed (rc=${rc})"
|
||||
if [ -z "$files" ]; then
|
||||
warn "HEAD is ahead of ${BASE} but the three-dot diff is empty (self-cancelling commits?)"
|
||||
warn " -> genuinely empty touched-requirement set; base WAS established."
|
||||
exit 0
|
||||
fi
|
||||
local files reqs=""
|
||||
files="$(changed_files)"
|
||||
[ -n "$files" ] || { log "no changed files vs base; empty touched-requirement set"; exit 0; }
|
||||
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
@@ -225,18 +109,9 @@ main() {
|
||||
|
||||
# Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
|
||||
# `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
|
||||
# `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
|
||||
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
|
||||
# YAML/shell comments (rule 3 in changed_diff's header) BEFORE the R-ID grep — a
|
||||
# requirement cited only in a `#` comment is prose, not a declaration.
|
||||
local diff_txt
|
||||
diff_txt="$(changed_diff)" || rc=$?
|
||||
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff -U0 ${BASE}...HEAD failed (rc=${rc})"
|
||||
reqs+=" $(printf '%s\n' "$diff_txt" \
|
||||
reqs+=" $(changed_diff \
|
||||
| grep -E '^[+-]' \
|
||||
| grep -vE '^(\+\+\+|---)' \
|
||||
| sed -E 's/^[+-]//' \
|
||||
| grep -vE '^[[:space:]]*#' \
|
||||
| grep -hoE 'R-[A-Z]+-[0-9]+' \
|
||||
| tr '\n' ' ' || true)"
|
||||
|
||||
|
||||
+2
-2
@@ -9,7 +9,7 @@
|
||||
# whatever platform/kyverno and conformance/ pin — see integration notes in README.
|
||||
|
||||
# L0 policy gate (Kyverno CLI) + ServiceDefinition validation (kyverno-json)
|
||||
KYVERNO_VERSION="1.19.0"
|
||||
KYVERNO_VERSION="1.13.4"
|
||||
KYVERNO_JSON_VERSION="0.0.3"
|
||||
|
||||
# L1/L2/L3 declarative runner (Chainsaw) — invoked via conformance/ entrypoint,
|
||||
@@ -18,7 +18,7 @@ CHAINSAW_VERSION="0.2.12"
|
||||
|
||||
# Lint / manifest tooling
|
||||
YAMLLINT_VERSION="1.35.1"
|
||||
KUBECTL_VERSION="1.36.4"
|
||||
KUBECTL_VERSION="1.31.4"
|
||||
YQ_VERSION="4.44.6"
|
||||
|
||||
# ServiceDefinition JSON-Schema validation (schemas/validate.sh). Pip-installed;
|
||||
|
||||
+2
-28
@@ -29,9 +29,8 @@ clusters/
|
||||
|
||||
## How the overlay model satisfies R-ENV-02
|
||||
|
||||
The base authors the reconciliation graph **once**. `platform.yaml` and the app-wave Kustomizations
|
||||
(`apps-wave-0/1/2.yaml` — the single `apps.yaml` was retired; `apps-vms.yaml` deliberately omits it,
|
||||
see below) declare `postBuild.substituteFrom` the `cluster-vars` ConfigMap. Platform and
|
||||
The base authors the reconciliation graph **once**. `platform.yaml` and `apps.yaml` (Flux
|
||||
`Kustomization`s) both declare `postBuild.substituteFrom` the `cluster-vars` ConfigMap. Platform and
|
||||
app manifests use `${TOKEN}` placeholders for every value that legitimately differs between
|
||||
environments (issuer, replica counts, storage sizes, retention, node placement).
|
||||
|
||||
@@ -40,31 +39,6 @@ they add no resources of their own and never copy/fork a base manifest. Promotio
|
||||
([`docs/02`](../../home-network-k3s/docs/02-environments-and-provisioning.md)) is a change to the
|
||||
prod overlay's variables, nothing else.
|
||||
|
||||
### LAN addressing is a knob, not a literal (fixed 2026-08-31)
|
||||
|
||||
Until 2026-08-31 the base manifests hardcoded the **dev** LAN: the k3s API host Cilium dials, the
|
||||
Cilium LB IP pool range, the pinned crafty/unifi L4 VIPs and the off-cluster S3 backup endpoint. That
|
||||
broke R-ENV-02 outright — `clusters/prod` would have rendered dev addresses (192.168.1.x) onto the
|
||||
prod LAN (192.168.2.x). They are now the `cluster-vars` "LAN addressing" block
|
||||
(`K3S_API_HOST`, `CILIUM_LB_POOL_START`/`_STOP`, `UNIFI_L4_VIP`, `CRAFTY_L4_VIP`,
|
||||
`S3_BACKUP_ENDPOINT`, alongside the pre-existing `GATEWAY_VIP`), set explicitly by BOTH overlays.
|
||||
|
||||
Two rules when touching that block:
|
||||
|
||||
1. **Substitution coverage.** A `${TOKEN}` only resolves in a path whose Flux Kustomization declares
|
||||
`postBuild.substituteFrom: cluster-vars`; anywhere else the literal string `${TOKEN}` ships to the
|
||||
cluster. `apps-vms` was the one path without it and now has it — which means every shell `${...}`
|
||||
in a VM manifest's cloud-init must be escaped `$${...}` (see `clusters/base/apps-vms.yaml` for the
|
||||
full envsubst escape contract).
|
||||
2. **`GATEWAY_VIP` must be inside `CILIUM_LB_POOL_START..STOP`,** and in practice equal to START: the
|
||||
`shared` Gateway pins no address, so Cilium LB-IPAM gives it the pool's first free IP while
|
||||
split-horizon CoreDNS answers with `${GATEWAY_VIP}`. A VIP outside the pool blackholes in-cluster
|
||||
hairpin traffic while the Gateway still looks healthy.
|
||||
|
||||
Values the operator has not decided are `TBD-PROD-*` placeholders, not guesses: prod inherits the
|
||||
DEV base defaults if an overlay op is simply omitted, so an invalid value that fails loudly is
|
||||
strictly safer than a plausible one that quietly comes up on the wrong network.
|
||||
|
||||
## How Flux is pointed at an environment
|
||||
|
||||
`flux bootstrap` (run by WS-INFRA in Phase 1) targets one environment's path per cluster:
|
||||
|
||||
@@ -23,21 +23,7 @@ spec:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./apps/vms
|
||||
# postBuild.substituteFrom ENABLED 2026-08-31 (R-ENV-02). It was previously omitted because Flux's
|
||||
# envsubst runs over the ENTIRE manifest — including VM cloud-init shell scripts — and would mangle
|
||||
# every `${...}`; the omission was safe only while no VM manifest referenced a cluster-var. It now
|
||||
# does: apps/vms/unifi/50-service-l4.yaml pins the UniFi L4 VIP from ${UNIFI_L4_VIP} instead of the
|
||||
# hardcoded dev-LAN literal that would have shipped 192.168.1.54 onto the PROD LAN.
|
||||
#
|
||||
# The escape contract the old comment prescribed is now in force and MUST be honoured by every VM
|
||||
# manifest under this path (verified with `flux envsubst`):
|
||||
# `${VAR}` -> substituted (empty when unset) => escape shell braces as `$${VAR}`, which
|
||||
# envsubst emits back as a literal `${VAR}`.
|
||||
# `$VAR` -> left alone by envsubst => bare shell vars need NO escaping.
|
||||
# `$(cmd)` -> left alone by envsubst => command substitution needs NO escaping.
|
||||
# `${ARR[0]}` (e.g. ${PIPESTATUS[0]}) -> HARD ERROR "missing closing brace", which fails this
|
||||
# whole Kustomization. NEVER use an indexed expansion here, escaped or not.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# NO postBuild.substituteFrom: Flux's envsubst would run over the ENTIRE manifest, including the
|
||||
# VM cloud-init shell scripts, and mangle every `${...}` (e.g. ${UOS_URL}, ${PIPESTATUS[0]}) —
|
||||
# corrupting the in-guest install. VM manifests reference no cluster-vars, so substitution is
|
||||
# unneeded here. A future VM that needs a cluster-var must escape its shell `${...}` as `$${...}`.
|
||||
@@ -1,60 +0,0 @@
|
||||
---
|
||||
# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
|
||||
# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
|
||||
#
|
||||
# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
|
||||
#
|
||||
# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
|
||||
# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
|
||||
# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
|
||||
# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
|
||||
#
|
||||
# * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
|
||||
# * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
|
||||
# tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
|
||||
# `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
|
||||
# the valve the 2026-09-01 substitution incident wanted and did not have.
|
||||
# * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
|
||||
# `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
|
||||
#
|
||||
# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
|
||||
# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
|
||||
# expressed as an explicit edge instead of an accident of co-tenancy.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: cilium
|
||||
namespace: flux-system
|
||||
spec:
|
||||
# A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
|
||||
# kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
|
||||
interval: 1h
|
||||
retryInterval: 2m
|
||||
# 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
|
||||
# rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
|
||||
timeout: 10m
|
||||
prune: true
|
||||
# TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
|
||||
wait: true
|
||||
# Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
|
||||
# by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
|
||||
# is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
|
||||
# revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
|
||||
# `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
|
||||
dependsOn:
|
||||
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/cilium
|
||||
# MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
|
||||
# kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
|
||||
# this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
|
||||
# renders empty). That is the intended fail-closed behaviour — but it means the stanza is
|
||||
# load-bearing for the CNI coming up at all. Do not remove it.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
|
||||
# oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).
|
||||
@@ -21,56 +21,13 @@ data:
|
||||
DNS_ZONE: "keithjackson.dev" # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
|
||||
# ExternalDNS domain-filter must be the zone, not the subdomain
|
||||
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
||||
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
||||
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
|
||||
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
|
||||
# takes the pool's first free IP, and coredns-custom answers
|
||||
# *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
|
||||
# hairpins through the Gateway (correct SNI -> wildcard cert).
|
||||
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
|
||||
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
|
||||
# forgets this knob at least lands on a REAL network). A prod overlay
|
||||
# MUST still set the prod LAN VIP alongside the prod L2 pool range.
|
||||
|
||||
# --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply -------------
|
||||
# These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint:
|
||||
# `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24)
|
||||
# — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values
|
||||
# (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the
|
||||
# dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block
|
||||
# before `flux bootstrap --path=clusters/prod`.
|
||||
#
|
||||
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
|
||||
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
|
||||
# (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
|
||||
# velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
|
||||
# ship the literal string into the cluster.
|
||||
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
|
||||
# [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
|
||||
# main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
|
||||
# reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
|
||||
K3S_API_HOST: "192.168.1.50"
|
||||
|
||||
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
|
||||
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
|
||||
# tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in
|
||||
# practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's
|
||||
# first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and
|
||||
# the flagged block in clusters/prod/kustomization.yaml.)
|
||||
CILIUM_LB_POOL_START: "192.168.1.53"
|
||||
CILIUM_LB_POOL_STOP: "192.168.1.60"
|
||||
|
||||
# Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the
|
||||
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
||||
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
||||
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
||||
|
||||
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
|
||||
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
|
||||
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
|
||||
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR.
|
||||
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
|
||||
GATEWAY_VIP: "10.73.0.240" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
||||
# ingress VIP of the `shared` Gateway (first IP of the Cilium
|
||||
# LB IP pool 10.73.0.240-.250, platform-config/cilium/l2-ip-pool.yaml).
|
||||
# coredns-custom answers *.home.keithjackson.dev with THIS so pod->
|
||||
# public-hostname traffic hairpins through the Gateway (correct SNI ->
|
||||
# wildcard cert). Base default = dev VIP; a prod overlay must set the
|
||||
# prod LAN VIP alongside the prod L2 pool range when hardware is built.
|
||||
|
||||
# --- storage (ADR-0003) ---
|
||||
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
||||
|
||||
@@ -5,13 +5,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- flux-system.yaml # GitRepository (reconciliation source)
|
||||
# Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
|
||||
# of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
|
||||
# Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
|
||||
# `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
|
||||
# interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
|
||||
# freeze the other eleven platform components. See clusters/base/cilium.yaml.
|
||||
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
||||
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
||||
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
||||
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
|
||||
|
||||
@@ -11,53 +11,8 @@ spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
# Cilium was moved OUT of this tier into its own Kustomization (clusters/base/cilium.yaml)
|
||||
# on 2026-09-01. `prune: true` is restored here now that the ownership transfer is VERIFIED
|
||||
# live: Namespace/kube-system, the Cilium HelmRelease and the HelmRepository all carry
|
||||
# `kustomize.toolkit.fluxcd.io/name=cilium`, this Kustomization's inventory no longer lists
|
||||
# any of the three, and the `cilium` Kustomization's inventory holds exactly those three.
|
||||
#
|
||||
# LESSON WORTH KEEPING: `platform/cilium/namespace.yaml` was the ONLY declaration of
|
||||
# Namespace/kube-system under ./platform. Moving that path out of a prune:true Kustomization
|
||||
# made the stale set `delete namespace kube-system` — coredns, metrics-server, local-path AND
|
||||
# Cilium. Before moving ANY platform/* component in future, check whether it is the sole
|
||||
# declarer of a shared Namespace; that turns a component-scoped prune into a cluster-scoped one.
|
||||
prune: true
|
||||
prune: true # drift correction: delete resources removed from git (R-GITOPS-03)
|
||||
wait: true # block `apps` until platform is healthy
|
||||
# ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
|
||||
#
|
||||
# `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root
|
||||
# `flux-system` Kustomization (clusters/<env>/sync.yaml), not by this one. Without an edge here,
|
||||
# `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a
|
||||
# NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING.
|
||||
# That is how `k8sServiceHost: ""` reached the Cilium HelmRelease.
|
||||
#
|
||||
# `dependsOn` is REVISION-gated, not merely Ready-gated (verified against
|
||||
# kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version):
|
||||
# when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile
|
||||
# revision N until the dependency has APPLIED revision N. Applying revision N is exactly what
|
||||
# writes a newly-added `cluster-vars` knob — so this one edge closes the race.
|
||||
#
|
||||
# `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting
|
||||
# consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty,
|
||||
# velero) chains transitively through it. One edge therefore protects the whole graph.
|
||||
#
|
||||
# TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest
|
||||
# ANYWHERE under the root's path (clusters/<env>, which includes clusters/base) fails the root
|
||||
# Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge
|
||||
# the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken
|
||||
# entrypoint should stop the line, not let consumers render against a half-applied variable set.
|
||||
# No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never
|
||||
# health-gates its own children (see clusters/dev/sync.yaml).
|
||||
dependsOn:
|
||||
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
|
||||
# THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
|
||||
# (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
|
||||
# suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
|
||||
# nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
|
||||
# is healthy. It is also revision-gated, which is what makes the prune-safety argument above
|
||||
# deterministic rather than a race.
|
||||
- name: cilium
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
|
||||
@@ -148,44 +148,12 @@ patches:
|
||||
- op: replace
|
||||
path: /data/ACME_CLUSTER_ISSUER
|
||||
value: "letsencrypt-prod"
|
||||
# --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
|
||||
# EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
|
||||
# equals the base default. These values are what the LIVE dev cluster is running right now, so
|
||||
# the substituted output is byte-identical to the pre-parameterization hardcoded manifests
|
||||
# (verified 2026-08-31). Never let this block drift from the live cluster.
|
||||
#
|
||||
# Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
|
||||
# (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
|
||||
# (base default 10.73.0.240 was the old libvirt VIP). coredns-custom answers
|
||||
# *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
|
||||
- op: replace
|
||||
path: /data/GATEWAY_VIP
|
||||
value: "192.168.1.53"
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials — the primary control-plane node
|
||||
# (k3s-1). [parity] equals the Ansible inventory's k3s_api_host for dev.
|
||||
- op: replace
|
||||
path: /data/K3S_API_HOST
|
||||
value: "192.168.1.50"
|
||||
# Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
|
||||
# (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_START
|
||||
value: "192.168.1.53"
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_STOP
|
||||
value: "192.168.1.60"
|
||||
# Pinned L4 VIPs out of that pool (lbipam.cilium.io/ips). Changing either MOVES a live device-
|
||||
# facing endpoint: unifi is the inform/STUN target every adopted UniFi device holds.
|
||||
- op: replace
|
||||
path: /data/UNIFI_L4_VIP
|
||||
value: "192.168.1.54"
|
||||
- op: replace
|
||||
path: /data/CRAFTY_L4_VIP
|
||||
value: "192.168.1.55"
|
||||
# Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
|
||||
# barmanObjectStore and Velero's backupStorageLocation write to.
|
||||
- op: replace
|
||||
path: /data/S3_BACKUP_ENDPOINT
|
||||
value: "http://192.168.1.9:9000"
|
||||
# dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
||||
|
||||
@@ -5,144 +5,11 @@
|
||||
# When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
|
||||
# here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
|
||||
# values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
|
||||
#
|
||||
# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
|
||||
# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
|
||||
# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
|
||||
# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
|
||||
# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
|
||||
# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
|
||||
# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
|
||||
# done here: prod is design-only and its root object is the prod bring-up owner's call.)
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- ../base
|
||||
- sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
|
||||
patches:
|
||||
# --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
|
||||
# Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
|
||||
# The `sops-age` Secret is created OUT OF BAND on the prod cluster
|
||||
# (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
|
||||
# Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: platform
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-wave-0
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-wave-1
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-wave-2
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: headlamp
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: crafty
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: velero
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
|
||||
#
|
||||
# *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
|
||||
#
|
||||
# Two independent reasons, both consequences of the split-horizon design:
|
||||
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
|
||||
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
|
||||
# prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
|
||||
# 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
|
||||
# answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
|
||||
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
|
||||
# gitea.home.keithjackson.dev to itself and never reach a repository.
|
||||
#
|
||||
# Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
|
||||
# mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
|
||||
# sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
|
||||
#
|
||||
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
|
||||
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
|
||||
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
|
||||
- patch: |-
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: GitRepository
|
||||
metadata:
|
||||
name: flux-system
|
||||
namespace: flux-system
|
||||
spec:
|
||||
url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
||||
ref:
|
||||
branch: main
|
||||
- target:
|
||||
kind: ConfigMap
|
||||
name: cluster-vars
|
||||
@@ -182,68 +49,8 @@ patches:
|
||||
- op: replace
|
||||
path: /data/VELERO_RETENTION
|
||||
value: "720h"
|
||||
# ==========================================================================================
|
||||
# PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12.
|
||||
# ==========================================================================================
|
||||
# These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would
|
||||
# have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and
|
||||
# this overlay supplies every one of them. Anything the operator has NOT yet decided stays an
|
||||
# INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED
|
||||
# op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*`
|
||||
# is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY.
|
||||
#
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table.
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# nodes 192.168.2.10 / .11 / .12 (Ansible inventory)
|
||||
# control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here)
|
||||
# ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below)
|
||||
# Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below)
|
||||
#
|
||||
# CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev
|
||||
# (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml)
|
||||
# pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and
|
||||
# apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a
|
||||
# matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30
|
||||
# proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept
|
||||
# answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing
|
||||
# in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy.
|
||||
# KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either.
|
||||
# The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and
|
||||
# LB-IPAM must never be able to hand it to a Service.
|
||||
- op: replace
|
||||
path: /data/GATEWAY_VIP
|
||||
value: "192.168.2.20"
|
||||
# Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above).
|
||||
# Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one.
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_START
|
||||
value: "192.168.2.20"
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_STOP
|
||||
value: "192.168.2.30"
|
||||
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
|
||||
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned
|
||||
# Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in
|
||||
# platform/cilium/helmrelease.yaml. Not yet decided.
|
||||
- op: replace
|
||||
path: /data/K3S_API_HOST
|
||||
value: "192.168.2.10"
|
||||
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
|
||||
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
|
||||
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
|
||||
# collide with the Gateway VIP or the node IPs (.10/.11/.12).
|
||||
- op: replace
|
||||
path: /data/UNIFI_L4_VIP
|
||||
value: "192.168.2.21"
|
||||
- op: replace
|
||||
path: /data/CRAFTY_L4_VIP
|
||||
value: "192.168.2.22"
|
||||
# Off-cluster DR target. Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the
|
||||
# network — a prod backup written there is not off-cluster DR and would mix two environments'
|
||||
# PITR streams in one bucket. Operator supplies the prod endpoint (full URL incl. scheme+port).
|
||||
- op: replace
|
||||
path: /data/S3_BACKUP_ENDPOINT
|
||||
value: "TBD-PROD-S3-BACKUP-ENDPOINT"
|
||||
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
|
||||
# TODO(WS-NET, Phase 7): set GATEWAY_VIP to the prod LAN ingress VIP (add a `replace` op here)
|
||||
# at the same time the prod Cilium LB IP pool range is supplied — the split-horizon
|
||||
# coredns-custom (platform/coredns) answers *.home.keithjackson.dev with ${GATEWAY_VIP}, so
|
||||
# the prod value must match the prod pool's Gateway address. Base default is the dev VIP.
|
||||
@@ -1,32 +0,0 @@
|
||||
---
|
||||
# Self-managing ROOT Kustomization for PROD (mirrors clusters/dev/sync.yaml).
|
||||
#
|
||||
# Reconciles ./clusters/prod from the GitRepository so the ENTIRE prod graph — the cluster-vars
|
||||
# ConfigMap, the GitRepository source, platform / platform-config / the app waves, and THIS object
|
||||
# itself — is applied from git on every push. dev/sync.yaml carried the note "a prod overlay needs
|
||||
# its own sync.yaml -> ./clusters/prod"; this is it.
|
||||
#
|
||||
# WHY IT IS NOW LOAD-BEARING, not just symmetry: `clusters/base/platform.yaml` gained
|
||||
# `dependsOn: [flux-system]` (2026-09-01) to close the post-build substitution race — a dependent
|
||||
# cannot reconcile revision N until the ROOT has APPLIED revision N. Without this object there is no
|
||||
# `flux-system` Kustomization on prod, so `platform` would block forever on a dependency that does
|
||||
# not exist, and NOTHING downstream would ever reconcile.
|
||||
#
|
||||
# `wait: false` is deliberate and is what keeps the dependsOn edge deadlock-free: the root reaches
|
||||
# Ready on apply success without health-gating its children (platform does its own wait/health
|
||||
# gating). Do not "harden" this to true.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: flux-system
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./clusters/prod
|
||||
@@ -5,25 +5,3 @@ _artifacts/
|
||||
|
||||
# Downloaded pinned tool binaries (run.sh may fetch chainsaw/kubectl here).
|
||||
.bin/
|
||||
|
||||
# Python bytecode from lib/ (status.py imports coverage.py); never committed.
|
||||
__pycache__/
|
||||
|
||||
# --- THE ONE EXCEPTION: the committed run ledger (runs/) -------------------------------
|
||||
# `runs/<UTC>-<short-sha>/{report.junit.xml,run.log}` + `runs/ledger.jsonl` are the
|
||||
# machine-readable proof-of-pass record that lib/status.py's docstring asks for. They are
|
||||
# SOURCE, not regenerated artifacts: a run's evidence cannot be reconstructed after the
|
||||
# fact, so it must survive in git.
|
||||
#
|
||||
# The negations are needed because the ROOT .gitignore ignores `*.junit.xml` globally
|
||||
# (and `coverage/`). A deeper .gitignore overrides a shallower one, and nothing ignores
|
||||
# the `conformance/runs/` directory itself, so re-including the files here works (git
|
||||
# cannot re-include a file whose PARENT DIRECTORY is excluded — that is why this must not
|
||||
# live under _artifacts/).
|
||||
#
|
||||
# The run LOG is the load-bearing half: Chainsaw's JUnit carries no stdout, and the L2/L3
|
||||
# tests self-skip GREEN, so the sentinel token in run.log is the only thing separating
|
||||
# "proven" from "skipped". Never commit a report.junit.xml without its run.log.
|
||||
!runs/
|
||||
!runs/**
|
||||
!runs/**/*.junit.xml
|
||||
@@ -18,11 +18,9 @@ behavioral / L3 fault-injection** (this harness).
|
||||
./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||
./run.sh --layer l2 # one layer: l1 | l2 | l3 | meta | all
|
||||
./run.sh --coverage-only # regenerate the coverage report (no cluster)
|
||||
./run.sh --status # derived requirement-STATUS report (no cluster)
|
||||
./run.sh --lint # structural-validate test defs (no cluster)
|
||||
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
./run.sh --require-all # phase-exit gate: every requirement must be covered
|
||||
./run.sh --record # run AND capture the evidence (see "Proof of pass")
|
||||
```
|
||||
|
||||
**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
|
||||
@@ -39,33 +37,6 @@ behavioral / L3 fault-injection** (this harness).
|
||||
used to be here; it now has a real L3 test — `tests/l3-fault-injection/node-loss` — and
|
||||
is covered, not exempted.)
|
||||
|
||||
### Coverage vs status — two different questions
|
||||
|
||||
`coverage.md` answers **"does a requirement-tagged test EXIST for R-x?"** (R-TEST-04).
|
||||
`status.md` (`./run.sh --status`, generator `lib/status.py`) answers the operator's
|
||||
question — **"what is the real state of each requirement given what is actually in the
|
||||
repo?"** — deriving three orthogonal axes per requirement:
|
||||
|
||||
1. **implementation** — is there a tracked artifact naming the requirement?
|
||||
2. **reconciliation** — is that artifact reachable by walking `resources:` from
|
||||
`clusters/<env>/kustomization.yaml` and expanding every Flux `Kustomization`'s
|
||||
`spec.path`? This is what separates **authored** from **actually reconciled**: a
|
||||
commented-out `resources:` entry (e.g. `platform/image-automation`) is correctly
|
||||
reported as in-git-but-not-on-any-cluster.
|
||||
3. **verification** — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts
|
||||
both as "covered"; status does not. TODO markers in a body always outrank an
|
||||
`executable: "true"` annotation, so ambiguity never upgrades a test.
|
||||
|
||||
**It deliberately refuses to derive proof-of-pass.** Nothing tracked in git records a test
|
||||
result — `_artifacts/` and `results/` are git-ignored and DECISIONS-LOG.md prose is not a
|
||||
signal — so every requirement's proof state is reported as `not-derivable` rather than
|
||||
defaulting to green. A status report that overstates completion is worse than none
|
||||
(`agents/definition-of-done.md`). Closing that gap needs a **committed, machine-readable
|
||||
run ledger**; until then the strongest claim the report makes is *wired and testable*.
|
||||
`--strict-coverage` also applies status's hygiene gate (registry↔spec drift,
|
||||
annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold).
|
||||
Exit codes follow `ci/lib.sh`: `0` pass / `1` fail / `3` skipped — a skip is never green.
|
||||
|
||||
`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
|
||||
the *same* script with the same config, so results are identical.
|
||||
|
||||
@@ -75,8 +46,6 @@ the *same* script with the same config, so results are identical.
|
||||
| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
|
||||
| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
|
||||
| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
|
||||
| `_artifacts/status/status.md` | Human requirement-STATUS report derived from repo state | operator / PM |
|
||||
| `_artifacts/status/status.json` | Machine status: implementation / reconciliation / verification per requirement | tooling |
|
||||
|
||||
## Reproducibility contract (R-TEST-02 — what the harness guarantees)
|
||||
- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
|
||||
@@ -105,90 +74,13 @@ requirements/registry.yaml mirror of EVERY spec R-* ID (coverage enumerates fr
|
||||
lib/
|
||||
common.sh tagging convention + test discovery/selection helpers
|
||||
coverage.py requirement-coverage generator (stdlib only)
|
||||
status.py requirement-STATUS generator (stdlib only; --selftest)
|
||||
record_run.py run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
|
||||
emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster)
|
||||
runs/
|
||||
ledger.jsonl TRACKED append-only proof-of-pass ledger (one row per
|
||||
testcase per run) — the only source status.py trusts
|
||||
<UTC>-<short-sha>/ TRACKED per-run evidence: run.log + report.junit.xml
|
||||
tests/
|
||||
l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md)
|
||||
incluster/
|
||||
chainsaw-job.yaml rbac.yaml in-cluster L2/L3 Job + scoped test SA (ADR-0011)
|
||||
```
|
||||
|
||||
## Proof of pass — the run ledger (`runs/`)
|
||||
|
||||
`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
|
||||
proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
|
||||
root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
|
||||
only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
|
||||
asked for — **a committed, machine-readable run ledger**.
|
||||
|
||||
```bash
|
||||
./run.sh --record # runs normally, and captures the evidence
|
||||
# -> conformance/runs/<UTC>-<short-sha>/run.log (tee'd, carries the sentinels)
|
||||
# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml (that run's JUnit)
|
||||
# -> conformance/runs/ledger.jsonl (one appended row per testcase)
|
||||
python3 lib/record_run.py --run-dir runs/<id> [--dry-run] # or record an existing run dir
|
||||
python3 lib/record_run.py --selftest # deterministic, no cluster
|
||||
```
|
||||
|
||||
**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
|
||||
that silently records itself is how a bad run becomes a permanent false proof. **Commit
|
||||
the run dir and the ledger** — an uncommitted ledger proves nothing.
|
||||
|
||||
### Why a committed JUnit alone would be WORSE than nothing
|
||||
|
||||
The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
|
||||
`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
|
||||
and `exit 0`, and their `check:` accepts either outcome —
|
||||
|
||||
```yaml
|
||||
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
|
||||
|| contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
|
||||
```
|
||||
|
||||
So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
|
||||
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
|
||||
A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
|
||||
R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.
|
||||
|
||||
**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
|
||||
`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
|
||||
`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
|
||||
verdict token), so the table below is never hand-maintained:
|
||||
|
||||
| Test | pass token (proof) | skip token (never proof) |
|
||||
|---|---|---|
|
||||
| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
|
||||
| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
|
||||
| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
|
||||
| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
|
||||
| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
|
||||
| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
|
||||
| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |
|
||||
|
||||
A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
|
||||
token present in that test's log lines, and **no** skip token anywhere in them. A skip
|
||||
token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
|
||||
`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
|
||||
non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
|
||||
skips as JUnit `<skipped/>`, never as a green pass.
|
||||
|
||||
### Staleness — a proof is a claim about a tree, not a badge
|
||||
|
||||
`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
|
||||
last commit touching that requirement's implementing manifests or its verifying test**.
|
||||
Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
|
||||
dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
|
||||
it never counts. Re-run with `--record` to refresh.
|
||||
|
||||
A requirement reads `proven` only when **every** verifying test does; one skipped, failed
|
||||
or unrecorded test drags the whole requirement down. With an empty ledger `--status`
|
||||
behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.
|
||||
|
||||
## How CI invokes it
|
||||
CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
|
||||
|
||||
|
||||
@@ -64,22 +64,6 @@ rules:
|
||||
- apiGroups: [""]
|
||||
resources: [pods/log]
|
||||
verbs: [get] # read fault/probe Job logs on failure
|
||||
# ---- Storage-tier precondition gate (READ-ONLY). Every storage test self-skips on the
|
||||
# absence of the `longhorn` StorageClass rather than failing: data-survival (R-STATE-01),
|
||||
# node-loss (R-STATE-02), dr-restore (R-STATE-03), vm-data-survival / vm-lifecycle /
|
||||
# vm-default-deny (R-PLAT-03). Without this read the gate cannot tell "absent" from
|
||||
# "forbidden" and would skip for the wrong reason.
|
||||
- apiGroups: [storage.k8s.io]
|
||||
resources: [storageclasses]
|
||||
verbs: [get, list]
|
||||
# ---- Controller-readiness preconditions (READ-ONLY). dr-restore (R-STATE-03) skips cleanly
|
||||
# unless the Velero Deployment and its node-agent DaemonSet (kopia File System Backup, the
|
||||
# volume-DATA path) are actually ready — a backup path that is not up is an absent
|
||||
# precondition, not a failure. No write verbs: tests never scale or patch a platform
|
||||
# controller.
|
||||
- apiGroups: [apps]
|
||||
resources: [deployments, daemonsets]
|
||||
verbs: [get, list, watch]
|
||||
# ---- Probe / fault Jobs (R-TLS-01/03 tls-probe; R-NET-06 tunnel stop/start)
|
||||
# `create` cannot be constrained by resourceNames, and the fault Job names are
|
||||
# fixed per-test (tunnel-stop/tunnel-start/tunnel-start-finally); scoped by the
|
||||
@@ -119,29 +103,10 @@ rules:
|
||||
# resources: [imagepolicies, imagerepositories]
|
||||
# verbs: [get, patch]
|
||||
#
|
||||
# ---- R-STATE-03 (DR restore, tests/l3-fault-injection/dr-restore): the drill creates a
|
||||
# Velero Backup of its OWN scratch namespace, destroys that namespace + its PV, creates a
|
||||
# Restore into a fresh namespace, and (in `finally`) issues a DeleteBackupRequest so the
|
||||
# objects it wrote are removed from the off-cluster bucket again. `create` cannot be
|
||||
# constrained by resourceNames; the blast radius is bounded by the fact that every object
|
||||
# the test creates or deletes is named `<its-own-chainsaw-namespace>-dr-*`.
|
||||
- apiGroups: [velero.io]
|
||||
resources: [backups, restores, deletebackuprequests]
|
||||
verbs: [create, delete, get, list, watch]
|
||||
# Read-only evidence: the BSL is the off-cluster target whose Available phase + endpoint the
|
||||
# test asserts; schedules prove backups are SCHEDULED (not one-off); podvolumebackups /
|
||||
# podvolumerestores prove the kopia snapshot of the volume DATA reached/returned from the
|
||||
# remote repository, and are dumped on failure for diagnosis.
|
||||
- apiGroups: [velero.io]
|
||||
resources: [backupstoragelocations, schedules, podvolumebackups, podvolumerestores]
|
||||
verbs: [get, list, watch]
|
||||
# R-STATE-03 needs `delete` on the PV, not merely read: the proof that the restore came from
|
||||
# OFF-cluster is that no in-cluster copy of the data survived. The test asserts its own PV is
|
||||
# gone, and force-deletes it if a Retain policy or a stuck finalizer left it behind. It only
|
||||
# ever names the PV bound to the PVC it created itself.
|
||||
- apiGroups: [""]
|
||||
resources: [persistentvolumes]
|
||||
verbs: [get, list, delete]
|
||||
# ---- R-STATE-03 (DR restore): trigger a Velero restore
|
||||
# - apiGroups: [velero.io]
|
||||
# resources: [restores, backups]
|
||||
# verbs: [create, get, list, watch]
|
||||
#
|
||||
# ---- R-GITOPS-03 (drift): mutate a reconciled resource + observe Flux revert
|
||||
# - apiGroups: [kustomize.toolkit.fluxcd.io]
|
||||
|
||||
@@ -47,7 +47,7 @@ conf_dirs_for_requirements() {
|
||||
done < <(conf_all_test_files) | sort -u
|
||||
}
|
||||
|
||||
# Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
|
||||
# Directories for a given layer (subtree): l1 | l2 | l3 | meta | all.
|
||||
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
|
||||
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
|
||||
# this is what lets `run.sh --layer l2,l3` (the ci/conformance.sh L2/L3 stage) select both
|
||||
@@ -58,7 +58,6 @@ conf_dirs_for_layer() {
|
||||
for layer in ${layers//,/ }; do
|
||||
case "$layer" in
|
||||
all) roots=("${CONF_ROOT}/tests"); break ;;
|
||||
l0) roots+=("${CONF_ROOT}/tests/l0-static") ;;
|
||||
l1) roots+=("${CONF_ROOT}/tests/l1-rendering") ;;
|
||||
l2) roots+=("${CONF_ROOT}/tests/l2-behavioral") ;;
|
||||
l3) roots+=("${CONF_ROOT}/tests/l3-fault-injection") ;;
|
||||
|
||||
@@ -115,9 +115,7 @@ def parse_test(path: Path) -> dict:
|
||||
# Layer inferred from path as a fallback / cross-check.
|
||||
rel = str(path)
|
||||
if not layer:
|
||||
if "l0-static" in rel:
|
||||
layer = "L0"
|
||||
elif "l1-rendering" in rel:
|
||||
if "l1-rendering" in rel:
|
||||
layer = "L1"
|
||||
elif "l2-behavioral" in rel:
|
||||
layer = "L2"
|
||||
|
||||
@@ -33,8 +33,7 @@ def collect(path: Path) -> tuple[str, list[str], str]:
|
||||
reqs = sorted({m.group(1).upper() for m in
|
||||
(LABEL_LINE_RE.match(l) for l in text.splitlines()) if m})
|
||||
rel = str(path)
|
||||
layer = ("L0" if "l0-static" in rel
|
||||
else "L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
||||
layer = ("L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
||||
else "L3" if "l3-fault-injection" in rel else "META")
|
||||
return name, reqs, layer
|
||||
|
||||
|
||||
@@ -1,949 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Append a conformance RUN LEDGER row per testcase — the committed proof-of-pass record.
|
||||
|
||||
WHY THIS EXISTS
|
||||
---------------
|
||||
`lib/status.py` reports PROOF as `not-derivable` for every requirement, and names its own
|
||||
fix in its docstring: *"a committed, machine-readable run ledger"*. There was none, because
|
||||
`conformance/_artifacts/` is git-ignored and the root `.gitignore` kills `*.junit.xml` and
|
||||
`coverage/` globally, so nothing about a green run ever reached repo state. This tool is
|
||||
that ledger's writer: it reads one run's JUnit + run log out of `conformance/runs/<run-id>/`
|
||||
and appends one JSON object per testcase to `conformance/runs/ledger.jsonl`, which IS
|
||||
tracked (see conformance/.gitignore for the negations that let it survive).
|
||||
|
||||
THE TRAP THIS TOOL EXISTS TO AVOID — READ BEFORE CHANGING ANYTHING
|
||||
------------------------------------------------------------------
|
||||
The L2/L3 tests are RUNNABLE / SELF-SKIPPING: when a precondition is absent (no `longhorn`
|
||||
StorageClass, no Velero CRDs, <3 Ready nodes) they echo a skip token and `exit 0`, and their
|
||||
`check:` block is deliberately written to accept either outcome, e.g. tests/l3-fault-
|
||||
injection/dr-restore:
|
||||
|
||||
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
|
||||
|| contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
|
||||
|
||||
So a SELF-SKIP exits 0 and Chainsaw records `passed` EXACTLY as a real run does. And
|
||||
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`
|
||||
(verified against the committed reports). Therefore:
|
||||
|
||||
*** A COMMITTED JUnit ALONE WOULD RECORD `DR_RESTORE_SKIPPED_OK` AS PROOF OF R-STATE-03. ***
|
||||
|
||||
AND THE SECOND TRAP, WHICH IS THE MIRROR OF THE FIRST
|
||||
------------------------------------------------------
|
||||
Chainsaw echoes each script's SOURCE into an indented `=== COMMAND` block before running
|
||||
it, so `echo NODE_LOSS_SKIPPED_OK` — the skip branch that was NOT taken — is present in
|
||||
the log of every run of that test. A section-blind grep therefore reads a skip token out
|
||||
of the source of every skip-gated test and can never record `proven` for any of them: the
|
||||
ledger fails closed so completely that it records nothing and proves nothing. Tokens are
|
||||
consequently read from SECTIONS (see classify_sections): a pass token only from
|
||||
`=== STDOUT`, a skip token from any section that is not script source. Skip still beats
|
||||
pass — within genuine output, which is the only place either was ever evidence.
|
||||
|
||||
That is the fake-green this project refuses, and it would be worse than no ledger at all
|
||||
because it would look authoritative. THE SENTINEL TOKEN IS THE ONLY THING THAT SEPARATES
|
||||
PROVEN FROM SKIPPED, AND IT EXISTS ONLY IN THE RUN LOG. Hence: this tool requires BOTH
|
||||
files, and for the Chainsaw layers it will NEVER emit `proof: "proven"` on a JUnit `passed`
|
||||
unless the log carries that test's own pass token and carries NO skip token.
|
||||
|
||||
HOW THE TOKENS ARE OBTAINED (derived, never hand-listed)
|
||||
--------------------------------------------------------
|
||||
Hard-coding a token table would rot the moment a test is edited. Instead every token is
|
||||
parsed out of the test's own `chainsaw-test.yaml`: each `contains($stdout, 'TOKEN')` in a
|
||||
`check:` block is a declared sentinel. Classification is by the project's naming convention
|
||||
(`*_SKIP*_OK` is a skip token, anything else is a pass token), and the TERMINAL pass token
|
||||
— the last non-skip token in file order — is the test's verdict token. Edit a test's tokens
|
||||
and this tool follows automatically; delete them and the test simply becomes unprovable
|
||||
rather than silently proven.
|
||||
|
||||
node-loss pass NODE_LOSS_VOLUME_AVAILABLE_OK skip NODE_LOSS_SKIPPED_OK
|
||||
dr-restore pass DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK skip DR_RESTORE_SKIPPED_OK
|
||||
(+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK / DR_SCHEDULE_SKIPPED_OK)
|
||||
|
||||
WHY L0/L1 NEED NO SENTINEL (and why that is not a loophole)
|
||||
-----------------------------------------------------------
|
||||
The offline layers are not run by Chainsaw. `lib/run_l1_offline.py` shells out to each
|
||||
dir's `check.sh` / `render-check.sh` and maps exit 3 to a JUnit `<skipped/>` element and
|
||||
any other non-zero to `<failure>`. A self-skip there is ALREADY a JUnit skip, never a green
|
||||
pass, so the JUnit is a truthful record on its own. Only the Chainsaw layers (L2/L3/META)
|
||||
can express "skipped" as "passed", and only they require a sentinel — see
|
||||
SENTINEL_REQUIRED_LAYERS.
|
||||
|
||||
LEDGER ROW (conformance/runs/ledger.jsonl — append-only, one object per testcase per run)
|
||||
ts_utc run-dir timestamp, else the JUnit's mtime (when)
|
||||
git_sha `git rev-parse HEAD`, suffixed `-dirty` when tracked files were
|
||||
modified — a proof captured from a dirty tree is NOT honoured by
|
||||
status.py, because what ran is not what is committed (what)
|
||||
env --env / CONF_ENV (cluster overlay the run targeted)
|
||||
chainsaw_version CHAINSAW_VERSION from versions.env (the pin IS the truth)
|
||||
kube_version `kubectl version` server gitVersion, else "unknown"
|
||||
layer L0|L1|L2|L3|META, from the test's `layer` annotation / JUnit property
|
||||
test testcase name (== the test dir basename, asserted by convention)
|
||||
requirements[] the `conformance.home.k3s/R-*` labels on the test — the SAME
|
||||
authoritative source coverage.py uses, not the JUnit classname
|
||||
junit_result passed | failed | error | skipped
|
||||
sentinel the DECIDING token grepped from run.log, or null
|
||||
proof proven | skipped | failed | junit-skipped
|
||||
| unknown-no-sentinel-declared | unknown-sentinel-absent
|
||||
run_id the runs/<dir> name (provenance: which log backs this row)
|
||||
note one line saying WHY this row landed on that proof state
|
||||
|
||||
`proven` is emitted on exactly one path (see decide()); every other path is non-green.
|
||||
|
||||
RECORDING IS EXPLICIT, NEVER IMPLICIT
|
||||
-------------------------------------
|
||||
Invoke this tool by hand, or via `run.sh --record`. A run that silently records itself is
|
||||
how a bad run becomes a permanent false proof, so `run.sh` without `--record` writes no
|
||||
ledger row at all.
|
||||
|
||||
EXIT CODES (ci/lib.sh convention: 0 pass / 1 fail / 3 skipped-not-green)
|
||||
0 rows appended.
|
||||
1 a real failure: unparsable JUnit, unreadable test tree, unwritable ledger.
|
||||
3 the inputs a ledger row needs are absent (no report.junit.xml, or no run.log).
|
||||
NOTHING is recorded and this is NEVER green — a run we cannot evidence is not a run.
|
||||
|
||||
Pure standard library, matching the harness's no-pip-dependency posture (coverage.py,
|
||||
status.py). PyYAML lives in catalog/render.py; the harness does not depend on it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
|
||||
_LIB = Path(__file__).resolve().parent
|
||||
if str(_LIB) not in sys.path:
|
||||
sys.path.insert(0, str(_LIB))
|
||||
# Importing a sibling would otherwise litter conformance/lib/__pycache__/ into the working
|
||||
# tree, and the lint stage scans untracked files. Same guard as status.py.
|
||||
sys.dont_write_bytecode = True
|
||||
import coverage as cov # noqa: E402 — reuse the registry/test parsers, don't fork them
|
||||
|
||||
CONF_ROOT = _LIB.parent
|
||||
REPO_ROOT = CONF_ROOT.parent
|
||||
DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
|
||||
|
||||
# The ONLY layers exempt from needing a pass sentinel: the offline ones, whose skips
|
||||
# lib/run_l1_offline.py already renders as JUnit <skipped/> rather than as a green pass
|
||||
# (see docstring). Written as an EXEMPT set, not a REQUIRED set, so that an unrecognised
|
||||
# or new layer fails CLOSED — it needs a sentinel — instead of quietly becoming provable
|
||||
# by JUnit alone.
|
||||
SENTINEL_EXEMPT_LAYERS = frozenset({"L0", "L1"})
|
||||
|
||||
# Layer from the test's subtree. Authoritative over the annotation: coverage.py's layer
|
||||
# regex stops at the first non-[L/A/M/0-9] character, so `layer: "META"` parses as "M" —
|
||||
# harmless for its own report, but here a mis-read layer decides whether a sentinel is
|
||||
# required at all, so it is derived from the path exactly as emit_skipped_junit.py does.
|
||||
_LAYER_BY_SUBTREE = (("/l0-static/", "L0"), ("/l1-rendering/", "L1"),
|
||||
("/l2-behavioral/", "L2"), ("/l3-fault-injection/", "L3"),
|
||||
("/meta/", "META"))
|
||||
|
||||
|
||||
def layer_of(path: Path) -> str:
|
||||
rel = str(path)
|
||||
for frag, layer in _LAYER_BY_SUBTREE:
|
||||
if frag in rel:
|
||||
return layer
|
||||
return ""
|
||||
|
||||
# A declared sentinel: `contains($stdout, 'TOKEN')` inside a Chainsaw `check:` block.
|
||||
# Both quote styles are accepted; the repo uses single quotes throughout.
|
||||
_SENTINEL_RE = re.compile(r"contains\(\s*\$stdout\s*,\s*['\"]([A-Za-z0-9_]+)['\"]\s*\)")
|
||||
|
||||
# Naming convention for a SKIP token (tests/l3-fault-injection/*: NODE_LOSS_SKIPPED_OK,
|
||||
# DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK, VM_DATA_SURVIVAL_SKIPPED_OK, ...).
|
||||
# Deliberately broad: mis-classifying a PASS token as a skip only ever loses a proof;
|
||||
# mis-classifying a SKIP token as a pass would manufacture one.
|
||||
_SKIP_TOKEN_RE = re.compile(r"_SKIP")
|
||||
|
||||
# A Chainsaw log line header, as it really appears (Go `t.Log` prefixes the source site and
|
||||
# pads it with backspaces, and the fields are ANSI-coloured — both are stripped first):
|
||||
# l.go:52: | 23:51:18 | node-loss | volume-stays-available... | SCRIPT | LOG |
|
||||
# Field 2 is the test name; it is how output is attributed back to a testcase. The
|
||||
# `<file>.go:<line>:` prefix is optional so a redirected/plain log still attributes.
|
||||
_LOG_HEADER_RE = re.compile(
|
||||
r"^\s*(?:[\w./-]+\.go:\d+:\s*)?"
|
||||
r"\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
|
||||
|
||||
# ANSI colour + the backspace padding Go writes ahead of a `t.Log` continuation. Stripped
|
||||
# before ANY parsing; neither ever appears at the head of a line, so indentation survives.
|
||||
_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
|
||||
|
||||
|
||||
def _strip_noise(line: str) -> str:
|
||||
return _ANSI_RE.sub("", line).replace("\b", "")
|
||||
|
||||
|
||||
# ===================================================================== output sections
|
||||
# THE BUG THIS SECTION MACHINERY EXISTS TO KILL — READ BEFORE CHANGING ANYTHING
|
||||
# -----------------------------------------------------------------------------
|
||||
# Chainsaw echoes the script it is ABOUT TO RUN, verbatim, in an indented `=== COMMAND`
|
||||
# block, and only afterwards prints what the script actually wrote in `=== STDOUT`. Every
|
||||
# skip-gated test contains its own skip token as SOURCE TEXT (`echo NODE_LOSS_SKIPPED_OK`),
|
||||
# so a raw whole-log grep finds that token in EVERY run of that test — including runs that
|
||||
# never took the skip branch. Combined with decide()'s (correct) rule that a skip token
|
||||
# outranks a pass token, a section-blind scan can never record `proven` for ANY test that
|
||||
# has a skip gate: the ledger fails closed so hard it records nothing. Observed on
|
||||
# runs/20260901T235117Z-e9a4b66, where NODE_LOSS_SKIPPED_OK appears twice as source at
|
||||
# lines 51/62 (`=== COMMAND`) and never in output, yet the run genuinely proved R-STATE-02.
|
||||
#
|
||||
# So tokens are attributed to SECTIONS, never to raw lines:
|
||||
# l.go:52: | 23:51:18 | node-loss | step | CMD | RUN | <- header, no section
|
||||
# === COMMAND <- marker (indent 8)
|
||||
# /usr/bin/sh -c set -euo pipefail <- source, indent >= 8
|
||||
# echo NODE_LOSS_SKIPPED_OK <- SOURCE, not output
|
||||
# l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG | <- indent 4 => section ends
|
||||
# === STDOUT <- marker
|
||||
# NODE_LOSS_VOLUME_AVAILABLE_OK <- the REAL emission
|
||||
#
|
||||
# A marker is an INDENTED `=== NAME` on a line of its own. Go's own test markers
|
||||
# (`=== RUN chainsaw`, `=== PAUSE`, `=== CONT`) are at column 0 and carry a test name
|
||||
# after the verb, so they match neither the indent nor the end-of-line anchor; they are
|
||||
# additionally named in _GO_TEST_MARKERS so a future indented variant still cannot open a
|
||||
# section that would swallow the whole log.
|
||||
_SECTION_MARKER_RE = re.compile(r"^(\s+)=== ([A-Z][A-Z0-9_]*)\s*$")
|
||||
_GO_TEST_MARKERS = frozenset({"RUN", "PAUSE", "CONT"})
|
||||
|
||||
# Pseudo-section for a log that contains NO section markers at all (the plain/redirected
|
||||
# logs the earlier ledger rows were written from, and the selftest fixtures). Such a log
|
||||
# never echoes script source, so there is nothing to be confused by and the legacy
|
||||
# whole-log scan is exactly as safe as it was — narrowing is only needed where a COMMAND
|
||||
# echo exists. Lines OUTSIDE a section in a log that DOES use sections are attributed to
|
||||
# "" and are evidence for nothing.
|
||||
UNSECTIONED = "*"
|
||||
|
||||
# A pass token counts ONLY where Chainsaw's own `contains($stdout, ...)` check looks:
|
||||
# real stdout. Anything else — source echo, stderr, an unrecognised future section — can
|
||||
# never manufacture a proof.
|
||||
PASS_SECTIONS = frozenset({"STDOUT", UNSECTIONED})
|
||||
|
||||
# Sections that carry SOURCE rather than output. A skip token is honoured in every OTHER
|
||||
# section (STDOUT, STDERR, any section name added by a later Chainsaw release), because
|
||||
# over-reading a skip only ever COSTS a proof, while over-reading a pass would fabricate
|
||||
# one. Unknown section names therefore fail closed in both directions.
|
||||
SOURCE_SECTIONS = frozenset({"COMMAND"})
|
||||
|
||||
|
||||
def classify_sections(log_text: str) -> list[tuple[str, str]]:
|
||||
"""[(section, line)] for every line of a run log.
|
||||
|
||||
`section` is the enclosing `=== NAME` block, "" for a line inside no block, or
|
||||
UNSECTIONED for every line when the log declares no sections at all. A block runs from
|
||||
its marker until the first non-blank line indented LESS than the marker (blank lines do
|
||||
not end it) or until the next marker; the marker line itself is content of nothing.
|
||||
"""
|
||||
rows: list[tuple[str, str]] = []
|
||||
cur, cur_indent, sectioned = "", 0, False
|
||||
for raw in log_text.splitlines():
|
||||
line = _strip_noise(raw)
|
||||
m = _SECTION_MARKER_RE.match(line)
|
||||
if m and m.group(2) not in _GO_TEST_MARKERS:
|
||||
sectioned = True
|
||||
cur, cur_indent = m.group(2), len(m.group(1))
|
||||
rows.append(("", line))
|
||||
continue
|
||||
if cur:
|
||||
if not line.strip() or len(line) - len(line.lstrip()) >= cur_indent:
|
||||
rows.append((cur, line))
|
||||
continue
|
||||
cur = ""
|
||||
rows.append(("", line))
|
||||
if not sectioned:
|
||||
return [(UNSECTIONED, line) for _, line in rows]
|
||||
return rows
|
||||
|
||||
|
||||
def scan_texts(lines) -> tuple[str, str]:
|
||||
"""(text a PASS token may be read from, text a SKIP token may be read from).
|
||||
|
||||
Accepts either [(section, line)] pairs from classify_sections() or a plain list of
|
||||
strings, which is treated as an unsectioned log (legacy whole-log scanning).
|
||||
"""
|
||||
pass_lines, skip_lines = [], []
|
||||
for item in lines:
|
||||
section, line = item if isinstance(item, tuple) else (UNSECTIONED, item)
|
||||
if section in PASS_SECTIONS:
|
||||
pass_lines.append(line)
|
||||
if section and section not in SOURCE_SECTIONS:
|
||||
skip_lines.append(line)
|
||||
return "\n".join(pass_lines), "\n".join(skip_lines)
|
||||
|
||||
_RUN_ID_TS_RE = re.compile(r"^(\d{8}T\d{6}Z)-")
|
||||
|
||||
|
||||
# ========================================================================= sentinels
|
||||
def parse_sentinels(test_yaml: Path) -> dict:
|
||||
"""Declared sentinel tokens of one Chainsaw test, derived from the test file itself.
|
||||
|
||||
Returns {"pass": [...], "skip": [...], "terminal_pass": str|None}. `terminal_pass` is
|
||||
the LAST non-skip token in file order — the test's verdict token, i.e. the one its
|
||||
final `check:` block asserts (dr-restore -> DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK).
|
||||
A test with no `contains($stdout, ...)` assertion declares no sentinel and is reported
|
||||
unprovable rather than proven; see decide().
|
||||
"""
|
||||
try:
|
||||
text = test_yaml.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return {"pass": [], "skip": [], "terminal_pass": None}
|
||||
toks = _SENTINEL_RE.findall(text)
|
||||
passes, skips = [], []
|
||||
for t in toks:
|
||||
(skips if _SKIP_TOKEN_RE.search(t) else passes).append(t)
|
||||
# De-duplicate, preserving order (dr-restore names DR_RESTORE_SKIPPED_OK twice).
|
||||
def uniq(xs: list[str]) -> list[str]:
|
||||
return list(dict.fromkeys(xs))
|
||||
passes, skips = uniq(passes), uniq(skips)
|
||||
return {"pass": passes, "skip": skips,
|
||||
"terminal_pass": passes[-1] if passes else None}
|
||||
|
||||
|
||||
def sentinel_map(tests_dir: Path) -> dict[str, dict]:
|
||||
"""{test-name: sentinels} for every test in the tree.
|
||||
|
||||
Keyed on the Chainsaw `metadata.name`, which is what lands in the JUnit `<testcase
|
||||
name=...>`. This repo keeps that equal to the test's directory basename; the mapping
|
||||
is by name, so it stays correct if that ever drifts.
|
||||
"""
|
||||
out: dict[str, dict] = {}
|
||||
for f in sorted(tests_dir.rglob("chainsaw-test.yaml")):
|
||||
meta = cov.parse_test(f)
|
||||
name = meta["name"] or f.parent.name
|
||||
s = parse_sentinels(f)
|
||||
s["dir"] = str(f.parent)
|
||||
s["layer"] = layer_of(f) or meta["layer"]
|
||||
s["requirements"] = meta["labels"]
|
||||
out[name] = s
|
||||
return out
|
||||
|
||||
|
||||
# =============================================================================== log
|
||||
def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[tuple[str, str]]],
|
||||
bool]:
|
||||
"""Split a run log into per-testcase buckets of (section, line) pairs.
|
||||
|
||||
Chainsaw prefixes its log lines with `| HH:MM:SS | <test> | <step> | ...` and prints a
|
||||
script's sections underneath that header, so the test name is carried by a state
|
||||
machine: a header naming a KNOWN test switches the current bucket, every following line
|
||||
lands in it. Requiring the name to be known is what stops run.sh's own `[run.sh]`
|
||||
chatter (and any other pipe-shaped line) from opening a bogus bucket.
|
||||
|
||||
Section labels come from classify_sections() and travel WITH the lines, so a bucket
|
||||
still knows which of its lines were script source and which were real output.
|
||||
|
||||
Returns (buckets, attributed) where `attributed` is False when the log contained no
|
||||
recognisable header at all — the signal for the degraded global scan in decide().
|
||||
"""
|
||||
buckets: dict[str, list[tuple[str, str]]] = {}
|
||||
cur: str | None = None
|
||||
attributed = False
|
||||
for section, line in classify_sections(log_text):
|
||||
m = _LOG_HEADER_RE.match(line)
|
||||
if m and m.group(1) in known:
|
||||
cur = m.group(1)
|
||||
attributed = True
|
||||
if cur is not None:
|
||||
buckets.setdefault(cur, []).append((section, line))
|
||||
return buckets, attributed
|
||||
|
||||
|
||||
def unique_tokens(smap: dict[str, dict]) -> set[str]:
|
||||
"""Tokens claimed by EXACTLY ONE test in the tree.
|
||||
|
||||
Only these may be used by the degraded global scan: `DNS_OK` is echoed by two different
|
||||
tests, so in an unattributable log it can never be evidence for either; but
|
||||
`NODE_LOSS_VOLUME_AVAILABLE_OK` names precisely one test and still can.
|
||||
"""
|
||||
owners: dict[str, set[str]] = {}
|
||||
for name, s in smap.items():
|
||||
for t in s["pass"] + s["skip"]:
|
||||
owners.setdefault(t, set()).add(name)
|
||||
return {t for t, o in owners.items() if len(o) == 1}
|
||||
|
||||
|
||||
# ============================================================================ decide
|
||||
def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
|
||||
scope: str) -> tuple[str | None, str, str]:
|
||||
"""The ONE place a testcase becomes `proven` — or, far more often, does not.
|
||||
|
||||
Returns (sentinel, proof, note).
|
||||
|
||||
Ordering is worst-first ON PURPOSE, so that ambiguity can only ever cost a proof:
|
||||
1. JUnit failure/error -> failed (a red run is recorded as red)
|
||||
2. JUnit <skipped/> -> junit-skipped (never green; ci/lib.sh rule)
|
||||
3. ANY declared skip token seen -> skipped (BEATS a green JUnit and beats
|
||||
IN REAL OUTPUT the pass token: if any gated
|
||||
step self-skipped, the run did
|
||||
not prove the requirement)
|
||||
4. no sentinel declared -> proven only for L0/L1, whose skips are already
|
||||
JUnit skips; unknown-no-sentinel-declared for
|
||||
the Chainsaw layers
|
||||
5. EVERY declared pass token seen -> proven
|
||||
6. anything else -> unknown-sentinel-absent
|
||||
Step 5 requires ALL pass tokens, not just the terminal one: every Chainsaw step here
|
||||
asserts `pass || skip`, so a missing pass token means a step did not reach its proven
|
||||
branch, whatever the JUnit says.
|
||||
|
||||
Steps 3 and 5 read from SECTION-SCOPED text, never from raw lines: a pass token counts
|
||||
only in `=== STDOUT`, a skip token counts in any section that is not script source.
|
||||
That narrows WHERE a token is read from and changes nothing about what it means — skip
|
||||
still outranks pass, and it outranks it within genuine output, which is the only place
|
||||
either of them was ever evidence. `lines` may be [(section, line)] pairs or plain
|
||||
strings (an unsectioned log; see scan_texts / UNSECTIONED).
|
||||
"""
|
||||
if junit_result in ("failed", "error"):
|
||||
return (None, "failed", f"JUnit recorded {junit_result}")
|
||||
if junit_result == "skipped":
|
||||
return (None, "junit-skipped",
|
||||
"JUnit <skipped/> — the test did not execute; a skip is never green")
|
||||
|
||||
pass_text, skip_text = scan_texts(lines)
|
||||
seen_skip = [t for t in sent["skip"] if t in skip_text]
|
||||
if seen_skip:
|
||||
return (seen_skip[0], "skipped",
|
||||
f"run log OUTPUT carries the SELF-SKIP token {seen_skip[0]!r} — the test "
|
||||
f"exited 0 without exercising the behaviour, so this is NOT proof")
|
||||
|
||||
if not sent["pass"] and not sent["skip"]:
|
||||
if layer not in SENTINEL_EXEMPT_LAYERS:
|
||||
return (None, "unknown-no-sentinel-declared",
|
||||
f"{layer or 'unknown-layer'} test declares no contains($stdout, ...) "
|
||||
f"sentinel, so a green JUnit cannot be told apart from a self-skip — "
|
||||
f"refusing to call it proven")
|
||||
return (None, "proven",
|
||||
f"{layer} offline check exited 0; run_l1_offline.py renders its skips as "
|
||||
f"JUnit <skipped/>, so a green JUnit is truthful without a sentinel")
|
||||
|
||||
missing = [t for t in sent["pass"] if t not in pass_text]
|
||||
if not missing:
|
||||
return (sent["terminal_pass"], "proven",
|
||||
f"run log OUTPUT carries every declared pass token "
|
||||
f"({', '.join(sent['pass'])}) and no skip token [{scope}]")
|
||||
return (None, "unknown-sentinel-absent",
|
||||
f"run log OUTPUT lacks pass token(s) {', '.join(missing)} for this test "
|
||||
f"[{scope}] — refusing to infer proof from the JUnit alone")
|
||||
|
||||
|
||||
# ============================================================================= junit
|
||||
def junit_cases(path: Path) -> list[dict]:
|
||||
"""[{name, layer, requirements, result}] for every <testcase> in a JUnit report."""
|
||||
root = ET.parse(path).getroot()
|
||||
cases: list[dict] = []
|
||||
for c in root.iter("testcase"):
|
||||
if c.find("failure") is not None:
|
||||
result = "failed"
|
||||
elif c.find("error") is not None:
|
||||
result = "error"
|
||||
elif c.find("skipped") is not None:
|
||||
result = "skipped"
|
||||
else:
|
||||
result = "passed"
|
||||
props = {p.get("name"): (p.get("value") or "")
|
||||
for p in c.iter("property") if p.get("name")}
|
||||
reqs = sorted(set(cov.RID_RE.findall(props.get("requirements", ""))))
|
||||
layer = (props.get("layer") or "").upper()
|
||||
if not layer:
|
||||
layer = (c.get("classname") or "").split(":", 1)[0].upper()
|
||||
cases.append({"name": c.get("name") or "?", "layer": layer,
|
||||
"requirements": reqs, "result": result})
|
||||
return cases
|
||||
|
||||
|
||||
# =============================================================================== git
|
||||
def git_sha(repo: Path) -> str:
|
||||
"""HEAD, suffixed `-dirty` when TRACKED files differ from it.
|
||||
|
||||
Untracked files are deliberately ignored, matching status.py's `tracked_files()`
|
||||
posture (untracked scratch is not repo state) — and because a `--record` run creates
|
||||
untracked `runs/<id>/` files as it goes, which would otherwise mark every run dirty.
|
||||
A `-dirty` sha is not a valid rev, so status.py's ancestry check cannot resolve it and
|
||||
the proof degrades to `stale-proof`: correct, since what ran is not what is committed.
|
||||
"""
|
||||
try:
|
||||
sha = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
|
||||
capture_output=True, text=True, check=True).stdout.strip()
|
||||
except (subprocess.CalledProcessError, OSError):
|
||||
return "unknown"
|
||||
dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--"],
|
||||
capture_output=True).returncode != 0
|
||||
return f"{sha}-dirty" if dirty else sha
|
||||
|
||||
|
||||
def pinned_chainsaw_version(conf_root: Path) -> str:
|
||||
"""CHAINSAW_VERSION from versions.env — the pin is the single source of truth."""
|
||||
f = conf_root / "versions.env"
|
||||
if f.is_file():
|
||||
m = re.search(r'^CHAINSAW_VERSION="?([^"\n]+)"?', f.read_text(), re.M)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return "unknown"
|
||||
|
||||
|
||||
def server_kube_version() -> str:
|
||||
"""Server gitVersion via `kubectl version` (read-only), else "unknown"."""
|
||||
for args in (["kubectl", "version", "-o", "json"],
|
||||
["kubectl", "version", "--output=json"]):
|
||||
try:
|
||||
p = subprocess.run(args, capture_output=True, text=True, timeout=15)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
continue
|
||||
if p.returncode != 0 or not p.stdout.strip():
|
||||
continue
|
||||
try:
|
||||
v = json.loads(p.stdout).get("serverVersion") or {}
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if v.get("gitVersion"):
|
||||
return v["gitVersion"]
|
||||
return "unknown"
|
||||
|
||||
|
||||
def run_timestamp(run_dir: Path, junit: Path) -> str:
|
||||
"""UTC stamp: the run-dir name's own `<UTC>-<sha>` prefix, else the JUnit's mtime."""
|
||||
m = _RUN_ID_TS_RE.match(run_dir.name)
|
||||
if m:
|
||||
t = m.group(1)
|
||||
return f"{t[0:4]}-{t[4:6]}-{t[6:8]}T{t[9:11]}:{t[11:13]}:{t[13:15]}Z"
|
||||
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(junit.stat().st_mtime))
|
||||
|
||||
|
||||
# ============================================================================ record
|
||||
def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
|
||||
kube_version: str, sha: str) -> list[dict]:
|
||||
junit = run_dir / "report.junit.xml"
|
||||
log = run_dir / "run.log"
|
||||
smap = sentinel_map(tests_dir)
|
||||
log_text = log.read_text(errors="replace")
|
||||
buckets, attributed = attribute_log(log_text, set(smap))
|
||||
uniq = unique_tokens(smap)
|
||||
all_pairs = classify_sections(log_text)
|
||||
sectioned = any(sec not in ("", UNSECTIONED) for sec, _ in all_pairs)
|
||||
section_note = ("tokens read from real-output sections only"
|
||||
if sectioned else "log declares no === sections; whole-log scan")
|
||||
ts = run_timestamp(run_dir, junit)
|
||||
|
||||
rows: list[dict] = []
|
||||
for case in junit_cases(junit):
|
||||
name = case["name"]
|
||||
sent = smap.get(name, {"pass": [], "skip": [], "terminal_pass": None,
|
||||
"layer": case["layer"], "requirements": []})
|
||||
layer = sent.get("layer") or case["layer"] or "?"
|
||||
# Requirements come from the test's LABELS (coverage.py's authoritative source);
|
||||
# the JUnit property is the fallback for a testcase with no matching test file.
|
||||
reqs = sent.get("requirements") or case["requirements"]
|
||||
|
||||
if name in buckets:
|
||||
lines, scope = buckets[name], f"log scoped to this testcase; {section_note}"
|
||||
elif not attributed:
|
||||
# Degraded mode: the log had no recognisable Chainsaw header (a redirected or
|
||||
# reformatted log). Fall back to a whole-log scan, but ONLY for tokens owned by
|
||||
# exactly one test, so a token another test emitted can never prove this one.
|
||||
lines = all_pairs
|
||||
sent = dict(sent,
|
||||
**{"pass": [t for t in sent["pass"] if t in uniq],
|
||||
"skip": [t for t in sent["skip"] if t in uniq]})
|
||||
sent["terminal_pass"] = sent["pass"][-1] if sent["pass"] else None
|
||||
scope = ("DEGRADED: unattributable log, globally-unique tokens only; "
|
||||
+ section_note)
|
||||
else:
|
||||
lines, scope = [], "no log lines attributed to this testcase"
|
||||
|
||||
sentinel, proof, note = decide(layer, case["result"], sent, lines, scope)
|
||||
rows.append({
|
||||
"ts_utc": ts,
|
||||
"git_sha": sha,
|
||||
"env": env,
|
||||
"chainsaw_version": chainsaw_version,
|
||||
"kube_version": kube_version,
|
||||
"layer": layer,
|
||||
"test": name,
|
||||
"requirements": reqs,
|
||||
"junit_result": case["result"],
|
||||
"sentinel": sentinel,
|
||||
"proof": proof,
|
||||
"run_id": run_dir.name,
|
||||
"note": note,
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
def append_rows(ledger: Path, rows: list[dict]) -> None:
|
||||
"""Append-only, one compact JSON object per line. Never rewrites existing rows."""
|
||||
ledger.parent.mkdir(parents=True, exist_ok=True)
|
||||
with ledger.open("a", encoding="utf-8") as fh:
|
||||
for r in rows:
|
||||
fh.write(json.dumps(r, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
# ========================================================================== selftest
|
||||
def _selftest() -> int:
|
||||
"""Deterministic, no-cluster self-test of every decision rule (house pattern:
|
||||
`status.py --selftest`, `catalog/render.py --selftest`). Throwaway tree only.
|
||||
|
||||
The load-bearing case is `SKIP sentinel does NOT produce proven`: it is the single
|
||||
assertion standing between this ledger and a permanent false proof of R-STATE-03.
|
||||
"""
|
||||
import tempfile
|
||||
ok = True
|
||||
|
||||
def check(label: str, cond: bool) -> None:
|
||||
nonlocal ok
|
||||
if not cond:
|
||||
sys.stderr.write(f"record_run.py selftest: FAIL — {label}\n")
|
||||
ok = False
|
||||
|
||||
# --- token derivation, against the dr-restore/node-loss shape ---------------------
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
root = Path(td)
|
||||
tdir = root / "tests" / "l3-fault-injection" / "dr-restore"
|
||||
tdir.mkdir(parents=True)
|
||||
(tdir / "chainsaw-test.yaml").write_text(
|
||||
"apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n"
|
||||
" name: dr-restore\n labels:\n"
|
||||
" conformance.home.k3s/R-STATE-03: \"true\"\n annotations:\n"
|
||||
" conformance.home.k3s/requirements: \"R-STATE-03\"\n"
|
||||
" conformance.home.k3s/layer: \"L3\"\nspec:\n steps:\n"
|
||||
" - script:\n content: 'echo OFF_CLUSTER_TARGET_OK'\n"
|
||||
" check:\n"
|
||||
" (contains($stdout, 'OFF_CLUSTER_TARGET_OK')"
|
||||
" || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n"
|
||||
" - script:\n content: 'echo x'\n check:\n"
|
||||
" (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')"
|
||||
" || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n")
|
||||
s = parse_sentinels(tdir / "chainsaw-test.yaml")
|
||||
check("pass tokens derived from the test file",
|
||||
s["pass"] == ["OFF_CLUSTER_TARGET_OK",
|
||||
"DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"])
|
||||
check("skip token classified by the *_SKIP*_OK convention, de-duplicated",
|
||||
s["skip"] == ["DR_RESTORE_SKIPPED_OK"])
|
||||
check("terminal pass token is the LAST non-skip token",
|
||||
s["terminal_pass"] == "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK")
|
||||
|
||||
smap = sentinel_map(root / "tests")
|
||||
check("sentinel_map keys on metadata.name and carries layer + labels",
|
||||
smap["dr-restore"]["layer"] == "L3"
|
||||
and smap["dr-restore"]["requirements"] == ["R-STATE-03"])
|
||||
|
||||
# --- decide(): the SKIP-is-never-proven contract ----------------------------------
|
||||
l3 = {"pass": ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"],
|
||||
"skip": ["DR_RESTORE_SKIPPED_OK"],
|
||||
"terminal_pass": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"}
|
||||
|
||||
# *** THE CASE THIS WHOLE FILE EXISTS FOR ***
|
||||
sent, proof, _ = decide("L3", "passed", l3, ["DR_RESTORE_SKIPPED_OK"], "t")
|
||||
check("SKIP sentinel on a GREEN JUnit is 'skipped', NEVER 'proven'",
|
||||
proof == "skipped" and sent == "DR_RESTORE_SKIPPED_OK")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
|
||||
check("pass sentinel on a green JUnit is 'proven'", proof == "proven")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK",
|
||||
"DR_RESTORE_SKIPPED_OK"], "t")
|
||||
check("a skip token BEATS a pass token in the same log (worst wins)",
|
||||
proof == "skipped")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3, ["nothing relevant here"], "t")
|
||||
check("green JUnit with no sentinel in the log is unknown, not proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3, [], "t")
|
||||
check("green JUnit with NO log lines attributed is unknown, not proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
_, proof, _ = decide("L3", "failed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
|
||||
check("a failing JUnit outranks a pass token in the log", proof == "failed")
|
||||
|
||||
_, proof, _ = decide("L3", "skipped", l3, [], "t")
|
||||
check("JUnit <skipped/> records junit-skipped, never green",
|
||||
proof == "junit-skipped")
|
||||
|
||||
multi = {"pass": ["A_OK", "B_OK"], "skip": [], "terminal_pass": "B_OK"}
|
||||
_, proof, _ = decide("L3", "passed", multi, ["B_OK"], "t")
|
||||
check("every declared pass token must be present, not just the terminal one",
|
||||
proof == "unknown-sentinel-absent")
|
||||
_, proof, _ = decide("L3", "passed", multi, ["A_OK", "B_OK"], "t")
|
||||
check("all pass tokens present == proven", proof == "proven")
|
||||
|
||||
none = {"pass": [], "skip": [], "terminal_pass": None}
|
||||
_, proof, _ = decide("L2", "passed", none, ["whatever"], "t")
|
||||
check("a Chainsaw test declaring no sentinel is NOT proven by a green JUnit",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
_, proof, _ = decide("L1", "passed", none, [], "t")
|
||||
check("L0/L1 offline checks are proven by JUnit (their skips are JUnit skips)",
|
||||
proof == "proven")
|
||||
_, proof, _ = decide("L0", "skipped", none, [], "t")
|
||||
check("an L0 JUnit skip is still never green", proof == "junit-skipped")
|
||||
_, proof, _ = decide("META", "passed", none, [], "t")
|
||||
check("META (a Chainsaw layer) needs a sentinel like L2/L3",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
_, proof, _ = decide("", "passed", none, [], "t")
|
||||
check("an unknown/blank layer fails CLOSED — sentinel required",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
|
||||
# --- SECTION SCOPING: the COMMAND echo is SOURCE, never evidence -------------------
|
||||
# Regression cases for the ledger's first real L3 run, which recorded a genuine PASS as
|
||||
# `skipped` because `echo NODE_LOSS_SKIPPED_OK` — the skip branch NOT taken — is echoed
|
||||
# inside `=== COMMAND` on every run of a skip-gated test.
|
||||
nl = {"pass": ["NODE_LOSS_VOLUME_AVAILABLE_OK"],
|
||||
"skip": ["NODE_LOSS_SKIPPED_OK"],
|
||||
"terminal_pass": "NODE_LOSS_VOLUME_AVAILABLE_OK"}
|
||||
_CMD = (" l.go:52: | 23:51:18 | node-loss | step | CMD | RUN |\n"
|
||||
" === COMMAND\n"
|
||||
" /usr/bin/sh -c set -euo pipefail\n"
|
||||
" if ! kubectl get storageclass longhorn >/dev/null 2>&1; then\n"
|
||||
" echo NODE_LOSS_SKIPPED_OK\n"
|
||||
"\n"
|
||||
" exit 0\n"
|
||||
" fi\n"
|
||||
" echo NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |\n")
|
||||
|
||||
ran = classify_sections(_CMD + " === STDOUT\n"
|
||||
" ready nodes: 3\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" l.go:52: | 23:52:42 | node-loss | step | SCRIPT | DONE |\n")
|
||||
sent, proof, _ = decide("L3", "passed", nl, ran, "t")
|
||||
check("skip token ONLY in the COMMAND echo + pass token in STDOUT is PROVEN",
|
||||
proof == "proven" and sent == "NODE_LOSS_VOLUME_AVAILABLE_OK")
|
||||
|
||||
selfskipped = classify_sections(_CMD + " === STDOUT\n"
|
||||
" SKIP: 'longhorn' StorageClass absent\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
sent, proof, _ = decide("L3", "passed", nl, selfskipped, "t")
|
||||
check("skip token GENUINELY in STDOUT is still 'skipped' (the guarantee holds)",
|
||||
proof == "skipped" and sent == "NODE_LOSS_SKIPPED_OK")
|
||||
|
||||
neither = classify_sections(_CMD + " === STDOUT\n ready nodes: 3\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, neither, "t")
|
||||
check("both tokens in COMMAND and neither in STDOUT is NOT proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
both_out = classify_sections(" === STDOUT\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, both_out, "t")
|
||||
check("skip still BEATS pass when both are in genuine output", proof == "skipped")
|
||||
|
||||
stderr_skip = classify_sections(" === STDERR\n NODE_LOSS_SKIPPED_OK\n"
|
||||
" === STDOUT\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, stderr_skip, "t")
|
||||
check("a skip token in a non-source section (STDERR) still costs the proof",
|
||||
proof == "skipped")
|
||||
|
||||
stderr_pass = classify_sections(" === STDERR\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, stderr_pass, "t")
|
||||
check("a pass token OUTSIDE stdout cannot prove anything",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
# --- classify_sections: markers, extents, and Go's own `=== RUN` --------------------
|
||||
gorows = classify_sections("=== RUN chainsaw\n=== PAUSE chainsaw\n"
|
||||
"=== CONT chainsaw/node-loss\n"
|
||||
" === COMMAND\n echo NODE_LOSS_SKIPPED_OK\n"
|
||||
" l.go:52: | 1:00:00 | node-loss | s | SCRIPT | DONE |\n"
|
||||
"--- PASS: chainsaw/node-loss (131.09s)\n")
|
||||
secs = [sec for sec, _ in gorows]
|
||||
check("Go's column-0 `=== RUN/PAUSE/CONT` never open a section",
|
||||
secs[:3] == ["", "", ""])
|
||||
check("a section covers its indented body and ends at a less-indented line",
|
||||
secs[3] == "" and secs[4] == "COMMAND" and secs[5] == "" and secs[6] == "")
|
||||
check("a log with no === markers stays wholly scannable (UNSECTIONED)",
|
||||
{sec for sec, _ in classify_sections("plain line\n another\n")} == {UNSECTIONED})
|
||||
|
||||
# --- the preserved run itself, as a fixture ----------------------------------------
|
||||
# runs/20260901T235117Z-e9a4b66 IS the bug: NODE_LOSS_SKIPPED_OK twice in `=== COMMAND`
|
||||
# (source), never in output; NODE_LOSS_VOLUME_AVAILABLE_OK once in `=== STDOUT`.
|
||||
fixture = CONF_ROOT / "runs" / "20260901T235117Z-e9a4b66"
|
||||
if (fixture / "run.log").is_file() and (fixture / "report.junit.xml").is_file():
|
||||
rows = build_rows(fixture, CONF_ROOT / "tests", "dev", "v0.2.12", "v1.36.4", "sha")
|
||||
row = next((r for r in rows if r["test"] == "node-loss"), None)
|
||||
check("preserved real L3 run records node-loss/R-STATE-02 as proven",
|
||||
row is not None and row["proof"] == "proven"
|
||||
and row["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK"
|
||||
and row["requirements"] == ["R-STATE-02"])
|
||||
check("preserved real L3 run attributes (no DEGRADED fallback needed)",
|
||||
row is not None and "DEGRADED" not in row["note"])
|
||||
pairs = classify_sections((fixture / "run.log").read_text(errors="replace"))
|
||||
secs = {sec for sec, line in pairs if "NODE_LOSS_SKIPPED_OK" in line}
|
||||
check("every NODE_LOSS_SKIPPED_OK in the preserved log is COMMAND source",
|
||||
secs == {"COMMAND"})
|
||||
secs = {sec for sec, line in pairs
|
||||
if line.strip() == "NODE_LOSS_VOLUME_AVAILABLE_OK"}
|
||||
check("the real emission in the preserved log is in STDOUT", secs == {"STDOUT"})
|
||||
|
||||
# --- log attribution --------------------------------------------------------------
|
||||
log = ("[run.sh] Coverage report first\n"
|
||||
" | 20:41:32 | node-loss | step | SCRIPT | RUN |\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n"
|
||||
" | 20:44:01 | dr-restore | step | SCRIPT | RUN |\n"
|
||||
" DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK\n")
|
||||
b, attributed = attribute_log(log, {"node-loss", "dr-restore"})
|
||||
check("log lines are attributed to the testcase whose header preceded them",
|
||||
attributed
|
||||
and any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["node-loss"])
|
||||
and not any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["dr-restore"]))
|
||||
check("a pass token does NOT leak across testcase boundaries",
|
||||
not any("DR_RESTORED" in l for _, l in b["node-loss"]))
|
||||
check("an unsectioned log keeps whole-log scanning (nothing narrowed away)",
|
||||
all(sec == UNSECTIONED for sec, _ in b["node-loss"]))
|
||||
|
||||
# Chainsaw's real headers carry a Go `t.Log` source prefix and ANSI colour; the earlier
|
||||
# regex matched neither, so every genuine Chainsaw log silently fell into DEGRADED mode.
|
||||
real_hdr = (" l.go:52: \x08\x08\x08| 23:51:18 | \x1b[32;1mnode-loss\x1b[0;22m | "
|
||||
"\x1b[32;1mstep\x1b[0;22m | SCRIPT | LOG |\n"
|
||||
" === STDOUT\n NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
b2, attributed_real = attribute_log(real_hdr, {"node-loss"})
|
||||
check("a REAL Chainsaw header (go source prefix + ANSI) attributes, not degrades",
|
||||
attributed_real and "node-loss" in b2
|
||||
and ("STDOUT", " NODE_LOSS_VOLUME_AVAILABLE_OK") in b2["node-loss"])
|
||||
_, attributed2 = attribute_log("no chainsaw headers at all\n", {"node-loss"})
|
||||
check("an unattributable log is reported as such (degraded mode)", not attributed2)
|
||||
|
||||
smap2 = {"a": {"pass": ["DNS_OK", "A_ONLY_OK"], "skip": []},
|
||||
"b": {"pass": ["DNS_OK"], "skip": []}}
|
||||
check("only tokens owned by exactly one test may be used in degraded mode",
|
||||
unique_tokens(smap2) == {"A_ONLY_OK"})
|
||||
|
||||
# --- end-to-end: a self-skipped L3 run must land in the ledger as `skipped` --------
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
root = Path(td)
|
||||
tdir = root / "tests" / "l3-fault-injection" / "node-loss"
|
||||
tdir.mkdir(parents=True)
|
||||
(tdir / "chainsaw-test.yaml").write_text(
|
||||
"kind: Test\nmetadata:\n name: node-loss\n labels:\n"
|
||||
" conformance.home.k3s/R-STATE-02: \"true\"\n annotations:\n"
|
||||
" conformance.home.k3s/requirements: \"R-STATE-02\"\n"
|
||||
" conformance.home.k3s/layer: \"L3\"\nspec:\n steps:\n"
|
||||
" - script:\n content: 'echo hi'\n check:\n"
|
||||
" (contains($stdout, 'NODE_LOSS_VOLUME_AVAILABLE_OK')"
|
||||
" || contains($stdout, 'NODE_LOSS_SKIPPED_OK')): true\n")
|
||||
rd = root / "runs" / "20260901T120000Z-abc1234"
|
||||
rd.mkdir(parents=True)
|
||||
(rd / "report.junit.xml").write_text(
|
||||
'<testsuites><testsuite><testcase classname="L3:R-STATE-02" '
|
||||
'name="node-loss"><properties>'
|
||||
'<property name="requirements" value="R-STATE-02"/>'
|
||||
'<property name="layer" value="L3"/></properties></testcase>'
|
||||
"</testsuite></testsuites>")
|
||||
(rd / "run.log").write_text(
|
||||
" | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
|
||||
" SKIP: 'longhorn' StorageClass absent\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
|
||||
check("end-to-end: Chainsaw 'passed' + skip token => proof 'skipped'",
|
||||
len(rows) == 1 and rows[0]["junit_result"] == "passed"
|
||||
and rows[0]["proof"] == "skipped"
|
||||
and rows[0]["sentinel"] == "NODE_LOSS_SKIPPED_OK")
|
||||
check("end-to-end: row carries ts from the run-dir name and the R-ID label",
|
||||
rows[0]["ts_utc"] == "2026-09-01T12:00:00Z"
|
||||
and rows[0]["requirements"] == ["R-STATE-02"])
|
||||
|
||||
(rd / "run.log").write_text(
|
||||
" | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
|
||||
check("end-to-end: Chainsaw 'passed' + pass token => proof 'proven'",
|
||||
rows[0]["proof"] == "proven"
|
||||
and rows[0]["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK")
|
||||
|
||||
ledger = root / "ledger.jsonl"
|
||||
append_rows(ledger, rows)
|
||||
append_rows(ledger, rows)
|
||||
check("ledger is append-only JSONL (one parsable object per line)",
|
||||
len(ledger.read_text().strip().splitlines()) == 2
|
||||
and json.loads(ledger.read_text().splitlines()[0])["test"] == "node-loss")
|
||||
|
||||
if not ok:
|
||||
return 1
|
||||
sys.stderr.write("record_run.py selftest: PASS\n")
|
||||
return 0
|
||||
|
||||
|
||||
# ============================================================================== main
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--run-dir", type=Path,
|
||||
help="conformance/runs/<UTC>-<sha>/ holding report.junit.xml + run.log")
|
||||
ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
|
||||
help=f"append-only ledger (default {DEFAULT_LEDGER})")
|
||||
ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
|
||||
ap.add_argument("--repo", type=Path, default=REPO_ROOT)
|
||||
ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
|
||||
ap.add_argument("--chainsaw-version", default=None,
|
||||
help="override the versions.env pin (recorded verbatim)")
|
||||
ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
|
||||
help="override the `kubectl version` probe (recorded verbatim)")
|
||||
ap.add_argument("--dry-run", action="store_true",
|
||||
help="print the rows that WOULD be appended; write nothing")
|
||||
ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.selftest:
|
||||
return _selftest()
|
||||
if args.run_dir is None:
|
||||
sys.stderr.write("[record] --run-dir is required (or use --selftest)\n")
|
||||
return 1
|
||||
|
||||
run_dir = args.run_dir.resolve()
|
||||
junit, log = run_dir / "report.junit.xml", run_dir / "run.log"
|
||||
# BOTH files or nothing. The JUnit alone cannot tell a self-skip from a real pass, so
|
||||
# recording from it alone is exactly the false proof this tool exists to prevent.
|
||||
missing = [p.name for p in (junit, log) if not p.is_file()]
|
||||
if missing:
|
||||
sys.stderr.write(
|
||||
f"[record] {run_dir} is missing {', '.join(missing)} — nothing recorded. "
|
||||
f"The JUnit alone cannot distinguish a self-skip from a real pass "
|
||||
f"(see module docstring). SKIPPED (not green).\n")
|
||||
return 3
|
||||
|
||||
try:
|
||||
rows = build_rows(
|
||||
run_dir, args.tests, args.env,
|
||||
args.chainsaw_version or pinned_chainsaw_version(CONF_ROOT),
|
||||
args.kube_version or server_kube_version(),
|
||||
git_sha(args.repo))
|
||||
except ET.ParseError as e:
|
||||
sys.stderr.write(f"[record] {junit} is not parsable JUnit: {e}\n")
|
||||
return 1
|
||||
except OSError as e:
|
||||
sys.stderr.write(f"[record] cannot read the run inputs: {e}\n")
|
||||
return 1
|
||||
|
||||
if not rows:
|
||||
sys.stderr.write(f"[record] {junit} contains no <testcase> — nothing to record. "
|
||||
f"SKIPPED (not green).\n")
|
||||
return 3
|
||||
|
||||
for r in rows:
|
||||
print(f"[record] {r['layer']:4s} {r['proof']:28s} {r['test']} "
|
||||
f"({','.join(r['requirements']) or 'untagged'}) "
|
||||
f"junit={r['junit_result']} sentinel={r['sentinel'] or '—'}")
|
||||
counts: dict[str, int] = {}
|
||||
for r in rows:
|
||||
counts[r["proof"]] = counts.get(r["proof"], 0) + 1
|
||||
print(f"[record] {len(rows)} testcase(s): "
|
||||
+ ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
|
||||
|
||||
if args.dry_run:
|
||||
print("[record] --dry-run: ledger NOT written.")
|
||||
return 0
|
||||
try:
|
||||
append_rows(args.ledger, rows)
|
||||
except OSError as e:
|
||||
sys.stderr.write(f"[record] cannot append to {args.ledger}: {e}\n")
|
||||
return 1
|
||||
print(f"[record] appended {len(rows)} row(s) -> {args.ledger}")
|
||||
print("[record] COMMIT the run dir and the ledger; an uncommitted ledger proves nothing.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,24 +1,17 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the CLUSTER-INDEPENDENT conformance checks OFFLINE and emit JUnit.
|
||||
"""Run the L1 *rendering* conformance checks OFFLINE (no cluster) and emit JUnit.
|
||||
|
||||
docs/14 puts two layers in CI with no cluster:
|
||||
* L0 static/policy -- "manifests/ServiceDefinitions obey invariants" (tests/l0-static/,
|
||||
each dir carrying a `check.sh`).
|
||||
* L1 rendering -- ServiceDefinition -> rendered resources vs golden expectations
|
||||
(tests/l1-rendering/, each dir carrying a `render-check.sh`).
|
||||
Both are executed here so they report a REAL pass/fail instead of being skipped with the
|
||||
cluster-gated layers. run.sh invokes this for `--layer l0`, `--layer l1`, and as the
|
||||
offline leg of a full run.
|
||||
docs/14: "L1 rendering assertions run in CI without a cluster." L1 verifies the
|
||||
ServiceDefinition -> rendered-resources mapping against golden expectations; unlike L2/L3
|
||||
it needs no live cluster. Each l1-rendering test dir carries a `render-check.sh` (renderer
|
||||
+ golden diff + structural asserts). run.sh invokes this for `--layer l1` so those tests
|
||||
report a REAL pass/fail instead of being skipped with the cluster-gated layers.
|
||||
|
||||
Pure standard library (matches the harness's no-pip-dependency posture); the checks
|
||||
themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives
|
||||
there; L0 checks to git + PyYAML).
|
||||
Pure standard library (matches the harness's no-pip-dependency posture); the render checks
|
||||
themselves shell out to catalog/render.py (the renderer's PyYAML dependency lives there).
|
||||
|
||||
Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] <test-dir> [<test-dir> ...]
|
||||
Exit 0 iff every selected offline check passes.
|
||||
|
||||
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
|
||||
carries; a dir with neither is reported SKIPPED (never a pass).
|
||||
Exit 0 iff every selected render check passes.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -32,19 +25,6 @@ from xml.dom import minidom
|
||||
|
||||
LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
|
||||
|
||||
# The offline check script a test dir may carry, in preference order.
|
||||
CHECK_SCRIPTS = ("render-check.sh", "check.sh")
|
||||
|
||||
|
||||
def layer_of(d: Path) -> str:
|
||||
"""Layer for the JUnit properties, from the subtree the test lives in."""
|
||||
rel = str(d)
|
||||
if "l0-static" in rel:
|
||||
return "L0"
|
||||
if "l1-rendering" in rel:
|
||||
return "L1"
|
||||
return "OFFLINE"
|
||||
|
||||
|
||||
def collect_meta(test_yaml: Path, prefix: str):
|
||||
text = test_yaml.read_text() if test_yaml.exists() else ""
|
||||
@@ -64,11 +44,10 @@ def main() -> int:
|
||||
|
||||
cases = [] # (name, reqs, dir, rc, output)
|
||||
for d in args.dirs:
|
||||
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
|
||||
check = d / "render-check.sh"
|
||||
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
|
||||
if check is None:
|
||||
cases.append((name, reqs, d, 3,
|
||||
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
|
||||
if not check.exists():
|
||||
cases.append((name, reqs, d, 3, f"no render-check.sh in {d}"))
|
||||
continue
|
||||
proc = subprocess.run(
|
||||
["bash", str(check)], capture_output=True, text=True
|
||||
@@ -82,7 +61,7 @@ def main() -> int:
|
||||
suite = ET.Element(
|
||||
"testsuite",
|
||||
{
|
||||
"name": "home-k3s-conformance L0/L1 (offline)",
|
||||
"name": "home-k3s-conformance L1 (offline render)",
|
||||
"tests": str(len(cases)),
|
||||
"failures": str(failures),
|
||||
"errors": "0",
|
||||
@@ -93,21 +72,19 @@ def main() -> int:
|
||||
tc = ET.SubElement(
|
||||
suite,
|
||||
"testcase",
|
||||
{"classname": f"{layer_of(d)}:{','.join(reqs) if reqs else 'UNTAGGED'}",
|
||||
"name": name},
|
||||
{"classname": f"L1:{','.join(reqs) if reqs else 'UNTAGGED'}", "name": name},
|
||||
)
|
||||
props = ET.SubElement(tc, "properties")
|
||||
ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
|
||||
ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
|
||||
ET.SubElement(props, "property", {"name": "layer", "value": "L1"})
|
||||
ET.SubElement(props, "property", {"name": "path", "value": str(d)})
|
||||
if rc == 3:
|
||||
ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
|
||||
ET.SubElement(tc, "skipped", {"message": out or "no render-check.sh"})
|
||||
elif rc != 0:
|
||||
fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
|
||||
fail = ET.SubElement(tc, "failure", {"message": f"render-check exit {rc}"})
|
||||
fail.text = out
|
||||
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
|
||||
print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
|
||||
f"({','.join(reqs) or 'untagged'})",
|
||||
print(f"[l1-offline] {status:4s} {name} ({','.join(reqs) or 'untagged'})",
|
||||
file=sys.stderr)
|
||||
if rc not in (0, 3):
|
||||
print(out, file=sys.stderr)
|
||||
@@ -126,7 +103,7 @@ def main() -> int:
|
||||
args.junit.parent.mkdir(parents=True, exist_ok=True)
|
||||
xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent=" ")
|
||||
args.junit.write_text(xml)
|
||||
print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
|
||||
print(f"[l1-offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
|
||||
f"{args.junit}", file=sys.stderr)
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
+46
-190
@@ -8,51 +8,18 @@
|
||||
#
|
||||
# ./run.sh # full suite; skips L1/L2/L3 if no cluster
|
||||
# ./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||
# ./run.sh --layer l2 # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
|
||||
# ./run.sh --layer l2 # one layer
|
||||
# ./run.sh --requirements R-NET-01,R-NET-02 --layer l2 # intersect: tagged AND in layer
|
||||
# ./run.sh --coverage-only # regenerate coverage report only (no cluster)
|
||||
# ./run.sh --status # DERIVED STATUS report only (no cluster): per
|
||||
# # requirement, is it implemented, is it actually
|
||||
# # reconciled by Flux, is its test executable or a
|
||||
# # scaffold, and whether the committed run ledger
|
||||
# # (runs/ledger.jsonl) actually proves it — see
|
||||
# # lib/status.py. No ledger row => not-derivable.
|
||||
# ./run.sh --lint # structural-validate test defs (no cluster)
|
||||
# ./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
# ./run.sh --requirements R-NET-01 # ALSO asserts those IDs are covered (merge gate)
|
||||
# ./run.sh --require-all # phase-exit: every requirement must be covered
|
||||
# ./run.sh --record # RUN AND CAPTURE: tee the whole run to
|
||||
# # runs/<UTC>-<short-sha>/run.log, copy the JUnit
|
||||
# # beside it, and append one ledger row per
|
||||
# # testcase to runs/ledger.jsonl (lib/record_run.py)
|
||||
#
|
||||
# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
|
||||
# --------------------------------------------------------------------------
|
||||
# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
|
||||
# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
|
||||
# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
|
||||
# that — `conformance/runs/` (see conformance/.gitignore for the negations).
|
||||
#
|
||||
# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
|
||||
# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
|
||||
#
|
||||
# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
|
||||
# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
|
||||
# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
|
||||
# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
|
||||
# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
|
||||
#
|
||||
# Artifacts (always written under conformance/_artifacts/, gitignored):
|
||||
# _artifacts/report.junit.xml JUnit (Chainsaw output, or skipped-scaffold shim)
|
||||
# _artifacts/coverage/coverage.md human coverage report
|
||||
# _artifacts/coverage/coverage.json machine coverage report
|
||||
# _artifacts/status/status.md human requirement-status report (--status)
|
||||
# _artifacts/status/status.json machine requirement-status report (--status)
|
||||
#
|
||||
# Run ledger (TRACKED — commit these; --record only):
|
||||
# runs/<UTC>-<short-sha>/run.log full tee'd run output (carries the sentinels)
|
||||
# runs/<UTC>-<short-sha>/report.junit.xml the JUnit for that same run
|
||||
# runs/ledger.jsonl append-only, one row per testcase per run
|
||||
#
|
||||
# Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
|
||||
# execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
|
||||
@@ -69,34 +36,30 @@ source "${CONF_ROOT}/versions.env"
|
||||
|
||||
ARTIFACTS="${CONF_ROOT}/_artifacts"
|
||||
COVERAGE_OUT="${ARTIFACTS}/coverage"
|
||||
STATUS_OUT="${ARTIFACTS}/status"
|
||||
JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
|
||||
# under `tee`. Captured BEFORE parsing so nothing is normalized away.
|
||||
ORIG_ARGS=("$@")
|
||||
|
||||
LAYER="all"
|
||||
REQS=""
|
||||
MODE="run" # run | coverage-only | status | lint
|
||||
MODE="run" # run | coverage-only | lint
|
||||
STRICT_COVERAGE=0
|
||||
REQUIRE_ALL=0
|
||||
FORCE_NO_CLUSTER=0
|
||||
RECORD=0
|
||||
|
||||
# Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
|
||||
# owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
|
||||
# exemption is explicit and reviewable, not silent.
|
||||
#
|
||||
# EMPTY as of 2026-09-01. R-ENV-02 and R-GITOPS-01 were the last two entries — exempted
|
||||
# because they were unscaffolded while the coarse path->requirement map keys clusters/*
|
||||
# and ci/* to them, so any overlay/CI edit would have false-blocked. They now have a real
|
||||
# verifying L0 test (tests/l0-static/flux-postbuild-substitution), which asserts the
|
||||
# substitution mechanism that IS the "authored once, promoted" contract, so they are
|
||||
# covered rather than exempt and the completeness gate can hold them to it. That test
|
||||
# names its own limits in its header: it proves the STATIC property (no manifest renders
|
||||
# an undefined variable), not the live-cluster half of either requirement.
|
||||
# R-ENV-02 (dev->prod promote) — owned by WS-INFRA/WS-GITOPS; unscaffolded (a GAP in
|
||||
# coverage.md). The coarse path->requirement map keys
|
||||
# clusters/* AND ci/* to it and the R-ENV-02 doc token
|
||||
# appears in platform/kustomization.yaml's header, so ANY
|
||||
# infra/overlay/CI edit (e.g. enabling a component, bumping
|
||||
# SPEC_PIN) would otherwise false-block on it.
|
||||
# R-GITOPS-01 (state in git/Flux) — same: owned by WS-GITOPS, unscaffolded, and the ci/*
|
||||
# path-map keys to it, so editing any ci/ file (SPEC_PIN
|
||||
# adoption included) would false-block.
|
||||
# These are visible GAPs, not hidden — removing them from this list is the right move the
|
||||
# moment WS-TEST adds their verifying tests.
|
||||
#
|
||||
# FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header
|
||||
# tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in
|
||||
@@ -112,95 +75,31 @@ RECORD=0
|
||||
# R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra")
|
||||
# but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no
|
||||
# longer a gap and has been removed from this list; the coverage report counts it as covered.
|
||||
DOCUMENTED_GAPS=""
|
||||
DOCUMENTED_GAPS="R-ENV-02 R-GITOPS-01"
|
||||
|
||||
# Print the header block (line 2 through the closing `####` rule, which is dropped) as
|
||||
# help. Bounded by the rule rather than a fixed line number so the header can grow.
|
||||
usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
|
||||
usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--layer) LAYER="$2"; shift 2 ;;
|
||||
--requirements) REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
|
||||
--coverage-only) MODE="coverage-only"; shift ;;
|
||||
--status) MODE="status"; shift ;;
|
||||
--lint) MODE="lint"; shift ;;
|
||||
--strict-coverage) STRICT_COVERAGE=1; shift ;;
|
||||
--require-all) REQUIRE_ALL=1; shift ;;
|
||||
--no-cluster) FORCE_NO_CLUSTER=1; shift ;;
|
||||
--record) RECORD=1; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "unknown arg: $1" >&2; usage; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
|
||||
mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}"
|
||||
|
||||
log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
|
||||
|
||||
###############################################################################
|
||||
# --record: capture this run as committed evidence, then hand it to the ledger.
|
||||
#
|
||||
# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
|
||||
# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
|
||||
# operator-visible output — no second, differently-behaving "recording mode" to drift.
|
||||
# CONF_RECORD_CHILD guards against recursion.
|
||||
#
|
||||
# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
|
||||
# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
|
||||
# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
|
||||
###############################################################################
|
||||
if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
|
||||
case "${MODE}" in
|
||||
run) ;;
|
||||
# --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
|
||||
# test, so a ledger row from one of them would be a row about nothing.
|
||||
*) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
|
||||
esac
|
||||
RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
|
||||
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
|
||||
RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
|
||||
mkdir -p "${RUN_DIR}"
|
||||
# Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
|
||||
# ever capture a report THIS run produced. Copying a stale artifact would pair someone
|
||||
# else's result with this run's log — a fabricated pairing, which is precisely the class
|
||||
# of false proof the ledger exists to prevent.
|
||||
rm -f "${JUNIT_OUT}"
|
||||
# Strip --record from the argv handed to the child; everything else passes through.
|
||||
CHILD_ARGS=()
|
||||
for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
|
||||
[ "$a" = "--record" ] || CHILD_ARGS+=("$a")
|
||||
done
|
||||
log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
|
||||
set +e
|
||||
CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
|
||||
| tee "${RUN_DIR}/run.log"
|
||||
RUN_RC=${PIPESTATUS[0]}
|
||||
set -e
|
||||
if [ -f "${JUNIT_OUT}" ]; then
|
||||
cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
|
||||
else
|
||||
log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
|
||||
fi
|
||||
# Explicit, separate step: read the JUnit + the log and append one row per testcase.
|
||||
# rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
|
||||
# CONF_LEDGER exists so the harness's own tests can exercise this path without writing
|
||||
# to the real ledger; operators should never set it.
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
|
||||
--run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
|
||||
--ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
|
||||
log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
|
||||
log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
|
||||
exit "${RUN_RC}"
|
||||
fi
|
||||
|
||||
# --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
|
||||
gen_coverage() {
|
||||
# Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
|
||||
# requirement is exempt, and handing the flag an empty string would be a silent no-op
|
||||
# that reads like an exemption in the CLI log.
|
||||
local flags=()
|
||||
[ -n "${DOCUMENTED_GAPS}" ] && flags+=(--allow-gap "${DOCUMENTED_GAPS}")
|
||||
local flags=(--allow-gap "${DOCUMENTED_GAPS}")
|
||||
[ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict)
|
||||
# A tagged subset run is also the merge gate: the touched requirements MUST be covered.
|
||||
[ -n "${REQS}" ] && flags+=(--require "${REQS}")
|
||||
@@ -212,23 +111,6 @@ gen_coverage() {
|
||||
"${flags[@]}"
|
||||
}
|
||||
|
||||
# --- Derived STATUS report (no cluster) ------------------------------------------------
|
||||
# Coverage answers "does a tagged test EXIST?". Status answers the operator's question —
|
||||
# "what is the real state of each requirement given what is actually in the repo?" — by
|
||||
# deriving, per requirement: implementing artifact? reachable from a Flux Kustomization
|
||||
# (reconciled, not merely authored)? tagged test that is EXECUTABLE rather than a TODO
|
||||
# scaffold? It REFUSES to derive whether a test passed: nothing tracked in git records a
|
||||
# run result, and a status report that defaults to green is worse than none.
|
||||
gen_status() {
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/status.py" \
|
||||
--repo "$(cd "${CONF_ROOT}/.." && pwd)" \
|
||||
--registry "${CONF_ROOT}/requirements/registry.yaml" \
|
||||
--tests "${CONF_ROOT}/tests" \
|
||||
--out "${STATUS_OUT}" \
|
||||
--env "${CONF_ENV:-dev}" \
|
||||
"$@"
|
||||
}
|
||||
|
||||
# --- Structural lint (no cluster): parse each test def and check the tag convention ----
|
||||
lint_tests() {
|
||||
log "Linting test definitions (structural, no cluster)"
|
||||
@@ -372,40 +254,26 @@ run_chainsaw() {
|
||||
return 0
|
||||
}
|
||||
|
||||
# --- L0/L1: the cluster-INDEPENDENT layers (docs/14) ------------------------------------
|
||||
# docs/14 puts TWO layers in CI without a cluster:
|
||||
# L0 static/policy tests/l0-static/** each dir carries a `check.sh` (repo invariants)
|
||||
# L1 rendering tests/l1-rendering/** each dir carries a `render-check.sh` (render == golden)
|
||||
# Both are executed directly here and emit a REAL pass/fail JUnit, instead of being skipped
|
||||
# with the cluster-gated layers. Honors the same --layer/--requirements selection as the rest
|
||||
# of the suite (select_dirs), filtered to those two subtrees.
|
||||
OFFLINE_SUBTREES='/l0-static/|/l1-rendering/'
|
||||
|
||||
offline_selected_dirs() {
|
||||
select_dirs | grep -E "${OFFLINE_SUBTREES}" || true
|
||||
# --- L1 rendering: cluster-INDEPENDENT render checks (docs/14) --------------------------
|
||||
# L1 verifies ServiceDefinition -> rendered-resources against golden expectations; it needs
|
||||
# no live cluster. Each l1-rendering test dir carries a `render-check.sh` (renderer + golden
|
||||
# diff + structural asserts). We run those directly and emit a REAL pass/fail JUnit, instead
|
||||
# of skipping L1 with the cluster-gated layers (T-0.3). Honors the same --layer/--requirements
|
||||
# selection as the rest of the suite (select_dirs), filtered to the l1-rendering subtree.
|
||||
l1_selected_dirs() {
|
||||
select_dirs | grep '/l1-rendering/' || true
|
||||
}
|
||||
|
||||
# True when EVERY requested layer is cluster-independent, so the offline result is the
|
||||
# whole result and run.sh must exit on it (the ci/l0.sh + ci/l1.sh merge-gate paths).
|
||||
layers_are_offline_only() {
|
||||
local l
|
||||
[ "${LAYER}" = all ] && return 1
|
||||
for l in ${LAYER//,/ }; do
|
||||
case "$l" in l0|l1) ;; *) return 1 ;; esac
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
run_offline_checks() {
|
||||
local -a dirs
|
||||
mapfile -t dirs < <(offline_selected_dirs)
|
||||
if [ "${#dirs[@]}" -eq 0 ]; then
|
||||
log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
|
||||
run_l1_offline() {
|
||||
local -a l1dirs
|
||||
mapfile -t l1dirs < <(l1_selected_dirs)
|
||||
if [ "${#l1dirs[@]}" -eq 0 ]; then
|
||||
log "No L1 render tests match the selection (layer=${LAYER} reqs='${REQS}')."
|
||||
return 0
|
||||
fi
|
||||
log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
|
||||
log "Running ${#l1dirs[@]} L1 render check(s) OFFLINE (no cluster; docs/14)."
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
|
||||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
|
||||
--junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${l1dirs[@]}"
|
||||
}
|
||||
|
||||
###############################################################################
|
||||
@@ -419,17 +287,6 @@ case "${MODE}" in
|
||||
log "coverage-only: done. See ${COVERAGE_OUT}/coverage.md"
|
||||
exit 0
|
||||
;;
|
||||
status)
|
||||
# rc: 0 report written, 1 a derivation/strict failure, 3 a check could not run
|
||||
# (SKIPPED, never counted as green — ci/lib.sh convention). Propagated as-is.
|
||||
rc=0
|
||||
# --strict-coverage is the harness's hygiene switch; apply it to status too.
|
||||
status_flags=()
|
||||
[ "${STRICT_COVERAGE}" = 1 ] && status_flags+=(--strict)
|
||||
gen_status ${status_flags[@]+"${status_flags[@]}"} || rc=$?
|
||||
log "status: done. See ${STATUS_OUT}/status.md"
|
||||
exit "${rc}"
|
||||
;;
|
||||
lint)
|
||||
lint_tests
|
||||
log "lint: done."
|
||||
@@ -437,30 +294,29 @@ case "${MODE}" in
|
||||
;;
|
||||
esac
|
||||
|
||||
# L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
|
||||
# checks and treat their result as authoritative REGARDLESS of cluster availability. An
|
||||
# offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
||||
OFFLINE_RC=0
|
||||
run_offline_checks || OFFLINE_RC=$?
|
||||
if layers_are_offline_only; then
|
||||
# Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
|
||||
if [ "${OFFLINE_RC}" -eq 0 ]; then
|
||||
log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||||
# L1 rendering is cluster-independent (docs/14): run the selected L1 render checks offline
|
||||
# and treat their result as authoritative REGARDLESS of cluster availability. A render-check
|
||||
# failure is a real failure (nothing cluster-gated about it), so it always blocks.
|
||||
L1_RC=0
|
||||
run_l1_offline || L1_RC=$?
|
||||
if [ "${LAYER}" = l1 ]; then
|
||||
# L1-only run (the ci/l1.sh merge-gate path): the offline render result IS the result.
|
||||
if [ "${L1_RC}" -eq 0 ]; then
|
||||
log "Done (L1 render). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
|
||||
else
|
||||
log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
|
||||
log "L1 render checks FAILED (rc=${L1_RC}). JUnit: ${JUNIT_OUT}"
|
||||
fi
|
||||
exit "${OFFLINE_RC}"
|
||||
exit "${L1_RC}"
|
||||
fi
|
||||
if [ "${OFFLINE_RC}" -ne 0 ]; then
|
||||
log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
|
||||
exit "${OFFLINE_RC}"
|
||||
if [ "${L1_RC}" -ne 0 ]; then
|
||||
log "L1 render checks FAILED (rc=${L1_RC}) — blocking (render failure is not cluster-gated)."
|
||||
exit "${L1_RC}"
|
||||
fi
|
||||
|
||||
# run mode: need a cluster for L2/L3. Without one (Phase 0), emit skipped JUnit for the
|
||||
# cluster-gated layers. (L1 already ran offline above and passed.)
|
||||
if [ "${FORCE_NO_CLUSTER}" = 1 ] || ! conf_cluster_reachable; then
|
||||
log "No reachable cluster — emitting SKIPPED JUnit scaffold for L2/L3 (Phase 0 behavior)."
|
||||
log "NOTE: the offline L0/L1 results above already ran for real; this shim covers L2/L3 only."
|
||||
log "Also running structural lint so scaffolds stay valid."
|
||||
lint_tests || { log "lint failed"; exit 1; }
|
||||
"${PYTHON}" "${CONF_ROOT}/lib/emit_skipped_junit.py" \
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
|
||||
<testcase name="node-loss" classname="" time="131.090121" />
|
||||
</testsuite>
|
||||
</testsuites>
|
||||
@@ -1,232 +0,0 @@
|
||||
[1m[run.sh][0m Coverage report first (always)
|
||||
[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
|
||||
[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
|
||||
[1m[run.sh][0m No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
|
||||
[1m[run.sh][0m LIGHT pass: no tests selected.
|
||||
[1m[run.sh][0m HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
|
||||
Version: 0.2.12
|
||||
Loading config (/work/conformance/config/chainsaw-config.yaml)...
|
||||
- Using test file: chainsaw-test
|
||||
- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
|
||||
- SkipDelete false
|
||||
- FailFast false
|
||||
- ReportFormat 'JUNIT-TEST'
|
||||
- ReportName 'chainsaw-report-heavy'
|
||||
- ReportPath '/work/conformance/_artifacts'
|
||||
- Namespace ''
|
||||
- FullName false
|
||||
- IncludeTestRegex ''
|
||||
- ExcludeTestRegex ''
|
||||
- ApplyTimeout 1m30s
|
||||
- AssertTimeout 3m0s
|
||||
- CleanupTimeout 2m0s
|
||||
- DeleteTimeout 2m0s
|
||||
- ErrorTimeout 1m0s
|
||||
- ExecTimeout 3m0s
|
||||
- DeletionPropagationPolicy Background
|
||||
- Parallel 1
|
||||
- Template true
|
||||
- NoCluster false
|
||||
- PauseOnFailure false
|
||||
Loading tests...
|
||||
- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
|
||||
Loading values...
|
||||
Running tests...
|
||||
=== RUN chainsaw
|
||||
=== PAUSE chainsaw
|
||||
=== CONT chainsaw
|
||||
=== RUN chainsaw/node-loss
|
||||
=== PAUSE chainsaw/node-loss
|
||||
=== CONT chainsaw/node-loss
|
||||
l.go:52: | 23:51:18 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mCREATE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-fond-marlin
|
||||
l.go:52: | 23:51:18 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 23:51:18 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -euo pipefail
|
||||
NS="$NAMESPACE"
|
||||
|
||||
# -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
|
||||
READY="$(kubectl get nodes \
|
||||
-o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
|
||||
| grep -c '^True$' || true)"
|
||||
echo "ready nodes: ${READY}"
|
||||
if [ "${READY:-0}" -lt 3 ]; then
|
||||
echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
|
||||
echo NODE_LOSS_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
CANARY="node-loss-canary-$(date +%s%N)"
|
||||
|
||||
# -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
|
||||
# writer pod that writes the canary and flushes it to the replicated volume. --
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: survivor
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
apply_pod() {
|
||||
kubectl -n "$NS" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: $1
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 3600"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: survivor
|
||||
EOF
|
||||
}
|
||||
|
||||
apply_pod writer
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
|
||||
printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
|
||||
echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
|
||||
|
||||
# The victim node is the one the attached volume currently lives on.
|
||||
VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
|
||||
[ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
|
||||
echo "victim node (currently hosts the attached volume): $VICTIM"
|
||||
|
||||
# Best-effort precondition log: if the Longhorn Volume CR is readable, show the
|
||||
# volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
|
||||
# runner is least-privilege and may not have longhorn.io read rights) — the
|
||||
# data-availability assertion below is the authoritative proof, not this.
|
||||
PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
if [ -n "$PVNAME" ]; then
|
||||
kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
|
||||
2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
|
||||
fi
|
||||
|
||||
# -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
|
||||
# Label it with THIS test's unique namespace so `finally` can always find and
|
||||
# uncordon exactly this node; then cordon it; then evict ONLY our own writer
|
||||
# pod, forcing the volume to detach from the victim. (No `drain` — unrelated
|
||||
# pods on the node are never touched.)
|
||||
kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
|
||||
kubectl cordon "$VICTIM"
|
||||
echo "cordoned + labelled victim node: $VICTIM"
|
||||
kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
|
||||
echo "evicted writer off the victim node; volume detached from $VICTIM"
|
||||
|
||||
# -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
|
||||
# so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
|
||||
# the volume there from its replicas on the other nodes. --
|
||||
apply_pod reader
|
||||
kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
|
||||
READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
|
||||
echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
|
||||
if [ "$READER_NODE" = "$VICTIM" ]; then
|
||||
echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
[ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
|
||||
echo "volume available + data intact from a surviving node"
|
||||
|
||||
# Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
|
||||
# (bounded) for robustness to settle back to available (healthy|degraded — both
|
||||
# serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
|
||||
# Non-fatal for the same least-privilege reason as the pre-fault log.
|
||||
if [ -n "$PVNAME" ]; then
|
||||
for i in $(seq 1 30); do
|
||||
ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
|
||||
-o jsonpath='{.status.robustness}' 2>/dev/null || true)"
|
||||
[ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
|
||||
echo "post-fault robustness[$i]=$ROB"
|
||||
case "$ROB" in healthy|degraded) break ;; esac
|
||||
sleep 10
|
||||
done
|
||||
fi
|
||||
|
||||
echo NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
ready nodes: 3
|
||||
persistentvolumeclaim/survivor created
|
||||
pod/writer created
|
||||
pod/writer condition met
|
||||
wrote canary: node-loss-canary-1788306678898809444
|
||||
victim node (currently hosts the attached volume): k3s-dev-2
|
||||
pre-fault robustness=healthy replicas=3
|
||||
node/k3s-dev-2 labeled
|
||||
node/k3s-dev-2 cordoned
|
||||
cordoned + labelled victim node: k3s-dev-2
|
||||
pod "writer" deleted from chainsaw-fond-marlin namespace
|
||||
evicted writer off the victim node; volume detached from k3s-dev-2
|
||||
pod/reader created
|
||||
pod/reader condition met
|
||||
reader scheduled on surviving node: k3s-dev-1 (victim was k3s-dev-2)
|
||||
recovered='node-loss-canary-1788306678898809444' expected='node-loss-canary-1788306678898809444'
|
||||
volume available + data intact from a surviving node
|
||||
post-fault robustness[1]=healthy
|
||||
NODE_LOSS_VOLUME_AVAILABLE_OK
|
||||
l.go:52: | 23:52:42 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mTRY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 23:52:42 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c set -uo pipefail
|
||||
for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
|
||||
kubectl uncordon "${n#node/}" || true
|
||||
kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
|
||||
echo "restored node to service: ${n#node/}"
|
||||
done
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
node/k3s-dev-2 uncordoned
|
||||
node/k3s-dev-2 unlabeled
|
||||
restored node to service: k3s-dev-2
|
||||
l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mCMD[0;22m | [36;1mRUN[0;22m |
|
||||
=== COMMAND
|
||||
/usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
|
||||
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mSCRIPT[0;22m | [36;1mLOG[0;22m |
|
||||
=== STDOUT
|
||||
pod "reader" deleted from chainsaw-fond-marlin namespace
|
||||
persistentvolumeclaim "survivor" deleted from chainsaw-fond-marlin namespace
|
||||
cleanup best-effort; chainsaw namespace teardown finalizes
|
||||
l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1mvolume-stays-available-across-node-loss[0;22m | [32;1mSCRIPT[0;22m | [32;1mDONE[0;22m |
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1mvolume-stays-available-across-node-loss[0;22m | [36;1mFINALLY[0;22m | [36;1mEND[0;22m |
|
||||
l.go:52: | 23:52:43 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mBEGIN[0;22m |
|
||||
l.go:52: | 23:52:43 | [32;1mnode-loss[0;22m | [32;1m@chainsaw [0;22m | [32;1mDELETE[0;22m | [32;1mOK[0;22m | v1/Namespace @ chainsaw-fond-marlin
|
||||
l.go:52: | 23:53:29 | [36;1mnode-loss[0;22m | [36;1m@chainsaw [0;22m | [36;1mCLEANUP[0;22m | [36;1mEND[0;22m |
|
||||
--- PASS: chainsaw (0.00s)
|
||||
--- PASS: chainsaw/node-loss (131.09s)
|
||||
PASS
|
||||
Tests Summary...
|
||||
- Passed tests 1
|
||||
- Failed tests 0
|
||||
- Skipped tests 0
|
||||
Done.
|
||||
merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
|
||||
[1m[run.sh][0m Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md
|
||||
@@ -1 +0,0 @@
|
||||
{"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "57d4df8c4f992732a49f3e2daddcace866778ab0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260901T235117Z-e9a4b66", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-01T23:51:17Z"}
|
||||
@@ -4,7 +4,6 @@ Every test is a **Chainsaw `Test`** in its own directory:
|
||||
|
||||
```
|
||||
tests/
|
||||
l0-static/ L0 — whole-repo invariants over the tracked tree (no cluster)
|
||||
l1-rendering/ L1 — ServiceDefinition -> rendered resources match golden
|
||||
l2-behavioral/ L2 — real runtime behavior (probes against a cluster)
|
||||
l3-fault-injection/ L3 — break something; assert fail-closed / no-data-loss
|
||||
@@ -15,35 +14,6 @@ tests/
|
||||
The layer a test belongs to is its subtree **and** its `.../layer` annotation; the
|
||||
coverage generator cross-checks the two.
|
||||
|
||||
### L0 static (`l0-static/`) — what belongs here, and what does not
|
||||
|
||||
docs/14's L0 row is *"Pre-apply, in CI · manifests/ServiceDefinitions obey invariants"*.
|
||||
Kyverno (`platform/kyverno/`) covers the invariants expressible over a **single
|
||||
resource**. This subtree covers the L0 invariants that are **properties of the repository
|
||||
as a whole** and that no per-resource policy can state — e.g. *no plaintext secret exists
|
||||
anywhere in the tracked tree*, *no tracked RoleBinding grants a workload identity a
|
||||
privileged role*, *the spec-immutability gate actually blocks a modified spec*.
|
||||
|
||||
Each `l0-static/<name>/` dir carries a **`check.sh`**: a self-contained offline assertion
|
||||
script (git + Python stdlib/PyYAML; no cluster, no network) that exits 0/1. It is executed:
|
||||
|
||||
- by `conformance/run.sh --layer l0` (and as the offline leg of a full run), via
|
||||
`lib/run_l1_offline.py`, producing a REAL pass/fail JUnit entry — never a skip; and
|
||||
- by the merge gate's **`ci/l0.sh`** stage, so these assertions run on every PR; and
|
||||
- by the wrapping Chainsaw `Test`, which runs the SAME `check.sh`, so an in-cluster run
|
||||
executes byte-identical assertions.
|
||||
|
||||
Two authoring rules specific to this layer:
|
||||
|
||||
1. **State the gap.** A statically-provable half of a requirement is still only half.
|
||||
Every `check.sh` header ends with an explicit *"what this does NOT prove"* list naming
|
||||
the cluster-layer remainder. Covering a requirement at L0 does not finish it.
|
||||
2. **Exemptions must be self-policing.** Where a check must tolerate something (a
|
||||
deliberately-plaintext policy fixture, an `.example` template), the exemption is
|
||||
conditioned on a property that would break if the exemption were abused — e.g. the
|
||||
plaintext Secret fixture is exempt *only while* a `kyverno-test.yaml` asserts the
|
||||
policy rejects it. Never a bare path allowlist.
|
||||
|
||||
## Requirement-tagging convention (authoritative)
|
||||
|
||||
A test declares the requirement(s) it verifies **three** ways; the label is the source
|
||||
@@ -58,7 +28,7 @@ metadata:
|
||||
annotations:
|
||||
# Human-readable canonical list — MUST equal the label set.
|
||||
conformance.home.k3s/requirements: "R-NET-01"
|
||||
# Declared layer (L0|L1|L2|L3|META) — must match the subtree.
|
||||
# Declared layer (L1|L2|L3|META) — must match the subtree.
|
||||
conformance.home.k3s/layer: "L2"
|
||||
# true => R-TEST-05 requires this to be (or have) an L3 fault-injection test.
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
@@ -95,7 +65,4 @@ and coverage extraction exact and robust.
|
||||
the PM — do not relax the assertion or the requirement.
|
||||
|
||||
Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
|
||||
later roadmap phase delivers the capability under test. `lib/status.py` treats such a
|
||||
marker — in the `chainsaw-test.yaml`, in `render-check.sh`, or in `check.sh` — as
|
||||
outranking any `conformance.home.k3s/executable: "true"` annotation, so a placeholder body
|
||||
can never be reported as an executable test.
|
||||
later roadmap phase delivers the capability under test.
|
||||
@@ -1,60 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — Flux postBuild substitution is complete and fail-closed-safe
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# R-ENV-02 ("authored once, promoted dev -> prod") is implemented by ONE mechanism:
|
||||
# `postBuild.substituteFrom cluster-vars` injecting `${TOKEN}`s into the manifests each
|
||||
# Flux Kustomization renders. That mechanism FAILS OPEN — an undefined variable becomes
|
||||
# the empty string while the Kustomization still reports Ready — which emptied Cilium's
|
||||
# k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01).
|
||||
#
|
||||
# GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom,
|
||||
# WHEN each `spec.path` is resolved to the exact file set it renders (following
|
||||
# kustomization.yaml `resources:` transitively, not globbing the repo),
|
||||
# THEN every `${...}` in those files is either DEFINED in clusters/base/cluster-vars.yaml
|
||||
# or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default
|
||||
# operator on an undefined name (the silent-empty class strict mode does NOT catch);
|
||||
# no unescaped expansion is unparseable by envsubst (which fails the WHOLE
|
||||
# Kustomization); every path rendering a knob actually substitutes cluster-vars;
|
||||
# every overlay-patched knob has a base default; and the check is wired into the
|
||||
# merge gate as a blocking, unconditionally-run stage.
|
||||
#
|
||||
# This is the prerequisite proof for enabling kustomize-controller's
|
||||
# StrictPostBuildSubstitutions feature gate, and the permanent guard that keeps it safe.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library
|
||||
# only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
|
||||
# Test wraps the SAME script.
|
||||
#
|
||||
# NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests
|
||||
# a path includes — a deliberate superset of the post-`kustomize build` stream. It does
|
||||
# not render, does not prove the feature gate is enabled on the live kustomize-controller,
|
||||
# and does not validate knob VALUES. R-GITOPS-01 is tagged for the narrow static slice
|
||||
# only: what git says a Kustomization renders is fully determined, with no variable
|
||||
# silently emptied on the way in. The live-state half is L2, untested today.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: flux-postbuild-substitution
|
||||
labels:
|
||||
conformance.home.k3s/R-ENV-02: "true"
|
||||
conformance.home.k3s/R-GITOPS-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every variable a substituted Flux path renders is defined in cluster-vars or
|
||||
escaped, so post-build substitution cannot silently empty a value and enabling
|
||||
StrictPostBuildSubstitutions cannot fail a legitimate manifest.
|
||||
spec:
|
||||
steps:
|
||||
- name: audit-every-substituted-flux-path
|
||||
description: >-
|
||||
Resolve each postBuild.substituteFrom Kustomization to the files it renders and
|
||||
assert every ${...} is defined or escaped, then assert the gate wires it blocking.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,369 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — flux-postbuild-substitution
|
||||
# Serves: R-ENV-02 ("Same GitOps config promotes dev/test -> prod (authored once)."
|
||||
# — spec docs/02-environments-and-provisioning.md)
|
||||
# R-GITOPS-01 ("All desired state in git, reconciled by Flux."
|
||||
# — spec docs/05-gitops-and-delivery.md)
|
||||
#
|
||||
# WHY THIS EXISTS — Flux post-build substitution FAILS OPEN
|
||||
# --------------------------------------------------------
|
||||
# R-ENV-02 is implemented by exactly one mechanism: `clusters/base/cluster-vars.yaml`
|
||||
# is injected into every `${TOKEN}` of the manifests each Flux Kustomization renders,
|
||||
# via `postBuild.substituteFrom`. That mechanism is silently lossy by default: an
|
||||
# UNDEFINED variable becomes the EMPTY STRING and the Kustomization still reports
|
||||
# Ready. On 2026-09-01 that emptied `k8sServiceHost` in the Cilium HelmRelease and put
|
||||
# cilium into Init:Error on 2 of 3 dev nodes (DECISIONS-LOG "substitution race that
|
||||
# fails OPEN"). "Authored once, promoted" is therefore only true while every token a
|
||||
# substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
|
||||
# the repo asserted that. This check does.
|
||||
#
|
||||
# ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
|
||||
# `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
|
||||
# ${DEFINED} -> substituted with the cluster-vars value.
|
||||
# ${UNDEFINED} -> EMPTY STRING, silently. `--strict` / the cluster-wide
|
||||
# StrictPostBuildSubstitutions feature gate turns this into a
|
||||
# loud Kustomization failure ("variable not set (strict mode)").
|
||||
# ${UNDEFINED:-d} -> the DEFAULT is applied AT RECONCILE TIME. Strict mode does
|
||||
# NOT catch this (it exempts default-operator forms by design),
|
||||
# so a shell guard written `[ -z "${X:-}" ]` inside a rendered
|
||||
# manifest is pre-resolved to `[ -z "" ]` and its else-branch
|
||||
# becomes unreachable in the container. Silent, and permanent.
|
||||
# $${VAR}, $${V:-d}, $${ARR[0]} -> emitted as the LITERAL ${...} for a downstream
|
||||
# shell/cloud-init consumer. Escaping is total: it neutralises
|
||||
# even the indexed form.
|
||||
# $VAR, $(cmd) -> left alone by envsubst; they need no escaping.
|
||||
# ${...}, ${a b}, ${ARR[0]} (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
|
||||
# variable name" / "missing closing brace") which fails the
|
||||
# WHOLE Kustomization — every resource in the path, not just
|
||||
# the offending one.
|
||||
# The escape contract this enforces is the one already documented in
|
||||
# `clusters/base/apps-vms.yaml`; this check is that comment made executable.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# WHEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom
|
||||
# is resolved to the exact file set its `spec.path` renders (by following
|
||||
# kustomization.yaml `resources:` transitively — a path, not a glob),
|
||||
# THEN (1) every `${...}` in those files is DEFINED in cluster-vars, or ESCAPED `$${...}`;
|
||||
# (2) no UNESCAPED expansion carries a default/length/substitution operator on an
|
||||
# undefined name (the silent-empty class strict mode does not catch);
|
||||
# (3) no UNESCAPED expansion is unparseable by envsubst (`${...}`, `${ARR[0]}`) —
|
||||
# that fails the entire Kustomization, so it is fatal wherever it appears;
|
||||
# (4) every Kustomization that renders a cluster-vars token actually declares
|
||||
# `substituteFrom` a ConfigMap named cluster-vars (else the literal token
|
||||
# ships into the cluster);
|
||||
# (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
|
||||
# the "add the knob with a base default BEFORE referencing it" rule that the
|
||||
# incident's interim discipline depends on;
|
||||
# AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and
|
||||
# ci/gate.sh sets the merge block when the l0 stage fails.
|
||||
# Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
|
||||
# preserves body comments into the stream envsubst reads (verified: a `$`-carrying
|
||||
# comment survives into the built apps/wave-1 output), so a braced expression in a
|
||||
# comment is a live hazard, not prose.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
|
||||
# - It does NOT render the manifests. It scans the SOURCE files a path includes,
|
||||
# which is a deliberate SUPERSET of the post-`kustomize build` stream (kustomize
|
||||
# drops some comments). A finding in a comment may not reach envsubst TODAY; it is
|
||||
# still reported, because whether a given comment survives the build is a kustomize
|
||||
# implementation detail that changes with the pinned version, and the failure it
|
||||
# causes is total (assertion 3). The end-to-end proof — `kustomize build <path> |
|
||||
# flux envsubst --strict` for every substituted path — needs kustomize + flux,
|
||||
# which the L0 CI toolchain does not install (.gitea/workflows/scripts/install-tools.sh
|
||||
# l0), so it cannot run here without permanently SKIPPING. It was run by hand and
|
||||
# recorded in DECISIONS-LOG; this check is the reproducible static half.
|
||||
# - It does NOT prove the cluster's live state matches git, nor that the
|
||||
# StrictPostBuildSubstitutions feature gate is actually enabled on
|
||||
# kustomize-controller. Both are cluster facts (L2), untested here. Tagging
|
||||
# R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
|
||||
# is fully determined, with no variable silently emptied on the way in.
|
||||
# - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
|
||||
# that an overlay's LAN block is self-consistent). Presence, not correctness.
|
||||
# - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
|
||||
# several are deliberately reference-only and say so in cluster-vars.yaml
|
||||
# (PIA_REGION "retained, not substituted"; CLUSTER_POD_CIDR "Documents ..."), so
|
||||
# failing on them would be a false block on documented intent.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green) if the thing under test is not present at all.
|
||||
if [ ! -d "${ROOT}/clusters" ] || [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then
|
||||
echo "check SKIPPED: clusters/ or clusters/base/cluster-vars.yaml absent — nothing to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit of Flux postBuild substitution safety. Pure standard library.
|
||||
|
||||
No PyYAML: the file shapes parsed here are line-constrained and the parse contracts are
|
||||
documented at each parser, matching lib/coverage.py + lib/status.py. The kustomization
|
||||
`resources:` parser is IMPORTED from lib/status.py rather than forked, so there is one
|
||||
definition of "what a Flux path actually renders" in the harness.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
sys.path.insert(0, str(ROOT / "conformance" / "lib"))
|
||||
import status as st # noqa: E402 — reuse the walk, don't fork it
|
||||
|
||||
FAILURES: list[str] = []
|
||||
ADVISORIES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ cluster-vars
|
||||
# Parse contract: `data:` block of a single ConfigMap, one ` KEY: "value"` per line
|
||||
# (every entry in clusters/base/cluster-vars.yaml is a quoted scalar; comments and
|
||||
# continuation-comment lines are indented further and never match).
|
||||
_DATA_KEY_RE = re.compile(r'^ ([A-Za-z_][A-Za-z0-9_]*):\s*(?:"|\')')
|
||||
|
||||
|
||||
def cluster_vars_keys(path: Path) -> set[str]:
|
||||
keys: set[str] = set()
|
||||
in_data = False
|
||||
for raw in path.read_text().splitlines():
|
||||
if re.match(r"^data:\s*$", raw):
|
||||
in_data = True
|
||||
continue
|
||||
if not in_data:
|
||||
continue
|
||||
if st._TOP_KEY_RE.match(raw): # a new top-level key ends the data block
|
||||
break
|
||||
m = _DATA_KEY_RE.match(raw)
|
||||
if m:
|
||||
keys.add(m.group(1))
|
||||
return keys
|
||||
|
||||
|
||||
VARS_FILE = ROOT / "clusters" / "base" / "cluster-vars.yaml"
|
||||
KEYS = cluster_vars_keys(VARS_FILE)
|
||||
check(len(KEYS) > 0, f"no data keys parsed from {VARS_FILE} — the parser or the file shape changed")
|
||||
|
||||
# ------------------------------------------- Flux Kustomizations with substituteFrom
|
||||
# Parse contract mirrors lib/status.py's flux_paths(): docs split on a lone `---`, a doc
|
||||
# is a Flux entrypoint only if it carries the kustomize.toolkit apiVersion AND
|
||||
# `kind: Kustomization` AND a spec-level `path:`. Inline `patch: |-` bodies in the
|
||||
# overlays carry Flux Kustomizations WITHOUT a path and are correctly skipped.
|
||||
_SUBFROM_RE = re.compile(r"^(\s*)substituteFrom:\s*$", re.M)
|
||||
_SUBITEM_KIND_RE = re.compile(r"^\s*-\s*kind:\s*(\S+)")
|
||||
_SUBITEM_NAME_RE = re.compile(r"^\s*(?:-\s*)?name:\s*([^\s#]+)")
|
||||
|
||||
|
||||
def substitute_sources(doc: str) -> list[tuple[str, str]]:
|
||||
"""[(kind, name)] declared under postBuild.substituteFrom in this doc."""
|
||||
m = _SUBFROM_RE.search(doc)
|
||||
if not m:
|
||||
return []
|
||||
indent = len(m.group(1))
|
||||
out: list[tuple[str, str]] = []
|
||||
kind = ""
|
||||
for raw in doc[m.end():].splitlines():
|
||||
if raw.strip() == "" or raw.lstrip().startswith("#"):
|
||||
continue
|
||||
if len(raw) - len(raw.lstrip()) <= indent and not raw.lstrip().startswith("-"):
|
||||
break # dedented out of the list
|
||||
km = _SUBITEM_KIND_RE.match(raw)
|
||||
if km:
|
||||
kind = km.group(1)
|
||||
continue
|
||||
nm = _SUBITEM_NAME_RE.match(raw)
|
||||
if nm and kind:
|
||||
out.append((kind, nm.group(1)))
|
||||
kind = ""
|
||||
return out
|
||||
|
||||
|
||||
kustomizations: list[dict] = []
|
||||
for yfile in sorted((ROOT / "clusters").rglob("*.yaml")):
|
||||
text = yfile.read_text()
|
||||
if st.FLUX_API not in text:
|
||||
continue
|
||||
for doc in st._DOC_SPLIT_RE.split(text):
|
||||
if st.FLUX_API not in doc or not re.search(r"^kind:\s*Kustomization\s*$", doc, re.M):
|
||||
continue
|
||||
pm = st._PATH_RE.search(doc)
|
||||
if not pm:
|
||||
continue
|
||||
nm = st._NAME_RE.search(doc)
|
||||
srcs = substitute_sources(doc)
|
||||
if not srcs:
|
||||
continue # no substitution => nothing to audit
|
||||
kustomizations.append({
|
||||
"file": str(yfile.relative_to(ROOT)),
|
||||
"name": nm.group(1) if nm else "?",
|
||||
"path": pm.group(1),
|
||||
"sources": srcs,
|
||||
})
|
||||
|
||||
check(len(kustomizations) > 0,
|
||||
"no Flux Kustomization with postBuild.substituteFrom found under clusters/ — "
|
||||
"the parser or the repo layout changed; this check would silently audit nothing")
|
||||
|
||||
# ------------------------------------------------------------------ token scanning
|
||||
# One `$`-run followed by a braced expression. `[^{}]*` is exhaustive for this tree:
|
||||
# a repo-wide sweep found no nested/unclosed brace expression in any manifest (only
|
||||
# Gitea Actions `${{ ... }}` in .gitea/, which no Flux Kustomization renders).
|
||||
TOKEN_RE = re.compile(r"(\$+)\{([^{}]*)\}")
|
||||
NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
# An expansion whose name is followed by a POSIX parameter operator. envsubst EVALUATES
|
||||
# these, so an undefined name yields the default/empty AT RECONCILE TIME.
|
||||
OP_RE = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*)(:[-=?+].*|:[0-9].*|/.*|#.*|%.*|\^.*|,.*)$")
|
||||
LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")
|
||||
|
||||
|
||||
def classify(dollars: int, inner: str) -> tuple[str, str] | None:
|
||||
"""Return (severity-tag, remediation) for a hazardous expansion, else None."""
|
||||
if dollars >= 2:
|
||||
return None # escaped: emitted as a literal
|
||||
if NAME_RE.match(inner):
|
||||
if inner in KEYS:
|
||||
return None # defined knob — the happy path
|
||||
return ("UNDEFINED",
|
||||
f"strict mode fails the Kustomization; today it renders EMPTY. Either add "
|
||||
f"'{inner}' to clusters/base/cluster-vars.yaml with a base default, or "
|
||||
f"escape it as $${{{inner}}} if it is meant for a shell/cloud-init consumer")
|
||||
m = OP_RE.match(inner) or LEN_RE.match(inner)
|
||||
if m:
|
||||
base = m.group(1)
|
||||
if base in KEYS:
|
||||
return None # defined knob with an operator
|
||||
return ("SILENT-EMPTY",
|
||||
f"envsubst resolves this AT RECONCILE TIME because '{base}' is not a "
|
||||
f"cluster-var, so the rendered manifest carries the default, not the "
|
||||
f"expression. StrictPostBuildSubstitutions does NOT catch this form. "
|
||||
f"Escape it as $${{{inner}}} if the CONTAINER's shell is meant to expand it")
|
||||
return ("FATAL-PARSE",
|
||||
"envsubst cannot parse this and fails the ENTIRE Kustomization (every resource "
|
||||
"in the path). Escape it as $${...} or rewrite the prose without braces")
|
||||
|
||||
|
||||
def rendered_files(path_spec: str) -> tuple[list[str], bool]:
|
||||
"""Repo-relative files the Flux `spec.path` renders, resolving `resources:` transitively."""
|
||||
rel = os.path.normpath(path_spec.lstrip("./")) if path_spec not in (".", "./") else "."
|
||||
if not (ROOT / rel).is_dir():
|
||||
return [], False
|
||||
applied, _ = st.reconciled_files(ROOT, rel)
|
||||
return sorted(applied), True
|
||||
|
||||
|
||||
findings: list[str] = []
|
||||
referenced: dict[str, set[str]] = {}
|
||||
inventory: list[str] = []
|
||||
|
||||
for k in kustomizations:
|
||||
files, exists = rendered_files(k["path"])
|
||||
check(exists, f"{k['file']}: Kustomization '{k['name']}' renders path '{k['path']}' "
|
||||
f"which is not a directory in this repo")
|
||||
has_cluster_vars = ("ConfigMap", "cluster-vars") in k["sources"]
|
||||
n_def = n_esc = n_bad = 0
|
||||
for rel in files:
|
||||
f = ROOT / rel
|
||||
try:
|
||||
text = f.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
continue
|
||||
for lineno, line in enumerate(text.splitlines(), 1):
|
||||
for m in TOKEN_RE.finditer(line):
|
||||
dollars, inner = len(m.group(1)), m.group(2)
|
||||
verdict = classify(dollars, inner)
|
||||
if verdict is None:
|
||||
if dollars >= 2:
|
||||
n_esc += 1
|
||||
else:
|
||||
n_def += 1
|
||||
referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
|
||||
set()).add(k["name"])
|
||||
# (4) a DEFINED knob is only substituted if this Kustomization
|
||||
# actually pulls cluster-vars in; otherwise the literal ships.
|
||||
check(has_cluster_vars,
|
||||
f"{rel}:{lineno}: renders cluster-var ${{{inner}}} but "
|
||||
f"Kustomization '{k['name']}' ({k['file']}) does not "
|
||||
f"substituteFrom ConfigMap/cluster-vars — the literal token "
|
||||
f"would ship into the cluster")
|
||||
continue
|
||||
n_bad += 1
|
||||
tag, fix = verdict
|
||||
findings.append(f" [{tag}] {rel}:{lineno} {m.group(0)}\n"
|
||||
f" rendered by Flux Kustomization '{k['name']}' "
|
||||
f"({k['file']}, path {k['path']})\n"
|
||||
f" {fix}\n"
|
||||
f" | {line.strip()[:120]}")
|
||||
inventory.append(f" {k['name']:<16} path={k['path']:<22} files={len(files):<3} "
|
||||
f"defined={n_def:<3} escaped={n_esc:<3} hazards={n_bad}")
|
||||
|
||||
check(not findings,
|
||||
f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")
|
||||
|
||||
# --------------------------------------------- (5) overlay knobs exist in the base
|
||||
# Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
|
||||
# `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
|
||||
# header) is that a knob is ADDED TO THE BASE with a default BEFORE any overlay or
|
||||
# manifest references it — that ordering is what closes the define-then-use race.
|
||||
_OP_PATH_RE = re.compile(r"^\s*(?:-\s*)?path:\s*/data/([A-Za-z_][A-Za-z0-9_]*)\s*$", re.M)
|
||||
for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
|
||||
for key in set(_OP_PATH_RE.findall(overlay.read_text())):
|
||||
check(key in KEYS,
|
||||
f"{overlay.relative_to(ROOT)}: patches /data/{key} but '{key}' has no BASE "
|
||||
f"default in clusters/base/cluster-vars.yaml — a second overlay would render "
|
||||
f"${{{key}}} empty (R-ENV-02: author once, supply per env)")
|
||||
|
||||
# ------------------------------------------------------- (6) the check is load-bearing
|
||||
L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
|
||||
GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
|
||||
check("flux-postbuild-substitution/check.sh" in L0,
|
||||
"ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped "
|
||||
"conformance selection can filter this check out of a gate run, so l0.sh must invoke "
|
||||
"it directly and unconditionally or the gate is not actually enforcing it")
|
||||
check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
|
||||
"ci/gate.sh does not set the merge block when the l0 stage fails")
|
||||
|
||||
# ------------------------------------------------------------- advisory: dead knobs
|
||||
for key in sorted(KEYS - set(referenced)):
|
||||
ADVISORIES.append(f"cluster-vars knob '{key}' is referenced by NO reconciled manifest "
|
||||
f"(dead knob, or documentation-only — advisory, not a failure)")
|
||||
|
||||
# ------------------------------------------------------------------------- report
|
||||
print("Flux postBuild substitution inventory (per Kustomization):")
|
||||
for line in inventory:
|
||||
print(line)
|
||||
print(f"\ncluster-vars knobs defined: {len(KEYS)} referenced: {len(referenced)}")
|
||||
|
||||
if ADVISORIES:
|
||||
print("\nADVISORIES (not failures):")
|
||||
for a in ADVISORIES:
|
||||
print(f" - {a}")
|
||||
|
||||
if findings:
|
||||
print("\nHAZARDS — unescaped expansions that are not cluster-vars knobs:", file=sys.stderr)
|
||||
for f in findings:
|
||||
print(f, file=sys.stderr)
|
||||
|
||||
print()
|
||||
if not FAILURES:
|
||||
print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "
|
||||
f"(R-ENV-02, R-GITOPS-01)")
|
||||
raise SystemExit(0)
|
||||
print(f"CHECK FAIL (flux-postbuild-substitution): {len(FAILURES)} failed, "
|
||||
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" FAIL {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
@@ -1,44 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — least privilege for workload identities and network reach (R-SEC-02)
|
||||
# Catalog: docs/14 L0 row, "Manifests/ServiceDefinitions obey invariants" — runs in
|
||||
# CI WITHOUT a cluster.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked RBAC object and every least-network-reach policy is inspected,
|
||||
# THEN no in-repo role grants a wildcard or a privilege-escalation verb, no workload
|
||||
# ServiceAccount is bound to a privileged role, the conformance runner's own
|
||||
# privileged test identity stays scoped to its authored ClusterRole, and each
|
||||
# least-network-reach policy both Enforces and is proven by an L0 negative row
|
||||
# (or a generated-resource row) to actually close reach.
|
||||
#
|
||||
# STATUS: EXECUTABLE — and DELIBERATELY PARTIAL. `check.sh` is a pure offline assertion
|
||||
# script (git ls-files + PyYAML); conformance/run.sh --layer l0 and ci/l0.sh run it
|
||||
# directly, and this Chainsaw Test wraps the SAME script. It proves the STATIC half of
|
||||
# both clauses. The RUNTIME half — that the API server actually denies an unrequested
|
||||
# verb, and that pod-to-pod traffic is actually dropped — needs a cluster and is NOT
|
||||
# asserted under this requirement; see the gap list in check.sh's header. R-SEC-02 is
|
||||
# therefore covered, not finished.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: least-privilege-rbac
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Workload identities carry no wildcard, no escalation verb and no privileged
|
||||
binding; the least-network-reach policies are enforced and proven to reject.
|
||||
spec:
|
||||
steps:
|
||||
- name: scan-rbac-and-least-reach-policies
|
||||
description: >-
|
||||
Assert the authored least-privilege posture of every tracked RBAC object and
|
||||
the enforced, test-proven status of the least-network-reach policies.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,269 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — least-privilege-rbac
|
||||
# Serves: R-SEC-02 ("The platform MUST follow least-privilege for workload
|
||||
# identities and network reach." — spec docs/00-vision-and-scope.md;
|
||||
# detail in docs/07 "least privilege ... least network reach").
|
||||
#
|
||||
# The requirement has TWO halves and this check covers the STATIC half of each,
|
||||
# at the layer docs/14 puts them ("L0 Static/policy | Pre-apply, in CI").
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked RBAC object and every least-network-reach policy is inspected,
|
||||
# THEN — workload identity —
|
||||
# (1) no in-repo Role/ClusterRole grants a wildcard apiGroup, resource or verb,
|
||||
# and none grants escalate/bind/impersonate (the privilege-escalation verbs);
|
||||
# (2) no (Cluster)RoleBinding binds a ServiceAccount — a WORKLOAD identity — to
|
||||
# `cluster-admin` or to any in-repo role holding wildcard rules. Human/Group
|
||||
# subjects are out of scope: R-SEC-02 constrains workload identities, and the
|
||||
# operator's own admin path is an OIDC group binding, deliberately;
|
||||
# (3) every ServiceAccount subject is namespaced (no cluster-wide ambiguity);
|
||||
# (4) the conformance runner's own SA — this workstream's PRIVILEGED TEST
|
||||
# IDENTITY, whose least-privilege authoring docs/14 explicitly delegates to
|
||||
# WS-TEST — is bound only to its own scoped ClusterRole, carries no wildcard
|
||||
# and no privilege-escalation verb, and grants nothing over RBAC objects
|
||||
# themselves (no rbac.authorization.k8s.io rules);
|
||||
# — network reach —
|
||||
# (5) the policies that implement least network reach exist, and each is Enforce
|
||||
# (or carries the Flux token that substitutes to Enforce): the default-deny
|
||||
# baseline generator, the baseline SHAPE validator, the host-namespace/
|
||||
# privilege restriction, and the egress-class label requirement;
|
||||
# (6) each of those policies is actually EXERCISED by the L0 Kyverno suite with a
|
||||
# NEGATIVE row (result: fail) — a policy nobody proves rejects anything is a
|
||||
# declaration, not a gate.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gap, cluster layer — R-SEC-02 is NOT fully covered):
|
||||
# - RUNTIME least privilege of workload identities: that the API server actually
|
||||
# denies the conformance SA (or any workload SA) the verbs it was not granted.
|
||||
# That needs a live cluster (L2) and no test asserts it today.
|
||||
# - RUNTIME least network reach: that pod-to-pod traffic is actually denied. The
|
||||
# behaviour is exercised by tests/l2-behavioral/{default-deny,deny-undeclared},
|
||||
# but those are tagged R-NET-01/R-NET-02 and are cluster-gated (skipped in Phase 0);
|
||||
# nothing proves it under an R-SEC-02 tag.
|
||||
# - Helm-chart-generated RBAC (e.g. what a HelmRelease's chart creates in-cluster) is
|
||||
# not in the repo, so it is outside this static scan entirely.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import os, re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def tracked() -> list[str]:
|
||||
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout
|
||||
return [p for p in out.split("\0") if p]
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
|
||||
if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
FILES = tracked()
|
||||
YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
|
||||
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
|
||||
|
||||
ESCALATION_VERBS = {"escalate", "bind", "impersonate"}
|
||||
PRIVILEGED_ROLES = {"cluster-admin", "admin", "edit"}
|
||||
|
||||
roles: dict[tuple[str, str], dict] = {} # (kind, name) -> {"file":…, "wildcard":bool, …}
|
||||
bindings: list[dict] = []
|
||||
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
kind = d.get("kind")
|
||||
name = (d.get("metadata") or {}).get("name")
|
||||
if kind in ("Role", "ClusterRole"):
|
||||
wildcard, escalating, rbac_over_rbac = False, set(), False
|
||||
for r in d.get("rules") or []:
|
||||
if not isinstance(r, dict):
|
||||
continue
|
||||
for field in ("apiGroups", "resources", "verbs"):
|
||||
if "*" in (r.get(field) or []):
|
||||
wildcard = True
|
||||
escalating |= set(v for v in (r.get("verbs") or [])) & ESCALATION_VERBS
|
||||
if "rbac.authorization.k8s.io" in (r.get("apiGroups") or []):
|
||||
rbac_over_rbac = True
|
||||
roles[(kind, name)] = {"file": rel, "wildcard": wildcard,
|
||||
"escalating": escalating, "rbac": rbac_over_rbac,
|
||||
"doc": d}
|
||||
# (1) no wildcards, no privilege-escalation verbs.
|
||||
check(not wildcard,
|
||||
f"{rel}: {kind}/{name} grants a wildcard apiGroup/resource/verb — that is "
|
||||
"not least-privilege (R-SEC-02)")
|
||||
check(not escalating,
|
||||
f"{rel}: {kind}/{name} grants privilege-escalation verb(s) "
|
||||
f"{sorted(escalating)} (R-SEC-02)")
|
||||
elif kind in ("RoleBinding", "ClusterRoleBinding"):
|
||||
bindings.append({"file": rel, "name": name, "doc": d})
|
||||
|
||||
check(len(roles) > 0, "no Role/ClusterRole found in the repo — the RBAC scan is not looking at it")
|
||||
check(len(bindings) > 0, "no RoleBinding/ClusterRoleBinding found — the RBAC scan is not looking at it")
|
||||
|
||||
for b in bindings:
|
||||
d, rel, name = b["doc"], b["file"], b["name"]
|
||||
ref = d.get("roleRef") or {}
|
||||
ref_kind, ref_name = ref.get("kind"), ref.get("name")
|
||||
sa_subjects = [s for s in (d.get("subjects") or [])
|
||||
if isinstance(s, dict) and s.get("kind") == "ServiceAccount"]
|
||||
local = roles.get((ref_kind, ref_name))
|
||||
ref_is_privileged = (ref_name in PRIVILEGED_ROLES) or bool(local and local["wildcard"])
|
||||
# (2) a WORKLOAD identity must never hold a privileged role.
|
||||
check(not (sa_subjects and ref_is_privileged),
|
||||
f"{rel}: {d.get('kind')}/{name} binds ServiceAccount(s) "
|
||||
f"{[s.get('name') for s in sa_subjects]} to the privileged role {ref_name!r} — "
|
||||
"workload identities must be least-privilege (R-SEC-02)")
|
||||
# (3) SA subjects must name their namespace (a ClusterRoleBinding without one is
|
||||
# ambiguous and, in a RoleBinding, silently defaults to the binding's namespace).
|
||||
for s in sa_subjects:
|
||||
check(bool(s.get("namespace")) or d.get("kind") == "RoleBinding",
|
||||
f"{rel}: {d.get('kind')}/{name} has a ServiceAccount subject "
|
||||
f"{s.get('name')!r} with no namespace (R-SEC-02)")
|
||||
|
||||
# --------------------------------- (4) our own privileged test identity, in detail
|
||||
RBAC_FILE = "conformance/incluster/rbac.yaml"
|
||||
check(RBAC_FILE in FILES, f"{RBAC_FILE} is missing — the conformance runner has no authored RBAC")
|
||||
runner_role = roles.get(("ClusterRole", "conformance-runner"))
|
||||
check(runner_role is not None,
|
||||
f"{RBAC_FILE}: no ClusterRole/conformance-runner — the test identity's grants are undefined")
|
||||
if runner_role:
|
||||
check(runner_role["file"] == RBAC_FILE,
|
||||
f"ClusterRole/conformance-runner is defined in {runner_role['file']}, not {RBAC_FILE}")
|
||||
check(not runner_role["wildcard"], "ClusterRole/conformance-runner uses a wildcard (R-SEC-02)")
|
||||
check(not runner_role["escalating"],
|
||||
"ClusterRole/conformance-runner grants a privilege-escalation verb (R-SEC-02)")
|
||||
check(not runner_role["rbac"],
|
||||
"ClusterRole/conformance-runner grants rights over RBAC objects themselves — a test "
|
||||
"identity that can rewrite RBAC is not least-privilege (R-SEC-02)")
|
||||
|
||||
runner_bindings = [b for b in bindings
|
||||
if any(s.get("kind") == "ServiceAccount" and s.get("name") == "conformance-runner"
|
||||
for s in (b["doc"].get("subjects") or []))]
|
||||
check(len(runner_bindings) >= 1, "the conformance-runner ServiceAccount has no binding at all")
|
||||
for b in runner_bindings:
|
||||
ref = b["doc"].get("roleRef") or {}
|
||||
check(ref.get("name") == "conformance-runner",
|
||||
f"{b['file']}: conformance-runner is bound to {ref.get('name')!r} instead of its own "
|
||||
"scoped ClusterRole (R-SEC-02)")
|
||||
for s in b["doc"].get("subjects") or []:
|
||||
if s.get("kind") == "ServiceAccount":
|
||||
check(s.get("namespace") == "conformance",
|
||||
f"{b['file']}: conformance-runner subject is in namespace "
|
||||
f"{s.get('namespace')!r}, expected 'conformance' (R-SEC-02)")
|
||||
|
||||
# --------------------- (5)/(6) least NETWORK REACH: the policies exist, Enforce, and
|
||||
# are proven to reject something at L0.
|
||||
# Validating policies: must Enforce AND be proven to REJECT something at L0.
|
||||
REQUIRED_POLICIES = {
|
||||
"validate-default-deny-shape": "a baseline that leaves a direction open is rejected",
|
||||
"restrict-host-namespaces": "host namespaces / privileged containers are rejected",
|
||||
"require-egress-class-label": "workloads must declare their egress class",
|
||||
}
|
||||
# Generating policies have no validationFailureAction and no negative row: the proof
|
||||
# that they close network reach is that the L0 suite asserts the resource they GENERATE.
|
||||
REQUIRED_GENERATE_POLICIES = {
|
||||
"require-default-deny-baseline": "every namespace gets a default-deny baseline",
|
||||
}
|
||||
policies: dict[str, dict] = {}
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") in ("ClusterPolicy", "Policy"):
|
||||
policies[(d.get("metadata") or {}).get("name")] = {"file": rel, "doc": d}
|
||||
|
||||
for pname, why in REQUIRED_POLICIES.items():
|
||||
p = policies.get(pname)
|
||||
check(p is not None, f"policy {pname!r} is missing — nothing enforces: {why} (R-SEC-02)")
|
||||
if not p:
|
||||
continue
|
||||
action = str((p["doc"].get("spec") or {}).get("validationFailureAction", ""))
|
||||
check(action == "Enforce" or action.startswith("${"),
|
||||
f"{p['file']}: policy {pname!r} validationFailureAction is {action!r} — a least-reach "
|
||||
"policy in Audit mode blocks nothing (R-SEC-02)")
|
||||
|
||||
negatives: dict[str, int] = {}
|
||||
suites = [f for f in FILES if os.path.basename(f) == "kyverno-test.yaml"]
|
||||
check(len(suites) > 0, "no kyverno-test.yaml — the L0 policy gate is not exercised at all")
|
||||
for suite in suites:
|
||||
for d in docs_of(suite):
|
||||
for r in d.get("results") or []:
|
||||
if isinstance(r, dict) and r.get("result") == "fail":
|
||||
negatives[r.get("policy")] = negatives.get(r.get("policy"), 0) + 1
|
||||
|
||||
for pname, why in REQUIRED_POLICIES.items():
|
||||
check(negatives.get(pname, 0) > 0,
|
||||
f"no L0 test row asserts that policy {pname!r} REJECTS a violating resource "
|
||||
f"(result: fail) — unproven gate for: {why} (R-SEC-02)")
|
||||
|
||||
# The generating half: the policy must exist, and the L0 suite must assert the resource
|
||||
# it generates (a `generatedResource` row that passes) — otherwise nothing proves a new
|
||||
# namespace actually arrives with its reach closed.
|
||||
for pname, why in REQUIRED_GENERATE_POLICIES.items():
|
||||
p = policies.get(pname)
|
||||
check(p is not None, f"policy {pname!r} is missing — nothing enforces: {why} (R-SEC-02)")
|
||||
generated_rows = []
|
||||
for suite in suites:
|
||||
for d in docs_of(suite):
|
||||
for r in d.get("results") or []:
|
||||
if isinstance(r, dict) and r.get("policy") == pname \
|
||||
and r.get("generatedResource") and r.get("result") == "pass":
|
||||
generated_rows.append((suite, r.get("generatedResource")))
|
||||
check(len(generated_rows) > 0,
|
||||
f"no L0 test row asserts the resource generated by {pname!r} — unproven gate for: "
|
||||
f"{why} (R-SEC-02)")
|
||||
for suite, gen in generated_rows:
|
||||
gp = os.path.normpath(os.path.join(os.path.dirname(suite), str(gen)))
|
||||
check(gp in FILES, f"{suite}: generatedResource {gen!r} does not resolve to a tracked file")
|
||||
golden = "\n".join(l for l in (ROOT / gp).read_text(errors="replace").splitlines()
|
||||
if not l.lstrip().startswith("#")) if gp in FILES else ""
|
||||
check("NetworkPolicy" in golden and re.search(r"policyTypes:", golden),
|
||||
f"{gp}: the asserted default-deny baseline is not a shaped NetworkPolicy (R-SEC-02)")
|
||||
|
||||
# The host-namespace policy must prove BOTH of its rules reject, not just one.
|
||||
host_rules = set()
|
||||
for suite in suites:
|
||||
for d in docs_of(suite):
|
||||
for r in d.get("results") or []:
|
||||
if isinstance(r, dict) and r.get("policy") == "restrict-host-namespaces" \
|
||||
and r.get("result") == "fail":
|
||||
host_rules.add(r.get("rule"))
|
||||
check({"disallow-host-namespaces", "disallow-privileged-containers"} <= host_rules,
|
||||
"the L0 suite does not prove BOTH restrict-host-namespaces rules reject a violating pod "
|
||||
f"(negative rows seen for: {sorted(host_rules)}) — R-SEC-02")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (least-privilege-rbac): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: least-privilege-rbac — {CHECKS} assertions over {len(roles)} role(s), "
|
||||
f"{len(bindings)} binding(s), {len(REQUIRED_POLICIES)} least-reach policies")
|
||||
PY
|
||||
@@ -1,43 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — no plaintext secrets in git (R-SEC-01)
|
||||
# Catalog: docs/14 L0 row, "Manifests/ServiceDefinitions obey invariants ...
|
||||
# no plaintext secrets" — runs in CI WITHOUT a cluster.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked file is inspected,
|
||||
# THEN every committed Secret is SOPS ciphertext, every ansible group_vars vault is
|
||||
# ansible-vault encrypted, no private key material is tracked, decryption is
|
||||
# delegated to an in-cluster SOPS key, and the L0 policy that rejects plaintext
|
||||
# Secrets exists, Enforces, and is proven by a negative test row to reject one.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files +
|
||||
# PyYAML, no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it
|
||||
# directly, and this Chainsaw Test wraps the SAME script so an in-cluster run executes
|
||||
# the identical assertions. Encryption at rest is necessary but NOT sufficient: the
|
||||
# runtime half of "decrypted only in-cluster", and a scan of git HISTORY, are named as
|
||||
# explicit gaps in check.sh's header and are NOT claimed here.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: no-plaintext-secrets
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
No secret is stored in plaintext in git: committed Secrets are SOPS ciphertext,
|
||||
ansible vaults are encrypted, no private key material is tracked, and the
|
||||
no-plaintext-secrets policy gate is enforced and proven to reject.
|
||||
spec:
|
||||
steps:
|
||||
- name: scan-tracked-tree-for-plaintext-secrets
|
||||
description: >-
|
||||
Assert over the whole tracked tree that no secret material is stored in
|
||||
plaintext and that decryption is delegated to the in-cluster SOPS key.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,287 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — no-plaintext-secrets
|
||||
# Serves: R-SEC-01 ("Secrets MUST NOT be stored in plaintext in git; they MUST be
|
||||
# encrypted at rest in the repo and decrypted only in-cluster."
|
||||
# — spec docs/00-vision-and-scope.md; detail in docs/07.)
|
||||
#
|
||||
# docs/14 puts exactly this invariant at L0 ("Static/policy | Pre-apply, in CI |
|
||||
# ... no plaintext secrets"), so it is asserted here over the WHOLE tracked tree,
|
||||
# with no cluster. Run by `conformance/run.sh --layer l0` and by ci/l0.sh.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked file is inspected,
|
||||
# THEN (1) SOPS is configured at the repo root with a PUBLIC age recipient;
|
||||
# (2) every committed Kubernetes Secret is SOPS-encrypted — a top-level
|
||||
# `sops:` block with a mac AND every data/stringData value is ENC[...];
|
||||
# (3) the only two exempt classes are narrow and self-policing:
|
||||
# a. `*.example` templates, which must NOT be referenced by any
|
||||
# kustomization and whose secret-named values must be placeholders;
|
||||
# b. the Kyverno L0 negative fixture, which is exempt ONLY while it is
|
||||
# actually wired into a kyverno-test.yaml that asserts
|
||||
# `disallow-plaintext-secrets` REJECTS it (result: fail);
|
||||
# (4) every ansible-vault group_vars file is `$ANSIBLE_VAULT;`-encrypted, and
|
||||
# no secret-valued ansible key carries a literal outside a vault;
|
||||
# (5) no private key material of any kind is tracked (PEM private keys, an
|
||||
# age SECRET key, kubeconfig client-key-data);
|
||||
# (6) decryption is delegated to the cluster: a Flux Kustomization declares
|
||||
# `decryption.provider: sops` with a secretRef, and the private half of
|
||||
# that key pair is nowhere in the repo (5).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gap, cluster layer):
|
||||
# - that the decrypted Secret only ever materialises in-cluster (the runtime half
|
||||
# of "decrypted only in-cluster") — needs a live cluster (L2), untested today.
|
||||
# - that no plaintext secret exists in git HISTORY. This walks the WORKING TREE at
|
||||
# HEAD only; docs/07's history-scan acceptance gate needs a secret scanner over
|
||||
# all revisions and is NOT implemented here.
|
||||
# - value-level entropy: a placeholder-looking string that is in fact a real
|
||||
# credential is not detectable statically.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import os, re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def tracked() -> list[str]:
|
||||
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout
|
||||
return [p for p in out.split("\0") if p]
|
||||
|
||||
|
||||
FILES = tracked()
|
||||
check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout")
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
"""Parsed YAML documents of a tracked file, or [] when it is not parseable YAML."""
|
||||
p = ROOT / rel
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(p.read_text(errors="replace")) if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
YAMLISH = re.compile(r"\.(ya?ml)(\.example)?$")
|
||||
YAML_FILES = [f for f in FILES if YAMLISH.search(f)]
|
||||
|
||||
# ---------------------------------------------------------------- (1) SOPS config
|
||||
sops_cfg = ROOT / ".sops.yaml"
|
||||
check(sops_cfg.is_file(), ".sops.yaml is missing from the repository root (R-SEC-01)")
|
||||
if sops_cfg.is_file():
|
||||
cfg = yaml.safe_load(sops_cfg.read_text()) or {}
|
||||
rules = cfg.get("creation_rules") or []
|
||||
check(len(rules) > 0, ".sops.yaml declares no creation_rules — new secrets would be committed unencrypted")
|
||||
for r in rules:
|
||||
pr = r.get("path_regex", "<none>")
|
||||
recips = str(r.get("age", ""))
|
||||
check(bool(recips), f".sops.yaml creation_rule {pr!r} has no age recipient")
|
||||
check(all(x.startswith("age1") for x in re.split(r"[,\s]+", recips) if x),
|
||||
f".sops.yaml creation_rule {pr!r} recipient is not a PUBLIC age1... key")
|
||||
check("AGE-SECRET-KEY" not in recips.upper(),
|
||||
f".sops.yaml creation_rule {pr!r} contains an age PRIVATE key")
|
||||
|
||||
# --------------------------------------------- Kyverno negative-fixture exemption
|
||||
# A plaintext Secret is tolerated ONLY where it is the negative case that proves the
|
||||
# L0 policy rejects plaintext. We do not trust the path: the file must be listed in a
|
||||
# kyverno-test.yaml whose results assert disallow-plaintext-secrets returns `fail`.
|
||||
POLICY_NAME = "disallow-plaintext-secrets"
|
||||
negative_fixtures: set[str] = set()
|
||||
kyverno_suites = [f for f in FILES if os.path.basename(f) == "kyverno-test.yaml"]
|
||||
check(len(kyverno_suites) > 0, "no kyverno-test.yaml found — the L0 policy gate is not exercised at all")
|
||||
for suite in kyverno_suites:
|
||||
docs = docs_of(suite)
|
||||
if not docs:
|
||||
continue
|
||||
d = docs[0]
|
||||
results = d.get("results") or []
|
||||
rejects_plaintext = any(r.get("policy") == POLICY_NAME and r.get("result") == "fail"
|
||||
for r in results if isinstance(r, dict))
|
||||
if not rejects_plaintext:
|
||||
continue
|
||||
base = os.path.dirname(suite)
|
||||
for res in d.get("resources") or []:
|
||||
negative_fixtures.add(os.path.normpath(os.path.join(base, str(res))))
|
||||
|
||||
check(len(negative_fixtures) > 0,
|
||||
f"no kyverno-test.yaml asserts that {POLICY_NAME} REJECTS a plaintext Secret "
|
||||
"(result: fail) — the plaintext-secret gate is unproven, so nothing may be exempt")
|
||||
|
||||
# --------------------------------------------------- (2)/(3) committed Secret scan
|
||||
SECRET_KEY_RE = re.compile(r"(pass|secret|token|key|credential|cookie|cert)", re.I)
|
||||
PLACEHOLDER_RE = re.compile(r"^\s*(|<[^>]*>|replace[-_ ]?me.*|change[-_ ]?me.*|placeholder.*|x{3,})\s*$", re.I)
|
||||
|
||||
secrets_seen = 0
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") != "Secret":
|
||||
continue
|
||||
secrets_seen += 1
|
||||
values = []
|
||||
for field in ("data", "stringData"):
|
||||
for k, v in (d.get(field) or {}).items():
|
||||
values.append((k, "" if v is None else str(v)))
|
||||
|
||||
is_example = rel.endswith(".example")
|
||||
is_negative_fixture = os.path.normpath(rel) in negative_fixtures
|
||||
|
||||
if is_example:
|
||||
# Template: never applied, and its secret-named values must be placeholders.
|
||||
refs = [k for k in YAML_FILES
|
||||
if os.path.basename(k) == "kustomization.yaml"
|
||||
and os.path.basename(rel) in (ROOT / k).read_text(errors="replace")]
|
||||
check(not refs,
|
||||
f"{rel}: an unencrypted .example template is referenced by a kustomization "
|
||||
f"({refs}) — it would be applied as a plaintext Secret")
|
||||
for k, v in values:
|
||||
if SECRET_KEY_RE.search(k):
|
||||
check(bool(PLACEHOLDER_RE.match(v)),
|
||||
f"{rel}: template key {k!r} holds {v[:24]!r}, not a placeholder — "
|
||||
"a real credential may have been committed (R-SEC-01)")
|
||||
continue
|
||||
|
||||
if is_negative_fixture:
|
||||
# Exempt, but only as the proven negative case; assert it really is plaintext
|
||||
# (an encrypted 'bad' fixture would silently stop testing anything).
|
||||
continue
|
||||
|
||||
# Everything else: must be SOPS-encrypted at rest.
|
||||
sops_block = d.get("sops")
|
||||
check(isinstance(sops_block, dict) and bool(sops_block.get("mac")),
|
||||
f"{rel}: committed Secret {d.get('metadata', {}).get('name')!r} has no top-level "
|
||||
"`sops:` block with a mac — it is stored in plaintext (R-SEC-01)")
|
||||
for k, v in values:
|
||||
check(v.startswith("ENC["),
|
||||
f"{rel}: Secret value {k!r} is not SOPS ciphertext (expected ENC[...]) — "
|
||||
"plaintext secret material in git (R-SEC-01)")
|
||||
|
||||
check(secrets_seen > 0, "no Kubernetes Secret manifests found at all — the scan is not looking at the repo")
|
||||
|
||||
# ------------------------------------------------------------ (4) ansible vaults
|
||||
VAULT_HEADER = "$ANSIBLE_VAULT;"
|
||||
vault_files = [f for f in FILES
|
||||
if os.path.basename(f) in ("vault.yml", "vault.yaml")]
|
||||
check(len(vault_files) > 0, "no ansible vault.yml files found — the vault scan is not looking at the repo")
|
||||
for rel in vault_files:
|
||||
head = (ROOT / rel).read_text(errors="replace")[:len(VAULT_HEADER)]
|
||||
check(head == VAULT_HEADER,
|
||||
f"{rel}: an ansible group_vars vault file is NOT ansible-vault encrypted "
|
||||
"(must start with the $ANSIBLE_VAULT header) — R-SEC-01")
|
||||
|
||||
SECRET_ANSIBLE_KEYS = ("ansible_become_password", "ansible_become_pass", "ansible_password",
|
||||
"ansible_ssh_pass", "ansible_sudo_pass", "ansible_ssh_private_key_file")
|
||||
ASSIGN_RE = re.compile(
|
||||
r"^\s*(" + "|".join(SECRET_ANSIBLE_KEYS) + r")\s*:\s*(.*?)\s*$")
|
||||
for rel in FILES:
|
||||
p = ROOT / rel
|
||||
if not p.is_file() or p.is_symlink():
|
||||
continue
|
||||
try:
|
||||
text = p.read_text(errors="replace")
|
||||
except Exception:
|
||||
continue
|
||||
if text.startswith(VAULT_HEADER):
|
||||
continue # encrypted at rest: opaque, by design
|
||||
for lineno, line in enumerate(text.splitlines(), 1):
|
||||
if line.lstrip().startswith("#"):
|
||||
continue
|
||||
m = ASSIGN_RE.match(line)
|
||||
if not m:
|
||||
continue
|
||||
key, val = m.group(1), m.group(2).strip().strip("'\"")
|
||||
ok = (val.startswith("{{") or val.startswith("$")
|
||||
or bool(PLACEHOLDER_RE.match(val)) or val == "")
|
||||
check(ok, f"{rel}:{lineno}: {key} is assigned the literal {val[:24]!r} outside an "
|
||||
"ansible-vault file — a plaintext secret in git (R-SEC-01)")
|
||||
|
||||
# ------------------------------------------------------ (5) no private key material
|
||||
# NOTE: each marker is assembled from fragments so that THIS file never contains the
|
||||
# literal string it hunts for — otherwise the scanner would flag itself the moment it
|
||||
# is tracked, and the only way to silence it would be to exempt the scanner (which is
|
||||
# exactly the kind of self-exemption this suite exists to prevent).
|
||||
_DASH5 = "-" * 5
|
||||
PRIVKEY_PATTERNS = [
|
||||
(re.compile(_DASH5 + r"BEGIN (?:[A-Z ]+ )?PRIVATE" + r" KEY" + _DASH5),
|
||||
"a PEM private key block"),
|
||||
(re.compile("AGE" + "-SECRET" + "-KEY-1"), "an age SECRET (decryption) key"),
|
||||
(re.compile(r"^\s*client" + r"-key-data\s*:\s*\S"), "kubeconfig client-key-data"),
|
||||
]
|
||||
for rel in FILES:
|
||||
p = ROOT / rel
|
||||
if not p.is_file() or p.is_symlink():
|
||||
continue
|
||||
try:
|
||||
text = p.read_text(errors="replace")
|
||||
except Exception:
|
||||
continue
|
||||
if text.startswith(VAULT_HEADER):
|
||||
continue
|
||||
for rx, what in PRIVKEY_PATTERNS:
|
||||
for lineno, line in enumerate(text.splitlines(), 1):
|
||||
if line.lstrip().startswith("#"):
|
||||
continue
|
||||
if rx.search(line):
|
||||
check(False, f"{rel}:{lineno}: tracked file contains {what} (R-SEC-01)")
|
||||
|
||||
# ----------------------------------------- (6) decryption is delegated to the cluster
|
||||
sops_decryption = []
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") != "Kustomization":
|
||||
continue
|
||||
dec = d.get("spec", {}).get("decryption") if isinstance(d.get("spec"), dict) else None
|
||||
if isinstance(dec, dict) and dec.get("provider") == "sops" and dec.get("secretRef"):
|
||||
sops_decryption.append(rel)
|
||||
# Flux overlay patches carry the same block inside a patch body.
|
||||
for rel in YAML_FILES:
|
||||
text = (ROOT / rel).read_text(errors="replace")
|
||||
if re.search(r"^\s*decryption:\s*$", text, re.M) and re.search(r"^\s*provider:\s*sops\s*$", text, re.M) \
|
||||
and re.search(r"^\s*secretRef:\s*$", text, re.M):
|
||||
sops_decryption.append(rel)
|
||||
check(len(sops_decryption) > 0,
|
||||
"no Flux Kustomization declares `decryption: {provider: sops, secretRef: ...}` — nothing "
|
||||
"proves decryption happens in-cluster rather than from committed key material (R-SEC-01)")
|
||||
|
||||
# ------------------------------------------- (7) the L0 policy gate is real and strict
|
||||
policy_files = [f for f in FILES if f.endswith(f"{POLICY_NAME}.yaml")]
|
||||
check(len(policy_files) == 1, f"expected exactly one {POLICY_NAME} ClusterPolicy, found {policy_files}")
|
||||
for rel in policy_files:
|
||||
d = (docs_of(rel) or [{}])[0]
|
||||
action = str(d.get("spec", {}).get("validationFailureAction", ""))
|
||||
check(action == "Enforce" or action.startswith("${"),
|
||||
f"{rel}: validationFailureAction is {action!r} — the no-plaintext-secrets gate must "
|
||||
"Enforce (or carry the Flux token that substitutes to Enforce)")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (no-plaintext-secrets): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: no-plaintext-secrets — {CHECKS} assertions over {len(FILES)} tracked files, "
|
||||
f"{secrets_seen} committed Secret document(s), {len(vault_files)} ansible vault file(s)")
|
||||
PY
|
||||
@@ -1,46 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the specification is immutable to agents (R-TEST-06)
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# A governance rule is made testable by testing its ENFORCEMENT MECHANISM, not by
|
||||
# observing that no agent misbehaved today.
|
||||
#
|
||||
# GIVEN the enforcement mechanism (ci/spec-immutability.sh + ci/SPEC_PIN),
|
||||
# WHEN it is driven against hermetic THROWAWAY spec repositories in each state,
|
||||
# THEN a clean spec at the pinned SHA passes; an uncommitted edit under docs/ or
|
||||
# decisions/, an untracked spec addition, and an unadopted committed drift each
|
||||
# exit 1 and block the merge; an absent spec repo exits 3 (honest skip, never
|
||||
# green); an unpinned SPEC_PIN fails closed; and the merge gate runs the stage
|
||||
# and blocks on its failure.
|
||||
#
|
||||
# THE REAL SPEC REPO IS NEVER TOUCHED: every case runs against a git repo created under
|
||||
# mktemp with SPEC_REPO pointed at it and a copy of ci/ carrying a fixture SPEC_PIN.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script; conformance/run.sh
|
||||
# --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME script.
|
||||
# NOT proven (named in check.sh's header): that the Gitea server REQUIRES this status
|
||||
# check before merge — branch protection is operator configuration outside the repo.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: spec-immutability-enforced
|
||||
labels:
|
||||
conformance.home.k3s/R-TEST-06: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-TEST-06"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
A spec modification is detected and blocks the merge; an unverifiable spec skips
|
||||
honestly rather than passing; and the gate wires that stage as blocking.
|
||||
spec:
|
||||
steps:
|
||||
- name: drive-the-spec-immutability-gate
|
||||
description: >-
|
||||
Run the real ci/spec-immutability.sh against throwaway spec repos in every
|
||||
state and assert its exit code, then assert the gate wires it as blocking.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,165 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — spec-immutability-enforced
|
||||
# Serves: R-TEST-06 ("AI agents MUST NOT modify the specifications (this docs/ tree,
|
||||
# the requirements, or accepted ADRs). Specs are changed only by the human
|
||||
# operator/architect." — spec docs/00-vision-and-scope.md.)
|
||||
#
|
||||
# HOW A GOVERNANCE RULE IS MADE TESTABLE
|
||||
# -------------------------------------
|
||||
# "An agent must not do X" cannot be proven by observing that no agent did X today.
|
||||
# What CAN be proven — and is what actually protects the spec — is that the MECHANISM
|
||||
# which detects and blocks a spec modification works, and is wired into the merge gate
|
||||
# as a blocking stage. So this test drives the real `ci/spec-immutability.sh` against
|
||||
# HERMETIC THROWAWAY spec repositories and asserts its exit-code behaviour.
|
||||
#
|
||||
# THE REAL SPEC REPO IS NEVER TOUCHED. Every case runs against a fixture git repo under
|
||||
# mktemp, with SPEC_REPO pointed at it and a COPY of ci/ (so the fixture's SPEC_PIN can
|
||||
# be set without editing the tracked one). Nothing here writes outside its temp dir —
|
||||
# which is itself the R-TEST-06 discipline this test exists to defend.
|
||||
#
|
||||
# GIVEN the enforcement mechanism (ci/spec-immutability.sh + ci/SPEC_PIN),
|
||||
# WHEN it is run against a spec repo in each state,
|
||||
# THEN (1) clean and at the pinned SHA -> exit 0 (pass)
|
||||
# (2) an UNCOMMITTED edit under docs/ -> exit 1 (blocks the merge)
|
||||
# (3) an uncommitted edit under decisions/ -> exit 1 (accepted ADRs are spec too)
|
||||
# (4) a NEW COMMIT (HEAD drifted from pin) -> exit 1 (an "adopted" change must be
|
||||
# an explicit reviewed SPEC_PIN bump)
|
||||
# (5) an untracked file added to the spec -> exit 1 (porcelain catches additions)
|
||||
# (6) the spec repo absent -> exit 3 (honest SKIP, never green)
|
||||
# (7) an empty/comment-only SPEC_PIN -> exit 1 (an unpinned gate is no gate)
|
||||
# AND (8) the tracked ci/SPEC_PIN really is a 40-hex commit SHA, and
|
||||
# (9) the gate wires spec-immutability as a stage whose FAIL sets the merge block —
|
||||
# i.e. the mechanism above is load-bearing, not dead code.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gap):
|
||||
# - That the Gitea server actually REQUIRES this status check before merge. Branch
|
||||
# protection is operator configuration outside the repo; nothing in-repo can assert it.
|
||||
# - Intent: an agent could still open a spec PR. The rule is enforced by the gate
|
||||
# blocking it, which is exactly what (1)-(9) assert; it is not enforced by prediction.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
CI_SRC="${ROOT}/ci"
|
||||
|
||||
PASS=0
|
||||
FAILED=0
|
||||
note() { printf ' ok %s\n' "$*"; PASS=$((PASS + 1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*" >&2; FAILED=$((FAILED + 1)); }
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "${WORK}"' EXIT
|
||||
|
||||
# --- a hermetic stand-in for the spec repo -----------------------------------------
|
||||
# Mirrors the real spec's shape: a docs/ tree and a decisions/ tree, one commit.
|
||||
make_spec_repo() {
|
||||
local d="$1"
|
||||
mkdir -p "${d}/docs" "${d}/decisions"
|
||||
# Fixture prose deliberately carries NO R-* token: ci/touched-requirements.sh scans the
|
||||
# non-comment lines of changed files, so an example ID here would be attributed to the
|
||||
# change as a touched requirement that does not exist in the registry.
|
||||
printf 'normative spec text (fixture)\n' > "${d}/docs/00-vision-and-scope.md"
|
||||
printf 'accepted decision\n' > "${d}/decisions/adr-0001-example.md"
|
||||
git -C "$d" init -q -b main
|
||||
git -C "$d" config user.email [email protected]
|
||||
git -C "$d" config user.name spec
|
||||
git -C "$d" config commit.gpgsign false # throwaway: never sign, ignore host gpg config
|
||||
git -C "$d" add -A
|
||||
git -C "$d" commit -qm "spec baseline"
|
||||
}
|
||||
|
||||
# A COPY of ci/ whose SPEC_PIN we may set freely. The script resolves CI_DIR from its own
|
||||
# location and sources lib.sh + versions.env from there, so a copy is self-contained.
|
||||
make_ci_copy() {
|
||||
local dst="$1" pin="$2"
|
||||
mkdir -p "$dst"
|
||||
cp "${CI_SRC}/spec-immutability.sh" "${CI_SRC}/lib.sh" "${CI_SRC}/versions.env" "$dst/"
|
||||
printf '%s\n' "$pin" > "${dst}/SPEC_PIN"
|
||||
chmod +x "${dst}/spec-immutability.sh"
|
||||
}
|
||||
|
||||
# run_case <expected-rc> <description> <spec-repo|-> <pin>
|
||||
run_case() {
|
||||
local want="$1" desc="$2" spec="$3" pin="$4" rc=0
|
||||
local cidir; cidir="$(mktemp -d -p "$WORK" ci.XXXXXX)"
|
||||
make_ci_copy "$cidir" "$pin"
|
||||
SPEC_REPO="$spec" REPO_ROOT="$WORK" "${cidir}/spec-immutability.sh" >/dev/null 2>&1 || rc=$?
|
||||
if [ "$rc" = "$want" ]; then note "${desc} -> exit ${rc}"; else
|
||||
bad "${desc}: expected exit ${want}, got ${rc}"; fi
|
||||
}
|
||||
|
||||
for f in spec-immutability.sh lib.sh versions.env SPEC_PIN; do
|
||||
if [ -f "${CI_SRC}/${f}" ]; then note "ci/${f} present"; else bad "ci/${f} is missing"; fi
|
||||
done
|
||||
|
||||
SPEC="${WORK}/spec"
|
||||
make_spec_repo "$SPEC"
|
||||
PINNED="$(git -C "$SPEC" rev-parse HEAD)"
|
||||
|
||||
# (1) clean + at the pin -> pass.
|
||||
run_case 0 "clean spec at the pinned SHA passes" "$SPEC" "$PINNED"
|
||||
|
||||
# (2) an uncommitted edit under docs/ -> block.
|
||||
printf 'normative spec text REWRITTEN BY AN AGENT\n' > "${SPEC}/docs/00-vision-and-scope.md"
|
||||
run_case 1 "uncommitted edit under docs/ BLOCKS" "$SPEC" "$PINNED"
|
||||
git -C "$SPEC" checkout -q -- docs/00-vision-and-scope.md
|
||||
|
||||
# (3) an uncommitted edit to an accepted ADR -> block (decisions/ is spec too).
|
||||
printf 'reversed decision\n' > "${SPEC}/decisions/adr-0001-example.md"
|
||||
run_case 1 "uncommitted edit under decisions/ BLOCKS" "$SPEC" "$PINNED"
|
||||
git -C "$SPEC" checkout -q -- decisions/adr-0001-example.md
|
||||
|
||||
# (5) an untracked addition to the spec tree -> block.
|
||||
printf 'a spec doc an agent invented\n' > "${SPEC}/docs/99-invented.md"
|
||||
run_case 1 "untracked file added to the spec BLOCKS" "$SPEC" "$PINNED"
|
||||
rm -f "${SPEC}/docs/99-invented.md"
|
||||
|
||||
# clean again -> still passes (proves cases 2/3/5 failed for the edit, not permanently).
|
||||
run_case 0 "spec restored to clean passes again" "$SPEC" "$PINNED"
|
||||
|
||||
# (4) a committed spec change that the pin has not adopted -> block.
|
||||
printf 'an extra normative line added by an agent\n' >> "${SPEC}/docs/00-vision-and-scope.md"
|
||||
git -C "$SPEC" add -A
|
||||
git -C "$SPEC" commit -qm "agent edits the spec"
|
||||
DRIFTED="$(git -C "$SPEC" rev-parse HEAD)"
|
||||
run_case 1 "committed spec drift from the pinned SHA BLOCKS" "$SPEC" "$PINNED"
|
||||
# ...and adopting it EXPLICITLY (a reviewed SPEC_PIN bump) is the only way to green.
|
||||
run_case 0 "an explicit SPEC_PIN bump adopts the advance" "$SPEC" "$DRIFTED"
|
||||
|
||||
# (6) spec repo unavailable -> honest SKIP (exit 3), never a pass.
|
||||
run_case 3 "absent spec repo SKIPS (exit 3), never green" "${WORK}/no-such-spec" "$PINNED"
|
||||
|
||||
# (7) an unpinned gate is not a gate.
|
||||
run_case 1 "empty SPEC_PIN FAILS closed" "$SPEC" "# only a comment, no SHA"
|
||||
|
||||
# (8) the tracked pin is a real commit SHA.
|
||||
REAL_PIN="$(grep -vE '^\s*#' "${CI_SRC}/SPEC_PIN" | tr -d '[:space:]')"
|
||||
if printf '%s' "$REAL_PIN" | grep -Eq '^[0-9a-f]{40}$'; then
|
||||
note "ci/SPEC_PIN is a 40-hex commit SHA"
|
||||
else
|
||||
bad "ci/SPEC_PIN is not a 40-hex commit SHA: '${REAL_PIN}'"
|
||||
fi
|
||||
|
||||
# (9) the mechanism is load-bearing: the merge gate runs it as a stage AND a FAIL there
|
||||
# sets the block. Asserted against the gate source because running the whole gate
|
||||
# from inside a conformance check would recurse into this very check.
|
||||
GATE="${CI_SRC}/gate.sh"
|
||||
if grep -Eq '^[[:space:]]*run_stage[[:space:]]+spec-immutability' "$GATE"; then
|
||||
note "ci/gate.sh runs spec-immutability as a gate stage"
|
||||
else
|
||||
bad "ci/gate.sh does not run spec-immutability as a stage — the gate is not wired"
|
||||
fi
|
||||
if awk '/OUTCOME\[spec-immutability\]/,/^esac/' "$GATE" | grep -Eq '^[[:space:]]*fail\).*BLOCK=1'; then
|
||||
note "ci/gate.sh blocks the merge when spec-immutability FAILS"
|
||||
else
|
||||
bad "ci/gate.sh does not set the merge block on a spec-immutability failure"
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$FAILED" -eq 0 ]; then
|
||||
echo "check OK: spec-immutability-enforced — ${PASS} assertions (R-TEST-06)"
|
||||
exit 0
|
||||
fi
|
||||
echo "CHECK FAIL (spec-immutability-enforced): ${FAILED} failed, ${PASS} passed" >&2
|
||||
exit 1
|
||||
@@ -61,70 +61,6 @@ spec:
|
||||
image: nginx:1.27
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --upstream=http://127.0.0.1:8080
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: paas-demo-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: paas-demo-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: paas-demo-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
@@ -141,9 +77,6 @@ spec:
|
||||
- name: port-8080
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): paas-demo-allow-gateway-ingress
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -162,7 +95,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '8080'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): paas-demo-allow-internet-egress
|
||||
@@ -180,10 +113,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute paas-demo-8080 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -210,7 +139,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: paas-demo
|
||||
port: 4180
|
||||
port: 8080
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
@@ -233,7 +162,7 @@ spec:
|
||||
interval: 30s
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
apiVersion: image.toolkit.fluxcd.io/v1beta2
|
||||
kind: ImageRepository
|
||||
metadata:
|
||||
name: paas-demo
|
||||
@@ -243,7 +172,7 @@ spec:
|
||||
interval: 1h0m0s
|
||||
---
|
||||
# Flux ImagePolicy (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
apiVersion: image.toolkit.fluxcd.io/v1beta2
|
||||
kind: ImagePolicy
|
||||
metadata:
|
||||
name: paas-demo
|
||||
|
||||
@@ -93,7 +93,7 @@ spec:
|
||||
interval: 30s
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
apiVersion: image.toolkit.fluxcd.io/v1beta2
|
||||
kind: ImageRepository
|
||||
metadata:
|
||||
name: baseline-demo
|
||||
@@ -103,7 +103,7 @@ spec:
|
||||
interval: 1h0m0s
|
||||
---
|
||||
# Flux ImagePolicy (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
apiVersion: image.toolkit.fluxcd.io/v1beta2
|
||||
kind: ImagePolicy
|
||||
metadata:
|
||||
name: baseline-demo
|
||||
|
||||
+4
-75
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --upstream=http://127.0.0.1:8989
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-8989
|
||||
port: 8989
|
||||
targetPort: 8989
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): sonarr-allow-gateway-ingress
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -189,7 +122,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '8989'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
|
||||
@@ -207,10 +140,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute sonarr-8989 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -237,7 +166,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: sonarr
|
||||
port: 4180
|
||||
port: 8989
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
@@ -283,7 +212,7 @@ spec:
|
||||
description: 'sonarr has been unreachable for 5m (SLI: up).'
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
apiVersion: image.toolkit.fluxcd.io/v1beta2
|
||||
kind: ImageRepository
|
||||
metadata:
|
||||
name: sonarr
|
||||
@@ -293,7 +222,7 @@ spec:
|
||||
interval: 1h0m0s
|
||||
---
|
||||
# Flux ImagePolicy (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
apiVersion: image.toolkit.fluxcd.io/v1beta2
|
||||
kind: ImagePolicy
|
||||
metadata:
|
||||
name: sonarr
|
||||
|
||||
@@ -84,70 +84,6 @@ spec:
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
|
||||
- --upstream=http://127.0.0.1:8989
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
|
||||
- --cookie-domain=.home.keithjackson.dev
|
||||
- --whitelist-domain=.home.keithjackson.dev
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -168,9 +104,6 @@ spec:
|
||||
- name: port-8989
|
||||
port: 8989
|
||||
targetPort: 8989
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): sonarr-allow-gateway-ingress
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -189,7 +122,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '8989'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
|
||||
@@ -207,10 +140,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute sonarr-8989 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -237,7 +166,7 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: sonarr
|
||||
port: 4180
|
||||
port: 8989
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
@@ -283,7 +212,7 @@ spec:
|
||||
description: 'sonarr has been unreachable for 5m (SLI: up).'
|
||||
---
|
||||
# Flux ImageRepository (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
apiVersion: image.toolkit.fluxcd.io/v1beta2
|
||||
kind: ImageRepository
|
||||
metadata:
|
||||
name: sonarr
|
||||
@@ -293,7 +222,7 @@ spec:
|
||||
interval: 1h0m0s
|
||||
---
|
||||
# Flux ImagePolicy (R-UPD)
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
apiVersion: image.toolkit.fluxcd.io/v1beta2
|
||||
kind: ImagePolicy
|
||||
metadata:
|
||||
name: sonarr
|
||||
|
||||
@@ -13,16 +13,11 @@
|
||||
# podSelector:{} NetworkPolicy covers it). R-NET-01 for IaaS.
|
||||
#
|
||||
# METHOD (virtctl-free): the VM's ROOT is a CDI DataVolume on Longhorn (a mounted
|
||||
# PVC — NOT an ephemeral containerDisk), the spec-required, rebuild-surviving disk model
|
||||
# for VMs (docs/06, R-STATE-01), matching fixtures/vm.yaml. (Feature-gate note,
|
||||
# revalidated 2026-08: an earlier revision said the containerDisk boot path was broken on
|
||||
# this cluster — KubeVirt v1.9 serves the container-disk binary via a k8s `image` volume
|
||||
# gated by the ALPHA `ImageVolume` feature gate, which k3s v1.31 had DISABLED, so the
|
||||
# apiserver pruned it and the guest died at OCI init. That premise is now STALE: the
|
||||
# cluster is k3s v1.36.4, where ImageVolume is ENABLED by default
|
||||
# (kubernetes_feature_enabled{name="ImageVolume"}=1; a server-side dry-run RETAINS an
|
||||
# `image` volume rather than pruning it), so containerDisk roots would boot — DataVolume
|
||||
# is kept because it is the rebuild-surviving model.) cloud-init runs the two probes and writes a
|
||||
# PVC — NOT an ephemeral containerDisk, whose boot path is broken on this k3s cluster:
|
||||
# KubeVirt v1.9 serves the container-disk binary via a k8s `image` volume gated by the
|
||||
# ALPHA `ImageVolume` feature gate, disabled on k3s v1.31, so the apiserver prunes it
|
||||
# and the guest dies at OCI init. DataVolume roots bypass that path and are the
|
||||
# spec-required disk model — docs/06). cloud-init runs the two probes and writes a
|
||||
# PASS/FAIL verdict to a raw Longhorn block disk, then powers off (RerunOnFailure =>
|
||||
# stays down). A busybox reader pod reads the verdict back. The container default-deny
|
||||
# test proves the datapath fail-closed (default-deny-failclosed, L3); this asserts
|
||||
|
||||
@@ -21,25 +21,19 @@
|
||||
# (hermetic).
|
||||
#
|
||||
# DISK MODEL — CDI DataVolume on Longhorn (NOT an ephemeral containerDisk):
|
||||
# The SPEC requires VMs to use Longhorn-backed disks that survive rebuild (docs/06,
|
||||
# R-STATE-01), so this test — like fixtures/vm.yaml and the L3 vm-data-survival partner
|
||||
# — boots from a CDI DataVolume on a Longhorn PVC (CDI imports the Fedora image into it
|
||||
# out-of-band via its importer pod, so the VM never touches the container-disk binary /
|
||||
# image-volume path). Cost: a one-time CDI import per run (covered by the raised exec
|
||||
# timeout below).
|
||||
# FEATURE-GATE NOTE (revalidated 2026-08 — the original "containerDisk is broken here"
|
||||
# rationale is now STALE): KubeVirt v1.9 supplies the container-disk binary to the
|
||||
# `volumerootdisk` init container via a Kubernetes-native `image` volume
|
||||
# (VolumeSource.Image) mounted at /container-disk-binary, gated by the `ImageVolume`
|
||||
# feature gate. On the ORIGINAL k3s v1.31 cluster that ALPHA gate was DISABLED, so the
|
||||
# apiserver silently PRUNED the image volume + its mount and a containerDisk VM died at
|
||||
# OCI init ("/container-disk-binary/usr/bin/container-disk: no such file or directory").
|
||||
# The cluster is now k3s v1.36.4, where ImageVolume is ENABLED by default
|
||||
# (kubernetes_feature_enabled{name="ImageVolume"}=1; a server-side dry-run of a Pod with
|
||||
# an `image` volume RETAINS the volume + mount instead of pruning them). containerDisk
|
||||
# roots would therefore boot on this cluster now — but DataVolume is retained regardless,
|
||||
# because only it (not an ephemeral containerDisk) is the rebuild-surviving, spec-required
|
||||
# disk model.
|
||||
# KubeVirt v1.9 supplies the container-disk binary to the `volumerootdisk` init
|
||||
# container via a Kubernetes-native `image` volume (VolumeSource.Image) mounted at
|
||||
# /container-disk-binary. That k8s feature is gated by the ALPHA `ImageVolume`
|
||||
# feature gate, which is DISABLED on this k3s v1.31 cluster — so the apiserver
|
||||
# silently PRUNES the image volume + its mount, the binary is never present, and a
|
||||
# containerDisk VM dies at OCI init with:
|
||||
# exec: "/container-disk-binary/usr/bin/container-disk": no such file or directory
|
||||
# A DataVolume-backed root is a mounted Longhorn PVC (CDI imports the same Fedora
|
||||
# image into it out-of-band via its importer pod) and therefore does NOT touch the
|
||||
# container-disk binary / image-volume path at all — it boots on this cluster. It is
|
||||
# also the disk model the SPEC requires for VMs (docs/06, R-STATE-01: Longhorn-backed
|
||||
# disks that survive rebuild), so this mirrors the representative fixtures/vm.yaml.
|
||||
# Cost: a one-time CDI import per run (covered by the raised exec timeout below).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
|
||||
@@ -1,75 +1,17 @@
|
||||
# =============================================================================
|
||||
# L3 FAULT-INJECTION — DR restore from an OFF-CLUSTER backup (R-STATE-03)
|
||||
# *SAFETY-CRITICAL* [RUNNABLE / self-skips]
|
||||
# Catalog: docs/14-testing-and-conformance.md "DR restore (R-STATE-03, L3)"
|
||||
# DoD: agents/definition-of-done.md — "DR-critical data restores from an
|
||||
# **off-cluster** backup (drill for at least one service/phase)."
|
||||
# Spec: docs/04-storage-and-state.md; ADR-0003 (Longhorn + Velero).
|
||||
# Impl: platform/velero/** (Velero + kopia File System Backup -> the versitygw S3
|
||||
# endpoint on the Proxmox HOST), reconciled by clusters/base/backup.yaml.
|
||||
# L3 FAULT-INJECTION SCAFFOLD — DR restore (R-STATE-03) *SAFETY-CRITICAL*
|
||||
# Catalog: docs/14 "DR restore (R-STATE-03, L3)"
|
||||
#
|
||||
# GIVEN a DR-critical stateful workload with known data on the replicated `longhorn`
|
||||
# tier, and a Velero BackupStorageLocation whose object-store endpoint is
|
||||
# PROVABLY OUTSIDE this cluster,
|
||||
# WHEN a backup of that namespace + its volume data is written to that off-cluster
|
||||
# store, and then EVERY in-cluster copy of the data is DESTROYED (namespace,
|
||||
# PVC and the underlying PersistentVolume/Longhorn replicas all deleted),
|
||||
# THEN a Velero restore into a CLEAN, previously-nonexistent namespace brings the
|
||||
# workload back and the known data reads back byte-identical.
|
||||
# GIVEN an OFF-CLUSTER backup of a namespace + volumes,
|
||||
# WHEN the namespace + volumes are restored into a CLEAN state,
|
||||
# THEN the service returns with its data.
|
||||
#
|
||||
# R-TEST-05 (fault injection, not happy-path): the restore is only meaningful because
|
||||
# the source is annihilated first. After the destroy step there is no PVC, no PV and
|
||||
# no Longhorn volume left in the cluster holding this data — so the bytes that come
|
||||
# back can ONLY have come from the off-cluster object store. A restore that silently
|
||||
# rebound a surviving in-cluster PV is explicitly excluded: the test captures the
|
||||
# original PV name, asserts it is GONE, and asserts the restored PVC is bound to a
|
||||
# DIFFERENT, newly provisioned PV.
|
||||
# R-TEST-05: proves recoverability by restoring into a genuinely clean namespace
|
||||
# (not by reading the still-live original). Backup TARGET is OQ-BKP (OPEN) — the
|
||||
# restore mechanism is Velero (T-4.3/T-4.4); this scaffold is target-agnostic.
|
||||
#
|
||||
# HOW "OFF-CLUSTER" IS PROVEN (not assumed) — step `assert-backup-target-is-off-cluster`:
|
||||
# 1. The BackupStorageLocation reports phase `Available` — Velero itself has
|
||||
# successfully reached and validated that object store.
|
||||
# 2. Its endpoint host is resolved to an IPv4 address and asserted to be NONE of:
|
||||
# loopback; any Node InternalIP/ExternalIP; any Service ClusterIP; any Pod IP;
|
||||
# inside any node's `spec.podCIDR(s)`; inside any `ServiceCIDR` (best-effort).
|
||||
# It is also asserted not to be a cluster-internal DNS name (*.svc,
|
||||
# *.cluster.local, localhost). => the bytes leave the cluster's address space.
|
||||
# 3. The backup's PodVolumeBackup carries a non-empty `status.snapshotID` — the
|
||||
# kopia snapshot really landed in the remote repository, i.e. VOLUME DATA (not
|
||||
# merely Kubernetes resource JSON) was uploaded off-cluster.
|
||||
# 4. Plus the destroy-then-restore sequence above, which removes every in-cluster
|
||||
# alternative source for the data.
|
||||
#
|
||||
# KNOWN, DELIBERATE GAPS (asserted nowhere — do not read this test as claiming them):
|
||||
# * OFF-HOST is NOT proven, only OFF-CLUSTER. On the dev estate the k3s nodes are
|
||||
# VMs on the same Proxmox host that runs the versitygw S3 target (192.168.1.9),
|
||||
# so this drill does not survive loss of that hypervisor. Off-site/off-host
|
||||
# replication is an operator concern outside R-STATE-03's wording.
|
||||
# * The test does NOT read the backup objects out of the bucket with an S3 client.
|
||||
# Doing so would require handing the conformance runner the backup credential,
|
||||
# widening a privileged test identity for weaker evidence than (1)-(4) above.
|
||||
# * The test restores its OWN scratch workload, never a real service's data. A
|
||||
# per-service drill (Nextcloud/Gitea/CNPG) stays an operator exercise.
|
||||
# * Bare-metal DR (rebuild the cluster itself, then restore) is NOT exercised; this
|
||||
# asserts namespace+volume restore into a live cluster, which is what the catalog
|
||||
# entry and the DoD line ask for.
|
||||
# * Backup ENCRYPTION at rest and retention/GC behaviour are not asserted.
|
||||
#
|
||||
# RUNNABLE / SELF-SKIPPING (a skip is honest; a fake pass is a defect). Skips CLEANLY
|
||||
# (green) when a precondition is absent: no `longhorn` StorageClass; no Velero API
|
||||
# (CRDs absent); no BackupStorageLocation configured at all; Velero server or its
|
||||
# node-agent DaemonSet not ready. It runs the full drill the moment those exist — no
|
||||
# edit needed. RBAC denial is NOT a skip: it fails loudly as a harness defect.
|
||||
#
|
||||
# DESTRUCTIVE BY DESIGN, SCOPED BY CONSTRUCTION: every object it destroys is one it
|
||||
# created itself, in two namespaces it names after its own Chainsaw namespace
|
||||
# (`<ns>-dr-src`, `<ns>-dr-dst`) and labels `conformance.home.k3s/created-by=<ns>`.
|
||||
# It never touches a real service's namespace, PVC or PV, and it never deletes a
|
||||
# Backup it did not create.
|
||||
#
|
||||
# TEARDOWN ALWAYS RUNS (`finally`, even on failure/timeout/panic): both scratch
|
||||
# namespaces, any leftover PV, the Restore CR, and the Backup — the latter via a
|
||||
# Velero DeleteBackupRequest so the objects are removed from the bucket too, with a
|
||||
# short `ttl` on the Backup as a belt-and-braces backstop if that request is lost.
|
||||
# STATUS: scaffold — needs T-4.3 (Velero + off-cluster schedule) + T-4.4 (drill).
|
||||
# BLOCKED-BY spec open question OQ-BKP (off-cluster backup target) — see report.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
@@ -81,603 +23,31 @@ metadata:
|
||||
conformance.home.k3s/requirements: "R-STATE-03"
|
||||
conformance.home.k3s/layer: "L3"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
# executable => runs against a cluster; self-skips (green) until Velero + a
|
||||
# BackupStorageLocation + the longhorn tier exist.
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Seed known data on longhorn, back the namespace + volume data up to the Velero
|
||||
object store whose endpoint is proven to be outside this cluster, destroy every
|
||||
in-cluster copy (namespace, PVC and PV), restore into a clean namespace, and read
|
||||
the known data back from a newly provisioned volume. Skips cleanly until Velero,
|
||||
a BackupStorageLocation and a longhorn StorageClass exist.
|
||||
conformance.home.k3s/never-executed: >-
|
||||
AUTHORED, NOT YET EXECUTED. No workstation in this repo has cluster access, and
|
||||
the drill is destructive, so it must first be run by the operator against the dev
|
||||
cluster in a maintenance window before R-STATE-03 may be called proven. Repo-derived
|
||||
status deliberately reports proof as not-derivable (conformance/lib/status.py).
|
||||
Restore a namespace + volumes from an off-cluster backup into a clean state;
|
||||
the service returns with its data.
|
||||
conformance.home.k3s/blocked-by: "OQ-BKP (off-cluster backup target, spec-open)"
|
||||
spec:
|
||||
# A full backup -> destroy -> restore cycle over kopia File System Backup, plus
|
||||
# Longhorn provisioning on HDD, is far slower than any other test here. Bound it
|
||||
# generously but explicitly (no sleep-and-hope; every wait below is a bounded poll).
|
||||
# Worst-case bound of the drill step, summing every poll below (seed ~5m + stale-target
|
||||
# clean ~5m + backup 15m + namespace destroy 12m + PV destroy 5m + restore 15m + restored
|
||||
# workload Ready 10m ~= 67m). Declared at 75m so the step's own bounded polls, not this
|
||||
# timeout, are what fail first and report WHICH phase stalled. Nothing here sleeps blindly.
|
||||
timeouts:
|
||||
exec: 75m
|
||||
delete: 10m
|
||||
cleanup: 15m
|
||||
steps:
|
||||
# -----------------------------------------------------------------------------
|
||||
# Preconditions + the OFF-CLUSTER property of the backup target. This step makes
|
||||
# no changes to the cluster; it either skips cleanly or proves the target is
|
||||
# genuinely outside the cluster's address space.
|
||||
# -----------------------------------------------------------------------------
|
||||
- name: assert-backup-target-is-off-cluster
|
||||
- name: given-off-cluster-backup-exists
|
||||
try:
|
||||
- script:
|
||||
content: 'echo "TODO(T-4.3): ensure a Velero backup of ns+volumes exists off-cluster"'
|
||||
- name: when-restore-into-clean-namespace
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -eu
|
||||
|
||||
# ---- helpers -------------------------------------------------------
|
||||
# Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
|
||||
# fail loudly; an absent API is a genuine precondition gap and may skip.
|
||||
die_or_skip() {
|
||||
case "$2" in
|
||||
*orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
|
||||
echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
|
||||
echo "$2" >&2
|
||||
exit 1 ;;
|
||||
*)
|
||||
echo "SKIP: $1 not available on this cluster -- $2"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
ip2int() {
|
||||
_q="$(echo "$1" | tr '.' ' ')"
|
||||
set -- $_q
|
||||
[ $# -eq 4 ] || { echo -1; return 0; }
|
||||
echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
|
||||
}
|
||||
|
||||
# in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
|
||||
in_cidr() {
|
||||
case "$2" in *"/"*) ;; *) return 1 ;; esac
|
||||
_net="${2%/*}"; _len="${2#*/}"
|
||||
case "$_len" in ''|*[!0-9]*) return 1 ;; esac
|
||||
[ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
|
||||
_i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
|
||||
[ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
|
||||
[ "$_len" -eq 0 ] && return 0
|
||||
_d=1; _s=0
|
||||
while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
|
||||
[ $(( _i / _d )) -eq $(( _n / _d )) ]
|
||||
}
|
||||
|
||||
# ---- Gate 1: the replicated storage tier the DR-critical data lives on --
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
|
||||
2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
|
||||
if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
|
||||
echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "backup storage locations:"
|
||||
echo "$BSL_LINES"
|
||||
|
||||
VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
|
||||
|
||||
# ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
|
||||
# File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
|
||||
# deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
|
||||
# ready node-agent no volume data can be captured, so the drill is not runnable.
|
||||
VREADY=0; NAREADY=0
|
||||
for i in $(seq 1 12); do
|
||||
VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
|
||||
NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
|
||||
echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
|
||||
if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
|
||||
sleep 10
|
||||
done
|
||||
if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
|
||||
echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
|
||||
echo DR_RESTORE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ---- Pick the BSL this drill will use: prefer the default, and it MUST be
|
||||
# Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
|
||||
# gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
|
||||
# wait for first-validation on a cold start), it does not skip. --------
|
||||
BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
|
||||
for i in $(seq 1 12); do
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
|
||||
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
|
||||
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
|
||||
if [ -n "$CAND" ]; then
|
||||
BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
|
||||
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
|
||||
BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
|
||||
BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
|
||||
break
|
||||
fi
|
||||
echo "poll $i: no Available BackupStorageLocation yet"
|
||||
sleep 10
|
||||
done
|
||||
if [ -z "$BSL_NAME" ]; then
|
||||
echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
|
||||
echo " The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
|
||||
echo "$BSL_LINES" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "using BSL ${BSL_NS}/${BSL_NAME} bucket=${BSL_BUCKET} endpoint=${BSL_URL}"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
|
||||
if [ -z "$BSL_URL" ]; then
|
||||
echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
|
||||
echo " the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
|
||||
echo " test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
|
||||
exit 1
|
||||
fi
|
||||
HOST="${BSL_URL#*://}" # strip scheme
|
||||
HOST="${HOST%%/*}" # strip path
|
||||
HOST="${HOST%%:*}" # strip port
|
||||
echo "backup endpoint host: $HOST"
|
||||
|
||||
case "$HOST" in
|
||||
localhost|*.svc|*.svc.cluster.local|*.cluster.local)
|
||||
echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
|
||||
echo " An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
|
||||
exit 1 ;;
|
||||
esac
|
||||
|
||||
# Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
|
||||
# off-cluster property -- that is a failure, never a silent pass.
|
||||
case "$HOST" in
|
||||
*[!0-9.]*)
|
||||
# Endpoint given as a name. Try the resolvers that may exist in the
|
||||
# runner image, in order; if none is available or none resolves, the
|
||||
# test FAILS below rather than guessing.
|
||||
EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
|
||||
[ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
|
||||
[ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
|
||||
*) EPIP="$HOST" ;;
|
||||
esac
|
||||
if [ -z "$EPIP" ]; then
|
||||
echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
|
||||
echo " the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "backup endpoint resolves to: $EPIP"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
|
||||
case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
|
||||
|
||||
NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
for a in $NODE_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
|
||||
|
||||
SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
|
||||
for a in $SVC_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
|
||||
|
||||
POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
|
||||
for a in $POD_IPS; do
|
||||
if [ "$a" = "$EPIP" ]; then
|
||||
echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
|
||||
|
||||
POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
for c in $POD_CIDRS; do
|
||||
case "$c" in *:*) continue ;; esac # IPv6 prefixes are out of scope here
|
||||
if in_cidr "$EPIP" "$c"; then
|
||||
echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
|
||||
|
||||
# Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
|
||||
# runner may not read): the checks above are the authoritative ones.
|
||||
SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
|
||||
if [ -n "$SVC_CIDRS" ]; then
|
||||
for c in $SVC_CIDRS; do
|
||||
case "$c" in *:*) continue ;; esac
|
||||
if in_cidr "$EPIP" "$c"; then
|
||||
echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
|
||||
else
|
||||
echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
|
||||
fi
|
||||
|
||||
echo "OFF_CLUSTER_TARGET_OK bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
|
||||
check:
|
||||
# green on either a clean precondition skip or a proven off-cluster target.
|
||||
(contains($stdout, 'OFF_CLUSTER_TARGET_OK') || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# R-STATE-03's "scheduled" half: the backup is not a one-off human action.
|
||||
# Non-destructive, and skipped in lock-step with the step above.
|
||||
# -----------------------------------------------------------------------------
|
||||
- name: assert-backups-are-scheduled
|
||||
set -euo pipefail
|
||||
echo "TODO(T-4.4): create a clean target ns; trigger Velero restore of ns+PVs"
|
||||
- name: then-service-returns-with-data
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -eu
|
||||
SCHED="$(kubectl get schedules.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
|
||||
if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
|
||||
# Distinguish "Velero absent" (skip, in lock-step with step 1) from
|
||||
# "Velero present but nothing scheduled" (a real R-STATE-03 defect).
|
||||
if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
|
||||
if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
|
||||
echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
|
||||
echo DR_SCHEDULE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
|
||||
echo " R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
|
||||
echo DR_SCHEDULE_SKIPPED_OK
|
||||
exit 0
|
||||
fi
|
||||
echo "velero schedules:"
|
||||
echo "$SCHED"
|
||||
# A Schedule that exists but is paused/disabled backs nothing up.
|
||||
if echo "$SCHED" | grep -q 'phase=Enabled'; then
|
||||
echo BACKUPS_ARE_SCHEDULED_OK
|
||||
else
|
||||
echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
|
||||
exit 1
|
||||
fi
|
||||
set -euo pipefail
|
||||
echo "TODO: assert app Ready in restored ns AND canary data present"
|
||||
echo "DR_RESTORE_OK"
|
||||
check:
|
||||
(contains($stdout, 'BACKUPS_ARE_SCHEDULED_OK') || contains($stdout, 'DR_SCHEDULE_SKIPPED_OK')): true
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# The drill itself: seed -> back up off-cluster -> ANNIHILATE -> restore -> read.
|
||||
# -----------------------------------------------------------------------------
|
||||
- name: data-restores-from-the-off-cluster-backup
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
set -eu
|
||||
NS="$NAMESPACE"
|
||||
SRC="${NS}-dr-src" # scratch source namespace (destroyed by design)
|
||||
DST="${NS}-dr-dst" # clean restore target (must not pre-exist)
|
||||
BK="${NS}-dr-backup"
|
||||
RS="${NS}-dr-restore"
|
||||
OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
|
||||
|
||||
# ---- Re-derive the preconditions (steps do not share shell state). Every
|
||||
# skip below mirrors step 1 exactly, so the whole test skips or runs
|
||||
# as a unit -- it can never half-run. -----------------------------
|
||||
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
||||
echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
|
||||
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
|
||||
CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
|
||||
[ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
|
||||
if [ -z "$CAND" ]; then
|
||||
echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
|
||||
echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
VNS="$(echo "$CAND" | cut -d' ' -f1)"
|
||||
BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
|
||||
if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
|
||||
echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
|
||||
echo DR_RESTORE_SKIPPED_OK; exit 0
|
||||
fi
|
||||
echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
|
||||
|
||||
# A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
|
||||
# be free. If either is lying around from an aborted run, remove it before we
|
||||
# claim "clean" -- and fail loudly if it will not go, because a half-deleted
|
||||
# namespace would silently invalidate the destroy proof below.
|
||||
kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
|
||||
|
||||
# ============================ GIVEN ==============================
|
||||
# A DR-critical stateful workload with known data on the replicated tier.
|
||||
CANARY="dr-canary-$(date +%s%N)"
|
||||
kubectl create namespace "$SRC"
|
||||
kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
|
||||
|
||||
kubectl -n "$SRC" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: dr-data
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
EOF
|
||||
|
||||
kubectl -n "$SRC" apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: dr-workload
|
||||
labels:
|
||||
app: dr-workload
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: app
|
||||
image: busybox:1.36.1
|
||||
command: ["sh", "-c", "sleep 7200"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: dr-data
|
||||
EOF
|
||||
|
||||
kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
|
||||
printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
|
||||
echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
|
||||
|
||||
ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
|
||||
[ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
|
||||
echo "original in-cluster PersistentVolume: $ORIG_PV"
|
||||
|
||||
# ======================= WHEN (part 1): back up off-cluster =======
|
||||
# Created as a Velero Backup CR -- exactly what `velero backup create` does.
|
||||
# defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
|
||||
# the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
|
||||
# ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
|
||||
# own GC still removes this drill's objects from the bucket.
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: velero.io/v1
|
||||
kind: Backup
|
||||
metadata:
|
||||
name: ${BK}
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
includedNamespaces: [${SRC}]
|
||||
storageLocation: ${BSL_NAME}
|
||||
snapshotVolumes: false
|
||||
defaultVolumesToFsBackup: true
|
||||
ttl: 2h0m0s
|
||||
EOF
|
||||
|
||||
BPHASE=""
|
||||
for i in $(seq 1 90); do
|
||||
BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
echo "poll $i: backup phase=${BPHASE:-<pending>}"
|
||||
case "${BPHASE:-}" in
|
||||
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
|
||||
esac
|
||||
sleep 10
|
||||
done
|
||||
if [ "${BPHASE:-}" != "Completed" ]; then
|
||||
echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
|
||||
kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
echo "backup $BK Completed to ${BSL_NAME}"
|
||||
|
||||
# ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
|
||||
# kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
|
||||
PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
|
||||
-l velero.io/backup-name="$BK" \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
|
||||
echo "pod volume backups:"
|
||||
echo "$PVB"
|
||||
if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
|
||||
echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
|
||||
echo " Resource-only backups cannot restore DR-critical DATA." >&2
|
||||
exit 1
|
||||
fi
|
||||
# A line is "<name> <phase> <snapshotID>"; require Completed AND a
|
||||
# non-empty snapshot ID (an empty ID means nothing reached the repository).
|
||||
GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
|
||||
if [ "${GOOD_PVB:-0}" -lt 1 ]; then
|
||||
echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
|
||||
echo " the volume data did not reach the off-cluster repository." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
|
||||
|
||||
# ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
|
||||
# Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
|
||||
# underlying PersistentVolume (and with it the Longhorn replicas). After this
|
||||
# there is no in-cluster source the restore could quietly read from.
|
||||
kubectl delete namespace "$SRC" --wait=true --timeout=420s
|
||||
for i in $(seq 1 30); do
|
||||
if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
|
||||
echo "poll $i: waiting for namespace $SRC to disappear"
|
||||
sleep 10
|
||||
done
|
||||
if kubectl get namespace "$SRC" >/dev/null 2>&1; then
|
||||
echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "source namespace destroyed"
|
||||
|
||||
# The longhorn StorageClass reclaims Delete, so the PV should already be gone;
|
||||
# force it if a Retain policy (or a stuck finalizer) left it behind, because a
|
||||
# surviving PV would make the restore prove nothing about off-cluster recovery.
|
||||
kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
|
||||
for i in $(seq 1 30); do
|
||||
if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
|
||||
echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
|
||||
sleep 10
|
||||
done
|
||||
if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
|
||||
echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
|
||||
echo " An in-cluster copy of the data still exists, so a successful restore" >&2
|
||||
echo " would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
|
||||
|
||||
# Best-effort corroboration (the least-privilege runner may not read longhorn.io):
|
||||
# the Longhorn Volume CR and its replicas should be gone too.
|
||||
if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
|
||||
echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
|
||||
|
||||
# ============================ RESTORE ============================
|
||||
# Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
|
||||
# the catalog's "restored into a clean state".
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: velero.io/v1
|
||||
kind: Restore
|
||||
metadata:
|
||||
name: ${RS}
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
backupName: ${BK}
|
||||
includedNamespaces: [${SRC}]
|
||||
namespaceMapping:
|
||||
${SRC}: ${DST}
|
||||
existingResourcePolicy: none
|
||||
EOF
|
||||
|
||||
RPHASE=""
|
||||
for i in $(seq 1 90); do
|
||||
RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
echo "poll $i: restore phase=${RPHASE:-<pending>}"
|
||||
case "${RPHASE:-}" in
|
||||
Completed|Failed|PartiallyFailed|FailedValidation) break ;;
|
||||
esac
|
||||
sleep 10
|
||||
done
|
||||
if [ "${RPHASE:-}" != "Completed" ]; then
|
||||
echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
|
||||
kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
|
||||
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
echo "restore $RS Completed into $DST"
|
||||
|
||||
# ============================= THEN ==============================
|
||||
# The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
|
||||
# `restore-wait` init container that blocks until the node-agent has written the
|
||||
# volume contents back, so Ready here already implies the data landed.
|
||||
if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
|
||||
echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
|
||||
kubectl -n "$DST" describe pod dr-workload >&2 || true
|
||||
kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
|
||||
echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
|
||||
[ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
|
||||
if [ "$NEW_PV" = "$ORIG_PV" ]; then
|
||||
echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
|
||||
echo " That is an in-cluster survival, not an off-cluster restore." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
|
||||
|
||||
GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
|
||||
echo "recovered='$GOT' expected='$CANARY'"
|
||||
if [ "$GOT" != "$CANARY" ]; then
|
||||
echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
|
||||
check:
|
||||
# green on either a clean precondition skip or a proven off-cluster restore.
|
||||
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK') || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
|
||||
(contains($stdout, 'DR_RESTORE_OK')): true
|
||||
finally:
|
||||
# ALWAYS RUNS -- on pass, fail, timeout or panic. This test creates namespaces
|
||||
# and Velero objects OUTSIDE its Chainsaw namespace, so Chainsaw's own namespace
|
||||
# teardown cannot reclaim them; everything below is keyed to names/labels derived
|
||||
# from THIS test's namespace, so it can never touch anything it did not create.
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
NS="$NAMESPACE"
|
||||
SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
|
||||
|
||||
VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
|
||||
|
||||
# Velero objects first: a DeleteBackupRequest makes Velero remove the backup
|
||||
# DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
|
||||
if [ -n "$VNS" ]; then
|
||||
kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
|
||||
if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
|
||||
kubectl apply -f - <<EOF || true
|
||||
apiVersion: velero.io/v1
|
||||
kind: DeleteBackupRequest
|
||||
metadata:
|
||||
name: ${BK}-delete
|
||||
namespace: ${VNS}
|
||||
labels:
|
||||
conformance.home.k3s/created-by: ${NS}
|
||||
spec:
|
||||
backupName: ${BK}
|
||||
EOF
|
||||
for i in $(seq 1 18); do
|
||||
kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
|
||||
sleep 10
|
||||
done
|
||||
# Backstop if Velero could not process the request (the 2h ttl also GCs it).
|
||||
kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
|
||||
kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
|
||||
fi
|
||||
echo "velero objects for $NS cleaned up (ns $VNS)"
|
||||
fi
|
||||
|
||||
# Scratch namespaces (both, in case the run aborted at any point).
|
||||
for n in "$DST" "$SRC"; do
|
||||
kubectl delete namespace "$n" --ignore-not-found --wait=false || true
|
||||
done
|
||||
# Anything left labelled as ours (defensive; the two names above are the only
|
||||
# namespaces this test ever creates).
|
||||
kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
|
||||
|
||||
# Released PVs from either namespace's PVCs would otherwise leak storage.
|
||||
for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
|
||||
kubectl delete pv "$pv" --ignore-not-found --wait=false || true
|
||||
echo "released leftover PersistentVolume: $pv"
|
||||
done
|
||||
|
||||
echo "dr-restore teardown complete (best-effort; every object was created by this test)"
|
||||
content: 'echo "TODO: delete the restore target namespace (self-cleaning)"'
|
||||
@@ -72,7 +72,7 @@ spec:
|
||||
# resolution — proving it advanced the selection with zero hand-authored change.
|
||||
# =========================================================================================
|
||||
kubectl -n "$NS" apply -f - <<'EOF'
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
apiVersion: image.toolkit.fluxcd.io/v1beta2
|
||||
kind: ImageRepository
|
||||
metadata:
|
||||
name: podinfo
|
||||
@@ -80,7 +80,7 @@ spec:
|
||||
image: ghcr.io/stefanprodan/podinfo
|
||||
interval: 1m
|
||||
---
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
apiVersion: image.toolkit.fluxcd.io/v1beta2
|
||||
kind: ImagePolicy
|
||||
metadata:
|
||||
name: podinfo-stable
|
||||
@@ -94,7 +94,7 @@ spec:
|
||||
semver:
|
||||
range: '>=6.0.0 <7.0.0'
|
||||
---
|
||||
apiVersion: image.toolkit.fluxcd.io/v1
|
||||
apiVersion: image.toolkit.fluxcd.io/v1beta2
|
||||
kind: ImagePolicy
|
||||
metadata:
|
||||
name: podinfo-pinned
|
||||
|
||||
@@ -11,19 +11,12 @@
|
||||
# (R-STATE-01 covers PaaS + IaaS).
|
||||
#
|
||||
# METHOD (virtctl-free, filesystem-agnostic, deterministic): a VM whose ROOT is a
|
||||
# CDI DataVolume on Longhorn (a mounted PVC — NOT an ephemeral containerDisk). A
|
||||
# containerDisk root is EPHEMERAL — discarded when the VM is destroyed — so it could
|
||||
# never prove data-survival; the spec-required disk model here is a Longhorn-backed
|
||||
# DataVolume that OUTLIVES the VM (docs/06/R-STATE-01), which is exactly what this test
|
||||
# destroys-and-rebuilds. (Feature-gate note, revalidated 2026-08: an earlier revision
|
||||
# of this comment also claimed the containerDisk BOOT path was broken on this cluster —
|
||||
# KubeVirt v1.9 serves the container-disk binary through a k8s `image` volume gated by
|
||||
# the ALPHA `ImageVolume` feature gate, which k3s v1.31 had DISABLED, so the apiserver
|
||||
# pruned the volume and the guest died at OCI init. That premise is now STALE: the
|
||||
# cluster is k3s v1.36.4, where ImageVolume is ENABLED by default
|
||||
# (kubernetes_feature_enabled{name="ImageVolume"}=1; a server-side dry-run RETAINS an
|
||||
# `image` volume rather than pruning it), so containerDisk roots would boot — DataVolume
|
||||
# is used purely because only it survives rebuild.) The VM also has a SECOND
|
||||
# CDI DataVolume on Longhorn (a mounted PVC — NOT an ephemeral containerDisk; the
|
||||
# containerDisk boot path is broken on this k3s cluster because KubeVirt v1.9 serves
|
||||
# the container-disk binary through a k8s `image` volume gated by the ALPHA
|
||||
# `ImageVolume` feature gate, which k3s v1.31 has disabled → the apiserver prunes the
|
||||
# volume and the guest dies at OCI init. DataVolume disks bypass that path and are the
|
||||
# spec-required disk model anyway — docs/06/R-STATE-01). The VM also has a SECOND
|
||||
# Longhorn BLOCK data disk; cloud-init writes a unique canary to that raw block device
|
||||
# and powers the guest off; runStrategy RerunOnFailure means
|
||||
# a clean shutdown STAYS down (VMI -> Succeeded), so the write is known-complete and
|
||||
|
||||
@@ -9,11 +9,11 @@ CHAINSAW_VERSION="v0.2.12"
|
||||
CHAINSAW_IMAGE="ghcr.io/kyverno/chainsaw:v0.2.12"
|
||||
|
||||
# kubectl — used by run.sh for cluster reachability preflight and by probe scripts.
|
||||
KUBECTL_VERSION="v1.36.4"
|
||||
KUBECTL_VERSION="v1.31.4"
|
||||
|
||||
# Kyverno CLI — L0 static/policy gate (owned by T-0.5, referenced here for a single
|
||||
# pinned toolchain view). Not invoked by run.sh yet.
|
||||
KYVERNO_VERSION="v1.19.0"
|
||||
KYVERNO_VERSION="v1.13.2"
|
||||
|
||||
# Python 3 — coverage generator + JUnit emitter use stdlib only (no pip deps), so the
|
||||
# only constraint is a modern interpreter.
|
||||
|
||||
@@ -1,268 +0,0 @@
|
||||
# Evaluation: Cilium 1.20.1 native Gateway API ExternalAuth vs. the oauth2-proxy design
|
||||
|
||||
**Status:** Research + plan only — NO live change proposed for tonight. Reviewable evaluation.
|
||||
**Author:** WS-NET/WS-SEC investigation (Claude Opus 4.8), 2026-08-31.
|
||||
**Scope:** Whether Cilium 1.20.1's native Gateway API `ExternalAuth` filter (GEP-1494) can replace
|
||||
(a) the hand-authored, **un-wired** forward-auth `CiliumClusterwideEnvoyConfig`
|
||||
(`platform/oauth2-proxy/forward-auth-cec.yaml`) and (b) Crafty's per-pod oauth2-proxy **sidecar**.
|
||||
**Answers the open item** from DECISIONS-LOG (2026-08-30): *"the Cilium ext_authz/forward-auth
|
||||
workaround relevance under 1.20.1."*
|
||||
|
||||
---
|
||||
|
||||
## TL;DR — Verdict: **DEFER (partial-feasible, not viable as-is tonight)**
|
||||
|
||||
Native `ExternalAuth` **exists in the Cilium 1.20.1 codebase** and *could* eventually replace both
|
||||
mechanisms with a single, Gateway-API-native, per-route filter. **But it is NOT usable on this
|
||||
cluster in its current state**, and adopting it is a non-trivial, elevated-risk change to the exact
|
||||
shared Gateway that SSO was just stabilized on. Recommendation: **keep the current design, do not
|
||||
adopt now.** Track native ExternalAuth as the long-term target and revisit after the blocking
|
||||
conditions below clear.
|
||||
|
||||
Three hard blockers, in order of severity:
|
||||
|
||||
1. **The cluster's Gateway API CRDs are the `standard` channel — the `ExternalAuth` HTTPRoute filter
|
||||
is not even installable.** Verified live: the `httproutes.gateway.networking.k8s.io` CRD is
|
||||
`gateway.networking.k8s.io/channel: standard`, bundle `v1.6.1`, and its `filters[].type` enum is
|
||||
exactly `[RequestHeaderModifier, ResponseHeaderModifier, RequestMirror, RequestRedirect,
|
||||
URLRewrite, ExtensionRef, CORS]` — **`ExternalAuth` is absent**. GEP-1494 is an **experimental**
|
||||
Gateway API feature; using it requires swapping the cluster-wide Gateway API CRDs to the
|
||||
**experimental channel** bundle. That is a breaking, cluster-wide CRD change, and it must be kept
|
||||
in lock-step with the Ansible-bootstrap ↔ Flux **parity contract**.
|
||||
2. **The feature is young and shipped with a fail-**open** security bug.** `ExternalAuth` landed in
|
||||
Cilium **1.20.0** (2026-07-29, PR #45739). Issue **#47178 "Basic externalauth setup fails open"**
|
||||
was filed against 1.20.0-rc.0 — the ext_authz filter silently produced no Envoy entry and let
|
||||
**unauthenticated traffic through**. The fix (PR #47265) was backported to the 1.20 branch
|
||||
(~2026-07-21) and should be in 1.20.0/1.20.1, but a fail-**open** regression in a brand-new
|
||||
security-critical filter is exactly the failure mode R-IDP-02 forbids. Confidence that today's
|
||||
filter is robust is low, and documentation is thin (Cilium ships **no** `external-auth` docs page
|
||||
in 1.20 — the dedicated doc URL 404s; the stable Gateway API doc page makes no mention of it).
|
||||
3. **`enforce_policy_on_l7lb` (the CVE-2025-30162 hairpin behavior) will bite the ext_authz hop too.**
|
||||
Commit `68690e5` just fixed cluster-wide SSO breakage: after the 1.16.5→1.20.1 jump, the shared
|
||||
Gateway's L7 proxy re-evaluates a pod's **egress policy against the *real backend* identity** when
|
||||
it hairpins (`enforce_policy_on_l7lb`), so every NetworkPolicy'd app needed an explicit
|
||||
`toEndpoints` egress allow to the Keycloak backend. A Gateway-level `ExternalAuth` filter makes the
|
||||
Gateway's Envoy hairpin to the **oauth2-proxy** backend cluster for the auth check — the identical
|
||||
class of policy re-evaluation. Expect to need new egress/identity allowances for the ext_authz
|
||||
path, and to re-validate it does not regress the freshly-fixed Keycloak egress.
|
||||
|
||||
Given SSO "was just painfully fixed," the cost/benefit does not favor touching it now.
|
||||
|
||||
---
|
||||
|
||||
## Current state (verified — live cluster + repo)
|
||||
|
||||
**Cluster:** Cilium `v1.20.1` (`quay.io/cilium/cilium:v1.20.1`), Gateway API CRDs bundle `v1.6.1`
|
||||
**standard** channel, k3s v1.36.4. Shared Gateway `gateway/shared` (class `cilium`, VIP
|
||||
`192.168.1.53`, Programmed), backed by the machine-owned `CiliumEnvoyConfig
|
||||
gateway/cilium-gateway-shared` (the only CEC/CCEC on the cluster). `cilium-config` has
|
||||
`enable-gateway-api: true` and **no** external-auth flag (`mesh-auth-*` is the unrelated SPIFFE
|
||||
mutual-auth feature and is disabled).
|
||||
|
||||
**Auth design today (three mechanisms, one of them dead):**
|
||||
|
||||
| Mechanism | Where | Apps | Status |
|
||||
|---|---|---|---|
|
||||
| **Native OIDC** (`auth.mode: oidc`) | in-app OIDC client → Keycloak | gitea, nextcloud, semaphore, grafana(native) | **Live, working.** Not in scope to change. |
|
||||
| **Native/other** | app-specific | seerr (own OIDC), plex | Live. Out of scope. |
|
||||
| **forward-auth CEC** (`auth.mode: forward-auth`) | `platform/oauth2-proxy/forward-auth-cec.yaml` — hand-authored `CiliumClusterwideEnvoyConfig` with an `ext_authz` HTTP filter → oauth2-proxy `/oauth2/auth` | jackett, pihole, qbittorrent, radarr, sonarr | **NOT DEPLOYED.** A TEMPLATE, **not** referenced by `platform/oauth2-proxy/kustomization.yaml`. Proven in a throwaway envoy pod (2026-08-28) but never wired. These 5 UIs are currently reachable on the shared Gateway **un-gated**. |
|
||||
| **oauth2-proxy sidecar** (per-pod reverse-proxy, "Option B") | `platform/crafty/deployment.yaml` container `oauth2-proxy` (`v7.8.1`, `:4180` → `https://127.0.0.1:8443`) | crafty | **Live, working.** In-data-path fail-closed gate. |
|
||||
|
||||
**The renderer contract (`catalog/render.py`):** `auth.mode ∈ {oidc, forward-auth, native, none}`
|
||||
(`AUTH_MODES`, line 53); default is `forward-auth` for any app with `ingressRoutes` else `none`
|
||||
(lines 76-77). `_auth_annotations()` (line 605) stamps
|
||||
`homelab.keithjackson.dev/auth-mode` + `/auth-groups` onto the HTTPRoute for `oidc`/`forward-auth`,
|
||||
and pods carry an `auth-mode` label enforced by a Kyverno `require-auth-mode-label` policy
|
||||
(R-IDP-02). Since commit `68690e5`, the renderer also emits a `toEndpoints` egress allow to the
|
||||
Keycloak backend for `oidc`/`forward-auth` apps (line 539). **No renderer field today expresses an
|
||||
ext_authz `backendRef`** — the forward-auth annotations are consumed by the (un-wired) platform
|
||||
enforcement, not by a Gateway-API filter.
|
||||
|
||||
**Why the CEC exists / why it's not on the Gateway (per its own header + README):** on Cilium 1.16.5
|
||||
there was no supported hook to inject `ext_authz` into the machine-regenerated shared-Gateway Envoy
|
||||
program, and native ExternalAuth "ships in NO released Cilium version yet (PoC only)." That premise
|
||||
is now **partially stale** — see below — which is precisely why this evaluation exists.
|
||||
|
||||
---
|
||||
|
||||
## Does Cilium 1.20.1 support native Gateway API ExternalAuth? (research)
|
||||
|
||||
**Yes, in the code — as an experimental Gateway API feature that is not enabled on this cluster.**
|
||||
|
||||
- **Shipped in 1.20.0.** The v1.20.0 release notes list: *"🔐 External Authorization: HTTPRoute
|
||||
requests can now be authenticated and authorized through an external service before they reach the
|
||||
application, using the Gateway API `ExternalAuth` filter from GEP-1494."* (PR
|
||||
[cilium/cilium#45739](https://github.com/cilium/cilium/pull/45739)). Isovalent's 1.20 launch post
|
||||
headlines "Gateway API ExternalAuth."
|
||||
- **Shape (GEP-1494 experimental filter).** An HTTPRoute rule gains a filter
|
||||
`type: ExternalAuth` with `externalAuth.protocol: GRPC|HTTP`, a `backendRef` to the auth service,
|
||||
and `grpc.allowedHeaders` / `http.{path,allowedHeaders,allowedResponseHeaders}` / `forwardBody`.
|
||||
It maps onto Envoy's `ext_authz` protocol — the **same** mechanism the hand-authored CEC uses,
|
||||
but declared per-route on the standard shared Gateway instead of a separate listener. Example:
|
||||
```yaml
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: HTTPRoute
|
||||
spec:
|
||||
rules:
|
||||
- filters:
|
||||
- type: ExternalAuth
|
||||
externalAuth:
|
||||
protocol: GRPC # or HTTP (oauth2-proxy /oauth2/auth is HTTP)
|
||||
backendRef: { kind: Service, name: oauth2-proxy, port: 80 }
|
||||
http: { path: /oauth2/auth, allowedHeaders: [cookie, authorization] }
|
||||
backendRefs: [{ name: <app>, port: <port> }]
|
||||
```
|
||||
- **It is EXPERIMENTAL-channel.** GEP-1494 status is **Experimental**
|
||||
([gateway-api.sigs.k8s.io/geps/gep-1494](https://gateway-api.sigs.k8s.io/geps/gep-1494/)). Its
|
||||
fields live only in the Gateway API **experimental** CRD bundle. **This cluster runs the `standard`
|
||||
channel** (verified), so the filter type is not present in the HTTPRoute schema — a manifest using
|
||||
it would be rejected at admission today.
|
||||
- **Maturity risk.** Fail-open bug [#47178](https://github.com/cilium/cilium/issues/47178) → fix
|
||||
[#47265](https://github.com/cilium/cilium/pull/47265) (backported to 1.20, ~2026-07-21). Cilium
|
||||
ships **no dedicated ExternalAuth documentation page** in 1.20 (the `/en/v1.20/.../external-auth/`
|
||||
and `/en/latest/.../external-auth/` URLs both 404; the stable Gateway API support page does not
|
||||
mention it). Feature is real but young and thinly documented.
|
||||
|
||||
**Bottom line:** the CEC/sidecar READMEs' claim that native ExternalAuth "ships in no released Cilium
|
||||
version" is now outdated — it ships in 1.20.x — **but** it is experimental, off by default here
|
||||
(wrong CRD channel), and not battle-tested.
|
||||
|
||||
---
|
||||
|
||||
## Would it cleanly replace the two mechanisms?
|
||||
|
||||
**(a) The un-wired forward-auth CEC — YES, in principle, and more cleanly.** This is the strongest
|
||||
case for native ExternalAuth: the CEC is a hand-authored TLS listener + routes + EDS clusters on a
|
||||
*separate* VIP that requires moving hosts off the shared Gateway and repointing DNS. A native
|
||||
`ExternalAuth` filter would instead be one filter block on each app's **existing** HTTPRoute on the
|
||||
**existing** shared Gateway/VIP/DNS — no second ingress, no DNS churn, no hand-authored TLS. Since
|
||||
the CEC is not deployed, there is **no rip-out risk** here: adopting native ExternalAuth for the 5
|
||||
forward-auth apps (jackett/pihole/qbittorrent/radarr/sonarr) would *complete* an unfinished story
|
||||
rather than replace a working one. The renderer already carries the `auth-mode`/`auth-groups` signal;
|
||||
it would need a new emitter that turns `auth.mode: forward-auth` into the HTTPRoute `ExternalAuth`
|
||||
filter (see plan).
|
||||
|
||||
**(b) Crafty's sidecar — PARTIAL, and lower value.** Native ExternalAuth could drop the per-pod
|
||||
oauth2-proxy container and gate `crafty.home` at the Gateway like the others. **But** Crafty's UI is
|
||||
HTTPS **self-signed on :8443**; the sidecar terminates that in-pod (`--ssl-upstream-insecure-skip-
|
||||
verify`). Moving the gate to the Gateway means the shared Gateway's Envoy must speak HTTPS to Crafty's
|
||||
self-signed `:8443` upstream — requiring a `BackendTLSPolicy` with `insecureSkipVerify` (or a real
|
||||
cert), and the `crafty` Service would target `:8443` directly. That is net-new Gateway plumbing to
|
||||
remove a small, **working**, fail-closed sidecar. Low priority; do Crafty last, if at all.
|
||||
|
||||
Native ExternalAuth **cannot** and **should not** touch the `oidc` apps (gitea/nextcloud/semaphore/
|
||||
grafana) — those use in-app OIDC and are out of scope.
|
||||
|
||||
---
|
||||
|
||||
## Proposed migration plan (do NOT execute now — for review)
|
||||
|
||||
Phased, reversible, lowest-blast-radius first. **Precondition gates P1-P3 must clear before ANY
|
||||
cluster change.** Each step names the files it touches.
|
||||
|
||||
### Phase 0 — Preconditions (research/verify only, no cluster change)
|
||||
- **P1. Confirm the fail-open fix is in 1.20.1.** Verify PR #47265 shipped in the running `v1.20.1`
|
||||
(changelog / `git tag --contains`), and reproduce a **fail-closed** ext_authz on a throwaway route
|
||||
(the 2026-08-28 CEC proof, redone with the *native filter* on a scratch HTTPRoute) BEFORE trusting
|
||||
it for R-IDP-02.
|
||||
- **P2. Decide the Gateway API channel swap.** Adopting GEP-1494 requires the **experimental** CRD
|
||||
bundle cluster-wide. Assess blast radius (every Gateway/HTTPRoute revalidated against experimental
|
||||
schemas) and reconcile the **Ansible-bootstrap ↔ Flux parity contract** — the Ansible bootstrap
|
||||
that installs Gateway API CRDs must install the *same* channel. Files: wherever the Gateway API CRD
|
||||
bundle is declared for both bootstrap and Flux (search `gateway.networking.k8s.io` CRD source;
|
||||
`platform/gateway/` + the Ansible bootstrap CRD step). This is the single biggest change and its
|
||||
own reviewable sub-plan.
|
||||
- **P3. Confirm the Cilium enablement path.** Determine whether Cilium honors the `ExternalAuth`
|
||||
filter on experimental-CRD presence alone or needs an agent/helm flag; if a flag exists it is a
|
||||
`[parity]` value in `platform/cilium/helmrelease.yaml` and the Ansible Cilium values.
|
||||
|
||||
### Phase 1 — One canary app (pihole), additive, reversible
|
||||
- Install experimental Gateway API CRDs (P2) + any Cilium flag (P3).
|
||||
- Add the `ExternalAuth` filter to **pihole's** HTTPRoute only, `backendRef` → `oauth2-proxy`
|
||||
Service, `http.path: /oauth2/auth`, header allow-list mirroring the CEC's proven stanza. Keep
|
||||
`failureMode`/deny semantics fail-**closed**.
|
||||
Files: `catalog/pihole.yaml` (unchanged intent) + a new renderer emitter in
|
||||
`catalog/render.py` (`_httproute()` path) that renders the filter from `auth.mode: forward-auth`;
|
||||
re-render pihole golden. Add the ext_authz **egress allow** to the oauth2-proxy backend for the app
|
||||
(the `enforce_policy_on_l7lb` mitigation, mirroring the Keycloak fix in `render.py`).
|
||||
- **Browser test** (the human): incognito `pihole.home` → 302 to Keycloak → login `platform-admins`
|
||||
→ PiHole UI; non-`platform-admins` → 403; oauth2-proxy down → **denied**, never the bare UI.
|
||||
- **Rollback:** delete the filter block (revert the HTTPRoute) → back to un-gated shared Gateway. No
|
||||
data, no DNS, no VIP involved.
|
||||
|
||||
### Phase 2 — Roll out to the remaining forward-auth apps
|
||||
- Repeat for jackett, qbittorrent, radarr, sonarr via the renderer (they already declare
|
||||
`auth.mode: forward-auth`; the emitter does it uniformly). Re-render 4 goldens.
|
||||
- **Delete `platform/oauth2-proxy/forward-auth-cec.yaml`** and its README section describing the
|
||||
separate-VIP interim — now obsolete. (It was never wired, so this is doc/template cleanup.)
|
||||
|
||||
### Phase 3 — Crafty (optional, last)
|
||||
- Add `ExternalAuth` to `crafty`'s HTTPRoute; point the `crafty` Service at `:8443`; add a
|
||||
`BackendTLSPolicy` for the self-signed upstream; **remove the oauth2-proxy sidecar** from
|
||||
`platform/crafty/deployment.yaml` and its SOPS creds. Verify game-port L4 path (192.168.1.55)
|
||||
untouched. Higher effort for marginal gain — defer unless consolidating.
|
||||
|
||||
---
|
||||
|
||||
## Risks & rollback
|
||||
|
||||
| Risk | Severity | Mitigation / Rollback |
|
||||
|---|---|---|
|
||||
| **Fail-open regression** (a new/edge ext_authz bug lets traffic through un-authed — R-IDP-02 violation) | **High** | P1 reproduction on a scratch route before any real app; canary (pihole) first with explicit browser deny-path test; keep un-gated-but-internal apps behind LAN until proven. |
|
||||
| **Experimental-channel CRD swap breaks existing Gateways/HTTPRoutes** | **High** | Treat P2 as its own reviewed change; snapshot/backup CRDs; parity with Ansible bootstrap; validate all 16 existing HTTPRoutes still Programmed after the swap. |
|
||||
| **`enforce_policy_on_l7lb` blocks the ext_authz hop** (Gateway hairpins to oauth2-proxy backend; NetworkPolicy'd apps 403 the auth check — the CVE-2025-30162 pattern from commit `68690e5`) | **Med-High** | Emit a `toEndpoints` egress allow to the oauth2-proxy backend for forward-auth apps, exactly as done for Keycloak; re-validate the Keycloak egress is not regressed. |
|
||||
| **Parity-contract drift** (Flux vs Ansible disagree on CRD channel / Cilium flag) | **Med** | Land CRD-channel + Cilium-flag changes in both bootstrap and Flux in one PR; gate (`make -C ci gate`). |
|
||||
| **Thin docs / churn** (experimental API fields change across Gateway API/Cilium minors) | **Med** | Pin bundle version; Renovate watch; keep the CEC template in git history as a fallback recipe. |
|
||||
| **Regressing freshly-fixed SSO** | **High** | Do nothing to `oidc` apps; canary + reversible filter-only changes; never touch the shared Gateway CEC directly. |
|
||||
|
||||
**Global rollback:** every proposed change is additive at the HTTPRoute level and reverts by removing
|
||||
the filter block. The nuclear fallback if native ExternalAuth proves flaky is the **already-proven
|
||||
Option B** (per-app oauth2-proxy reverse-proxy, as Crafty uses today) — Gateway-API-native, no CRD
|
||||
channel swap, just a `backendRef` to a per-app proxy.
|
||||
|
||||
---
|
||||
|
||||
## Recommendation
|
||||
|
||||
**DEFER.** Native ExternalAuth is the right long-term target and would let us delete both the dead CEC
|
||||
template and (eventually) Crafty's sidecar in favor of one uniform, Gateway-API-native filter. But
|
||||
**tonight it is not viable**: the enabling CRD channel isn't installed, the feature is experimental
|
||||
and recently fail-open-buggy, it is undocumented in 1.20, and it would perturb the shared Gateway and
|
||||
the just-fixed `enforce_policy_on_l7lb` egress that SSO now depends on.
|
||||
|
||||
Concretely:
|
||||
- **Do NOT** adopt now. **Keep** Crafty's working sidecar and leave the 5 forward-auth UIs as-is.
|
||||
- **When revisiting** (next maintenance window, not tonight): execute **Phase 0** preconditions,
|
||||
then a **pihole-only canary** (Phase 1). Only after a clean fail-closed browser proof, roll out
|
||||
Phase 2 and retire the CEC template.
|
||||
- **Cheaper interim if gating the 5 UIs becomes urgent before then:** adopt **Option B** (per-app
|
||||
oauth2-proxy reverse-proxy, the Crafty pattern) — no experimental CRDs, fully reversible — rather
|
||||
than the separate-VIP CEC or a rushed native-ExternalAuth switch.
|
||||
|
||||
This resolves the DECISIONS-LOG open item: **native ExternalAuth is relevant and code-present under
|
||||
1.20.1, but blocked on an experimental CRD-channel swap + maturity/fail-open concerns; the interim
|
||||
workaround stays.**
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- Cilium v1.20.0 release notes — "External Authorization" / GEP-1494 (PR #45739):
|
||||
https://github.com/cilium/cilium/releases/tag/v1.20.0
|
||||
- Cilium v1.20.1 release: https://github.com/cilium/cilium/releases/tag/v1.20.1
|
||||
- Isovalent — "Cilium 1.20: Gateway API ExternalAuth…": https://isovalent.com/blog/post/cilium-1-20/
|
||||
- GEP-1494 (HTTP Auth in Gateway API), status **Experimental**:
|
||||
https://gateway-api.sigs.k8s.io/geps/gep-1494/
|
||||
- Issue #47178 "Basic externalauth setup fails open": https://github.com/cilium/cilium/issues/47178
|
||||
- Fix PR #47265 (backported to 1.20): https://github.com/cilium/cilium/pull/47265
|
||||
- Issue #45704 (GEP-1494 filter support, closed as dup of the implementing PR):
|
||||
https://github.com/cilium/cilium/issues/45704
|
||||
- Cilium 1.20.1 Gateway API support page (no ExternalAuth mention — feature undocumented in 1.20):
|
||||
https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/
|
||||
- Live cluster (read-only): Cilium `v1.20.1`; HTTPRoute CRD `channel: standard`, bundle `v1.6.1`,
|
||||
`filters[].type` enum lacks `ExternalAuth`; `cilium-config` has no external-auth flag.
|
||||
- Repo: `platform/oauth2-proxy/forward-auth-cec.yaml` (+ README), `platform/crafty/deployment.yaml`
|
||||
(+ README), `catalog/render.py` (`AUTH_MODES`, `_auth_annotations`, keycloak-egress emitter),
|
||||
commit `68690e5` (CVE-2025-30162 / `enforce_policy_on_l7lb` egress fix).
|
||||
</content>
|
||||
</invoke>
|
||||
@@ -1,588 +0,0 @@
|
||||
# Cluster upgrade plan — k3s 1.31.5 → 1.36.x + node OS (dev cluster)
|
||||
|
||||
> ## ✅ COMPLETED / historical — executed 2026-08-30, see git log
|
||||
>
|
||||
> **This plan has been carried out.** The dev cluster now runs **k3s v1.36.4+k3s1** on **Debian 13
|
||||
> (trixie)** with Cilium 1.20.1 and Longhorn 1.12.1. The steps, gates, and rollbacks below are
|
||||
> retained as the record of how the upgrade was performed — do NOT re-execute them against the live
|
||||
> cluster. Current state is tracked in `infra/PHASE1-STATUS.md` and the 2026-08-30 DECISIONS-LOG entry.
|
||||
|
||||
**Scope:** the `home-network-k3s-impl` **dev** cluster (`k3s-dev-1/2/3` @ 192.168.1.50-52), 3× HA
|
||||
embedded-etcd servers behind a kube-vip VIP, Debian 12 bookworm. Upgrade Kubernetes/k3s from
|
||||
**v1.31.5+k3s1 → v1.36.x+k3s1** (five minors) and the node OS (Debian 12 → 13), with an
|
||||
etcd-snapshot / Velero / CNPG backup gate and explicit PASS/STOP criteria at **every** step, and a
|
||||
defined rollback for each hop.
|
||||
|
||||
> Status: **PLAN ONLY.** Nothing here has been executed. All version pins marked "confirm at
|
||||
> execution" must be re-checked against the upstream release page on the day of the hop — this doc
|
||||
> was written 2026-08-29 and homelab component versions move weekly (Renovate).
|
||||
|
||||
> Companion runbooks that this plan reuses wholesale:
|
||||
> - `docs/plans/ssd-etcd-execution-runbook.md` — the **proven safe-bounce** procedure (Longhorn
|
||||
> rebuild-freeze + one-node-at-a-time + leader-last + per-node verify gates). Every k3s hop below
|
||||
> is a parameterised repeat of that runbook.
|
||||
> - `docs/plans/ssd-etcd-isolation.md` — why etcd is fragile on this cluster (slow HDD; etcd now on
|
||||
> a dedicated SSD `virtio2`) and how a single member can be reset+rejoined from the other two.
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR / operator decisions needed
|
||||
|
||||
| # | Decision | Recommendation | Needs operator sign-off |
|
||||
|---|---|---|---|
|
||||
| D1 | k3s upgrade **method** | Scripted **in-place per-node** binary upgrade driven by Ansible, one node at a time, leader-last, with the Longhorn rebuild-freeze. NOT system-upgrade-controller (its auto-drain fights this cluster's slow-HDD Longhorn and removes the human gate between nodes). | **Yes** — SUC vs scripted-in-place |
|
||||
| D2 | Node **OS** upgrade | **Immutable re-provision** via tofu (bump the cloud image to Debian 13 trixie, replace one node at a time, rejoin etcd), run as a **separate campaign** from the k3s hops. Not in-place `apt` dist-upgrade. | **Yes** — Debian 13 now, or stay on 12 |
|
||||
| D3 | Do OS + k3s on the **same** node bounce? | **No.** Never combine. Finish the k3s ladder to a stable target, then run the OS campaign (or vice-versa), one variable per bounce. | Confirm |
|
||||
| D4 | Sequencing granularity | One k3s **minor** per maintenance window (5 windows). Cilium + Longhorn + Gateway-API CRDs advance in **lockstep** inside the same window (they are the pacing blockers). | Confirm |
|
||||
| D5 | Flux `image.toolkit.fluxcd.io/v1beta2` → `/v1` migration | **Must** be done before Flux is upgraded to v2.7+ (which is required for k8s 1.36). Do it early, decoupled. | Confirm timing |
|
||||
|
||||
**Headline blockers (do these or the hop fails):**
|
||||
1. **Cilium 1.16.5 is already at/near its ceiling** — Cilium 1.16 tops out around k8s 1.30-1.31.
|
||||
Cilium **must** step 1.16→1.17→1.18→1.19→1.20 in lockstep with the k8s minors. It is the single
|
||||
most urgent item.
|
||||
2. **Longhorn 1.7.3** must step 1.7→1.8→1.9→1.10→1.11→1.12 (one minor at a time) to reach a
|
||||
1.36-validated release (1.12.x).
|
||||
3. **Flux v2.4.0 → v2.7+** is required for k8s 1.36, and v2.7 **removed** the image-automation
|
||||
`v1beta2` API the repo still uses.
|
||||
4. **KubeVirt v1.9.0 / CDI v1.66.0 are already 1.36-ready** — a pleasant surprise; no VM-stack
|
||||
upgrade needed, just keep them where they are.
|
||||
|
||||
---
|
||||
|
||||
## 1. Confirmed current-state inventory
|
||||
|
||||
### 1.1 Nodes (live, verified 2026-08-29 via `dev.sh run kubectl get nodes -o wide`)
|
||||
|
||||
| Node | IP | Roles | k8s | OS | Kernel | Runtime |
|
||||
|---|---|---|---|---|---|---|
|
||||
| k3s-dev-1 | 192.168.1.50 | control-plane,etcd,master | v1.31.5+k3s1 | Debian 12 bookworm | 6.1.0-52-cloud | containerd 1.7.23-k3s2 |
|
||||
| k3s-dev-2 | 192.168.1.51 | control-plane,etcd,master | v1.31.5+k3s1 | Debian 12 bookworm | 6.1.0-52-cloud | containerd 1.7.23-k3s2 |
|
||||
| k3s-dev-3 | 192.168.1.52 | control-plane,etcd,master | v1.31.5+k3s1 | Debian 12 bookworm | 6.1.0-52-cloud | containerd 1.7.23-k3s2 |
|
||||
|
||||
- 3 servers, embedded etcd, **all** control-plane (no dedicated workers). Quorum = 2/3.
|
||||
- kube-vip control-plane VIP **192.168.1.49** (fronted through pfSense NAT at 192.168.2.50:6443).
|
||||
- etcd datastore is on the **dedicated SSD `virtio2` (`/dev/vdc`, LABEL=k3s-etcd)** mounted at
|
||||
`/var/lib/rancher/k3s/server/db` (migrated this session). Longhorn data disk = `virtio1`
|
||||
(`/dev/vdb`, `/var/lib/longhorn`). OS = `virtio0` (`/dev/vda`).
|
||||
- **18** Longhorn volumes, all `healthy` (the count is the post-step invariant).
|
||||
|
||||
### 1.2 Where versions are pinned in this repo
|
||||
|
||||
| Thing | Pin location | Current value |
|
||||
|---|---|---|
|
||||
| **k3s version** | `infra/ansible/roles/k3s_server/defaults/main.yml` → `k3s_version`; applied via `INSTALL_K3S_VERSION` in `roles/k3s_server/tasks/main.yml` | `v1.31.5+k3s1` |
|
||||
| k3s install guard | `roles/k3s_server/tasks/main.yml` — install task has `creates: /etc/rancher/k3s/k3s.yaml` | **idempotent → will NOT upgrade a running node on a version bump** (see §4) |
|
||||
| kubectl/helm/flux (toolbox) | `infra/devcontainer/versions.env`, `ci/versions.env` | kubectl 1.31.4, flux 2.4.0 |
|
||||
| **Cilium (ongoing)** | `platform/cilium/helmrelease.yaml` → `version: "1.16.5"` | 1.16.5 |
|
||||
| Cilium (bootstrap) + **Gateway API CRDs** | `infra/ansible/roles/cilium/defaults/main.yml` → `gateway_api_version: "v1.2.1"`, `gateway_api_crd_urls` | v1.2.1 |
|
||||
| **Longhorn** | `platform/longhorn/helmrelease.yaml` → `version: ">=1.7.0 <1.8.0"` (live 1.7.3) | 1.7.3 |
|
||||
| **KubeVirt + CDI** | `platform/kubevirt/kustomization.yaml` (remote release manifests) | KubeVirt v1.9.0, CDI v1.66.0 |
|
||||
| **kube-vip** | `infra/ansible/roles/kube_vip/defaults/main.yml` → `kube_vip_version` | v0.8.9 |
|
||||
| **external-snapshotter** | `platform/external-snapshotter/kustomization.yaml` (`ref=v8.2.0`) | v8.2.0 |
|
||||
| **cert-manager** | `platform/cert-manager/helmrelease.yaml` → `">=1.16.0 <1.17.0"` (live 1.16.5) | 1.16.5 |
|
||||
| **CNPG** | `platform/cnpg/helmrelease.yaml` → `">=0.22.0 <1.0.0"` (live chart 0.29.0) | operator ≈1.26/1.27 era |
|
||||
| **kube-prometheus-stack** | `platform/monitoring/kube-prometheus-stack.yaml` → `">=65.0.0 <66.0.0"` (live 65.8.1) | 65.8.1 |
|
||||
| **Loki** | `platform/monitoring/loki.yaml` → `">=6.0.0 <7.0.0"` (live 6.55.0) | 6.55.0 |
|
||||
| **Alloy** | `platform/monitoring/alloy.yaml` → `">=0.10.0 <1.0.0"` (live 0.12.6) | 0.12.6 |
|
||||
| **Velero** | `platform/velero/helmrelease.yaml` → `">=8.0.0 <9.0.0"` (live 8.7.2, Velero v1.15.x) | 8.7.2 |
|
||||
| **Kyverno** | `platform/kyverno/admission/helmrelease.yaml` → `version: "3.3.x"` (live 3.3.9) | 3.3.9 |
|
||||
| **oauth2-proxy** | `platform/oauth2-proxy/helmrelease.yaml` → `">=7.0.0 <8.0.0"` (live 7.18.0) | 7.18.0 |
|
||||
| **external-dns** | `platform/externaldns/helmrelease.yaml` → `">=1.15.0 <1.16.0"` (live 1.15.2) | 1.15.2 |
|
||||
| **headlamp** | `platform/headlamp/helmrelease.yaml` → `">=0.41.0 <0.42.0"` (live 0.41.0) | 0.41.0 |
|
||||
| **Flux** | installed by `flux bootstrap` (toolbox v2.4.0); controllers live in `flux-system` | v2.4.0 |
|
||||
| **tofu cloud image** | `infra/tofu/variables.tf` → `image_source`, `image_file_name` | `debian-12-generic-amd64.qcow2` (bookworm) |
|
||||
|
||||
### 1.3 Live platform component versions (verified 2026-08-29)
|
||||
|
||||
Flux 2.4.0 (source-controller v1.4.1, kustomize-controller v1.4.0, helm-controller v1.1.0,
|
||||
image-automation v0.39.0, image-reflector v0.33.0, notification v1.4.0) · cert-manager v1.16.5 ·
|
||||
CNPG chart 0.29.0 (PostgreSQL 16.4, 3 single-instance clusters: keycloak-db, nextcloud-db,
|
||||
semaphore-db) · Cilium 1.16.5 · Kyverno 3.3.9 · Longhorn 1.7.3 · Alloy 0.12.6 ·
|
||||
kube-prometheus-stack 65.8.1 · Loki 6.55.0 · oauth2-proxy 7.18.0 · Velero 8.7.2 (+plugin-for-aws
|
||||
v1.11.1) · KubeVirt v1.9.0 · CDI v1.66.0 · snapshot-controller v8.2.0 · kube-vip v0.8.9 ·
|
||||
Gateway API CRDs v1 (v1.2.1 line).
|
||||
|
||||
### 1.4 CRD / API surface the repo actually uses (grep of `platform/ apps/ catalog/ clusters/`)
|
||||
|
||||
`v1`, `apps/v1`, `networking.k8s.io/v1`, `rbac.authorization.k8s.io/v1`, `storage.k8s.io/v1`,
|
||||
`gateway.networking.k8s.io/v1`, `cilium.io/v2`, `monitoring.coreos.com/v1`,
|
||||
`helm.toolkit.fluxcd.io/v2`, `source.toolkit.fluxcd.io/v1`, `kustomize.toolkit.fluxcd.io/v1`,
|
||||
`image.toolkit.fluxcd.io/v1beta2` ⚠, `postgresql.cnpg.io/v1`, `kubevirt.io/v1`,
|
||||
`cdi.kubevirt.io/v1beta1`, `cert-manager.io/v1`, `kyverno.io/v1`, `k8s.keycloak.org/v2alpha1`,
|
||||
`snapshot.storage.k8s.io/v1`, `client.authentication.k8s.io/v1beta1` (kubeconfig exec plugin).
|
||||
|
||||
**None of these are Kubernetes *core* APIs removed in 1.32-1.36.** The only removal that bites is a
|
||||
**Flux** CRD (`image.toolkit.fluxcd.io/v1beta2`) — see §3.
|
||||
|
||||
---
|
||||
|
||||
## 2. k3s upgrade path (sequential, one minor at a time)
|
||||
|
||||
### 2.1 The path and why skipping is unsafe
|
||||
|
||||
```
|
||||
1.31.5+k3s1 → 1.32.z → 1.33.z → 1.34.z → 1.35.z → 1.36.z (+k3s1 each)
|
||||
```
|
||||
|
||||
**Skipping minors is not supported.** Kubernetes' version-skew policy allows the kube-apiserver and
|
||||
components to differ by at most **one minor** during an upgrade, and a downgrade of the etcd
|
||||
datastore is not supported. Jumping >1 minor risks an apiserver that can't read objects written by
|
||||
an intermediate storage version and control-plane components outside the skew window. On k3s (all
|
||||
three nodes are servers/apiserver), the whole control plane moves together per hop, so the rule is:
|
||||
**one minor per campaign, all three nodes brought to the same minor before the next.**
|
||||
Source: <https://kubernetes.io/releases/version-skew-policy/>.
|
||||
|
||||
### 2.2 Target patch per minor (confirm at execution)
|
||||
|
||||
Pick the **latest patch of each minor** from <https://github.com/k3s-io/k3s/releases> (or the k3s
|
||||
channel server `https://update.k3s.io/v1-release/channels`, where channel `v1.3X` resolves to the
|
||||
latest patch of that minor). Known-latest as of writing:
|
||||
|
||||
| Hop | Target (min) | Notes / source |
|
||||
|---|---|---|
|
||||
| → 1.32 | latest `v1.32.z+k3s1` | confirm on releases page |
|
||||
| → 1.33 | latest `v1.33.z+k3s1` | k3s 1.33 release notes: <https://docs.k3s.io/release-notes/v1.33.X> |
|
||||
| → 1.34 | latest `v1.34.z+k3s1` | confirm |
|
||||
| → 1.35 | latest `v1.35.z+k3s1` | confirm |
|
||||
| → 1.36 | **`v1.36.3+k3s1`** (Aug 4 2026) or newer (`v1.36.4+k3s1` reported) | <https://docs.k3s.io/release-notes/v1.36.X> |
|
||||
|
||||
### 2.3 k3s-specific breaking changes across the path
|
||||
|
||||
- **containerd major bump 1.7 → 2.x.** k3s 1.36.3 ships **containerd v2.3.2-k3s2** (current node is
|
||||
1.7.23-k3s2). The containerd v1→v2 migration happens somewhere in the 1.32-1.34 window. It is
|
||||
transparent for standard workloads, but: (a) `config.toml` schema changed — this repo sets
|
||||
`device_ownership_from_security_context` via `--nonroot-devices` (k3s config.yaml), which k3s
|
||||
regenerates into the new containerd config on restart; verify it survives (KubeVirt/CDI Block-mode
|
||||
PVCs depend on it — see `roles/k3s_server/tasks/main.yml`). (b) Any node with custom
|
||||
`/etc/rancher/k3s/config.toml.tmpl` would need review — this cluster has none.
|
||||
- **etcd 3.5 → 3.6.** k3s 1.36.3 ships **etcd v3.6.14-k3s1**. etcd datastore version bumps are
|
||||
**one-way** (no downgrade). This is the single strongest reason for the pre-hop etcd snapshot gate
|
||||
(§6) and defines the point-of-no-return (§7).
|
||||
- **Traefik chart v40 breaking change** in k3s ≥1.36.2 (ingress-nginx provider rename). **N/A** —
|
||||
this cluster disables Traefik (`k3s_disable: "traefik,servicelb,local-storage"`); Cilium Gateway
|
||||
API owns ingress.
|
||||
- **kube-proxy / flannel / network-policy already disabled** (`--disable-kube-proxy`,
|
||||
`--flannel-backend=none`, `--disable-network-policy`) so k8s 1.36's **IPVS-mode kube-proxy
|
||||
deprecation is N/A** — Cilium is the kube-proxy replacement.
|
||||
- Always read the **Kubernetes urgent-upgrade notes** for each minor before the hop:
|
||||
<https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/> (`CHANGELOG-1.3X.md` →
|
||||
"Urgent Upgrade Notes").
|
||||
|
||||
---
|
||||
|
||||
## 3. Removed / deprecated API audit (1.32 → 1.36)
|
||||
|
||||
### 3.1 Kubernetes core removals/deprecations per minor
|
||||
|
||||
| k8s | Removed / deprecated | Affects this repo? |
|
||||
|---|---|---|
|
||||
| **1.32** | **`flowcontrol.apiserver.k8s.io/v1beta3`** (FlowSchema, PriorityLevelConfiguration) **removed** — use `flowcontrol.apiserver.k8s.io/v1` (GA since 1.29). Field rename `assuredConcurrencyShares`→`nominalConcurrencyShares`. | **No** — repo defines no FlowSchema/PriorityLevelConfiguration CRs. |
|
||||
| **1.33** | `status.nodeInfo.kubeProxyVersion` **field removed** from Node. | **No** — nothing reads it. |
|
||||
| **1.34** | **AppArmor** annotations path deprecations continue; `AppArmor` GA'd earlier. | No repo AppArmor annotations. |
|
||||
| **1.35** | Minor graduations; no removal that touches this repo's API set. | No. |
|
||||
| **1.36** | **`gitRepo` volume permanently disabled**; **Service `externalIPs` deprecated**; **kube-proxy IPVS** on a deprecation clock. | **No** — no gitRepo volumes, no `externalIPs`, kube-proxy already replaced by Cilium. |
|
||||
|
||||
Sources: <https://kubernetes.io/docs/reference/using-api/deprecation-guide/> ·
|
||||
<https://www.kubernetes.io/blog/2026/03/30/kubernetes-v1-36-sneak-peek/> ·
|
||||
per-minor `CHANGELOG-1.3X.md`.
|
||||
|
||||
**Conclusion: the repo's manifests use no Kubernetes core API removed in 1.32-1.36.** (Grep for
|
||||
`flowcontrol.apiserver`, `gitRepo:`, `externalIPs:`, `policy/v1beta1`, `batch/v1beta1`,
|
||||
`autoscaling/v2beta*`, `extensions/v1beta1`, `PodSecurityPolicy` returns **nothing**.)
|
||||
|
||||
### 3.2 The one real removal that bites — Flux `image.toolkit.fluxcd.io/v1beta2`
|
||||
|
||||
**Flux v2.7+ removed `image.toolkit.fluxcd.io/v1beta2`** (ImageRepository, ImagePolicy,
|
||||
ImageUpdateAutomation → graduated to `image.toolkit.fluxcd.io/v1`). Flux v2.7+ is required for k8s
|
||||
1.36 (§4.1). The repo still uses `v1beta2` in:
|
||||
- `platform/image-automation/image-update-automation.yaml`
|
||||
- `platform/image-automation/per-service/pihole.example.yaml`
|
||||
(plus any per-service ImageRepository/ImagePolicy generated from that template.)
|
||||
|
||||
**Required migration (do BEFORE upgrading Flux to v2.7+):** follow the official
|
||||
"Upgrade Procedure for Flux v2.7+" — migrate stored objects off v1beta2, then update every manifest
|
||||
in Git from `image.toolkit.fluxcd.io/v1beta2` → `image.toolkit.fluxcd.io/v1`.
|
||||
Source: <https://github.com/fluxcd/flux2/discussions/5572> ·
|
||||
<https://fluxcd.io/flux/components/image/imagerepositories/>.
|
||||
|
||||
### 3.3 Detection tooling — run BEFORE each hop
|
||||
|
||||
Add a read-only pre-hop gate using **both** a live-cluster scan and a manifest scan:
|
||||
|
||||
```bash
|
||||
# pluto — scans live cluster AND Helm releases for deprecated/removed apiVersions, target minor
|
||||
./infra/devcontainer/dev.sh run bash -lc 'pluto detect-all-in-cluster --target-versions k8s=v1.3X.0 -owide'
|
||||
# or scan the rendered manifests in git:
|
||||
./infra/devcontainer/dev.sh run bash -lc 'pluto detect-files -d platform -d apps --target-versions k8s=v1.3X.0'
|
||||
|
||||
# kubent (kube-no-trouble) — quick live scan
|
||||
./infra/devcontainer/dev.sh run bash -lc 'kubent --target-version 1.3X'
|
||||
```
|
||||
|
||||
- `pluto` and `kubent` detect **built-in** deprecated APIs from their bundled version data; they do
|
||||
**not** know about third-party CRD removals (e.g. the Flux `v1beta2` above) unless given a custom
|
||||
deprecation list. So pair the tool pass with the manual §3.2 check and a `grep` of the repo for
|
||||
each component's deprecated CRD versions before its own upgrade.
|
||||
- Neither tool is currently in the toolbox image; add to `infra/devcontainer/Containerfile` +
|
||||
`versions.env`, or run once via `go install`/release binary inside the devcontainer.
|
||||
- Sources: <https://github.com/FairwindsOps/pluto> · <https://github.com/doitintl/kube-no-trouble>.
|
||||
|
||||
---
|
||||
|
||||
## 4. Component compatibility matrix & upgrade sequencing
|
||||
|
||||
### 4.1 Kubernetes-version support per component (target = k8s 1.36)
|
||||
|
||||
| Component | Current | k8s-1.36 target (confirm patch) | k8s support fact | Must move… | Source |
|
||||
|---|---|---|---|---|---|
|
||||
| **Cilium** | 1.16.5 | **1.20.x** | 1.16:≤~1.30 · 1.17:1.29-1.32 · 1.18:1.30-1.33 · 1.19:1.32-1.35 · 1.20:1.33-1.36 | **in lockstep, every hop** (pacing blocker #1) | compatibility.fyi/cilium; docs.cilium.io/…/compatibility |
|
||||
| **Gateway API CRDs** | v1.2.1 | **v1.6.x** (Cilium 1.20 targets v1.6.1 — verify) | too old for Cilium 1.20 | with each Cilium hop | docs.cilium.io/en/v1.20/…/gateway-api |
|
||||
| **Longhorn** | 1.7.3 | **1.12.x** (1.12.1) | 1.7:≤~1.31 · 1.11.1:1.32-1.35 · 1.12:adds 1.36; min k8s stays v1.21 | one minor at a time (pacing blocker #2) | endoflife.date/longhorn; longhorn.io best-practices; SUSE support matrix |
|
||||
| **KubeVirt** | **v1.9.0** | **v1.9.x (already there)** | v1.9 validates 1.34/1.35/1.36 (N-2) | **no upgrade needed** | github.com/kubevirt/sig-release k8s-support-matrix |
|
||||
| **CDI** | **v1.66.0** | keep paired w/ KubeVirt v1.9 | same window as KubeVirt | no upgrade needed (keep pair) | kubevirt/containerized-data-importer releases |
|
||||
| **external-snapshotter / snapshot-controller** | v8.2.0 | **latest v8.x (≥8.6)** | v8 tracks current k8s; snapshot CRDs fwd/back-compatible | loose — opportunistic | github.com/kubernetes-csi/external-snapshotter releases |
|
||||
| **cert-manager** | 1.16.5 | **1.21.x** | 1.21 supports 1.33-1.36 | before/with the later hops | cert-manager.io/docs/releases |
|
||||
| **CNPG** | chart 0.29.0 (op ≈1.26/1.27) | **1.30.x** | 1.30 supports 1.34/1.35/1.36; 1.29 supports 1.33-1.35 | before the 1.36 hop | cloudnative-pg.io/docs/devel/supported_releases |
|
||||
| **Flux** | v2.4.0 | **v2.7+ (latest, ~v2.8.x)** | Flux N-2; v2.7+ needed for 1.36 | after §3.2 v1beta2→v1 migration | fluxcd.io/flux/releases |
|
||||
| **kube-prometheus-stack** | 65.8.1 | **~88.x+** (supports kubectl/k8s 1.36) | chart 88.x tested on 1.36 | before/with later hops; **apply updated CRDs separately** | github.com/prometheus-community/helm-charts |
|
||||
| **Loki (chart)** | 6.55.0 | **latest 6.x/newer** | uses stable core APIs — loose | opportunistic | grafana/helm-charts |
|
||||
| **Alloy (chart)** | 0.12.6 | **latest** | loose (DaemonSet, stable APIs) | opportunistic | grafana/alloy |
|
||||
| **Velero** | chart 8.7.2 (v1.15) | **latest (v1.17+/chart ~10.x)** | v1.16 tested→1.33, v1.17 tested→1.34; "1.18-latest" expected | before the 1.36 hop | velero.io support-matrix; compatibility.fyi/velero |
|
||||
| **Kyverno** | 3.3.9 | **latest 3.x (N-2 for 1.36)** | Kyverno follows k8s N-2; 3.3.x too old for 1.36 | before/with later hops | kyverno.io/docs/installation/releases |
|
||||
| **oauth2-proxy** | 7.18.0 | **latest 7.x** | loose | opportunistic | oauth2-proxy chart |
|
||||
| **external-dns** | 1.15.2 | **latest 1.x** | loose | opportunistic | kubernetes-sigs/external-dns |
|
||||
| **kube-vip** | v0.8.9 | **latest (k8s.io 0.36.x-based)** | leader-election uses coordination.k8s.io/v1 (stable); latest bumps client-go to 0.36 | before/with the 1.36 hop | github.com/kube-vip/kube-vip releases |
|
||||
| **headlamp** | 0.41.0 | latest | loose UI | opportunistic | headlamp chart |
|
||||
|
||||
### 4.2 The pacing blockers, and the per-hop lockstep
|
||||
|
||||
**Cilium and Longhorn are the two items that force the pace** — each supports only ~3-4 k8s minors
|
||||
and each upgrades **one minor at a time (no skipping, pre-flight required)**. KubeVirt/CDI are
|
||||
already at target, so the VM stack is *not* a blocker this cycle. The mapping:
|
||||
|
||||
| Window | k8s target | Cilium (HelmRelease) | Gateway API CRDs | Longhorn (HelmRelease) | Other components landed in this window |
|
||||
|---|---|---|---|---|---|
|
||||
| **W1** | 1.32 | **1.17** (covers 1.29-1.32) | bump toward Cilium 1.17's required version | **1.8** | — |
|
||||
| **W2** | 1.33 | **1.18** (1.30-1.33) | bump | **1.9** | cert-manager→1.18/1.19; kube-prometheus-stack CRDs refresh |
|
||||
| **W3** | 1.34 | **1.19** (1.32-1.35) | bump | **1.10** | CNPG→1.28/1.29; Velero→latest |
|
||||
| **W4** | 1.35 | 1.19 still covers 1.35 (optional →1.20 early) | as needed | **1.11** | Kyverno→latest; **Flux v1beta2→v1 migration + Flux→v2.7+** |
|
||||
| **W5** | 1.36 | **1.20** (1.33-1.36) | **v1.6.x** | **1.12** | cert-manager→1.21; CNPG→1.30; kube-vip→latest; snapshot-controller→latest v8.x |
|
||||
|
||||
Ordering rule **inside** each window (critical): **CNI/storage compatibility first, then the k3s
|
||||
control-plane hop, then verify.** Concretely per window:
|
||||
1. Land the **component** upgrades that must *precede* the new k8s minor (Cilium minor, Gateway API
|
||||
CRDs, Longhorn minor, and any component whose *current* version won't run on the new k8s). Let
|
||||
Flux reconcile; verify healthy on the **still-current** k8s.
|
||||
2. Run the **k3s minor hop** (§5) node-by-node.
|
||||
3. Verify the components again on the **new** k8s; then land the "opportunistic/loose" bumps.
|
||||
|
||||
> Rationale for "Cilium before the apiserver hop": if the apiserver moves to a minor the running
|
||||
> Cilium doesn't support, CNI/agent may fail to start on that node and the node never goes Ready.
|
||||
> Cilium's own guidance is to run its **pre-flight check** DaemonSet before each Cilium minor bump
|
||||
> and to upgrade Cilium **before or alongside** the k8s minor within its supported window.
|
||||
|
||||
---
|
||||
|
||||
## 5. Upgrade method — evaluation & recommendation (D1)
|
||||
|
||||
### 5.1 Options
|
||||
|
||||
**(a) Rancher system-upgrade-controller (SUC) with Plans.** Canonical k3s upgrade tool. A `Plan`
|
||||
targets nodes by label, concurrency 1, HA-aware, drains → replaces the k3s binary → uncordons.
|
||||
- **Pros:** official, declarative, handles server-vs-agent ordering; could live in GitOps.
|
||||
- **Cons for THIS cluster:** SUC's default **drain evicts pods**, which on this slow-HDD Longhorn
|
||||
triggers replica **rebuild** storms that starve etcd WAL fsync and flap the VIP — the exact
|
||||
failure `ssd-etcd-isolation.md` documents. You'd have to disable drain (`drain: {enabled:false}`)
|
||||
and still manually apply the Longhorn freeze, at which point SUC's main value (drain automation) is
|
||||
gone. Worse, SUC **automates away the human gate between nodes** — the user's hard requirement is
|
||||
an explicit backup + PASS/STOP verification *between* every node, which SUC does not pause for.
|
||||
|
||||
**(b) Scripted in-place per-node binary upgrade (Ansible-driven).** Re-run the k3s install script
|
||||
with the new `INSTALL_K3S_VERSION` on one node at a time; k3s replaces its binary and restarts,
|
||||
etcd re-joins at the new version. This is exactly the `ssd-etcd-execution-runbook.md` bounce shape.
|
||||
- **Pros:** reuses the **proven** freeze + leader-last + per-node-verify pattern; no drain (no
|
||||
rebuild storm — replicas *reattach*); the human gate sits naturally between nodes; minimal moving
|
||||
parts. **Note the `creates:` guard:** the existing install task self-skips on a running node, so
|
||||
the upgrade path must invoke the install script *explicitly* (a dedicated play/tag, or an ad-hoc
|
||||
`curl … | INSTALL_K3S_VERSION=<tag> sh -s - server`), not rely on bumping the var alone.
|
||||
- **Cons:** does not touch the OS (handled separately, §6/D2); slightly more manual than SUC.
|
||||
|
||||
**(c) Immutable re-provision via tofu.** Bump `image_source` + `k3s_version`, `tofu replace` one VM
|
||||
at a time, node rejoins etcd fresh and Longhorn rebuilds that node's replicas.
|
||||
- **Pros:** **also solves the OS upgrade** in the same motion; clean, reproducible cattle nodes.
|
||||
- **Cons:** destroys the node's etcd member + **all** its Longhorn replicas → full **rebuild** (not
|
||||
reattach) on the slow HDD per node (throttled but heavy, ~hours for 18 volumes' replicas across
|
||||
three rebuilds); the etcd SSD `virtio2` is wiped on replace (fine for a 3-node HA etcd — the new
|
||||
member syncs from the other two). Heavier and slower than (b) for a *pure k8s* hop.
|
||||
|
||||
### 5.2 Recommendation
|
||||
|
||||
- **k3s minor hops (W1-W5): choose (b), scripted in-place per-node.** It matches the cluster's
|
||||
proven safe-bounce, avoids rebuild storms, and preserves the mandatory per-node human gate. Pin
|
||||
the new tag in `roles/k3s_server/defaults/main.yml` (`k3s_version`) **and** drive the actual
|
||||
upgrade with an explicit install-script invocation (bypassing the `creates:` guard), `serial: 1`,
|
||||
leader-last.
|
||||
- **Node OS (Debian 12→13): choose (c), immutable re-provision**, as a **separate** campaign (§8).
|
||||
A clean trixie image beats an in-place `apt` dist-upgrade for reproducibility, and the nodes are
|
||||
already cattle (cloud-init + `node_baseline` rebuild them). Accept the per-node Longhorn rebuild
|
||||
there (that campaign *wants* rebuild, throttled at limit=1, since replicas are genuinely gone).
|
||||
- SUC remains a reasonable future option **once the cluster is on fast disks** (the prod/mini-PC
|
||||
Phase-7 cluster) where drain-driven rebuild is cheap; note it for later.
|
||||
|
||||
---
|
||||
|
||||
## 6. Data-validation gates at EVERY step (hard requirement)
|
||||
|
||||
Every **node bounce** (there are 3 per k3s window; 3 per OS window) is wrapped by a pre-step backup
|
||||
gate and a post-step verification gate. Every **window** additionally takes a full cluster backup
|
||||
before the first node.
|
||||
|
||||
### 6.1 PRE-STEP backup gate (before the first node of a window, and re-confirm before each node)
|
||||
|
||||
```bash
|
||||
# 1) etcd on-demand snapshot (k3s native) — the primary rollback artifact. Run on the leader.
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible <leader> -b -m shell \
|
||||
-a "k3s etcd-snapshot save --name pre-1.3X-$(date +%Y%m%d%H%M)"'
|
||||
# verify it exists + is copied off-node (snapshots land in /var/lib/rancher/k3s/server/db/snapshots;
|
||||
# also archived to S3 if configured). Confirm size > 0 and note the name.
|
||||
|
||||
# 2) Velero backup (cluster resources + PVC data via kopia FSB → MinIO 192.168.1.9:9000)
|
||||
./infra/devcontainer/dev.sh run bash -lc 'velero backup create pre-1.3X-$(date +%Y%m%d%H%M) --wait'
|
||||
./infra/devcontainer/dev.sh run bash -lc 'velero backup describe <name> --details' # Phase=Completed, 0 errors
|
||||
|
||||
# 3) CNPG logical backup + verify, for each of keycloak-db / nextcloud-db / semaphore-db
|
||||
# (they already run continuous WAL + daily base backups to MinIO via barmanObjectStore).
|
||||
# Trigger an on-demand base backup and confirm it completes:
|
||||
./infra/devcontainer/dev.sh run kubectl -n <ns> create -f - <<'EOF'
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Backup
|
||||
metadata: { name: pre-upg-<ns>, namespace: <ns> }
|
||||
spec: { method: barmanObjectStore, cluster: { name: <cluster> } }
|
||||
EOF
|
||||
./infra/devcontainer/dev.sh run kubectl -n <ns> get backup pre-upg-<ns> -o jsonpath='{.status.phase}' # completed
|
||||
# capture an integrity baseline to compare after:
|
||||
./infra/devcontainer/dev.sh run kubectl -n <ns> exec <cluster>-1 -- psql -tAc \
|
||||
"select (select count(*) from pg_database), (select sum(('x'||substr(md5(t.oid::text),1,8))::bit(32)::bigint) from pg_class t)"
|
||||
```
|
||||
|
||||
**PASS to proceed:** etcd snapshot present (>0 bytes, name recorded) **AND** Velero backup
|
||||
`Completed` with 0 errors **AND** all three CNPG on-demand backups `completed` with baseline
|
||||
captured.
|
||||
**STOP (do not start the hop):** any backup fails, errors, or cannot be verified restorable.
|
||||
|
||||
### 6.2 The per-node bounce itself (reuse `ssd-etcd-execution-runbook.md`)
|
||||
|
||||
Per node `NODE=k3s-dev-N`, `VMID`, **leader last** (re-verify leader before each node):
|
||||
|
||||
```bash
|
||||
# leader check
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell \
|
||||
-a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_is_leader"'
|
||||
|
||||
# 1) FREEZE Longhorn rebuilds (reattach, not rebuild) — the proven safe-bounce
|
||||
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io \
|
||||
concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"0"}'
|
||||
|
||||
# 2) Upgrade k3s ON THIS NODE ONLY (explicit install, bypassing the creates: guard):
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible $NODE -b -m shell \
|
||||
-a "curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.3X.z+k3s1 sh -s - server"'
|
||||
# (k3s replaces its binary and restarts; embedded etcd re-joins at the new version)
|
||||
|
||||
# 3) VERIFY this node fully before touching the next (see §6.3), THEN
|
||||
# 4) THAW rebuilds only after 18/18 healthy:
|
||||
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io \
|
||||
concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"1"}'
|
||||
```
|
||||
|
||||
### 6.3 POST-STEP verification gate (after each node; full set after the last node of a window)
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run kubectl get nodes -o wide # NODE Ready, VERSION = new minor
|
||||
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd # ok
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell \
|
||||
-a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader"' # all = 1
|
||||
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io \
|
||||
-o custom-columns=R:.status.robustness --no-headers | grep -c healthy # 18
|
||||
./infra/devcontainer/dev.sh run kubectl -n kube-system rollout status ds/cilium --timeout=300s
|
||||
./infra/devcontainer/dev.sh run bash -lc 'flux get kustomizations -A && flux get helmreleases -A' # all Ready
|
||||
# etcd snapshot restorability spot-check (list; a scratch restore is the W-level drill, §7)
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible <leader> -b -m shell \
|
||||
-a "k3s etcd-snapshot ls"'
|
||||
# CNPG integrity vs the §6.1 baseline
|
||||
./infra/devcontainer/dev.sh run kubectl -n <ns> exec <cluster>-1 -- pg_isready # accepting connections
|
||||
# re-run the row-count/checksum query and diff against baseline (must match)
|
||||
```
|
||||
|
||||
**App smoke tests (the QA-sweep URL set)** — after each *window* completes, expect HTTP 200/302
|
||||
(or the app's healthy code) from:
|
||||
`auth` · `crafty` · `gitea` · `grafana` · `headlamp` · `jackett` · `keycloak`
|
||||
(`/realms/homelab/.well-known/openid-configuration`) · `nextcloud` · `pihole` · `plex` ·
|
||||
`qbittorrent` · `radarr` · `seerr` · `semaphore` · `sonarr` · `unifi` — all under
|
||||
`*.home.keithjackson.dev`. Plus one KubeVirt VM check (a VMI Running / console reachable).
|
||||
|
||||
**PASS to advance to the next node/window:** all nodes Ready at the expected version · etcd
|
||||
`/readyz/etcd`=ok and 3/3 have a leader · **18/18** Longhorn healthy (reattached, no rebuild storm)
|
||||
· Cilium DaemonSet rolled out · all Flux Kustomizations + HelmReleases Ready · CNPG `pg_isready` +
|
||||
row-count/checksum unchanged vs baseline · QA-sweep all green.
|
||||
**STOP (halt, leave freeze=0, do not advance):** etcd `/readyz` not ok · a member without a leader ·
|
||||
VIP `192.168.2.50:6443` flapping · a node not rejoining/not Ready · any Longhorn volume Faulting ·
|
||||
k3s crash-looping · a CNPG DB failing `pg_isready` or checksum drift · a Flux HR stuck not-Ready ·
|
||||
any QA-sweep URL down. Recovery per §7.
|
||||
|
||||
---
|
||||
|
||||
## 7. Rollback per step & point-of-no-return
|
||||
|
||||
### 7.1 Per-node (before etcd majority is on the new minor)
|
||||
|
||||
The safest rollback for a single bad node is to **re-pin the old k3s version on that node** and
|
||||
re-run the install script for the old tag, then verify it rejoins at the old minor; or reset the
|
||||
member and rejoin from the other two (2/3 quorum holds), per `ssd-etcd-isolation.md` §4.5. Because
|
||||
you go **leader-last and one node at a time**, at any moment ≤1 node is on the new minor until the
|
||||
2nd node upgrades.
|
||||
|
||||
### 7.2 Point-of-no-return per hop
|
||||
|
||||
**When the *second* server reaches the new minor, etcd majority (2/3) is on the new datastore
|
||||
version** (etcd 3.5→3.6 across the path is one-way). Past that point, "roll back the node" is no
|
||||
longer enough — a true rollback is an **etcd snapshot restore** of the whole cluster to the pre-hop
|
||||
snapshot:
|
||||
|
||||
```bash
|
||||
# cluster-wide restore from the pre-hop snapshot (DISRUPTIVE — full control-plane stop):
|
||||
# on ONE server, with k3s stopped on all three:
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -b -m shell -a "systemctl stop k3s"'
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible <node1> -b -m shell \
|
||||
-a "k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/<pre-1.3X-snap>"'
|
||||
# then wipe db on the other two and rejoin them (standard k3s cluster-reset-restore flow),
|
||||
# and PIN k3s_version back to the old tag before they re-join.
|
||||
```
|
||||
|
||||
After an etcd restore, **Longhorn volume data is recovered by Velero (kopia FSB) / CNPG PITR** for
|
||||
anything written between snapshot and restore — this is why the §6.1 triad (etcd + Velero + CNPG)
|
||||
is mandatory, not just etcd.
|
||||
|
||||
### 7.3 Window-level DR drill (do once, in W1, to prove the path)
|
||||
|
||||
Before the first real hop, prove restorability end-to-end: `velero restore create --from-backup
|
||||
<pre-snap> --namespace-mappings <ns>:<ns>-scratch` into a scratch namespace and confirm data lands
|
||||
(this is the existing Phase-4 R-STATE-03 gate). If the drill fails, **do not begin the upgrade.**
|
||||
|
||||
### 7.4 OS-campaign rollback
|
||||
|
||||
Re-provision is inherently reversible per node: if a re-provisioned trixie node misbehaves,
|
||||
`tofu` re-apply with `image_source` reverted to bookworm re-creates it on Debian 12; the node
|
||||
rejoins etcd and rebuilds replicas as with any replacement. Point-of-no-return is the same etcd
|
||||
majority rule if OS re-provision is (against advice) combined with a k3s bump — which is why D3 says
|
||||
keep them separate.
|
||||
|
||||
---
|
||||
|
||||
## 8. Node OS upgrade — Debian 12 → 13 (trixie)
|
||||
|
||||
- **Debian 13 "trixie" released 2025-08-09, ships Linux 6.12 LTS** (up from bookworm's 6.1).
|
||||
Support to 2028-08 (LTS 2030-08). Sources: <https://www.debian.org/News/2025/20250809> ·
|
||||
<https://www.debian.org/releases/trixie/>.
|
||||
- **Kernel implications (positive):** 6.12 is newer than Cilium's/Longhorn's minimums and improves
|
||||
eBPF/BPF feature coverage for Cilium and has current `iscsi_tcp`/`nfs`/`dm_crypt`/`overlay`/
|
||||
`br_netfilter` modules Longhorn needs (all already pinned in
|
||||
`roles/node_baseline/defaults/main.yml` → `node_baseline_kernel_modules`). No known regression for
|
||||
this stack; `open-iscsi`/`nfs-common` packages exist on trixie.
|
||||
- **Method (D2): immutable re-provision.** Bump `infra/tofu/variables.tf`:
|
||||
- `image_source` → `https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2`
|
||||
- `image_file_name` → `debian-13-generic-amd64.qcow2`
|
||||
Then `tofu apply -replace=proxmox_virtual_environment_vm.node["k3s-dev-N"]` **one node at a time**,
|
||||
let `node_baseline` + `k3s_server` (secondary/join) rebuild it, and let it rejoin etcd. Because a
|
||||
replaced node loses its Longhorn replicas, run this campaign with rebuild **throttled at 1** (not
|
||||
frozen at 0 — you *want* the rebuild here), wait for 18/18 healthy before the next node, and keep
|
||||
the same per-node backup + verify gates (§6).
|
||||
- **Coordinate with k3s (D3):** run the OS campaign **after** the cluster is stable on a chosen k3s
|
||||
minor (either finish the whole 1.36 ladder first, or interleave one full campaign between k3s
|
||||
windows) — **never** re-provision to trixie *and* bump k3s in the same node bounce; that entangles
|
||||
two failure domains and defeats per-step rollback.
|
||||
- **In-place `apt` dist-upgrade** is the fallback if re-provision is undesirable, but it is discarded
|
||||
here: it mutates cattle nodes outside the tofu/cloud-init source of truth, risks half-migrated
|
||||
states across three nodes, and violates the repo's "changes go through IaC" principle.
|
||||
|
||||
---
|
||||
|
||||
## 9. Sequencing & HA safety
|
||||
|
||||
- **Quorum:** 3 servers, embedded etcd, quorum 2/3. Exactly **one** node down at a time keeps
|
||||
quorum. Never bounce two simultaneously.
|
||||
- **Leader last:** re-verify the etcd leader before each node (`etcd_server_is_leader` on :2381) and
|
||||
upgrade followers first, leader last, so leadership only moves once and late.
|
||||
- **VIP stability:** kube-vip floats 192.168.1.49 to the current leader; with one node down the VIP
|
||||
fails over to a surviving leader and the API stays reachable through the pfSense NAT
|
||||
(192.168.2.50:6443). Watch for VIP flapping as a STOP signal.
|
||||
- **Rebuild-freeze (reattach, not rebuild):** for k3s in-place hops set
|
||||
`concurrent-replica-rebuild-per-node-limit=0` around each bounce so a briefly-down node's replicas
|
||||
**reattach** instead of triggering a rebuild storm that starves etcd WAL fsync (the whole reason
|
||||
etcd is on a dedicated SSD). Thaw to `1` only after 18/18 healthy. (For the OS re-provision
|
||||
campaign use `1`, not `0`, since replicas are genuinely gone and must rebuild.)
|
||||
- **CNPG note:** all three DBs are **single-instance** (`instances: 1`) — each incurs a brief
|
||||
outage when its pod reschedules during a node bounce. Order/announce accordingly; the app
|
||||
smoke-tests (§6.3) will blip while a DB pod moves. (Consider `instances: 3` for the prod cluster.)
|
||||
- **KubeVirt/CDI:** already 1.36-ready; VMs without `evictionStrategy: LiveMigrate` simply
|
||||
Evict+restart on a node bounce (we do **not** drain in method (b), so VMs are only affected on
|
||||
the node actually bouncing).
|
||||
|
||||
**Estimated maintenance windows:** per k3s minor ≈ **60-120 min** (3 nodes × ~15-30 min bounce +
|
||||
verify + freeze/thaw + component reconcile), so **5 windows ≈ 5-10 h** total spread across days,
|
||||
plus the W-level backups. OS re-provision campaign ≈ **2-4 h per node** (dominated by Longhorn
|
||||
rebuild of that node's share of 18 volumes on HDD) → ~1-1.5 days, one node per session.
|
||||
|
||||
---
|
||||
|
||||
## 10. Dev-first, then prod (Phase-7 carry-over)
|
||||
|
||||
This dev run is the rehearsal for the eventual **prod / mini-PC** cluster. What carries over:
|
||||
- The **method** (scripted in-place per-node for k3s; immutable re-provision for OS), the
|
||||
**leader-last + freeze + per-node-gate** discipline, and the **backup triad** gate are
|
||||
cluster-agnostic and become the prod runbook verbatim.
|
||||
- The **component lockstep table** (§4.2) is the reusable dependency map; re-confirm exact patch
|
||||
numbers at prod time (they will have moved).
|
||||
- **Differences to expect on prod:** fast disks (NVMe/SSD) make Longhorn rebuild cheap → **SUC
|
||||
becomes viable** there (drain no longer dangerous), and multi-instance CNPG (`instances: 3`)
|
||||
removes the single-DB outage blip. Prod may also run dedicated workers, changing SUC Plan
|
||||
server-vs-agent ordering.
|
||||
- The **Flux v1beta2→v1 migration** (§3.2) is a one-time repo change shared by both clusters — do it
|
||||
once, both benefit.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A — pre-flight checklist per window (copy/paste)
|
||||
|
||||
```
|
||||
[ ] Read k3s release notes for the target minor + k8s CHANGELOG "Urgent Upgrade Notes"
|
||||
[ ] pluto/kubent scan (live + manifests) against target minor → clean (or Flux v1beta2 handled)
|
||||
[ ] Component pre-reqs landed & Ready on CURRENT k8s (Cilium minor, Gateway CRDs, Longhorn minor, …)
|
||||
[ ] Cilium pre-flight check DaemonSet passed for the Cilium minor bump
|
||||
[ ] PRE-STEP backup gate PASS: etcd snapshot + Velero Completed + 3× CNPG completed + baselines captured
|
||||
[ ] (W1 only) Velero restore DR drill into scratch ns PASS
|
||||
[ ] Leader identified; node order = followers → leader
|
||||
[ ] For each node: freeze(0) → upgrade → §6.3 verify PASS → thaw(1) → 18/18 healthy → next
|
||||
[ ] After last node: full §6.3 + QA-sweep all green + KubeVirt VM check
|
||||
[ ] Land opportunistic/loose component bumps; Flux all Ready
|
||||
[ ] Pin the new k3s_version in roles/k3s_server/defaults/main.yml; PR → main (never commit to main)
|
||||
[ ] Record outcome in DECISIONS-LOG.md
|
||||
```
|
||||
|
||||
## Appendix B — sources
|
||||
|
||||
- k3s releases & notes: <https://github.com/k3s-io/k3s/releases> · <https://docs.k3s.io/release-notes/v1.36.X> · <https://docs.k3s.io/release-notes/v1.33.X> · channels <https://update.k3s.io/v1-release/channels>
|
||||
- k8s version-skew: <https://kubernetes.io/releases/version-skew-policy/> · deprecation guide <https://kubernetes.io/docs/reference/using-api/deprecation-guide/> · v1.36 sneak peek <https://www.kubernetes.io/blog/2026/03/30/kubernetes-v1-36-sneak-peek/>
|
||||
- Cilium compat: <https://docs.cilium.io/en/stable/network/kubernetes/compatibility/> · <https://compatibility.fyi/projects/cilium/> · Gateway API <https://docs.cilium.io/en/v1.20/network/servicemesh/gateway-api/gateway-api/>
|
||||
- Longhorn: <https://endoflife.date/longhorn> · <https://longhorn.io/docs/> · <https://www.suse.com/suse-longhorn/support-matrix/> · <https://github.com/longhorn/longhorn/wiki/Version-Update-Policy>
|
||||
- KubeVirt/CDI: <https://github.com/kubevirt/sig-release/blob/main/releases/k8s-support-matrix.md> · <https://github.com/kubevirt/kubevirt/blob/main/docs/kubernetes-compatibility.md>
|
||||
- external-snapshotter: <https://github.com/kubernetes-csi/external-snapshotter/releases>
|
||||
- cert-manager: <https://cert-manager.io/docs/releases/>
|
||||
- CNPG: <https://cloudnative-pg.io/docs/devel/supported_releases/>
|
||||
- Flux: <https://fluxcd.io/flux/releases/> · v2.7 upgrade <https://github.com/fluxcd/flux2/discussions/5572> · image API <https://fluxcd.io/flux/components/image/imagerepositories/>
|
||||
- kube-prometheus-stack: <https://github.com/prometheus-community/helm-charts>
|
||||
- Velero: <https://velero.io/docs/main/> · <https://compatibility.fyi/projects/velero/>
|
||||
- Kyverno: <https://kyverno.io/docs/installation/releases/>
|
||||
- kube-vip: <https://github.com/kube-vip/kube-vip/releases> · <https://kube-vip.io/docs/upgrade/>
|
||||
- Debian 13: <https://www.debian.org/News/2025/20250809> · <https://www.debian.org/releases/trixie/>
|
||||
- Tools: pluto <https://github.com/FairwindsOps/pluto> · kubent <https://github.com/doitintl/kube-no-trouble>
|
||||
```
|
||||
@@ -1,185 +0,0 @@
|
||||
# L2/L3 conformance CI wiring — implementation plan
|
||||
|
||||
**Status:** PLAN — deferred, awaiting operator decision on the trigger-credential model. No
|
||||
cluster changes made.
|
||||
**Author:** WS-GITOSS agent, 2026-08-31 (Hand-off 2).
|
||||
**Scope:** wire the L2 (behavioural) + L3 (fault-injection) conformance layers into the LIVE
|
||||
Gitea Actions merge gate, running them as an **in-cluster Chainsaw Job** so CI holds **no broad
|
||||
kubeconfig** (docs/14 "Where each layer runs", ADR-0011, R-SEC-02).
|
||||
|
||||
> This companions the landed Hand-off 1 (commit `27bc024`: `make -C ci gate` parity + the
|
||||
> comment-only R-ID scan fix). The `conformance` job in `.gitea/workflows/gate.yml` is still an
|
||||
> honest `if: false` placeholder — this plan is what turns it on **safely**.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this is a written plan, not a landed change
|
||||
|
||||
The intended seam is real and mostly scaffolded, but finishing it autonomously would mean
|
||||
handing a shared, **live** cluster a new privileged identity reachable from CI, plus building
|
||||
several missing functional pieces. The hand-off's rule is explicit: *if wiring real cluster
|
||||
access from CI is too risky/ambiguous to finish autonomously, STOP and write a concrete plan —
|
||||
do NOT hand CI a broad kubeconfig.* The blocker is a genuine **privilege-escalation** design
|
||||
question (§4), not just plumbing, and the cluster is being changed concurrently by sibling
|
||||
agents. So the credential model must be decided by the operator before anything is applied.
|
||||
|
||||
---
|
||||
|
||||
## 2. Current state (what exists today)
|
||||
|
||||
| Piece | File | State |
|
||||
|---|---|---|
|
||||
| Live gate workflow | `.gitea/workflows/gate.yml` | LIVE. `gate` job runs `make -C ci gate`. `conformance` job is `if: false` placeholder. |
|
||||
| Dormant full blueprint | `.gitea/workflows/merge-gate.yaml` | Shows the intended shape: `conformance` job `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `secrets.CONFORMANCE_TRIGGER_KUBECONFIG`, `make -C ci conformance`, then a `merge-gate` decision job (`scripts/decide.sh`). Manual-only (`workflow_dispatch`). |
|
||||
| CI conformance stage | `ci/conformance.sh` | SKIPS (exit 3) unless `CLUSTER_AVAILABLE=true`. When "on", it currently runs `conformance/run.sh --layer l2,l3` **locally in the CI job** and collects `_artifacts/`. Its own header carries `TODO(Phase 1): drive the in-cluster Chainsaw Job under the narrow CONFORMANCE_TRIGGER_KUBECONFIG` — i.e. the trigger path is **not implemented**. |
|
||||
| Conformance runner | `conformance/run.sh` + `conformance/lib/`, `conformance/tests/` | Works locally/in-cluster. `--layer l2,l3 [--requirements ...]`, writes `_artifacts/report.junit.xml` + `coverage/`. Needs bash + python3 + chainsaw (+ kubectl for `conf_cluster_reachable`). |
|
||||
| In-cluster Job skeleton | `conformance/incluster/chainsaw-job.yaml` | Skeleton: `generateName: conformance-run-`, ns `conformance`, `serviceAccountName: conformance-runner`, image `ghcr.io/kyverno/chainsaw:v0.2.12`, `command: ["./run.sh"]`, args default `--layer all --strict-coverage`. **`workspace` is an empty `emptyDir` with no init-container to populate it**, and `results` is an `emptyDir`. |
|
||||
| Scoped runner RBAC | `conformance/incluster/rbac.yaml` | ns `conformance` + SA `conformance-runner` + a **privileged** ClusterRole (namespaces create/delete/patch, pods+pods/exec, jobs create/delete, **nodes get/list/patch** for R-STATE-02 cordon, pods/log). Least-privilege, per-test-justified. **Not reconciled by Flux** (no kustomization references `conformance/incluster/`). |
|
||||
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
|
||||
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
|
||||
|
||||
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
|
||||
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
|
||||
|
||||
---
|
||||
|
||||
## 3. Gaps to close (functional)
|
||||
|
||||
1. **Runner image.** `ghcr.io/kyverno/chainsaw` is a minimal image; `run.sh` needs **bash +
|
||||
python3 + chainsaw + kubectl**, and the repo-clone init needs **git**. The Job cannot run
|
||||
`./run.sh` on the stock chainsaw image. → Build a pinned `conformance-runner` image (chainsaw
|
||||
`v0.2.12` + kubectl `v1.36.4` + git + python3 + bash), push to the Gitea container registry,
|
||||
pin its digest in `conformance/versions.env` (`CONFORMANCE_RUNNER_IMAGE`). Keep
|
||||
`CHAINSAW_VERSION` as the single source of truth the image is built from.
|
||||
2. **Repo at the commit-under-test.** The Job must run the suite at the PR head SHA. → Add an
|
||||
**init-container** (`git clone --depth 1` of the impl repo, then `git checkout <SHA>`) writing
|
||||
into the shared `workspace` volume, using a **read-only** Gitea deploy token. SHA + the touched
|
||||
`--requirements` are injected by the CI trigger step as Job env/args.
|
||||
3. **Results collection without giving CI exec/cp.** The Job writes `_artifacts/` to the `results`
|
||||
volume; CI holds only trigger/read rights (§4). → Job's last step prints a delimited
|
||||
base64 tar of `_artifacts/` to stdout (`=== ARTIFACTS-TGZ-BASE64 ===`…); CI decodes it from
|
||||
`kubectl logs`. No `pods/exec`, no `kubectl cp`, no results PVC exposure required. (A results
|
||||
PVC + a read-only sidecar is the durable alternative if log size becomes a problem.)
|
||||
4. **`ci/conformance.sh` rewrite.** Replace the local `run.sh` call with: render the Job manifest
|
||||
(inject image digest, SHA, `--requirements <touched>`), `kubectl --kubeconfig=$CONFORMANCE_TRIGGER_KUBECONFIG
|
||||
-n conformance create -f -`, `kubectl wait --for=condition=complete/failed`, stream logs,
|
||||
decode artifacts into `results/junit` + `reports/coverage`, map Job success→0 / failure→1 /
|
||||
no-cred→3 (skip). Keep the touched-subset selection it already computes.
|
||||
5. **Flux wiring.** Reconcile `conformance/incluster/` (ns + `conformance-runner` SA/RBAC + the
|
||||
new trigger SA/Role + the admission policy of §4) via a **non-gating** Kustomization
|
||||
(`wait:false`, `dependsOn: platform-config`), mirroring `apps-vms` — a conformance identity
|
||||
must never gate the app tier.
|
||||
6. **Gitea provisioning (operator, out-of-GitOps).** Set repo/org `vars.CLUSTER_AVAILABLE=true`
|
||||
and `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` via the Gitea API
|
||||
(`PUT /repos/{owner}/{repo}/actions/{variables,secrets}/{name}`). The kubeconfig embeds the
|
||||
trigger SA's token only. SOPS-seal the SA token in-repo (`secrets/conformance-trigger.sops.yaml`)
|
||||
for reproducibility/rotation; the Gitea secret is derived from it.
|
||||
|
||||
---
|
||||
|
||||
## 4. The security crux — trigger credential must NOT grant `conformance-runner`'s power
|
||||
|
||||
**Problem.** Kubernetes places no `use` check on a Pod/Job's `serviceAccountName` at creation.
|
||||
So **any principal that can `create jobs` in namespace `conformance` can create a Job with
|
||||
`serviceAccountName: conformance-runner` and an arbitrary `command`** — i.e. run arbitrary code
|
||||
with the privileged runner SA (nodes/patch, pods/exec, jobs, namespaces/patch). A CI-held token
|
||||
with plain `create jobs` therefore *is* the privileged identity. That breaks R-SEC-02 ("CI holds
|
||||
no broad cluster access") even though the token's RBAC "looks" narrow.
|
||||
|
||||
**Recommended model — narrow trigger SA + an admission constraint (uses the existing Kyverno L0
|
||||
stack):**
|
||||
|
||||
- SA `conformance-trigger` (ns `conformance`), Role: `create` on `jobs`; `get,list,watch` on
|
||||
`jobs`; `get` on `pods` and `pods/log`. Nothing else. Bound only in the `conformance` ns.
|
||||
- A **Kyverno ClusterPolicy (validate, enforce, fail-closed)** that, *for Jobs created by
|
||||
`system:serviceaccount:conformance:conformance-trigger`*, ADMITS only Jobs that match the
|
||||
committed shape: image == the pinned `CONFORMANCE_RUNNER_IMAGE` digest,
|
||||
`serviceAccountName == conformance-runner`, `command == ["./run.sh"]`, `args` matching
|
||||
`^--(layer|requirements|strict-coverage)`, `securityContext` non-root/no-privilege/no host
|
||||
namespaces, and no extra volumes beyond `workspace`/`results`. Any other Job from that SA is
|
||||
DENIED. This is the real security boundary: it bounds CI to "run exactly the committed
|
||||
conformance suite, nothing else." (Equivalent: a k8s ValidatingAdmissionPolicy bound to that SA.)
|
||||
- The `chainsaw-conf-*` per-test namespaces the runner creates are already flagged in
|
||||
`rbac.yaml` as needing exclusion from any cluster-wide disallow-privileged/host-namespaces
|
||||
policy for R-NET-06 — carry that exception through, documented and audited.
|
||||
|
||||
**Alternative model — GitOps-native, zero CI cluster credential (more work, strongest):** CI does
|
||||
not reach the cluster at all. A `ConformanceRun` marker (a CR or a commit to a watched path) is
|
||||
reconciled by an in-cluster controller/Flux that creates the Job from the in-repo template; the
|
||||
Job publishes results back as a Gitea commit-status / a fetchable artifact that CI polls. Removes
|
||||
the trigger credential entirely, at the cost of a controller and a trigger/ordering mechanism that
|
||||
must complete before the merge decision. Recommended only if even the constrained trigger SA of
|
||||
the primary model is deemed unacceptable.
|
||||
|
||||
**Do NOT:** give CI a `cluster-admin`/broad kubeconfig; give the trigger SA unconstrained
|
||||
`create jobs`; mount an SA token into the act_runner pod; or reuse `conformance-runner` as the
|
||||
trigger identity.
|
||||
|
||||
---
|
||||
|
||||
## 5. Concrete step sequence
|
||||
|
||||
Land these as separate reviewable PRs, gate-green at each step. Steps 1–5 are in-repo/GitOps and
|
||||
safe to land incrementally *without* turning L2/L3 on; only step 7 (the operator setting the Gitea
|
||||
var) actually enables it.
|
||||
|
||||
1. **Runner image.** `conformance/incluster/Containerfile` (chainsaw+kubectl+git+python3+bash);
|
||||
build+push to the Gitea registry; pin `CONFORMANCE_RUNNER_IMAGE=<registry>/conformance-runner@sha256:…`
|
||||
in `conformance/versions.env`. Add a note tying it to `CHAINSAW_VERSION`.
|
||||
2. **Job manifest.** Rewrite `chainsaw-job.yaml`: swap to `CONFORMANCE_RUNNER_IMAGE`; add the
|
||||
git-clone init-container (SHA + repo URL + read token env); parametrise `args` (CI injects
|
||||
`--layer l2,l3 --requirements <touched>`); keep `restartPolicy: Never`, `backoffLimit: 0`,
|
||||
non-root/seccomp/readonly-rootfs; add sane CPU/mem requests+limits (heavy L3 etcd load —
|
||||
see `run.sh` HEAVY set + `config/chainsaw-config.yaml`).
|
||||
3. **Trigger SA/RBAC + admission policy.** Add `conformance-trigger` SA + Role/RoleBinding to
|
||||
`rbac.yaml`; add the Kyverno ClusterPolicy (new file under `platform/kyverno/` or
|
||||
`conformance/incluster/`) constraining that SA's Jobs (§4). Add an L0 Kyverno test fixture for
|
||||
the policy (admit the good Job, deny a tampered one) so `make -C ci l0` covers it.
|
||||
4. **Flux Kustomization.** New `clusters/<cluster>/conformance.yaml` (or an `apps-conformance`
|
||||
entry) — `wait:false`, `dependsOn: platform-config`, `prune:true`, pointing at
|
||||
`conformance/incluster/`. No `postBuild.substituteFrom` unless needed. Verify it reconciles
|
||||
Ready with the SA/RBAC/policy present and NO Job running (Jobs are created only on trigger).
|
||||
5. **`ci/conformance.sh` rewrite.** Implement the trigger→wait→collect→map flow of §3.4 behind
|
||||
`CONFORMANCE_TRIGGER_KUBECONFIG`; keep exit 3 when the cred is absent (honest skip). Update
|
||||
`ci/README.md`. `make -C ci gate` still SKIPS L2/L3 locally (no cred) — unchanged for humans.
|
||||
6. **Workflow.** Replace the `if: false` placeholder in `gate.yml`'s `conformance` job with the
|
||||
blueprint shape: `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}`, `needs: [gate]`, install the
|
||||
`conformance` toolchain, `env: CONFORMANCE_TRIGGER_KUBECONFIG: ${{ secrets.CONFORMANCE_TRIGGER_KUBECONFIG }}`,
|
||||
`run: make -C ci conformance`. Decide: either keep the single-`gate` model (aggregate `gate.sh`
|
||||
already SKIPS conformance and reports UNVERIFIED) and add conformance as a **second required
|
||||
check**, or adopt the blueprint's separate `merge-gate` decision job (`scripts/decide.sh`).
|
||||
Keep the job honest: skipped when the var is off, never faked green (HARD RULE 4). Add the
|
||||
conformance check to branch protection only after step 7 proves it green.
|
||||
7. **Enable (operator).** Provision the SOPS-sealed trigger SA token; set the Gitea
|
||||
`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` and `vars.CLUSTER_AVAILABLE=true`. First run on a
|
||||
throwaway PR; confirm the Job is created, runs the touched subset, results are collected, and
|
||||
the gate blocks on a real L2/L3 failure and passes on green. Then make it required.
|
||||
|
||||
---
|
||||
|
||||
## 6. Validation & rollback
|
||||
|
||||
- **Dry-run the admission policy** before granting the trigger SA: `kubectl auth can-i` matrix for
|
||||
`conformance-trigger`; attempt a *tampered* Job (different command) as that SA and confirm
|
||||
Kyverno DENIES it; attempt the committed Job and confirm ADMIT.
|
||||
- **Fail-closed check:** temporarily break an L3 test, confirm the Job fails and the gate BLOCKS
|
||||
(mirrors Hand-off 1's proof that `gate.sh` exit 1 fails the job).
|
||||
- **Rollback:** unset `vars.CLUSTER_AVAILABLE` (instant SKIP, gate reverts to L0/L1-only) and/or
|
||||
suspend the Flux Kustomization; delete the trigger SA token from Gitea secrets. No app-tier
|
||||
impact (conformance is a non-gating tier).
|
||||
|
||||
---
|
||||
|
||||
## 7. Open decisions for the operator
|
||||
|
||||
1. **Trigger model:** constrained trigger SA + Kyverno admission policy (§4 primary) vs.
|
||||
GitOps-native zero-credential (§4 alternative). This plan recommends the primary.
|
||||
2. **Runner image host:** Gitea container registry (in-network, matches the Flux source loop) vs.
|
||||
ghcr.io. Recommend Gitea registry, digest-pinned.
|
||||
3. **Gate topology:** conformance as a second required check on the single-job `gate.yml` vs.
|
||||
restoring the blueprint's multi-job `merge-gate.yaml` with the `decide.sh` decision job.
|
||||
4. **Results transport:** base64-tar-over-logs (simple, no extra rights) vs. a results PVC +
|
||||
read-only sidecar (durable). Recommend logs first; revisit if artifacts grow.
|
||||
5. **Which requirements go live first:** the R-NET/R-TLS/R-DNS/R-STATE tests already green against
|
||||
the live cluster (per DECISIONS-LOG) are the natural first `--requirements` subset to certify
|
||||
through the wired path before widening.
|
||||
@@ -1,468 +0,0 @@
|
||||
# Longhorn SSD tiering for the k3s dev cluster
|
||||
|
||||
**Status:** PROPOSAL — awaiting user approval. No storage, tofu, Longhorn, or live
|
||||
infrastructure has been changed. This is a read-only investigation + reviewable plan.
|
||||
**Author:** investigation agent (read-only), 2026-08-31
|
||||
**Problem:** All 18 Longhorn data volumes live on the slow/failing Seagate-class HDD
|
||||
pool (`Storage`). Every node reboot marks that node's replicas stale and triggers a
|
||||
full 18-volume rebuild that spikes Proxmox IO-delay to 30 %+ and runs for a long time,
|
||||
making every upgrade / resize / reboot expensive and risky. etcd was already isolated
|
||||
onto a dedicated per-node SSD disk (`virtio2`, `local-zfs`) for exactly this reason
|
||||
(`docs/plans/ssd-etcd-isolation.md`). The recorded blocker was that the SSD "does not
|
||||
fit" the 3 × 128 GiB Longhorn data disks, so Longhorn stayed on HDD "interim until the
|
||||
all-SSD mini-PCs land."
|
||||
|
||||
**Verdict (short):** The "doesn't fit" blocker was computed against the **provisioned**
|
||||
128 GiB disks. The **actual working set is ~26 GB per node (~79 GB across all three
|
||||
nodes)** — bulk media is on NFS, *not* Longhorn, so every Longhorn volume is a small
|
||||
config/DB volume. **A hot-tier move onto SSD is viable and recommended**; a full move
|
||||
is also feasible on space but re-couples rebuild I/O with etcd/host on the same SSD
|
||||
pool. Recommendation: **Phase 1 hot-tier move (Prometheus + the CNPG DBs + Loki +
|
||||
gitea) onto a tagged `longhorn-ssd` disk**, one node at a time.
|
||||
|
||||
---
|
||||
|
||||
## 1. Measured facts (all collected read-only, 2026-08-31)
|
||||
|
||||
### 1.1 The Longhorn working set — provisioned vs *actual*
|
||||
|
||||
`kubectl -n longhorn-system get volumes.longhorn.io` — capacity (`spec.size`) vs
|
||||
**actual consumed** (`status.actualSize`, the real on-disk footprint of one replica,
|
||||
snapshots included). All 18 volumes are `numberOfReplicas: 3`, so each node holds one
|
||||
replica of each and a node's on-disk footprint ≈ the actualSize column below.
|
||||
|
||||
| Volume / PVC | Namespace | App class | Provisioned | **Actual (per replica)** | Tier |
|
||||
|---|---|---|---:|---:|---|
|
||||
| prometheus-…-db-0 | monitoring | Prometheus TSDB | 8 GiB | **11.59 GB** | HOT |
|
||||
| unifi-rootdisk | unifi | KubeVirt VM disk (`longhorn-vm`) | 40 GiB | 6.20 GB | cold |
|
||||
| nextcloud-data | nextcloud | file data | 20 GiB | 1.44 GB | cold |
|
||||
| nextcloud-db-1 | nextcloud-db | CNPG Postgres | 5 GiB | 1.41 GB | HOT |
|
||||
| storage-loki-0 | monitoring | Loki chunks/index | 5 GiB | 1.24 GB | HOT |
|
||||
| keycloak-db-1 | keycloak | CNPG Postgres | 5 GiB | 0.93 GB | HOT |
|
||||
| gitea-data | gitea | git repos + SQLite/meta | 10 GiB | 0.90 GB | HOT |
|
||||
| semaphore-db-1 | semaphore-db | CNPG Postgres | 5 GiB | 0.79 GB | HOT |
|
||||
| crafty-data | crafty | Minecraft world | 20 GiB | 0.54 GB | cold |
|
||||
| plex-config | plex | Plex metadata/db | 8 GiB | 0.23 GB | cold |
|
||||
| seerr-config | seerr | config | 5 GiB | 0.17 GB | cold |
|
||||
| sonarr-config | sonarr | config db | 5 GiB | 0.15 GB | cold |
|
||||
| qbittorrent-config | qbittorrent | config | 5 GiB | 0.15 GB | cold |
|
||||
| radarr-config | radarr | config db | 5 GiB | 0.15 GB | cold |
|
||||
| pihole-config | pihole | config | 2 GiB | 0.10 GB | cold |
|
||||
| jackett-config | jackett | config | 2 GiB | 0.10 GB | cold |
|
||||
| seerr/act-runner-data | gitea-actions | CI cache | 1 GiB | 0.05 GB | cold |
|
||||
| semaphore-config | semaphore | config | 1 GiB | 0.05 GB | cold |
|
||||
| **TOTALS (per node / one replica)** | | | **152 GiB** | **~26.2 GB** | |
|
||||
|
||||
- **Per node working set: ~26.2 GB actual** vs **152 GiB provisioned** (× 3 replicas =
|
||||
456 GiB nominal, but Longhorn/ZFS are thin — nominal is not what lands on disk).
|
||||
- **Whole-cluster working set: ~79 GB** across the three nodes.
|
||||
- **Bulk media is not on Longhorn.** The StorageClass survey (§1.3) shows Plex media
|
||||
libraries and the *arr download shares use the `bulk` NFS class
|
||||
(`nfs.csi.k8s.io`), not Longhorn. So the "bulk stays on HDD while small/hot moves to
|
||||
SSD" framing mostly dissolves: **there is no large Longhorn volume.** The biggest is
|
||||
`unifi-rootdisk` at 6.2 GB actual, and it is a cold KubeVirt root disk.
|
||||
|
||||
### 1.2 Longhorn per-node disk (the current single HDD data disk)
|
||||
|
||||
`kubectl -n longhorn-system get nodes.longhorn.io` (all three nodes identical):
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Disk | `default-disk`, path `/var/lib/longhorn`, tags `[]`, `allowScheduling: true` |
|
||||
| `storageMaximum` | 124.9 GiB (the ext4 fs on the 128 GiB `virtio1` HDD disk) |
|
||||
| `storageAvailable` | 100.4 GiB free |
|
||||
| `storageScheduled` | **152.0 GiB** — over-scheduled vs the 124.9 GiB max |
|
||||
|
||||
The 152 GiB *scheduled* > 124.9 GiB *max* is Longhorn honouring
|
||||
`storage-over-provisioning-percentage: 200`. Actual usage is only ~24.5 GiB
|
||||
(124.9 − 100.4), matching §1.1.
|
||||
|
||||
### 1.3 StorageClasses in play
|
||||
|
||||
`kubectl get storageclass`:
|
||||
|
||||
| Class | Provisioner | Notes |
|
||||
|---|---|---|
|
||||
| `longhorn` | `driver.longhorn.io` | default class, 3 replicas, ext4 — **17 of 18 PVCs** |
|
||||
| `longhorn-vm` | `driver.longhorn.io` | 3 replicas, `migratable: true` — `unifi-rootdisk` only |
|
||||
| `longhorn-scratch` | `driver.longhorn.io` | 1 replica, best-effort locality (ephemeral) |
|
||||
| `longhorn-static` | `driver.longhorn.io` | static-provisioning class |
|
||||
| `bulk` | `nfs.csi.k8s.io` | NFS — bulk media (unresolved `REPLACE-ME` server/share) |
|
||||
|
||||
### 1.4 Longhorn settings relevant to a migration / rebuild storm
|
||||
|
||||
`kubectl -n longhorn-system get settings.longhorn.io …`:
|
||||
|
||||
| Setting | Value | Relevance |
|
||||
|---|---|---|
|
||||
| `concurrent-replica-rebuild-per-node-limit` | **1** | already throttled — one rebuild at a time per node (set during etcd work) |
|
||||
| `storage-over-provisioning-percentage` | 200 | dangerous on a *small* SSD disk — must lower for the SSD disk sizing |
|
||||
| `storage-minimal-available-percentage` | 25 | Longhorn stops scheduling when a disk is <25 % free — a safety floor |
|
||||
| `replica-auto-balance` | best-effort | will help spread replicas after a new disk appears |
|
||||
| `default-replica-count` | 3 | — |
|
||||
|
||||
Longhorn version: **v1.12.1**. **No backup target is configured**
|
||||
(`backuptargets.longhorn.io/default`: URL empty, `available: false`) — so any
|
||||
migration path that relies on Longhorn *backup → restore-to-new-SC* is **not
|
||||
available** until an S3/NFS backup target is set up. This constrains the migration
|
||||
mechanism (§4) to in-cluster options: CSI clone, per-volume `diskSelector` rebuild, or
|
||||
app-level redeploy.
|
||||
|
||||
### 1.5 The SSD pool (`local-zfs` = `rpool/data`)
|
||||
|
||||
Measured read-only on 2026-08-29 during the etcd-isolation work
|
||||
(`docs/plans/ssd-etcd-isolation.md` §1.1) and unchanged in substance since — the only
|
||||
addition was the three 16 GiB etcd disks on 2026-08-30, which are thin and hold ~83 MB
|
||||
of etcd data each (~250 MB total actual):
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Pool | `rpool`, mirror of 2 × Micron 5100/52x0 960 GB **enterprise SATA SSD w/ PLP** |
|
||||
| Health | 97 % lifetime remaining, 0 reallocated sectors — excellent |
|
||||
| `zpool` size / free | 888 G / **~350 G free** (60 % capacity) |
|
||||
| ZFS `AVAIL` to a new dataset/zvol | **~323 GiB** |
|
||||
| Proxmox host root on it | `rpool/ROOT/pve-1` = 430 G |
|
||||
| Storage entry | `local-zfs`, content `images,rootdir`, `sparse 1` (thin) |
|
||||
| Already carries | 3 × 16 GiB etcd zvols (thin, ~250 MB actual) |
|
||||
|
||||
> **Re-verify at execution time.** Before applying, re-run `zpool list rpool` and
|
||||
> `pvesm status` on the Proxmox host to confirm current free space (this figure is
|
||||
> dated 2026-08-29). The vault-gated ansible path (`proxmox_hosts` group) or a Proxmox
|
||||
> UI check both work.
|
||||
|
||||
### 1.6 Is the HDD *failing* — data-loss urgency?
|
||||
|
||||
The problem statement calls the Seagate HDD "slow/failing." **SMART was not
|
||||
re-collected in this investigation** (the Proxmox host query is vault-gated in the
|
||||
dev container). The etcd-isolation doc characterised `Storage` as 2 × WDC WUH721818
|
||||
18 TB (healthy) and `Storage2` as 2 × Seagate ST8000VN004 — the Longhorn `virtio1`
|
||||
disks are on `Storage` (the WD mirror), a ZFS **mirror**, so a single-disk SMART fault
|
||||
is not immediate data loss. **Action item for the executor:** collect
|
||||
`smartctl -a` for both `Storage`-pool members and `zpool status -x` before scheduling;
|
||||
if a member is degraded, that raises urgency and argues for moving the *hot* data off
|
||||
sooner. Treat "failing" as unconfirmed until SMART is read — but the mirror + 3-way
|
||||
Longhorn replication means the plan is not gated on it.
|
||||
|
||||
---
|
||||
|
||||
## 2. Capacity verdict — does it fit?
|
||||
|
||||
| Scenario | Per-node SSD footprint (actual) | ×3 across nodes | Provisioned to size the disk | Fits in ~323 GiB SSD `AVAIL`? |
|
||||
|---|---:|---:|---:|---|
|
||||
| **Hot tier** (Prometheus + 3 CNPG DBs + Loki + gitea) | **~16.9 GB** | **~50.6 GB** | 38 GiB/node → suggest a **48 GiB** SSD disk/node | **Yes, comfortably** (~145 GiB of thin zvols max, ~51 GB actual) |
|
||||
| **Full move** (all 18 Longhorn volumes) | ~26.2 GB | ~79 GB | 152 GiB/node nominal, but thin | **Yes on space** (~79 GB actual) — the "doesn't fit" blocker was a *provisioned*-number artefact |
|
||||
| All-SSD mini-PC path (no move now) | — | — | — | N/A — hardware-blocked, no committed date |
|
||||
|
||||
**The recorded blocker is overturned by the actual numbers.** 3 × 128 GiB = 384 GiB
|
||||
*provisioned* does not fit next to the 430 GiB host root on an 888 GiB pool — that is
|
||||
true and is why the earlier doc said "NO for Longhorn." But the **real data is ~79 GB**,
|
||||
and both ZFS (`sparse 1`) and Longhorn are thin, so what actually lands on the SSD is
|
||||
the working set, not the nominal disk size. Both a hot-tier move and a full move fit on
|
||||
free space today.
|
||||
|
||||
The reason to still prefer the **hot tier** over a full move is **not space** — it is
|
||||
**I/O coupling** (see §3.4 and §6): the only SSD pool is `rpool`, which now also carries
|
||||
the Proxmox host root *and* the freshly-isolated etcd disks. A full move puts every
|
||||
Longhorn rebuild back onto the same pool as etcd, partially undoing the isolation just
|
||||
completed. A hot-tier move keeps the SSD footprint and the rebuild-I/O burst small while
|
||||
capturing the single largest and churniest contributor (Prometheus = 11.6 GB = 44 % of
|
||||
the whole working set) and all the latency-sensitive Postgres DBs.
|
||||
|
||||
---
|
||||
|
||||
## 3. Options evaluated
|
||||
|
||||
### 3.1 Option A — Hot-tier move onto a tagged SSD Longhorn disk (RECOMMENDED)
|
||||
|
||||
Add a second, SSD-backed Longhorn data disk per node (a new `virtio3` zvol on
|
||||
`local-zfs`), tag it `ssd`, tag the existing HDD disk `hdd`, publish a `longhorn-ssd`
|
||||
StorageClass with `diskSelector: ["ssd"]`, and migrate the hot volumes onto it. Cold
|
||||
config volumes stay on the HDD `default-disk`.
|
||||
|
||||
- **Hot set:** `prometheus-…-db-0` (monitoring), `storage-loki-0` (monitoring),
|
||||
`keycloak-db-1`, `nextcloud-db-1`, `semaphore-db-1` (the three CNPG Postgres DBs),
|
||||
`gitea-data`. ~16.9 GB/node actual, 38 GiB provisioned.
|
||||
- **Why these:** Prometheus is the biggest and churniest (TSDB compaction + snapshot
|
||||
bloat — 11.6 GB actual on an 8 GiB nominal volume). Postgres DBs are fsync-latency
|
||||
sensitive and small. Loki and gitea are moderate-churn. Moving these removes the bulk
|
||||
of rebuild bytes *and* the latency-sensitive I/O from the HDD.
|
||||
- **Footprint:** small and bounded → the I/O-coupling risk on `rpool` is minimal.
|
||||
- **Downside:** the ~12 cold config volumes (~9 GB/node) still rebuild on the HDD on a
|
||||
reboot — but they are tiny and low-churn, so the storm shrinks from 18 volumes /
|
||||
~26 GB to ~12 volumes / ~9 GB, and the SSD ones rebuild fast.
|
||||
|
||||
### 3.2 Option B — Full move (all 18 Longhorn volumes to SSD)
|
||||
|
||||
Size the SSD disk per node for the full working set (e.g. 96 GiB thin, lower
|
||||
over-provisioning) and repoint the default StorageClass / migrate every volume.
|
||||
|
||||
- **Pro:** eliminates HDD Longhorn rebuild entirely — reboots no longer touch the HDD
|
||||
for storage; SSD rebuild is fast and low-IO-delay.
|
||||
- **Con:** re-couples *all* Longhorn rebuild I/O with etcd + host root on `rpool`,
|
||||
partially undoing `docs/plans/ssd-etcd-isolation.md`. SSD IOPS make this far less
|
||||
harmful than the HDD case, but it is a real regression of the isolation just done.
|
||||
Also consumes more of the small SSD pool's headroom.
|
||||
- **Verdict:** feasible, but hold it as a *Phase 2* only if the hot-tier move proves
|
||||
insufficient. Not the opening move.
|
||||
|
||||
### 3.3 Option C — Do nothing; wait for the all-SSD mini-PCs
|
||||
|
||||
Zero work, zero risk, but leaves the cluster fragile to every node bounce
|
||||
indefinitely — the mini-PC migration has no committed date. What that path *requires*:
|
||||
new hardware (all-SSD mini-PCs), a fresh Proxmox/k3s build, and a full Longhorn data
|
||||
migration anyway. Option A is a cheap, reversible bridge that the mini-PC path later
|
||||
supersedes — it does not waste that work (the `longhorn-ssd` class and tagging carry
|
||||
over).
|
||||
|
||||
### 3.4 The I/O-coupling caveat (applies to A and B)
|
||||
|
||||
`rpool` is the **only** SSD pool. It already carries the host root and the isolated
|
||||
etcd disks. Putting Longhorn rebuild I/O back on it re-introduces *some* contention
|
||||
with etcd's WAL fsync — the exact thing the etcd disk was meant to escape. Two facts
|
||||
keep Option A safe: (1) SSD random IOPS are ~2–3 orders of magnitude above the HDD, so
|
||||
concurrent fsync + rebuild is tolerable where it was fatal on spinning rust; (2) the
|
||||
hot-tier footprint is small so rebuild bursts are short. **Mitigation:** keep
|
||||
`concurrent-replica-rebuild-per-node-limit: 1`, and watch
|
||||
`etcd_disk_wal_fsync_duration_seconds` (Prometheus) during the first SSD rebuild — if
|
||||
p99 fsync climbs materially, stop and reassess before proceeding to the next node.
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommended implementation — Option A, phased, one node at a time
|
||||
|
||||
> Every step below is a *proposal*. Nothing here has been executed. Apply per node,
|
||||
> lowest-quorum-risk node first, verifying Longhorn/etcd health between nodes.
|
||||
|
||||
### Phase 0 — Pre-flight (read-only, do before touching anything)
|
||||
|
||||
1. Re-verify SSD free space: on the Proxmox host `zpool list rpool` and `pvesm status`
|
||||
(confirm ≳ 100 GiB `AVAIL` remains after the etcd zvols).
|
||||
2. Collect HDD SMART: `smartctl -a` on both `Storage`-pool members + `zpool status -x`.
|
||||
If a member is degraded, note the added urgency but the mirror + 3× replication mean
|
||||
the plan is unchanged.
|
||||
3. Confirm Longhorn is fully healthy (all volumes `robustness: healthy`) and no rebuild
|
||||
is in flight. Do not start during a rebuild.
|
||||
4. Snapshot the CNPG DBs (CNPG can take an on-demand base backup) and note that gitea /
|
||||
Prometheus / Loki data is reproducible or non-critical — belt-and-braces before any
|
||||
volume move, since there is no Longhorn backup target.
|
||||
|
||||
### Phase 1 — Add an SSD Longhorn disk per node (tofu, mirrors the etcd-disk pattern)
|
||||
|
||||
The VMs are bpg/OpenTofu-managed (`infra/tofu/main.tf`,
|
||||
`resource "proxmox_virtual_environment_vm" "node"`, `for_each = local.nodes`). Disks
|
||||
are `disk {}` blocks; `virtio0` (OS) and `virtio1` (Longhorn HDD) are on `var.datastore`
|
||||
(`Storage`), `virtio2` (etcd) is on `var.etcd_ssd_datastore` (`local-zfs`). Add a
|
||||
**fourth disk on the SSD datastore**, kept last so bpg does not reorder existing disks.
|
||||
|
||||
1. New variables in `infra/tofu/variables.tf`:
|
||||
```hcl
|
||||
variable "longhorn_ssd_datastore" {
|
||||
description = "SSD-backed datastore for the tagged Longhorn SSD disk (local-zfs = rpool/data, Micron 5100 SATA SSD mirror w/ PLP)."
|
||||
type = string
|
||||
default = "local-zfs"
|
||||
}
|
||||
variable "longhorn_ssd_disk_gib" {
|
||||
description = "Per-node SSD Longhorn disk, GiB. Hot tier is ~17 GiB actual / 38 GiB provisioned; 48 GiB gives headroom without crowding rpool."
|
||||
type = number
|
||||
default = 48
|
||||
}
|
||||
```
|
||||
2. New `disk {}` block in the VM resource in `main.tf`, appended **after** the
|
||||
`virtio2` etcd block (ascending interface → `virtio3` → `/dev/vdd`):
|
||||
```hcl
|
||||
# Tagged SSD Longhorn disk (virtio3 -> /dev/vdd) for the hot tier (Prometheus, CNPG
|
||||
# DBs, Loki, gitea). See docs/plans/longhorn-ssd-tiering.md. Kept last so bpg won't
|
||||
# reorder virtio0/1/2. discard=on so ext4/Longhorn TRIM reclaims thin zvol space.
|
||||
disk {
|
||||
datastore_id = var.longhorn_ssd_datastore
|
||||
interface = "virtio3"
|
||||
size = var.longhorn_ssd_disk_gib
|
||||
discard = "on"
|
||||
cache = "none"
|
||||
iothread = false
|
||||
}
|
||||
```
|
||||
3. `tofu plan` **must** show an in-place *disk add* (update) for each node — **not** a
|
||||
replacement. If it shows replace, stop (never destroy a running etcd node).
|
||||
4. Apply **one node at a time** with `-target` on the name-keyed resource address,
|
||||
e.g. `-target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'`. A disk add is
|
||||
applied online by bpg, but treat the node as maintenance-affected regardless.
|
||||
|
||||
### Phase 2 — Format + mount the SSD disk (ansible `node_baseline`, mirrors etcd)
|
||||
|
||||
`roles/node_baseline` already formats/mounts the HDD Longhorn disk (`/dev/vdb` →
|
||||
`/var/lib/longhorn`) and the etcd disk (`/dev/vdc`). Add the same idempotent,
|
||||
data-safe pattern for `/dev/vdd`:
|
||||
|
||||
1. New defaults in `roles/node_baseline/defaults/main.yml`:
|
||||
```yaml
|
||||
node_baseline_longhorn_ssd_device: /dev/vdd
|
||||
node_baseline_longhorn_ssd_mount: /var/lib/longhorn-ssd
|
||||
node_baseline_longhorn_ssd_fs_label: longhorn-ssd
|
||||
```
|
||||
2. New tasks in `roles/node_baseline/tasks/main.yml` (copy the HDD Longhorn
|
||||
filesystem+mount pair; `community.general.filesystem` never reformats an existing
|
||||
fs, so re-runs are safe):
|
||||
```yaml
|
||||
- name: Create ext4 filesystem on the SSD Longhorn disk (data-safe; skips if present)
|
||||
community.general.filesystem:
|
||||
fstype: ext4
|
||||
dev: "{{ node_baseline_longhorn_ssd_device }}"
|
||||
opts: "-L {{ node_baseline_longhorn_ssd_fs_label }}"
|
||||
- name: Mount the SSD Longhorn disk
|
||||
ansible.posix.mount:
|
||||
path: "{{ node_baseline_longhorn_ssd_mount }}"
|
||||
src: "LABEL={{ node_baseline_longhorn_ssd_fs_label }}"
|
||||
fstype: ext4
|
||||
opts: "defaults,nofail,x-systemd.device-timeout=5s"
|
||||
state: mounted
|
||||
```
|
||||
Use `LABEL=` (not `/dev/vdd`) so the fstab entry survives device renames, and
|
||||
`nofail` so a missing SSD Longhorn disk does not block boot (unlike etcd, an absent
|
||||
Longhorn extra-disk should not wedge the node).
|
||||
|
||||
### Phase 3 — Register + tag the disks in Longhorn
|
||||
|
||||
Per node (via the Longhorn `nodes.longhorn.io` CR or UI):
|
||||
|
||||
1. **Tag the existing HDD disk** `default-disk` with `hdd` and set the new
|
||||
`longhorn-ssd` StorageClass's counterpart default class to select it (so cold
|
||||
volumes stay put and nothing hot accidentally lands on HDD once the SSD class
|
||||
exists).
|
||||
2. **Add the SSD disk** at `/var/lib/longhorn-ssd` as a new Longhorn disk on each node,
|
||||
`allowScheduling: true`, tag `ssd`.
|
||||
```yaml
|
||||
# patch each nodes.longhorn.io/<node> .spec.disks
|
||||
disk-ssd:
|
||||
path: /var/lib/longhorn-ssd
|
||||
allowScheduling: true
|
||||
tags: ["ssd"]
|
||||
storageReserved: 0
|
||||
```
|
||||
3. Confirm each node now reports two disks with correct `storageMaximum` (~48 GiB for
|
||||
the SSD one) and `Schedulable` condition true.
|
||||
|
||||
### Phase 4 — Publish the `longhorn-ssd` StorageClass
|
||||
|
||||
New GitOps manifest (wherever the Longhorn StorageClasses are declared in
|
||||
`platform/` — mirror the existing `longhorn` class and add the selector):
|
||||
|
||||
```yaml
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: longhorn-ssd
|
||||
provisioner: driver.longhorn.io
|
||||
allowVolumeExpansion: true
|
||||
parameters:
|
||||
numberOfReplicas: "3"
|
||||
staleReplicaTimeout: "30"
|
||||
fsType: ext4
|
||||
dataLocality: disabled
|
||||
diskSelector: "ssd" # only schedules replicas on ssd-tagged disks
|
||||
disableRevisionCounter: "true"
|
||||
unmapMarkSnapChainRemoved: "ignored"
|
||||
```
|
||||
|
||||
Optionally add a matching `diskSelector: "hdd"` to the existing default `longhorn`
|
||||
class so newly-created cold volumes stay on HDD.
|
||||
|
||||
### Phase 5 — Migrate the hot volumes (no Longhorn backup target → in-cluster methods)
|
||||
|
||||
Because no backup target exists (§1.4), use one of these, **one volume at a time**,
|
||||
respecting `concurrent-replica-rebuild-per-node-limit: 1`:
|
||||
|
||||
- **CNPG DBs (keycloak / nextcloud / semaphore) — cleanest via the operator.** Change
|
||||
the CNPG `Cluster` `.spec.storage.storageClass` to `longhorn-ssd` and let CNPG do a
|
||||
rolling replica re-creation (it provisions a fresh instance on the new class, streams
|
||||
from the primary, then promotes) — no manual data copy, no downtime beyond a
|
||||
switchover. Do one CNPG cluster at a time; verify replication caught up before the
|
||||
next.
|
||||
- **Prometheus / Loki / gitea — per-volume `diskSelector` rebuild (in place, no PVC
|
||||
swap).** Patch the existing Longhorn volume to select the SSD disk and let Longhorn
|
||||
rebuild the replicas onto it, then drop the HDD replicas:
|
||||
```
|
||||
kubectl -n longhorn-system patch volumes.longhorn.io <vol> --type merge \
|
||||
-p '{"spec":{"diskSelector":["ssd"]}}'
|
||||
```
|
||||
Then trigger a controlled rebuild by deleting **one** HDD replica at a time (Longhorn
|
||||
reschedules onto an `ssd`-tagged disk; `replica-auto-balance: best-effort` helps).
|
||||
Watch the volume return to `robustness: healthy` before touching the next replica or
|
||||
the next volume. This avoids recreating PVCs and keeps the app running.
|
||||
- Alternative if the in-place rebuild misbehaves: scale the app to 0, CSI-**clone**
|
||||
the PVC onto `longhorn-ssd` (`dataSource` = existing PVC), repoint the workload to
|
||||
the clone, scale back up. More disruptive; keep as fallback.
|
||||
|
||||
### Phase 6 — Validate
|
||||
|
||||
1. All hot volumes `robustness: healthy`, replicas only on `ssd`-tagged disks
|
||||
(`kubectl -n longhorn-system get replicas.longhorn.io -o wide`).
|
||||
2. SSD Longhorn disk usage ≈ 17 GiB/node; HDD disk down to the cold ~9 GiB/node.
|
||||
3. Deliberately reboot **one** node and confirm the rebuild storm is materially smaller
|
||||
(fewer volumes, SSD-fast) and Proxmox IO-delay stays low. Confirm etcd fsync p99 did
|
||||
not regress during the SSD rebuild (§3.4).
|
||||
|
||||
---
|
||||
|
||||
## 5. Rebuild-storm mitigation (built into the plan)
|
||||
|
||||
- **One node at a time** for every phase (tofu `-target`, ansible `--limit`, one CNPG
|
||||
cluster / one replica at a time) — etcd quorum is 3-node, never take two down.
|
||||
- **`concurrent-replica-rebuild-per-node-limit: 1`** already set — a rebooted node
|
||||
rebuilds serially, capping IO-delay.
|
||||
- **Migrate during a quiet window**, never while another rebuild is in flight.
|
||||
- **Lower `storage-over-provisioning-percentage`** *for the small SSD disk sizing* —
|
||||
200 % on a 48 GiB disk would let Longhorn schedule ~96 GiB of nominal claims onto
|
||||
~17 GiB of real data; keep the hot set well under `storageMaximum`, and rely on
|
||||
`storage-minimal-available-percentage: 25` as the floor. (The setting is global —
|
||||
consider dropping it to ~120–150 % cluster-wide, or simply keep the SSD disk sized so
|
||||
200 % of the hot claims still fits.)
|
||||
- **Verify etcd fsync latency** during the first SSD rebuild before proceeding (§3.4).
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks and rollback
|
||||
|
||||
| Risk | Likelihood | Mitigation / rollback |
|
||||
|---|---|---|
|
||||
| SSD rebuild I/O re-starves etcd on shared `rpool` | Low (SSD IOPS) | rebuild-limit 1; watch `etcd_disk_wal_fsync_duration`; stop after node 1 if p99 regresses |
|
||||
| SSD pool fills (host root 430 G + etcd + Longhorn on 888 G) | Low (hot set ~51 GB) | keep SSD disk small (48 GiB), `storage-minimal-available-percentage: 25`; re-check `zpool list` between nodes |
|
||||
| tofu shows *replace* instead of disk-add | Medium if module drift | inspect `tofu plan` per node; abort on replace — never destroy a running etcd node |
|
||||
| Volume-migration data loss (no backup target) | Low | CNPG native rolling; per-volume rebuild keeps HDD replica until SSD healthy; CNPG base backup + reproducible data taken in Phase 0 |
|
||||
| Data disk device rename (`/dev/vdd` shifts) | Low | mount by `LABEL=longhorn-ssd`, not device path |
|
||||
| HDD actually failing mid-migration | Unknown (SMART unread) | 3× replication + mirror; Phase 0 SMART check; hot data lands on SSD first, reducing exposure |
|
||||
|
||||
**Rollback:** every phase is reversible. To back out: migrate the hot volumes back to
|
||||
the HDD `default-disk` (reverse the `diskSelector` patch / point CNPG back at
|
||||
`longhorn`), remove the `disk-ssd` from each Longhorn node, delete the `longhorn-ssd`
|
||||
StorageClass, then remove the `virtio3` disk block from tofu and re-apply one node at a
|
||||
time. The SSD zvols are thin and independent of the HDD data, so removal is clean.
|
||||
|
||||
---
|
||||
|
||||
## 7. Recommendation
|
||||
|
||||
**Adopt Option A (hot-tier move), phased, one node at a time.** The recorded "SSD
|
||||
doesn't fit Longhorn" blocker was based on the 384 GiB *provisioned* figure; the
|
||||
**actual working set is ~79 GB cluster-wide (~26 GB/node)** because all bulk media
|
||||
lives on NFS and every Longhorn volume is a small config/DB volume. The hot tier
|
||||
(Prometheus + the three CNPG Postgres DBs + Loki + gitea) is **~51 GB actual /
|
||||
114 GiB provisioned** across three nodes and fits the ~323 GiB SSD `AVAIL` with wide
|
||||
headroom, while capturing the single largest and churniest rebuild contributor
|
||||
(Prometheus, 44 % of the working set) and the latency-sensitive databases.
|
||||
|
||||
A **full move is feasible on space** and would fully eliminate HDD Longhorn rebuilds,
|
||||
but it re-couples all rebuild I/O with etcd + host root on the only SSD pool, partially
|
||||
undoing the just-completed etcd isolation — hold it as an optional Phase 2 gated on the
|
||||
hot-tier move proving insufficient. **Option C (wait for mini-PCs) is not a solution**
|
||||
tonight — no committed date, and the tagging + `longhorn-ssd` class built here carry
|
||||
over to that migration anyway.
|
||||
|
||||
**Do this first, before any apply:** re-verify live SSD free space (`zpool list
|
||||
rpool`, `pvesm status`) and read HDD SMART (`smartctl -a`, `zpool status -x`). If the
|
||||
Seagate/`Storage` pool shows a degraded member, the urgency rises and the hot-tier move
|
||||
should be scheduled sooner — but the mirror + 3× Longhorn replication mean the plan is
|
||||
not gated on the disk's health.
|
||||
|
||||
> This document is a plan only. No storage, tofu, Longhorn, StorageClass, or live
|
||||
> infrastructure has been modified.
|
||||
@@ -1,104 +0,0 @@
|
||||
# Phase 2 execution runbook — SSD-backed etcd migration (interactive/manual)
|
||||
|
||||
> ## ✅ COMPLETED / historical — executed 2026-08-30, see git log
|
||||
>
|
||||
> **This runbook has been executed.** Every k3s node's etcd now lives on its dedicated SSD `virtio2`
|
||||
> disk. The command sequence below is retained as the record of how the migration was performed — do
|
||||
> NOT re-run it against the live cluster. Current state is tracked in `infra/PHASE1-STATUS.md` and the
|
||||
> 2026-08-30 DECISIONS-LOG entry.
|
||||
|
||||
Companion to `ssd-etcd-isolation.md`. This is the **exact command sequence** to migrate
|
||||
each k3s node's etcd onto its dedicated SSD `virtio2` disk. It must be run where live-infra
|
||||
mutations are permitted — an **interactive `claude` session** (default permission mode, which
|
||||
prompts for approval) or **by hand**. The autonomous "auto mode" session hard-denies
|
||||
`tofu apply` and node-level ssh mutations, so it cannot run this.
|
||||
|
||||
## Preconditions (verified 2026-08-29, cluster healthy)
|
||||
- Run everything from the staged worktree that carries the tofu disk block:
|
||||
`cd /home/jakman85/source/home-network-k3s-impl/.claude/worktrees/agent-ssd-etcd`
|
||||
(branch `feature/ssd-etcd-isolation` @ `af5fc5b`).
|
||||
- All tooling via the devcontainer: `./infra/devcontainer/dev.sh run <cmd>`.
|
||||
- `./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh init` once (done).
|
||||
- Ansible must run from `infra/ansible/` (for the vault-pass path): wrap as
|
||||
`./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible ...'`.
|
||||
- **Run mutation commands STANDALONE** (not chained with `&&`/pipes) — the classifier is
|
||||
more likely to deny a compound command that embeds a mutation.
|
||||
|
||||
## ORDER (leader last — RE-VERIFY before each node)
|
||||
Current leader = **dev-3**, so: **dev-2 (vmid 102) → dev-1 (vmid 105) → dev-3 (vmid 101)**.
|
||||
dev-3 is last and additionally gets the memory 16384→24576 bump (needs a full stop/start).
|
||||
Re-check the leader before each node and keep the leader last:
|
||||
```
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_is_leader"'
|
||||
```
|
||||
|
||||
## Pre-flight before EACH node (read-only — all must pass)
|
||||
```
|
||||
./infra/devcontainer/dev.sh run kubectl get nodes
|
||||
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd # expect: ok
|
||||
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy # expect: 18
|
||||
```
|
||||
|
||||
## Per-node procedure (substitute NODE=k3s-dev-N, VMID)
|
||||
|
||||
### 1. Freeze rebuilds (STANDALONE)
|
||||
```
|
||||
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"0"}'
|
||||
```
|
||||
|
||||
### 2. Add the SSD disk via tofu (STANDALONE). dev-2/dev-1 = disk only; dev-3 also memory.
|
||||
```
|
||||
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]' # confirm: disk ADD, in-place, NOT replace
|
||||
./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh apply -auto-approve '-target=proxmox_virtual_environment_vm.node["k3s-dev-N"]'
|
||||
```
|
||||
Confirm the guest sees the new disk (hot-plug):
|
||||
```
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "lsblk /dev/vdc"'
|
||||
```
|
||||
If `/dev/vdc` is NOT present (no hot-plug), power-cycle the VM (freeze is holding):
|
||||
```
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm stop VMID"' # wait until stopped
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible proxmox -m shell -a "qm start VMID"'
|
||||
```
|
||||
(For dev-3 the memory bump requires this stop/start regardless.)
|
||||
|
||||
### 3. Migrate etcd data (node commands, become=root). Do NOT skip the copy-before-mount order.
|
||||
```
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl stop k3s"'
|
||||
# mkfs ONLY if /dev/vdc has no filesystem yet (fresh disk):
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "blkid /dev/vdc || mkfs.ext4 -L k3s-etcd /dev/vdc"'
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak && mkdir /var/lib/rancher/k3s/server/db && mount LABEL=k3s-etcd /var/lib/rancher/k3s/server/db && rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/"'
|
||||
# Persist the mount (matches the node_baseline role: LABEL + defaults,noatime):
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m ansible.posix.mount -a "path=/var/lib/rancher/k3s/server/db src=LABEL=k3s-etcd fstype=ext4 opts=defaults,noatime dump=0 passno=2 state=mounted"'
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "systemctl start k3s"'
|
||||
```
|
||||
|
||||
### 4. VERIFY full recovery before touching the next node (all must pass)
|
||||
```
|
||||
./infra/devcontainer/dev.sh run kubectl get nodes # NODE Ready
|
||||
./infra/devcontainer/dev.sh run kubectl get --raw=/readyz/etcd # ok
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -m shell -a "findmnt /var/lib/rancher/k3s/server/db"' # source = /dev/vdc
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s_servers -m shell -a "curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader"' # all =1
|
||||
./infra/devcontainer/dev.sh run kubectl -n longhorn-system get volumes.longhorn.io -o custom-columns=R:.status.robustness --no-headers | grep -c healthy # 18
|
||||
# dev-3 only: confirm ~24Gi allocatable
|
||||
./infra/devcontainer/dev.sh run kubectl get node k3s-dev-3 -o jsonpath='{.status.allocatable.memory}'
|
||||
```
|
||||
|
||||
### 5. Thaw + clean up (STANDALONE), only after step 4 fully passes
|
||||
```
|
||||
./infra/devcontainer/dev.sh run kubectl -n longhorn-system patch settings.longhorn.io concurrent-replica-rebuild-per-node-limit --type=merge -p '{"value":"1"}'
|
||||
./infra/devcontainer/dev.sh run bash -lc 'cd infra/ansible && ansible k3s-dev-N -b -m shell -a "rm -rf /var/lib/rancher/k3s/server/db.bak"'
|
||||
```
|
||||
Wait for 18/18 healthy (no rebuild storm), re-verify leader, then proceed to the next node.
|
||||
|
||||
## STOP conditions (halt immediately, leave limit=0, do not advance)
|
||||
etcd `/readyz` not ok, a member without a leader, VIP `192.168.2.50:6443` flapping, a node
|
||||
not rejoining, any Longhorn volume Faulting, or k3s crashlooping. Recovery: keep rebuilds
|
||||
frozen; a botched single member can be reset and re-joined from the other two (plan §4.5).
|
||||
|
||||
## Close-out (after all 3)
|
||||
- All 3 nodes Ready; etcd 3/3 with a leader; `findmnt` shows `/dev/vdc` at the db path on each.
|
||||
- 18/18 Longhorn volumes healthy; rebuild-limit back to **1**.
|
||||
- `./infra/devcontainer/dev.sh run ./infra/tofu/tf.sh plan` → **No changes** (all disks present, dev-3 memory reconciled).
|
||||
- Merge `feature/ssd-etcd-isolation` → `main` and push (push is classifier-blocked in auto mode — run `git push gitea main` by hand) so Flux/tofu source records the disk + baseline.
|
||||
- Regression test: after approval, bounce ONE node and confirm etcd stays healthy / VIP steady — the whole point of the change.
|
||||
@@ -1,277 +0,0 @@
|
||||
# SSD-backed etcd isolation for the k3s dev cluster
|
||||
|
||||
> ## ✅ COMPLETED / historical — executed 2026-08-30, see git log
|
||||
>
|
||||
> **This proposal was approved and carried out.** Each node now has a dedicated `virtio2` SSD etcd
|
||||
> disk (`/dev/vdc`, `local-zfs`) hosting the etcd data dir. The **Status: PROPOSAL** line below and
|
||||
> the sizing/RAM figures are the original investigation record — retained for the reasoning, not as a
|
||||
> live to-do. Current state is tracked in `infra/PHASE1-STATUS.md` and the 2026-08-30 DECISIONS-LOG entry.
|
||||
|
||||
**Status:** PROPOSAL — awaiting user approval. No changes have been made.
|
||||
**Author:** investigation agent (read-only), 2026-08-29
|
||||
**Problem:** Longhorn replica-rebuild I/O storms on the HDD-backed VM disks starve
|
||||
etcd's WAL fsync, crash-looping k3s and flapping the kube-vip VIP
|
||||
(`192.168.2.50:6443`). Even `concurrent-replica-rebuild-per-node-limit=1` tips etcd.
|
||||
**Question:** Can the Proxmox host's ~1 TB system SSD (`rpool`) host dedicated
|
||||
SSD-backed storage for etcd (and/or Longhorn) to end these incidents?
|
||||
|
||||
**Verdict: YES for etcd (strongly recommended). NO for Longhorn.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Measured facts (all collected read-only on 2026-08-29)
|
||||
|
||||
### 1.1 `rpool` — the system/boot SSD pool
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Pool | `rpool`, **mirror** (`mirror-0`) of 2 devices |
|
||||
| Devices | `ata-MTFDDAK960TDS_...` × 2 — **Micron 5100/52x0 960 GB, 2.5" SATA** |
|
||||
| Class | **Enterprise SATA SSD with power-loss protection (PLP)** — safe for sync/fsync |
|
||||
| Health | `Percent_Lifetime_Remain` = **97%** (3% used), 13 904 power-on hours, 0 reallocated sectors, ~96 TB written. Excellent. |
|
||||
| Pool size / alloc / **free** | 888 G / 538 G / **350 G free** (`zpool list`, 60% cap) |
|
||||
| ZFS `AVAIL` | **~323 GiB** available to any new dataset/zvol |
|
||||
| System root | `rpool/ROOT/pve-1` = **430 G used** (Proxmox host `/`) |
|
||||
| Proxmox storage on it | `local-zfs` → pool `rpool/data`, `content images,rootdir`, **`sparse 1`** (thin), ~338 GiB available per `pvesm status` |
|
||||
| Pool settings | `compression on`, **`sync standard`** (inherited), `dedup off`, recordsize 128K |
|
||||
|
||||
`local-zfs` already advertises `images,rootdir` content, so **Proxmox can place VM
|
||||
disks on the SSD today with no storage-config change.**
|
||||
|
||||
### 1.2 Current VM disk backing — confirms the HDD assumption
|
||||
|
||||
VMs (bpg/OpenTofu-provisioned): **dev-1 = vmid 105, dev-2 = 102, dev-3 = 101**.
|
||||
Each has two disks, **both on the `Storage` datastore**:
|
||||
|
||||
- `Storage` = ZFS pool on **2× WDC WUH721818ALE6L4 18 TB HDD** (`sdc`/`sdd`, `ROTA=1`), mirror, 16.4 T.
|
||||
- Per node: `virtio0 = vm-<id>-disk-0` (64 G, OS/root) and `virtio1 = vm-<id>-disk-1` (128 G, Longhorn data). `volblocksize 16K`.
|
||||
|
||||
So **etcd, the OS, and Longhorn all currently sit on spinning rust.** That is the
|
||||
root cause: etcd WAL fsync competes with Longhorn rebuild I/O on the same HDD spindles.
|
||||
|
||||
(`Storage2` = 2× Seagate ST8000VN004 8 TB HDD is a separate HDD pool, not involved.)
|
||||
|
||||
### 1.3 Size of the hot data
|
||||
|
||||
- **etcd DB: ~82.6 MB.** Measured via apiserver metric
|
||||
`apiserver_storage_size_bytes{storage_cluster_id="etcd-0"} = 8.264e7`.
|
||||
A 4-day-old cluster running ~18 services. Even allowing generous WAL + growth
|
||||
headroom, **etcd fits in single-digit GB with room to spare — trivially ×3.**
|
||||
- **Longhorn: ~22 GB actual data per node right now**, but the data disk is
|
||||
**128 GiB** and Longhorn reports `storageScheduled ≈ 152 GiB` (thin-overcommitted)
|
||||
vs `storageMaximum ≈ 125 GiB` per node. 18 PVCs, `numberOfReplicas=3`.
|
||||
Provisioned total ≈ 152 GiB. Longhorn is the **churn source we want to isolate
|
||||
etcd from**, and it can grow toward 128 GiB/node.
|
||||
|
||||
### 1.4 Sizing conclusion
|
||||
|
||||
| Candidate | Per-node footprint | ×3 on rpool | Fits in 323 GiB avail? |
|
||||
|---|---|---|---|
|
||||
| **etcd** | ~83 MB now; size a 16 GiB disk | ~48 GiB reserved (thin) | **Yes, easily** |
|
||||
| **Longhorn** | ~22 GB now, up to 128 GiB provisioned | up to **~384 GiB** | **No — would fill/threaten the boot disk** |
|
||||
|
||||
---
|
||||
|
||||
## 2. Viability verdict
|
||||
|
||||
**etcd on SSD: viable and high-leverage.** etcd is ~83 MB; three 16 GiB SSD-backed
|
||||
disks cost ~48 GiB of thin allocation (actual ~250 MB) against 323 GiB free on an
|
||||
enterprise, PLP-equipped, 97%-healthy SSD mirror. This directly removes the WAL
|
||||
fsync path from the HDD contention domain — the smallest-footprint fix for the exact
|
||||
failure mode. **This is the recommendation.**
|
||||
|
||||
**Longhorn on SSD: not viable here.** Its provisioned footprint (~384 GiB across 3
|
||||
replicas at full size) does not fit alongside the 430 GiB Proxmox root on a 888 GiB
|
||||
pool, and co-locating heavy Longhorn rebuild I/O on the boot SSD reintroduces
|
||||
contention with the host itself. Keep Longhorn on HDD (`Storage`). Rejected.
|
||||
|
||||
---
|
||||
|
||||
## 3. Options evaluated
|
||||
|
||||
- **Option A — dedicated small SSD disk per node, etcd-only (RECOMMENDED).**
|
||||
Add a ~16 GiB SSD-backed virtual disk (zvol on `rpool/data` via `local-zfs`) to
|
||||
each of the 3 VMs, mount it at the k3s etcd directory
|
||||
(`/var/lib/rancher/k3s/server/db`), migrate etcd data onto it one node at a time.
|
||||
Isolates WAL fsync from HDD Longhorn churn. Minimal space, minimal blast radius.
|
||||
- **Option B — Longhorn replicas/disk on SSD.** Rejected (§2): does not fit and
|
||||
re-couples host I/O.
|
||||
- **Option C — do nothing; wait for the SSD mini-PCs.** Zero risk, zero work, but
|
||||
leaves the cluster fragile to every node bounce until the mini-PC migration lands
|
||||
(deferred, no committed date). Option A is a cheap, reversible bridge that Option C
|
||||
eventually supersedes (§7).
|
||||
|
||||
**Recommendation: Option A now, as an interim fix, carried by tofu.**
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommended implementation — Option A, step-by-step
|
||||
|
||||
### 4.1 Model it in tofu (preferred over manual)
|
||||
|
||||
The VMs are bpg/OpenTofu-managed (`infra/tofu/`). Disks are declared as `disk`
|
||||
blocks in `main.tf`; both current disks (`virtio0`, `virtio1`) point at
|
||||
`var.datastore` (`Storage`). Add a **third disk on the SSD datastore**:
|
||||
|
||||
1. **New variables** in `infra/tofu/variables.tf`:
|
||||
```hcl
|
||||
variable "etcd_ssd_datastore" {
|
||||
description = "SSD-backed datastore for the dedicated etcd disk (VERIFIED: local-zfs = rpool/data, ZFS mirror on Micron 5100 SATA SSDs w/ PLP)."
|
||||
type = string
|
||||
default = "local-zfs"
|
||||
}
|
||||
variable "etcd_disk_gib" {
|
||||
description = "Dedicated SSD etcd disk per node, GiB (etcd DB is ~83 MB; 16 GiB is generous headroom)."
|
||||
type = number
|
||||
default = 16
|
||||
}
|
||||
```
|
||||
2. **New disk block** in the VM resource in `main.tf`, appended AFTER the existing
|
||||
`virtio1` block (keep interface order ascending so bpg does not reorder):
|
||||
```hcl
|
||||
# Dedicated SSD etcd disk (virtio2 -> /dev/vdc). Isolates etcd WAL fsync from
|
||||
# the HDD Storage pool where the OS + Longhorn disks live.
|
||||
disk {
|
||||
datastore_id = var.etcd_ssd_datastore
|
||||
interface = "virtio2"
|
||||
size = var.etcd_disk_gib
|
||||
discard = "on"
|
||||
cache = "none" # honor guest fsync through to ZFS ZIL (PLP SSD = safe+fast)
|
||||
iothread = false
|
||||
}
|
||||
```
|
||||
3. **Plan and inspect before applying.** `tofu plan` MUST show an **in-place update
|
||||
(disk add)** for the targeted VM — NOT a replacement. If it shows replace, stop and
|
||||
reassess (do not destroy a running etcd node).
|
||||
4. **Apply one node at a time** with `-target` (see §4.3 for quorum ordering), e.g.
|
||||
`-target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'` (adjust to the actual
|
||||
resource address in this repo). A disk add is applied online by bpg, but treat the
|
||||
node as going down for the etcd migration regardless.
|
||||
|
||||
> **Coupling caveat — dev-3 memory drift.** dev-1/dev-2 are at `memory_mib = 24576`
|
||||
> but **dev-3 is live at 16384** (drift from tofu default 24576). A tofu apply that
|
||||
> touches dev-3 will also try to reset its memory to 24576. Per the Proxmox rule,
|
||||
> memory changes need a **full stop/start**, not a reboot. Decide deliberately:
|
||||
> either (a) let the same maintenance window fix the memory drift on dev-3 (stop →
|
||||
> set → start), or (b) `-target` narrowly and accept that dev-3's plan will include
|
||||
> the memory change. Do dev-3 in its own window with eyes on it.
|
||||
|
||||
**Manual alternative (if not using tofu):** on the Proxmox host,
|
||||
`qm set <vmid> -virtio2 local-zfs:16,discard=on,cache=none` then stop/start the VM.
|
||||
This drifts from tofu state and is discouraged — if done, import/reconcile into tofu
|
||||
afterward. Prefer the tofu path.
|
||||
|
||||
### 4.2 Optional space guarantee
|
||||
|
||||
`local-zfs` is `sparse` (thin), so the etcd zvols consume only real usage. To
|
||||
guarantee etcd can never be squeezed if the system disk fills, set a modest
|
||||
`refreservation` on each etcd zvol (e.g. `zfs set refreservation=16G
|
||||
rpool/data/vm-<id>-disk-2`). 3×16 GiB = 48 GiB reserved out of 323 GiB avail — safe.
|
||||
This is a host-side ZFS tweak, not expressible in bpg; apply manually if desired.
|
||||
|
||||
### 4.3 Migrate etcd onto the new disk — quorum-safe, ONE node at a time
|
||||
|
||||
k3s has no flag to relocate only the etcd directory (`--data-dir` moves everything),
|
||||
so the clean approach is to **mount the SSD disk at
|
||||
`/var/lib/rancher/k3s/server/db`** (this directory holds `etcd/` — member data, WAL,
|
||||
snapshots — and nothing else that matters).
|
||||
|
||||
**Pre-flight:** confirm all 3 etcd members healthy and the cluster is quiet (no active
|
||||
Longhorn rebuild). `kubectl get nodes` all Ready; check etcd member health.
|
||||
|
||||
**Per node, repeat for dev-3, then dev-2, then dev-1 (leader last):**
|
||||
|
||||
1. Verify 3/3 etcd members healthy BEFORE touching this node (losing one of three
|
||||
keeps quorum at 2/3; never take a second node until the previous is fully back).
|
||||
2. Stop k3s on the node: `systemctl stop k3s`. (Node leaves quorum; 2/3 remain.)
|
||||
3. Identify the new disk (`/dev/vdc` — the third virtio disk) and format it:
|
||||
`mkfs.ext4 -L k3s-etcd /dev/vdc` (ext4 or xfs; ext4 is fine and simple).
|
||||
4. Preserve and move existing data:
|
||||
```
|
||||
mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak
|
||||
mkdir /var/lib/rancher/k3s/server/db
|
||||
mount /dev/vdc /var/lib/rancher/k3s/server/db
|
||||
rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/
|
||||
```
|
||||
5. Persist the mount in `/etc/fstab` **by UUID** (`blkid /dev/vdc`), e.g.
|
||||
`UUID=<uuid> /var/lib/rancher/k3s/server/db ext4 defaults,noatime 0 2`, then
|
||||
`systemctl daemon-reload`.
|
||||
6. Start k3s: `systemctl start k3s`. Wait until the node is `Ready` AND its etcd
|
||||
member is healthy and has re-joined (member list shows 3 started, no learner
|
||||
stuck). Confirm the VIP `192.168.2.50:6443` is stable.
|
||||
7. Only after full recovery, remove the backup: `rm -rf
|
||||
/var/lib/rancher/k3s/server/db.bak`. Then proceed to the next node.
|
||||
|
||||
**Reproducibility for rebuilds:** the SSD disk + mount should also be baked into the
|
||||
node **cloud-init first-boot baseline** (the same template that already provisions the
|
||||
Longhorn `virtio1` disk and swap-off), so a `tofu`-rebuilt node comes up with
|
||||
`/dev/vdc` formatted and mounted at the etcd path automatically. Update that template
|
||||
in the same change set as the tofu disk block.
|
||||
|
||||
### 4.4 Validation
|
||||
|
||||
- `kubectl get nodes` — all 3 Ready; no flapping over a 30–60 min soak.
|
||||
- etcd: 3 members, all started/healthy, one leader; DB size sane (~tens–hundreds MB).
|
||||
- On each node, `df -h /var/lib/rancher/k3s/server/db` shows the new `/dev/vdc` mount.
|
||||
- **Regression test the original failure:** with the cluster healthy, trigger a
|
||||
controlled Longhorn replica rebuild (e.g. bounce ONE node after approval) and
|
||||
confirm etcd stays healthy and the VIP does not flap — the whole point of the change.
|
||||
- Host-side: `zpool status rpool` clean; `zfs list rpool/data` shows the 3 new
|
||||
`vm-*-disk-2` zvols with small real usage.
|
||||
|
||||
### 4.5 Rollback
|
||||
|
||||
Per node, fully reversible while `db.bak` still exists (before step 7):
|
||||
`systemctl stop k3s` → `umount .../server/db` → `rmdir` the empty mountpoint →
|
||||
`mv db.bak db` → remove the fstab line → `systemctl start k3s`. To fully revert the
|
||||
disk, detach `virtio2` (`qm set <vmid> -delete virtio2` after stop) or `tofu` revert
|
||||
the disk block, then stop/start. Because etcd is a replicated 3-member cluster, a
|
||||
botched single node can also simply be reset and re-joined from the other two.
|
||||
|
||||
---
|
||||
|
||||
## 5. Risks
|
||||
|
||||
1. **System-disk contention / exhaustion (top risk).** `rpool` is also the Proxmox
|
||||
boot/root disk (430 GiB used of 888 GiB). Mitigations: keep the etcd disks tiny
|
||||
(16 GiB, thin), optional `refreservation` (§4.2), and **never** put Longhorn here
|
||||
(§2). etcd's steady-state real usage is ~hundreds of MB; the marginal I/O it adds
|
||||
to the SSD is negligible next to what it removes from the HDD.
|
||||
2. **Consumer-SSD fsync/PLP caveat — N/A here, but assert it.** etcd correctness under
|
||||
power loss depends on honest fsync. These are **Micron 5100/52x0 enterprise SATA
|
||||
SSDs with power-loss protection**, so `sync=standard` + `cache=none` is both fast
|
||||
and durable. (If the disks were consumer SSDs without PLP this concern would be
|
||||
real — it is not, for this hardware.)
|
||||
3. **Quorum during migration.** Mitigated by strict one-node-at-a-time with full
|
||||
recovery gates (§4.3) and leader-last ordering. Never take a second node down
|
||||
before the previous rejoins.
|
||||
4. **tofu coupling with dev-3 memory drift + apply semantics (§4.1).** An apply
|
||||
touching dev-3 also resets memory 16384→24576 (needs stop/start). Verify `tofu
|
||||
plan` shows disk *add*, not VM *replace*, before applying. Use `-target`.
|
||||
5. **Backup/restore assumptions.** k3s etcd snapshots (`/var/lib/rancher/k3s/server/db/snapshots`)
|
||||
now live on the SSD too — fine, but confirm any snapshot/backup job still points at
|
||||
the (unchanged) path.
|
||||
|
||||
---
|
||||
|
||||
## 6. Recommendation summary
|
||||
|
||||
Implement **Option A**: a 16 GiB SSD-backed `virtio2` disk per k3s VM (zvol on
|
||||
`rpool/data` via `local-zfs`), etcd-only, migrated one node at a time with the
|
||||
quorum-safe procedure in §4.3, modeled in tofu (+ cloud-init baseline) for
|
||||
reproducibility. Do **not** move Longhorn to SSD. This is a small, reversible,
|
||||
high-leverage fix targeting the exact WAL-starvation failure.
|
||||
|
||||
---
|
||||
|
||||
## 7. Relationship to the deferred SSD-mini-PC migration
|
||||
|
||||
This is explicitly an **interim bridge** on the current Proxmox dev host
|
||||
(`192.168.2.102`), whose bulk storage is HDD. Once the k3s cluster migrates to the
|
||||
planned **all-SSD mini-PCs**, every I/O path (OS, etcd, Longhorn) is already on SSD
|
||||
and the HDD-contention failure mode disappears — the dedicated etcd disk becomes
|
||||
unnecessary. At that point the isolation can be either dropped or carried forward
|
||||
harmlessly (a dedicated etcd disk is still good hygiene). Because Option A is captured
|
||||
in tofu + cloud-init, it travels with the cluster definition and can be removed in one
|
||||
reviewed change when the mini-PCs land. Until then it stops the recurring incidents
|
||||
at minimal cost.
|
||||
+5
-15
@@ -1,19 +1,9 @@
|
||||
# Dev cluster — current state & reconnect guide
|
||||
|
||||
**Last verified: 2026-08-31.** Phase 1 is **complete and long since exited**; the cluster has been
|
||||
**Last verified: 2026-08-26.** Phase 1 is **complete and long since exited**; the cluster has been
|
||||
running the full platform (phases 2–6) for some time. This doc is how a session — human or agent —
|
||||
reconnects to it and re-establishes the toolchain.
|
||||
|
||||
> **New PM / agent taking over?** Read this doc (reconnect + toolchain), then the **latest
|
||||
> `DECISIONS-LOG.md` entry (2026-08-31)** — it is the current-state + gotchas + prioritized **Next
|
||||
> Steps** handoff — and the plans under `docs/plans/`. Start at Next Steps.
|
||||
|
||||
> **Phase 7 (prod) is OPEN as of 2026-08-31 — prep only.** The first prod node
|
||||
> (`k3s-prod-1`, `192.168.2.10`) is on the bench. This document covers the **dev** cluster;
|
||||
> for prod see [`PROD-PROVISIONING.md`](PROD-PROVISIONING.md) (T-7.1 checklist) and the
|
||||
> Phase-7 entry in `DECISIONS-LOG.md`. Nothing is installed on prod hardware until the
|
||||
> R-ENV-02 tokenization blockers close and the gate is green.
|
||||
|
||||
> **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation
|
||||
> (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster
|
||||
> now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`,
|
||||
@@ -25,12 +15,12 @@ reconnects to it and re-establishes the toolchain.
|
||||
| | |
|
||||
|---|---|
|
||||
| Substrate | Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0` |
|
||||
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 8 vCPU, 24 GiB, Debian 13 (trixie) |
|
||||
| k3s | **v1.36.4+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
|
||||
| CNI | Cilium 1.20.1 (kube-proxy replacement, L2 announcements, egress gateway) |
|
||||
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 6 vCPU, 16 GiB, Debian 12 (bookworm) |
|
||||
| k3s | **v1.31.5+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
|
||||
| CNI | Cilium 1.16.5 (kube-proxy replacement, L2 announcements, egress gateway) |
|
||||
| kube-API | kube-vip VIP **`192.168.1.49:6443`**, tls-san'd on the serving cert |
|
||||
| Ingress VIP | **`192.168.1.53`** (shared Gateway). Pool `192.168.1.53–.60` for LB VIPs; UniFi holds `.54` |
|
||||
| Storage | Longhorn 1.12.1 — classes `longhorn` (default), `longhorn-static`, `longhorn-vm` (RWX/Block, migratable), `longhorn-scratch`. `bulk` (NFS CSI) is a **non-provisioning stub** pending OQ-2 |
|
||||
| Storage | Longhorn 1.7.3 — classes `longhorn` (default), `longhorn-static`, `longhorn-vm` (RWX/Block, migratable), `longhorn-scratch`. `bulk` (NFS CSI) is a **non-provisioning stub** pending OQ-2 |
|
||||
| Nested virt | `cpu_type = host` → real KVM in-guest; KubeVirt + CDI run on it |
|
||||
|
||||
Everything above is provisioned by IaC: [`tofu/`](tofu/) creates the VMs,
|
||||
|
||||
@@ -1,362 +0,0 @@
|
||||
# Prod bare-metal provisioning checklist (T-7.1)
|
||||
|
||||
**Task:** T-7.1 · **Workstream:** WS-INFRA · **Serves:** R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03
|
||||
**Status:** Phase 7 **opened by the operator 2026-08-31** — *prep only*. Nothing is installed on prod
|
||||
hardware until the blockers in §7 are closed and `make -C ci gate` is green.
|
||||
|
||||
> **Entry-gate note (recorded, not waived).** `roadmap/phases.md` gates Phase 7 on *"Phases 1–6
|
||||
> proven in dev/test; operator declares prod hardware ready."* The operator has declared hardware
|
||||
> ready. The first condition is **not** met: conformance is 37/51, and the uncovered set includes
|
||||
> **R-ENV-01/02/03** — the very requirements this document serves. We proceed with prep work
|
||||
> knowingly; see `DECISIONS-LOG.md` (2026-08-31 Phase-7 entry).
|
||||
|
||||
Prod is **3× mini-PC bare metal** ([spec docs/02](../../home-network-k3s/docs/02-environments-and-provisioning.md)).
|
||||
The dev path (OpenTofu → Proxmox VM → cloud-init) **does not apply**: there is no hypervisor API to
|
||||
call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse
|
||||
the *same* Ansible roles and the *same* Flux base as dev — that reuse is the R-ENV-02 obligation.
|
||||
|
||||
---
|
||||
|
||||
## 0. Node inventory
|
||||
|
||||
Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
|
||||
|
||||
| | node 1 | node 2 | node 3 |
|
||||
|---|---|---|---|
|
||||
| Hostname | `k3s-prod-1` | `k3s-prod-2` | `k3s-prod-3` |
|
||||
| LAN IP | `192.168.2.10` | `192.168.2.11` | `192.168.2.12` |
|
||||
| Model | HP EliteDesk 800 G5 Desktop Mini | | |
|
||||
| CPU | i7-9700T, 8c/8t, VT-x | | |
|
||||
| RAM | 31 GiB | | |
|
||||
| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`) | | |
|
||||
| OS | Debian 13 (trixie), kernel 6.12 | | |
|
||||
| Firmware | HP R21 Ver. 02.26.00 | | |
|
||||
| `/dev/kvm` | present | | |
|
||||
|
||||
Dev nodes for comparison: 8 vCPU / 24 GiB / Debian 13, **two** dedicated data disks (Longhorn + etcd).
|
||||
|
||||
---
|
||||
|
||||
## 1. Firmware / BIOS (per node, before OS install)
|
||||
|
||||
- [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after
|
||||
boot with `/dev/kvm` present and `grep -cE 'vmx|svm' /proc/cpuinfo` non-zero.
|
||||
- [ ] **AC power recovery = Power On.** A homelab node must return after a power cut without a
|
||||
human pressing the button. (`docs/02` calls out wake/boot behaviour explicitly.)
|
||||
- [ ] **Wake-on-LAN enabled** on the onboard NIC (remote recovery when the box is off).
|
||||
- [ ] **Boot order** — internal disk first; USB above it only while imaging. Disable network boot
|
||||
unless PXE is adopted.
|
||||
- [ ] **Secure Boot** — record the setting. k3s itself is fine either way; note it so all three
|
||||
nodes match.
|
||||
- [ ] **Firmware version recorded** in §0 and consistent across all three nodes.
|
||||
|
||||
## 2. Disk layout — **OPEN DECISION, blocks the build**
|
||||
|
||||
`docs/02` requires the OS-vs-Longhorn device split to be **documented before prod build**. Dev went
|
||||
further and added a *third* device for etcd after HDD contention starved etcd WAL fsync
|
||||
(`docs/plans/ssd-etcd-isolation.md`).
|
||||
|
||||
Node 1 as-found is a **single** 477 GB SATA SSD carrying everything:
|
||||
|
||||
```
|
||||
sda 477G C400-MTFDDAC512M (SSD, ~2011-era Micron/Crucial)
|
||||
├─sda1 976M /boot/efi
|
||||
├─sda2 30G /
|
||||
├─sda3 12G /var
|
||||
├─sda4 977M swap
|
||||
└─sda5 433G /srv <- the only large free space
|
||||
```
|
||||
|
||||
Consequences to settle before imaging nodes 2 and 3:
|
||||
|
||||
- [x] **Drive health MEASURED 2026-09-01** (`smartctl -a /dev/sda`, via the baseline package
|
||||
floor — see §5). Raw attributes, so the numbers are auditable:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Model | Crucial/Micron RealSSD m4/C400/P400 (`C400-MTFDDAC512MAM`, fw `070H`) |
|
||||
| SMART self-assessment | **PASSED** — no attribute in `WHEN_FAILED` |
|
||||
| Power-on hours | 6,814 (~284 days) · 147 power cycles |
|
||||
| **Rated life used** | **31%** (`202`), i.e. 69% remaining (`173` VALUE 069) |
|
||||
| **Grown failing blocks** | **15** (`170`), reallocation events **15** (`196`) |
|
||||
| **Erase failures** | **187** (`172`) |
|
||||
| **Unexpected power losses** | **32** (`174`) |
|
||||
| Pending / offline-uncorrectable / reported-uncorrectable | **0 / 0 / 0** (`197`/`198`/`187`) |
|
||||
| UDMA CRC errors | 0 (`199`) — cabling is clean |
|
||||
|
||||
**Read attribute `5` carefully.** Its raw value is `61440`, which is NOT a bad-sector
|
||||
count: `61440 = 15 x 4096`, and both `170` and `196` independently read **15**. Its
|
||||
normalised VALUE is 100 against a THRESH of 10. So the drive has **15 grown bad blocks**,
|
||||
not sixty-one thousand bad sectors. The first `node_health` run FAILED on that raw value;
|
||||
the check now judges on normalised VALUE/THRESH, `WHEN_FAILED`, and the vendor-independent
|
||||
uncorrectable counters instead.
|
||||
|
||||
**Verdict: usable, but not for production state.** Nothing says imminent failure — no
|
||||
pending or uncorrectable sectors, SMART self-assessment passes. But this is a ~2011
|
||||
consumer MLC drive that has already consumed **31% of its rated life**, grown **15** bad
|
||||
blocks, logged **187** erase failures, and taken **32** unclean power losses **with no
|
||||
power-loss protection**, in a single failure domain. etcd is the worst possible tenant for
|
||||
it: etcd fsyncs constantly, and an unclean power loss on a drive without PLP is exactly how
|
||||
a member corrupts — this drive has already logged 32 of them.
|
||||
(Non-ECC RAM is a separate, **accepted** risk — operator decision 2026-09-01, see
|
||||
`DECISIONS-LOG.md`. It is not part of this verdict.)
|
||||
- [ ] **Decide the layout.**- [ ] **Decide the layout.** `roles/node_baseline` expects *block devices*
|
||||
(`node_baseline_longhorn_device`, `node_baseline_etcd_device`) and makes a filesystem on them.
|
||||
Its defaults `/dev/vdb` and `/dev/vdc` are **virtio names that do not exist on bare metal** —
|
||||
prod must override them in `group_vars`. On a single-disk node there is no second device at
|
||||
all, so one of:
|
||||
- **Add a second drive** (M.2 NVMe if the chassis has a free slot; verify physically — only a
|
||||
SATA controller is populated today). Cleanest: matches dev's isolation, gives etcd its own
|
||||
spindle.
|
||||
- **Use `/dev/sda5`** (the 433 GB `/srv` partition) as the Longhorn device, etcd stays on `/`.
|
||||
Buildable today, zero spend, but re-creates the shared-spindle contention dev was burned by.
|
||||
- **Repartition** into dedicated OS / etcd / Longhorn partitions. Better isolation, still one
|
||||
physical failure domain.
|
||||
- [ ] **All three nodes get the identical layout.** Longhorn replica placement assumes symmetry.
|
||||
- [ ] Record the chosen device paths here and set them in
|
||||
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` (they are deliberately
|
||||
`/dev/BLOCKED-pending-disk-decision-…` placeholders today — do **not** guess). A full
|
||||
`node-baseline` run against a node still carrying those placeholders now stops on an explicit
|
||||
assert naming this section, instead of failing obscurely at `mkfs` time.
|
||||
- [x] **The SMART-vs-smartmontools circular dependency is broken** (2026-08-31). `roles/node_baseline`
|
||||
is tagged `packages` / `system` / `disks`, so the package floor can land **without** the disk
|
||||
tasks this decision blocks — see §5's drive-health box for the exact command.
|
||||
|
||||
## 3. OS install (per node)
|
||||
|
||||
- [ ] **Debian 13 (trixie)**, matching dev exactly — same kernel series, same package baseline.
|
||||
- [ ] Hostname `k3s-prod-N`.
|
||||
- [ ] Partition per the §2 decision.
|
||||
- [ ] **No cloud-init needed.** Dev's first-boot work was deliberately moved into
|
||||
`roles/node_baseline` when provisioning became API-only (`infra/cloud-init/` is a README now),
|
||||
so the same role covers bare metal. Nothing to port.
|
||||
|
||||
## 4. Network (per node)
|
||||
|
||||
- [ ] **Static IP** on the prod LAN, recorded in §0. Node 1 is `192.168.2.10/24`, gateway
|
||||
`192.168.2.1`.
|
||||
- [x] **Prod address plan — DECIDED (operator, 2026-08-31).**
|
||||
|
||||
| Role | Address | Owned by |
|
||||
|---|---|---|
|
||||
| `k3s-prod-1` / `-2` / `-3` | `192.168.2.10` / `.11` / `.12` | Ansible inventory |
|
||||
| **kube-vip control-plane VIP** | `192.168.2.9` | Ansible (`group_vars/k3s_prod`) |
|
||||
| **Ingress Gateway VIP** (`GATEWAY_VIP`) | `192.168.2.20` | Flux (`clusters/prod`) |
|
||||
| **Cilium L2 announcement pool** | `192.168.2.20` – `.30` | Flux (`platform-config/cilium`) |
|
||||
|
||||
Two deliberate properties, both mirroring dev:
|
||||
- The **control-plane VIP sits outside the LB pool** (`.9`, below the nodes), so kube-vip and
|
||||
Cilium LB-IPAM can never contend for the same address. Dev achieves this the other way
|
||||
round (`.49` control-plane, pool `.53-.60`).
|
||||
- **`GATEWAY_VIP` is the FIRST address of the pool.** Cilium LB-IPAM can only allocate from a
|
||||
defined pool block, so a Gateway VIP outside every pool would not be allocatable — the pool
|
||||
was widened from `.21-.30` to `.20-.30` for exactly this reason. Dev has the same shape
|
||||
(`GATEWAY_VIP` `.53` = first of `.53-.60`).
|
||||
- [ ] **Reserve all of `192.168.2.9` and `.20-.30` outside any DHCP range** on the prod LAN before
|
||||
the build, and confirm nothing else on the network already answers on them.
|
||||
- [ ] **Resolver: SECOND UPSTREAM IS AN OPEN DECISION — blocks `node-baseline.yml --tags dns`.**
|
||||
Node 1 has a single IPv4 nameserver, `192.168.2.1`, which is also the LAN gateway. A single
|
||||
upstream is a known SPOF that has already bitten this project — it broke containerd image
|
||||
pulls and CoreDNS forwarding on dev.
|
||||
|
||||
The MECHANISM is now in the repo and needs no further work: the `dns`-tagged block in
|
||||
`roles/node_baseline` detects whether a node uses systemd-resolved (dev: writes
|
||||
`/etc/systemd/resolved.conf.d/99-k3s-dns.conf`, restarts resolved) or a directly-managed
|
||||
`/etc/resolv.conf` (prod's shape: writes the file, refusing if resolvconf/NetworkManager
|
||||
owns it), and configures the ordered upstream list either way. The earlier
|
||||
`fix/node-dns-resilient-upstream` branch was **dropped, not merged** — it wrote a netplan
|
||||
drop-in and ran `netplan apply`, and netplan is not installed on these Debian 13 trixie
|
||||
nodes, so it would have broken `node-baseline.yml` on every host.
|
||||
|
||||
What is still MISSING is the VALUE. `group_vars/k3s_prod/vars.yml` sets
|
||||
`node_baseline_dns_servers` to `192.168.2.1` followed by the deliberately invalid
|
||||
`BLOCKED-pending-second-upstream-decision-see-PROD-PROVISIONING-md-4`, and the role's assert
|
||||
stops the play on it before writing anything. Choosing the second upstream is a **network
|
||||
decision, not a default the role may take**: a public resolver (`8.8.8.8`, `1.1.1.1`) would
|
||||
resolve reliably but would also bypass any local zone and any filtering the gateway applies.
|
||||
Decide between — and record the choice HERE, in the same change that edits the group_vars:
|
||||
- a second on-LAN resolver on `192.168.2.x`;
|
||||
- the dev-side PiHole `192.168.1.18`, **if** prod can actually route to it (unverified —
|
||||
prod sits on the workstation side of the pfSense boundary);
|
||||
- an explicit, written-down acceptance of a public resolver as fallback-only.
|
||||
|
||||
Also decide whether node 1's current IPv6 nameservers (`2001:4860:4860::8888` / `::8844`)
|
||||
are kept: the direct-file write path replaces `/etc/resolv.conf` with exactly the listed
|
||||
servers, so keeping them means listing them explicitly.
|
||||
`node_baseline_dns_search` is `[]` (no search domain, node 1's behaviour today) until prod's
|
||||
local domain is decided.
|
||||
- [ ] Confirm the prod LAN's relationship to the dev LAN (`192.168.1.x`) and to the pfSense NAT —
|
||||
prod on `192.168.2.x` sits on the *workstation* side of the boundary dev lives behind.
|
||||
|
||||
## 5. Access (per node) — **blocks all Ansible**
|
||||
|
||||
- [ ] **`k3s` user with the `home_network` SSH key.** Working on node 1.
|
||||
- [ ] **Password-protected sudo, driven by Ansible `become` + a vaulted password.**
|
||||
**Prod does NOT use NOPASSWD sudo** (operator decision, 2026-08-31). Dev nodes do; prod
|
||||
deliberately keeps the sudo password in place and supplies it to Ansible as an
|
||||
**ansible-vault-encrypted** var, exactly like the existing `hypervisor` and `proxmox_hosts`
|
||||
groups:
|
||||
```
|
||||
inventory/group_vars/k3s_prod/
|
||||
README.md how to (re)create the vault — committed
|
||||
vault.yml ansible_become_password: "..." — committed, ENCRYPTED
|
||||
```
|
||||
The passphrase that decrypts it lives **outside the repo** at
|
||||
`~/.config/home-k3s/ansible-vault-pass` (`ansible.cfg :: vault_password_file`), so plays still
|
||||
run unattended without a plaintext secret ever touching git.
|
||||
```bash
|
||||
ansible-vault create inventory/group_vars/k3s_prod/vault.yml # ansible_become_password: ...
|
||||
```
|
||||
Prod nodes must therefore set `ansible_become: true` in their group_vars and must **not**
|
||||
inherit any passwordless-sudo assumption from `group_vars/all.yml`.
|
||||
- [ ] **Drive health baseline** (§2). **Do not hand-install the tooling** — package installation is
|
||||
the `node_baseline` role's job, never a manual `apt-get` on a node (operator ruling,
|
||||
2026-08-31; same click-ops-never principle as the rest of the repo). `smartmontools` is part
|
||||
of `node_baseline_packages`, so it arrives with the baseline play — but a *full* baseline
|
||||
would hit the disk tasks §2 has not decided yet. Run the **packages-only** baseline instead
|
||||
(the `disks`-tagged tasks, including the BLOCKED-device assert, are not selected at all, so
|
||||
the placeholder device paths are never even templated):
|
||||
```bash
|
||||
cd infra/ansible
|
||||
ansible-playbook playbooks/node-baseline.yml \
|
||||
-e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
|
||||
```
|
||||
(Needs `group_vars/k3s_prod/vault.yml` to exist first — the become password above.) Reading
|
||||
the drive is then:
|
||||
```bash
|
||||
sudo smartctl -a /dev/sda # Power_On_Hours, Percent_Lifetime_Remaining, reallocated sectors
|
||||
```
|
||||
Record the result in §2 and close the layout decision in the same change that replaces the
|
||||
two `node_baseline_*_device` placeholders.
|
||||
|
||||
## 6. Ansible baseline & cluster join
|
||||
|
||||
Reuses the dev roles **unchanged** — that reuse **is** R-ENV-02. Only inventory and group_vars
|
||||
differ, plus a target-group override on the command line.
|
||||
|
||||
### How prod is targeted (done, 2026-08-31)
|
||||
|
||||
Every node-host play used to hardcode the literal dev groups `k3s_servers` / `k3s_primary` /
|
||||
`k3s_secondary`, so nothing here could select prod at all. Two knobs now close that, deliberately
|
||||
kept separate because Ansible resolves them at different times:
|
||||
|
||||
| Layer | Knob | Set via |
|
||||
|---|---|---|
|
||||
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts` (each defaults to the dev group) | `-e` **extra vars only** — a play's `hosts:` is templated before any host exists, so `group_vars` is not readable there |
|
||||
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group`, `cluster_kubeconfig_name` | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml`; **no per-run flag** |
|
||||
|
||||
So `roles/k3s_server` finds the prod `--cluster-init` founder, builds the prod join target and
|
||||
tls-san list, and `roles/cilium` fetches the prod kubeconfig, with **no role fork**. Running with
|
||||
no extra vars is byte-for-byte the dev behaviour it always had (verified by a before/after
|
||||
`--list-hosts` diff on every playbook). Full detail: `infra/ansible/README.md` → **Targeting prod**.
|
||||
|
||||
- [x] Add a `k3s_prod` inventory group (`k3s_prod_primary` / `k3s_prod_secondary`). Prod is reached
|
||||
**directly** on the LAN — no NAT port-forward hop, unlike dev's `192.168.2.102:225x` mapping.
|
||||
(Nodes 2/3 stay commented out until they are physically on the bench.)
|
||||
- [x] `inventory/group_vars/k3s_prod/vars.yml`: `kube_vip_address` (`192.168.2.9`),
|
||||
`cluster_controller_api_server`, `k3s_api_host`, the group-name overrides above — **and** the
|
||||
§2 device overrides for `node_baseline_longhorn_device` / `node_baseline_etcd_device`, which
|
||||
are still BLOCKED placeholders.
|
||||
- [ ] `inventory/group_vars/k3s_prod/vault.yml` — the vaulted `ansible_become_password` (§5). Until
|
||||
it exists **no** prod play can escalate, and `--check` against prod cannot work either.
|
||||
- [ ] Run the **packages-only** baseline first, to unblock the §2 drive-health read:
|
||||
```bash
|
||||
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
|
||||
```
|
||||
- [ ] Run the **full** `playbooks/node-baseline.yml -e k3s_hosts=k3s_prod` → packages, kernel
|
||||
modules, sysctls, iscsid, data-disk filesystems + mounts. **Blocked until §2 is decided**: it
|
||||
asserts on the placeholder device paths and stops. (`qemu-guest-agent` is a no-op on bare
|
||||
metal — the role already only *starts* it when the virtio channel exists, so it degrades
|
||||
cleanly.)
|
||||
- [ ] Run `playbooks/k3s.yml -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`
|
||||
→ HA embedded etcd, flannel/kube-proxy/Traefik/servicelb disabled. **Only once all three
|
||||
nodes are present** — see §8.
|
||||
- [ ] Run `playbooks/kube-vip.yml -e k3s_hosts=k3s_prod` → control-plane VIP (`192.168.2.9`) on the
|
||||
prod LAN.
|
||||
- [ ] Run `playbooks/cilium.yml -e k3s_hosts=k3s_prod` → CNI at bootstrap; Flux's HelmRelease adopts
|
||||
it afterwards. The bootstrap values and the HelmRelease **must stay in parity**
|
||||
(`kubeProxyReplacement`, `k8sServiceHost`/`Port`) or the two layers fight over CNI config
|
||||
(`docs/02`). Note the prod kubeconfig is fetched to `.cache/prod.kubeconfig`, never over the
|
||||
dev one the dev container exports as `$KUBECONFIG`.
|
||||
- [ ] Run `playbooks/verify.yml -e k3s_hosts=k3s_prod`.
|
||||
- [ ] `playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod` **retargets but is not yet correct**:
|
||||
`roles/flux_bootstrap` shells out to `bootstrap/flux/bootstrap.sh`, which bootstraps
|
||||
`--path=clusters/dev`. Prod needs `--path=clusters/prod` — a `bootstrap/` change, not a
|
||||
target-group change. Tracked with the other repo blockers in §7.
|
||||
- [ ] Decide whether prod runs a VPN egress gateway node (dev uses `k3s-dev-3`, ADR-0015).
|
||||
`playbooks/vpn.egress.gateway.yml` takes `-e vpn_egress_gateway_hosts=<group>`, but there is
|
||||
deliberately no prod egress group in the inventory yet, and it would need its **own** vaulted
|
||||
PIA credentials — the existing vault is scoped to the dev group.
|
||||
|
||||
## 7. Repo blockers — **prod cannot be reconciled until these land**
|
||||
|
||||
`clusters/prod` is design-only today and **would build wrong**. R-ENV-02 promises dev and prod differ
|
||||
*only* by `cluster-vars` values; in practice dev LAN addresses are hardcoded into base manifests:
|
||||
|
||||
| Hardcoded value | Where | Prod needs |
|
||||
|---|---|---|
|
||||
| `k8sServiceHost: 192.168.1.50` | `platform/cilium/helmrelease.yaml:50`, mirrored in `values.yaml:16` | prod primary node IP |
|
||||
| LB pool `192.168.1.53–.60` | `platform-config/cilium/l2-ip-pool.yaml:15` | a prod-LAN pool |
|
||||
| `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
|
||||
| `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
|
||||
| `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |
|
||||
|
||||
- [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
|
||||
overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
|
||||
header comment in `clusters/base/cluster-vars.yaml`). `platform`, `platform-config`, the app
|
||||
waves, `apps-vms` and `crafty` all already carry `postBuild.substituteFrom: cluster-vars`, so
|
||||
the seam exists.
|
||||
- [ ] **`clusters/prod` must set `GATEWAY_VIP`.** It currently doesn't, so prod would inherit the
|
||||
base default `10.73.0.240` — an address from the **retired libvirt network**. The base file's
|
||||
own comment flags this as owed "when hardware is built."
|
||||
- [ ] **Dev overlay sets every new knob to today's exact value** so rendered dev output is
|
||||
byte-identical and the live cluster sees zero drift. Verify with `make -C ci render-golden`.
|
||||
- [ ] **`clusters/prod` is NOT bootstrappable as-is** — three gaps, found 2026-09-01 while landing
|
||||
the `dependsOn` fix. Only the first is now closed:
|
||||
- [x] **Root Kustomization** — `clusters/prod/sync.yaml` authored, mirroring dev. This became
|
||||
*load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries
|
||||
`dependsOn: [flux-system]`, so without a root object on prod, `platform` would block
|
||||
forever on a dependency that does not exist and nothing downstream would reconcile.
|
||||
- [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key
|
||||
as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev
|
||||
exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster
|
||||
(`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only.
|
||||
- [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision
|
||||
2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and
|
||||
will be decommissioned once prod is stable. Recorded consequence: once prod is live,
|
||||
**every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window.
|
||||
The URL is patched in but **cannot work on a cold prod cluster**, for two independent
|
||||
reasons found 2026-09-01:
|
||||
1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for
|
||||
decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod
|
||||
loses its source of truth. **This is the decommission plan's first task, not its last.**
|
||||
2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire
|
||||
`home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`:
|
||||
prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve
|
||||
`gitea.home.keithjackson.dev` to itself and never reach a repository.
|
||||
Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a
|
||||
cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own
|
||||
Phase-6 cutover.
|
||||
- [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
|
||||
the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).
|
||||
|
||||
## 8. Build sequence (operator decision, 2026-08-31: **wait for all three nodes**)
|
||||
|
||||
Node 1 is baselined now; the cluster is **not** initialized until nodes 2 and 3 are on the bench.
|
||||
This keeps prod spec-conformant (3-node HA embedded etcd) from its first boot, avoids a
|
||||
single-server-etcd window, and avoids a later member-add dance.
|
||||
|
||||
1. Per-node §1–§5 on each of the three boxes.
|
||||
2. §7 repo work lands on `main` with the gate green.
|
||||
3. `node-baseline` → `k3s` (all three) → `kube-vip` → `cilium`.
|
||||
4. `flux bootstrap ... --path=clusters/prod`.
|
||||
5. Conformance against prod (§9).
|
||||
|
||||
## 9. Exit criteria (Phase 7)
|
||||
|
||||
- [ ] Prod runs the same config as dev via the prod overlay, **no base edits** (R-ENV-02).
|
||||
- [ ] Node destroy/rebuild from IaC + Ansible without data loss (R-ENV-01, R-STATE-01).
|
||||
- [ ] Full **automated** prod DR drill (R-STATE-03) — `definition-of-done.md` requires this for prod
|
||||
specifically.
|
||||
- [ ] Monitoring and alerts live on prod.
|
||||
- [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).
|
||||
+1
-1
@@ -20,7 +20,7 @@ Proxmox LAN and are reached across a pfSense NAT:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50/.51/.52` — 6 vCPU, 24 GiB, Debian 13, k3s v1.36.4 |
|
||||
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50/.51/.52` — 6 vCPU, 16 GiB, Debian 12, k3s v1.31.5 |
|
||||
| kube-API | kube-vip VIP `192.168.1.49:6443`, reached from the workstation via NAT `192.168.2.50:6443` |
|
||||
| Node SSH | pfSense NAT `192.168.2.102:2250/2251/2252` → node `:22` (Ansible only) |
|
||||
| Ingress VIP | `192.168.1.53` (Cilium L2, shared Gateway); `.53–.60` reserved for LB VIPs |
|
||||
|
||||
@@ -1,13 +1,3 @@
|
||||
# Runtime cache written by the cilium/verify/flux plays (fetched kubeconfig,
|
||||
# rendered Cilium values). Never commit — the kubeconfig is a credential.
|
||||
.cache/
|
||||
|
||||
# Generated node-health reports (roles/node_health, playbooks/node-health.yml).
|
||||
# Regenerated on every run and they carry drive serial numbers — never commit. Same
|
||||
# convention as conformance/_artifacts/: generated output is an artifact, not source.
|
||||
_artifacts/
|
||||
|
||||
# Galaxy collections installed by `ansible-galaxy collection install -r
|
||||
# collections/requirements.yml -p collections/`. Pinned by requirements.yml, fetched at
|
||||
# setup time — vendored bytes are not source. ansible.cfg sets collections_path.
|
||||
collections/ansible_collections/
|
||||
+3
-115
@@ -4,9 +4,8 @@ The "Cluster-join" node lifecycle stage (docs/02 stage 3) and the Flux hand-off
|
||||
(T-1.4 call-in). Installs a 3-server HA k3s cluster (embedded etcd), Cilium as the
|
||||
CNI, verifies readiness, then invokes the `bootstrap/` entrypoints.
|
||||
|
||||
> **LIVE.** These plays have been executed against the running dev cluster (k3s
|
||||
> v1.36.4+k3s1, 3 nodes). This is no longer author-only scaffolding — treat it as the
|
||||
> operational bring-up + re-provision runbook. See **Re-provisioning a node** below.
|
||||
> **AUTHOR-ONLY — NOTHING HERE HAS BEEN EXECUTED.** No `ansible-playbook` run, no
|
||||
> `tofu apply`, no cluster contacted. The PM static-validates (see below).
|
||||
|
||||
Serves **R-ENV-01/02**, **R-PLAT-01**, ADR-0001 (k3s), ADR-0002 (Cilium),
|
||||
ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
|
||||
@@ -23,40 +22,14 @@ ADR-0004 (Flux hand-off), ADR-0007 (SOPS).
|
||||
## Run order
|
||||
```
|
||||
cd infra/ansible
|
||||
ansible-playbook playbooks/site.yml # core bring-up, in order (baseline→k3s→cilium→verify→flux)
|
||||
ansible-playbook playbooks/site.yml # everything, in order
|
||||
# — or step by step —
|
||||
ansible-playbook playbooks/node-baseline.yml # 0. packages, kernel modules, Longhorn disk, swap-off
|
||||
ansible-playbook playbooks/k3s.yml # 1. HA k3s (primary --cluster-init, then joins)
|
||||
ansible-playbook playbooks/cilium.yml # 2. Cilium CNI (cluster is NotReady until this)
|
||||
ansible-playbook playbooks/verify.yml # 3. gate #1: 3 nodes Ready, Hubble up, no kube-proxy
|
||||
ansible-playbook playbooks/flux-bootstrap.yml # 4. hand off to bootstrap/ (SOPS then Flux)
|
||||
```
|
||||
|
||||
> **`site.yml` is NOT everything.** It covers the core bring-up only. Two node-host
|
||||
> layers live OUTSIDE `site.yml` and must be run separately:
|
||||
> ```
|
||||
> ansible-playbook playbooks/kube-vip.yml # control-plane VIP (192.168.1.49); defaults to k3s_servers
|
||||
> ansible-playbook playbooks/vpn.egress.gateway.yml # node-host OpenVPN split-tunnel + kill-switch; defaults to k3s-dev-3
|
||||
> ```
|
||||
> (Both take the same target-group override as the rest — see **Targeting prod** below.)
|
||||
|
||||
## Re-provisioning a node (tofu -replace / fresh OS)
|
||||
A node rebuild wipes **everything on the host filesystem** — including host-level config
|
||||
that Flux/Helm do NOT manage. After `tofu apply`/`-replace` re-creates a node, re-run the
|
||||
node-host layers or the node comes back missing pieces:
|
||||
|
||||
| Re-run | Why it doesn't survive a rebuild |
|
||||
|---|---|
|
||||
| `node-baseline.yml` | packages (incl. `cryptsetup`), kernel modules, Longhorn disk, swap-off |
|
||||
| `k3s.yml` | the k3s server binary + config + join (secondaries re-join the VIP) |
|
||||
| `kube-vip.yml` | the kube-vip auto-deploy manifest on the server's host path |
|
||||
| `vpn.egress.gateway.yml` | **the OpenVPN (PIA) client, `tun0`, systemd units + nftables kill-switch on k3s-dev-3 — miss this and `egress-class=vpn` pods (sonarr/radarr/qbittorrent) are kill-switch-dropped with no internet** |
|
||||
|
||||
Everything Flux owns (platform + apps) self-heals once the node re-joins; only the
|
||||
node-host layers above need a manual re-run. Verify the VPN after: on k3s-dev-3,
|
||||
`systemctl is-active vpn-egress-tunnel.service` and `curl --interface tun0 https://ipinfo.io/ip`
|
||||
should return a PIA exit IP (not the home ISP).
|
||||
|
||||
## What each layer does
|
||||
| Playbook / role | Action |
|
||||
|---|---|
|
||||
@@ -89,91 +62,6 @@ expose different entrypoints, override the two vars in
|
||||
`roles/flux_bootstrap/defaults/main.yml`; the task graph is unchanged. Flagged
|
||||
for PM reconciliation.
|
||||
|
||||
## Targeting prod (R-ENV-02: same playbooks, same roles, scoped overrides)
|
||||
|
||||
`inventory/hosts.ini` carries a `[k3s_prod_primary]` / `[k3s_prod_secondary]` /
|
||||
`[k3s_prod:children]` section for the three planned bare-metal nodes (`k3s-prod-1` at
|
||||
`192.168.2.10` exists and is reachable today; `k3s-prod-2`/`-3` at `.11`/`.12` are addressed but
|
||||
not yet racked, so their inventory lines stay commented out), with its own `group_vars/k3s_prod/`
|
||||
(vaulted `ansible_become_password` + non-secret overrides).
|
||||
|
||||
**Nothing here is forked for prod.** Every node-host play takes its target group from a variable
|
||||
that defaults to the dev group, so the dev commands above are unchanged, and prod is the same
|
||||
command plus an override. Two separate mechanisms, for two separate reasons:
|
||||
|
||||
| Layer | Knob | Set via | Why |
|
||||
|---|---|---|---|
|
||||
| Which hosts a play **runs on** | `k3s_hosts`, `k3s_primary_hosts`, `k3s_secondary_hosts`, `vpn_egress_gateway_hosts` | `-e` **extra vars only** | Ansible templates a play's `hosts:` *before* it has any host, so it cannot see `group_vars` at all |
|
||||
| Which group a **role** means by "my primary" / "my servers" | `k3s_primary_group`, `k3s_servers_group` (+ `cluster_kubeconfig_name`) | `group_vars` — dev in `all.yml`, prod in `k3s_prod/vars.yml` | Host context exists by then, so each environment answers for itself with **no per-run flag** — same mechanism `k3s_api_host` already used |
|
||||
|
||||
`playbooks/node-health.yml` established the first pattern (`-e node_health_hosts=k3s_prod`); it
|
||||
keeps its own var name because it is a standalone diagnostic you point anywhere, not part of the
|
||||
bring-up chain.
|
||||
|
||||
```bash
|
||||
# dev — completely unchanged, no extra vars
|
||||
ansible-playbook playbooks/site.yml
|
||||
|
||||
# prod — one flag for the whole-cluster plays
|
||||
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
|
||||
ansible-playbook playbooks/cilium.yml -e k3s_hosts=k3s_prod
|
||||
ansible-playbook playbooks/kube-vip.yml -e k3s_hosts=k3s_prod
|
||||
ansible-playbook playbooks/verify.yml -e k3s_hosts=k3s_prod
|
||||
ansible-playbook playbooks/flux-bootstrap.yml -e k3s_hosts=k3s_prod # see caveat below
|
||||
|
||||
# prod — k3s.yml needs both halves of the primary/secondary split
|
||||
ansible-playbook playbooks/k3s.yml \
|
||||
-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
|
||||
|
||||
# prod — the whole chain (extra vars are global, so they reach imported playbooks)
|
||||
ansible-playbook playbooks/site.yml \
|
||||
-e k3s_hosts=k3s_prod \
|
||||
-e k3s_primary_hosts=k3s_prod_primary \
|
||||
-e k3s_secondary_hosts=k3s_prod_secondary
|
||||
```
|
||||
|
||||
What the roles then resolve **by themselves** for a prod host, from `group_vars/k3s_prod/vars.yml`:
|
||||
`k3s_primary_group=k3s_prod_primary` (the `--cluster-init` founder, the node-token `delegate_to`
|
||||
target, and the node Cilium fetches the kubeconfig from), `k3s_servers_group=k3s_prod` (the tls-san
|
||||
node-IP list), `k3s_api_host=192.168.2.10`, `kube_vip_address=192.168.2.9`, and
|
||||
`cluster_kubeconfig_name=prod.kubeconfig` — so a prod run can never overwrite the
|
||||
`.cache/dev.kubeconfig` that `infra/devcontainer/dev.sh` exports as `$KUBECONFIG`.
|
||||
|
||||
### Packages-only baseline (`--tags packages`)
|
||||
`roles/node_baseline` is tagged `packages` / `system` / `disks`. An untagged run is unchanged (all
|
||||
three, in order). `--tags packages` runs **only** the apt task:
|
||||
|
||||
```bash
|
||||
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages --limit k3s-prod-1
|
||||
```
|
||||
|
||||
This exists to break a circular dependency: prod's disk layout (`infra/PROD-PROVISIONING.md` §2) is
|
||||
blocked on a SMART read, SMART needs `smartmontools`, `smartmontools` ships in
|
||||
`node_baseline_packages` — but the role's disk tasks need block devices that decision hasn't chosen
|
||||
yet. With `--tags packages` no `disks`-tagged task is selected, so the deliberate
|
||||
`/dev/BLOCKED-pending-disk-decision-…` placeholders in `group_vars/k3s_prod/vars.yml` are never
|
||||
even templated, let alone handed to `mkfs`. A **full** run against a node whose devices are still
|
||||
placeholders stops on an explicit assert naming §2 instead of failing obscurely at mkfs time. Never
|
||||
hand-install packages on a node — that is this role's job (operator ruling, 2026-08-31).
|
||||
|
||||
### Still not runnable end to end
|
||||
- The **disk-layout** (§2) and the `node_baseline_*_device` overrides remain BLOCKED placeholders.
|
||||
- `group_vars/k3s_prod/vault.yml` (the become password) does not exist yet, so **no** prod play can
|
||||
escalate — and `--check` against prod fails for the same reason.
|
||||
- `flux-bootstrap.yml` retargets, but `roles/flux_bootstrap` shells out to
|
||||
`bootstrap/flux/bootstrap.sh`, which bootstraps `--path=clusters/dev`. Prod needs
|
||||
`--path=clusters/prod`: a `bootstrap/` change, not a target-group change (§7).
|
||||
- `vpn.egress.gateway.yml` is parameterised (`-e vpn_egress_gateway_hosts=…`) but prod has **no**
|
||||
egress group yet — whether prod runs one at all is an open §6 decision, and it would need its own
|
||||
vaulted PIA credentials.
|
||||
- The operator has ruled the prod cluster stays **uninitialized** until all three nodes are present
|
||||
(§8). `192.168.2.20` is prod's *ingress* Gateway VIP — a Flux/`cluster-vars` concern, not an
|
||||
Ansible one; the kube-vip **control-plane** VIP is `192.168.2.9` (`kube_vip_address`).
|
||||
|
||||
Everything above this section describes the **dev** runbook; the values in it (`192.168.1.x`,
|
||||
`.cache/dev.kubeconfig`, `clusters/dev`) are dev's, supplied by `group_vars`, not baked into the
|
||||
roles.
|
||||
|
||||
## Static validation (PM, in dev container — NOT run here)
|
||||
```
|
||||
yamllint -c ../../ci/yamllint.yaml infra/ansible
|
||||
|
||||
@@ -1,10 +1,6 @@
|
||||
[defaults]
|
||||
inventory = inventory/hosts.ini
|
||||
roles_path = roles
|
||||
# Galaxy collections live in the repo (gitignored), installed from
|
||||
# collections/requirements.yml. The devcontainer image does NOT ship them, so without
|
||||
# this every play dies at parse time on community.general.modprobe.
|
||||
collections_path = collections
|
||||
host_key_checking = False
|
||||
retry_files_enabled = False
|
||||
# SSH key the k3s nodes authorize for the `k3s` user (see inventory ansible_user).
|
||||
|
||||
@@ -4,35 +4,12 @@
|
||||
# cloud-init created this login user (tofu var login_user); it has NOPASSWD sudo.
|
||||
ansible_user: k3s
|
||||
|
||||
# --- environment group NAMES (R-ENV-02: one set of roles, scoped overrides) ---------
|
||||
# The GROUP NAMES a role uses to answer "who is MY environment's primary / MY
|
||||
# environment's servers?". Roles must never hardcode `groups['k3s_primary']`, or they
|
||||
# only ever work against dev. These two are the dev values; group_vars/k3s_prod/vars.yml
|
||||
# overrides them with k3s_prod_primary / k3s_prod, so a host in the prod group resolves
|
||||
# its own primary automatically — exactly the way k3s_api_host below is overridden there.
|
||||
#
|
||||
# These are GROUP NAMES (keys into `groups`), NOT host patterns. The play-level target
|
||||
# pattern is a separate, extra-vars-only knob (`k3s_hosts` / `k3s_primary_hosts` /
|
||||
# `k3s_secondary_hosts`, see playbooks/*.yml) because Ansible templates a play's `hosts:`
|
||||
# before it has any host context and so cannot see group_vars at all.
|
||||
k3s_primary_group: k3s_primary
|
||||
k3s_servers_group: k3s_servers
|
||||
|
||||
# Filename of the kubeconfig the controller-side roles (cilium, cluster_verify,
|
||||
# flux_bootstrap) fetch from the primary into infra/ansible/.cache/. Scoped per
|
||||
# environment so pointing those plays at prod can NEVER overwrite the dev kubeconfig
|
||||
# that infra/devcontainer/dev.sh exports as $KUBECONFIG — an operator's `kubectl get
|
||||
# nodes` silently answering from prod would be a genuinely dangerous mix-up.
|
||||
# group_vars/k3s_prod/vars.yml overrides this to prod.kubeconfig. (.cache/ is gitignored;
|
||||
# a kubeconfig is a credential.)
|
||||
cluster_kubeconfig_name: dev.kubeconfig
|
||||
|
||||
# The kube-apiserver endpoint the whole stack agrees on: the primary's REAL node IP.
|
||||
# NOTE: ansible_host is now the pfSense NAT SSH target (192.168.2.102:225x), NOT the node
|
||||
# IP — so we read the real 192.168.1.x from k3s_node_ip instead. Used for k3s --tls-san /
|
||||
# secondary --server AND for Cilium k8sServiceHost so the Ansible bootstrap and the Flux
|
||||
# HelmRelease stay in PARITY (docs/02 contract).
|
||||
k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['k3s_node_ip'] }}"
|
||||
k3s_api_host: "{{ hostvars[groups['k3s_primary'][0]]['k3s_node_ip'] }}"
|
||||
k3s_api_port: 6443
|
||||
|
||||
# Control-plane HA VIP floated by kube-vip (roles/kube_vip). A stable failover
|
||||
|
||||
@@ -1,68 +0,0 @@
|
||||
# group_vars for the `k3s_prod` group (prod k3s nodes — Phase 7 PREP ONLY)
|
||||
|
||||
Every node-host play that touches a k3s node (`playbooks/node-baseline.yml`, `k3s.yml`,
|
||||
`cilium.yml`, `kube-vip.yml`, ...) already runs with `become: true` at the play level. On **dev**
|
||||
that succeeds without a password (cloud-init's `login_user` gets NOPASSWD sudo). **Prod
|
||||
deliberately does NOT get that** — operator decision, 2026-08-31 (`DECISIONS-LOG.md`, "PHASE 7
|
||||
OPENED", finding 3; `infra/PROD-PROVISIONING.md` §5). Supply the sudo password as an
|
||||
**ansible-vault-encrypted** var here — it is encrypted, so it is safe to commit; the passphrase
|
||||
that decrypts it stays out of the repo, exactly like the existing `hypervisor` and `proxmox_hosts`
|
||||
groups.
|
||||
|
||||
## Two files (one committed-encrypted, one never committed)
|
||||
|
||||
1. **The vault passphrase** — a file **outside the repo**, referenced by `ansible.cfg`
|
||||
(`vault_password_file = ~/.config/home-k3s/ansible-vault-pass`). Skip this if you already made
|
||||
one for the `hypervisor` or `proxmox_hosts` vaults — it's the SAME passphrase file, shared by
|
||||
every vaulted group in this repo:
|
||||
```bash
|
||||
mkdir -p ~/.config/home-k3s
|
||||
install -m600 /dev/stdin ~/.config/home-k3s/ansible-vault-pass <<<'YOUR_VAULT_PASSPHRASE'
|
||||
```
|
||||
|
||||
2. **The encrypted become password** — `vault.yml` in **this directory** (committed, encrypted).
|
||||
`vault.yml.example` alongside it shows the exact shape (see it for the reasoning); copy its
|
||||
content, fill in the real password, then encrypt it in place — do not commit the `.example`
|
||||
content unencrypted as `vault.yml`:
|
||||
```bash
|
||||
cd infra/ansible
|
||||
ansible-vault create inventory/group_vars/k3s_prod/vault.yml
|
||||
# file content (single line):
|
||||
# ansible_become_password: "THE k3s USER'S SUDO PASSWORD ON k3s-prod-1"
|
||||
```
|
||||
(`ansible-vault create` uses the passphrase file above automatically — no prompt.)
|
||||
|
||||
Once `vault.yml` exists, prod plays run unattended — same as dev, just with a real sudo boundary
|
||||
behind them instead of NOPASSWD.
|
||||
|
||||
## Notes
|
||||
- `vault.yml` is skipped by the yamllint gate (`ci/yamllint.yaml` ignores `**/vault.yml`) because a
|
||||
vault blob is not parseable YAML. `vault.yml.example` is plain YAML and IS linted — it's a
|
||||
copy-paste scaffold, not a secret (same convention as
|
||||
`platform/backup/minio-credentials.sops.yaml.example`).
|
||||
- The k3s **dev** nodes keep passwordless sudo unchanged — this vault is scoped to the `k3s_prod`
|
||||
group only.
|
||||
- `ansible_become: true` for this group is set explicitly in `vars.yml` (this directory) — belt
|
||||
and braces alongside the playbooks' own `become: true`, so prod's posture is self-documenting
|
||||
and doesn't rely on inheriting anything from `group_vars/all.yml` by accident.
|
||||
- Non-secret prod vars live in `vars.yml`, not here — this README is scoped to the vault hand-off.
|
||||
`k3s_api_host` is set to a real value. `kube_vip_address`, `cluster_controller_api_server`,
|
||||
`node_baseline_longhorn_device` and `node_baseline_etcd_device` are all still-BLOCKED
|
||||
placeholders pending operator decisions (control-plane VIP and disk layout, respectively) — see
|
||||
`vars.yml`'s comments for what IS already decided (node addresses, the separate *ingress* VIP,
|
||||
which is a Flux/`cluster-vars` concern and does not live here) versus what is not.
|
||||
- This is the Ansible-bootstrap-layer secret (host credential), distinct from the platform's
|
||||
in-cluster **SOPS + age** secrets (ADR-0007) — different layer, different tool, same convention
|
||||
as `hypervisor`/`proxmox_hosts`.
|
||||
- **Targeting prod is solved** (2026-08-31): plays take `-e k3s_hosts=k3s_prod` (and, for `k3s.yml`,
|
||||
`-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`), while the
|
||||
roles resolve "my environment's primary/servers" from `k3s_primary_group` / `k3s_servers_group`
|
||||
in `vars.yml` here — no role fork, no per-run flag for that half. See
|
||||
`infra/ansible/README.md` → "Targeting prod".
|
||||
- **Do not run any prod playbook yet.** Phase 7 is prep-only: the vault above just makes the plays
|
||||
*able* to authenticate once the §7 repo blockers close and the disk-layout (§2) decision in
|
||||
`infra/PROD-PROVISIONING.md` is made. The ONE exception the operator has sanctioned is the
|
||||
packages-only baseline that unblocks the §2 drive-health read —
|
||||
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages` — which
|
||||
cannot touch the BLOCKED device placeholders because every disk task is `disks`-tagged and so is
|
||||
not selected. It still needs `vault.yml` (above) to exist first.
|
||||
@@ -1,172 +0,0 @@
|
||||
---
|
||||
# group_vars/k3s_prod/vars.yml — NON-secret prod-scoped overrides for the [k3s_prod]
|
||||
# umbrella group (inventory/hosts.ini: [k3s_prod_primary]/[k3s_prod_secondary] under
|
||||
# [k3s_prod:children]). Ansible auto-loads group_vars/<groupname>/*.yml for every host
|
||||
# that is a (transitive) member of that group. See infra/PROD-PROVISIONING.md (T-7.1)
|
||||
# §5/§6 for the checklist this file implements, and DECISIONS-LOG.md "PHASE 7 OPENED"
|
||||
# (2026-08-31) for the operator decisions behind it.
|
||||
#
|
||||
# Pairs with vault.yml in this same directory (ansible-vault-encrypted, operator-
|
||||
# created — see README.md here) for the one secret prod needs: ansible_become_password.
|
||||
#
|
||||
# PHASE 7 IS PREP ONLY. Five vars below (kube_vip_address, cluster_controller_api_server,
|
||||
# node_baseline_longhorn_device, node_baseline_etcd_device, and the SECOND entry of
|
||||
# node_baseline_dns_servers) are deliberately BROKEN
|
||||
# PLACEHOLDERS, not real values — see their comments. Do not "fix" them with a guess;
|
||||
# follow the referenced section of PROD-PROVISIONING.md, record the decision there,
|
||||
# and update this file in the same change.
|
||||
|
||||
# --- become posture (§5) ----------------------------------------------------------
|
||||
# Prod deliberately does NOT get dev's passwordless (NOPASSWD) sudo — operator
|
||||
# decision, 2026-08-31 (DECISIONS-LOG.md "PHASE 7 OPENED", finding 3). group_vars/
|
||||
# all.yml carries no explicit `ansible_become` of its own (dev's passwordless sudo
|
||||
# means become succeeds without a password prompt, not that become is switched off),
|
||||
# and every node-host playbook already sets `become: true` at the play level
|
||||
# (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml) — for dev that silently
|
||||
# succeeds because sudo needs no password there. Setting ansible_become explicitly
|
||||
# here is belt-and-braces: this group's posture is self-documenting and does not rely
|
||||
# on inheriting dev's assumption by accident.
|
||||
ansible_become: true
|
||||
# The password itself is ansible_become_password — ansible-vault-encrypted, in
|
||||
# vault.yml (this directory), never here in plaintext. See README.md to create it.
|
||||
|
||||
# --- environment group NAMES (R-ENV-02) -------------------------------------------
|
||||
# group_vars/all.yml defines these as the DEV group names (k3s_primary / k3s_servers);
|
||||
# here they name PROD's groups. Roles index `groups[k3s_primary_group]` /
|
||||
# `groups[k3s_servers_group]` instead of a literal, so roles/k3s_server and roles/cilium
|
||||
# resolve "my environment's primary" and "my environment's servers" correctly for prod
|
||||
# hosts with NO role fork and no per-run flag — the reuse R-ENV-02 promises.
|
||||
#
|
||||
# NOTE these are GROUP NAMES, not the play-level target pattern. Selecting which hosts a
|
||||
# play RUNS against is a separate extra-vars knob (`-e k3s_hosts=k3s_prod`, and for
|
||||
# k3s.yml `-e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary`),
|
||||
# because Ansible templates `hosts:` before any host — and therefore any group_vars —
|
||||
# exists. See infra/ansible/README.md "Targeting prod".
|
||||
k3s_primary_group: k3s_prod_primary
|
||||
k3s_servers_group: k3s_prod
|
||||
|
||||
# Keep the fetched prod kubeconfig in its OWN cache file. group_vars/all.yml defaults this
|
||||
# to dev.kubeconfig, which is the file infra/devcontainer/dev.sh exports as $KUBECONFIG —
|
||||
# a prod play must never clobber it (a `kubectl` that silently answers from prod is worse
|
||||
# than no kubeconfig at all). .cache/ is gitignored; a kubeconfig is a credential.
|
||||
cluster_kubeconfig_name: prod.kubeconfig
|
||||
|
||||
# --- k3s API host — safe to set now, the primary's real IP is already known -------
|
||||
# Dev's k3s_api_host (group_vars/all.yml) reads a separate k3s_node_ip hostvar because
|
||||
# dev's ansible_host is the pfSense NAT SSH target, not the node's real address. Prod
|
||||
# has NO NAT hop — ansible_host in hosts.ini IS already the node's real, routable LAN
|
||||
# address (see the KEY DIFFERENCE note there) — so point straight at it instead of
|
||||
# needing a k3s_node_ip split.
|
||||
k3s_api_host: "{{ hostvars[groups[k3s_primary_group][0]]['ansible_host'] }}"
|
||||
# cluster_controller_tls_server_name (group_vars/all.yml) derives from kube_vip_address
|
||||
# by template reference, so it needs no separate override here — it picks up the real
|
||||
# value set below automatically for prod hosts too.
|
||||
|
||||
# --- control-plane VIP + controller access — DECIDED (§4) -------------------------
|
||||
# Prod addressing, all decided by the operator 2026-08-31:
|
||||
# nodes k3s-prod-1/2/3 192.168.2.10 / .11 / .12 (hosts.ini)
|
||||
# control-plane VIP 192.168.2.9 (THIS value — kube-vip, Ansible-side)
|
||||
# ingress Gateway VIP 192.168.2.20 (Flux/cluster-vars — NOT here)
|
||||
# ingress L2 pool 192.168.2.20 - .30 (Flux/cluster-vars — NOT here)
|
||||
#
|
||||
# The control-plane VIP sits BELOW the node addresses and deliberately OUTSIDE the
|
||||
# Cilium LB pool (.20-.30), so kube-vip and Cilium LB-IPAM can never contend for the
|
||||
# same address. That separation is the point: dev learned the same shape the other way
|
||||
# round (control-plane .49, ingress pool .53-.60).
|
||||
#
|
||||
# This is the address kube-vip floats for the API and that k3s tls-san's onto its
|
||||
# serving cert; secondaries use it as their --server join target. Dev's equivalent is
|
||||
# kube_vip_address: 192.168.1.49 in group_vars/all.yml.
|
||||
#
|
||||
# HISTORY: an earlier pass here wrongly set this to 192.168.2.20 — that is the INGRESS
|
||||
# VIP, not the control-plane VIP. Corrected. Keep the two distinct.
|
||||
kube_vip_address: "192.168.2.9"
|
||||
# Derived from kube_vip_address above (not repeated), so fixing the one line above is
|
||||
# the only change needed once a real control-plane VIP is chosen. Prod has no NAT hop
|
||||
# (unlike dev's cluster_controller_api_server, the dev-container's NAT path to its VIP
|
||||
# — see group_vars/all.yml), so once real this collapses to a direct "https://<VIP>:6443".
|
||||
cluster_controller_api_server: "https://{{ kube_vip_address }}:6443"
|
||||
|
||||
# --- node_baseline data-disk devices — BLOCKED, open decision (§2) ----------------
|
||||
# roles/node_baseline defaults (node_baseline_longhorn_device: /dev/vdb,
|
||||
# node_baseline_etcd_device: /dev/vdc) assume tofu-provisioned virtio disks that DO
|
||||
# NOT EXIST on bare metal. Node 1 is a SINGLE SATA disk (/dev/sda) with no free block
|
||||
# device at all today; PROD-PROVISIONING.md §2 lists three candidate fixes (add a
|
||||
# second drive / repartition / reuse the /dev/sda5 "/srv" partition) that all wait on
|
||||
# a SMART health read of a ~15-year-old consumer SSD (operator decision, 2026-08-31).
|
||||
#
|
||||
# The overrides below are deliberately present-but-broken: leaving them UNSET would
|
||||
# let the role silently fall back to /dev/vdb//dev/vdc, which ALSO don't exist on this
|
||||
# hardware — a confusing failure that looks like a wiring bug. This value fails just
|
||||
# as hard, but says exactly why, and blocks a `find`/glob from accidentally matching
|
||||
# a real device. DO NOT replace with a guess — replace only once §2 is decided, and
|
||||
# record the chosen device paths in PROD-PROVISIONING.md §2 in the SAME change that
|
||||
# updates these two lines. All three nodes must end up with an identical layout
|
||||
# (Longhorn replica placement assumes symmetry) before any of this is used for real.
|
||||
node_baseline_longhorn_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
|
||||
node_baseline_etcd_device: "/dev/BLOCKED-pending-disk-decision-see-PROD-PROVISIONING-md-2"
|
||||
|
||||
# --- node_baseline DNS upstreams — SECOND UPSTREAM BLOCKED, open decision (§4) -----
|
||||
# roles/node_baseline defaults are DEV's list (192.168.1.18 PiHole, then 192.168.1.1
|
||||
# pfSense) — addresses on a network prod cannot even reach. Inherited verbatim they would
|
||||
# point every prod node at a PiHole that does not answer it, so they are overridden here,
|
||||
# in the same seam as kube_vip_address / k3s_api_host / cluster_kubeconfig_name above.
|
||||
# (NOT clusters/base/cluster-vars.yaml: that ConfigMap is consumed by Flux
|
||||
# postBuild.substituteFrom for platform/ and apps/ manifests, and no Ansible code reads it.)
|
||||
#
|
||||
# WHAT IS DECIDED: 192.168.2.1 — node 1's current sole nameserver, and the LAN gateway.
|
||||
# It stays FIRST, so whatever local zone it serves keeps working (the dev outage taught
|
||||
# that ordering is policy, not preference — see the role's defaults).
|
||||
#
|
||||
# WHAT IS NOT: the SECOND upstream. 192.168.2.1 is presently gateway AND only IPv4
|
||||
# resolver, and there is no second address on this LAN that is known to answer. Picking
|
||||
# one is a NETWORK decision, not a default this role may take: a public resolver
|
||||
# (8.8.8.8, 1.1.1.1) would resolve reliably but would ALSO bypass any local zone and any
|
||||
# network-level filtering the gateway applies — precisely the bypass the dev fix refused
|
||||
# to make. Candidates the operator must choose between and record in
|
||||
# infra/PROD-PROVISIONING.md §4: a second on-LAN resolver, the dev-side PiHole IF prod can
|
||||
# route to 192.168.1.18, or an explicit, written-down decision to accept a public resolver
|
||||
# as fallback-only.
|
||||
#
|
||||
# Deliberately present-but-broken, exactly like node_baseline_longhorn_device above:
|
||||
# leaving the list at one entry, or unset, would either configure the very SPOF this exists
|
||||
# to remove or silently inherit dev's PiHole. This value can never parse as an address, and
|
||||
# the `dns`-tagged assert in roles/node_baseline stops the play on the string BLOCKED
|
||||
# before a single file is written. DO NOT replace it with a guess — replace it only once
|
||||
# §4 is decided, and record the chosen resolver there in the SAME change.
|
||||
#
|
||||
# Note the two Google IPv6 servers currently in node 1's /etc/resolv.conf
|
||||
# (2001:4860:4860::8888/::8844) are NOT carried over: the direct-file write path replaces
|
||||
# the file with exactly this list, so if they are to be kept they must be listed here
|
||||
# explicitly — as part of that same recorded decision.
|
||||
node_baseline_dns_servers:
|
||||
# DECIDED (operator, 2026-09-01): prod's resolvers are PUBLIC and deliberately differ
|
||||
# from dev's. Dev is PiHole-first (192.168.1.18) because dev needs the local jackson.lan
|
||||
# zone AND split-horizon *.home.keithjackson.dev. Prod has no local resolver on its LAN,
|
||||
# so it takes two INDEPENDENT public upstreams — which resolves the single-upstream SPOF
|
||||
# (a single upstream broke containerd image pulls and CoreDNS forwarding on dev).
|
||||
- "8.8.8.8"
|
||||
- "1.1.1.1"
|
||||
# SCOPE — this file sets NODE resolution only: containerd image pulls, and the upstream
|
||||
# CoreDNS forwards `.` to. Three different resolvers answer *.home.keithjackson.dev, and
|
||||
# only ONE of them is configured here:
|
||||
#
|
||||
# pods (in-cluster) -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template`
|
||||
# plugin answers the WHOLE home.keithjackson.dev zone with ${GATEWAY_VIP}. Entirely
|
||||
# self-contained in the cluster; PiHole is NOT in this path. Prod is already correct
|
||||
# (GATEWAY_VIP = 192.168.2.20 in clusters/prod).
|
||||
# LAN clients -> PiHole. Not a cluster concern.
|
||||
# nodes -> THIS FILE. Public resolvers, so the zone resolves PUBLICLY.
|
||||
#
|
||||
# THE ONE THING THAT BREAKS: the k3s API SERVER is a node process and carries
|
||||
# `oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab`
|
||||
# (roles/k3s_server/tasks/main.yml). It reaches the issuer through the NODE resolver, so on
|
||||
# prod it would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl
|
||||
# OIDC SSO. Dev avoids this because its node resolver is PiHole, which answers the zone
|
||||
# with the dev Gateway VIP.
|
||||
#
|
||||
# NOT blocking today (prod has no cluster and no Keycloak yet). Options, in
|
||||
# PROD-PROVISIONING.md §4: a narrow Ansible-managed /etc/hosts entry pinning
|
||||
# keycloak.home.keithjackson.dev -> 192.168.2.20 on prod nodes; or a local resolver on the
|
||||
# prod LAN serving the zone; or run prod on the cert kubeconfig and defer OIDC.
|
||||
node_baseline_dns_search: []
|
||||
@@ -1,7 +0,0 @@
|
||||
$ANSIBLE_VAULT;1.1;AES256
|
||||
33643331386138666338353162353562613737343265376639646262663931663239333735366163
|
||||
3961623239313038313966326339336464383763616266310a313766653231636636393030306635
|
||||
37653764656235363738363535386539613938393839623462316134383631623464343930383133
|
||||
6433386532653564630a383036633266353965383565666639363733393531643162366265366537
|
||||
30313265343035653961666331663534313564333939383038306338666535316639333136363036
|
||||
6532313164613836663862303561646362303130316136306361
|
||||
@@ -1,25 +0,0 @@
|
||||
# =============================================================================
|
||||
# TEMPLATE — NOT ENCRYPTED, NOT LOADED (a plain `.example` file; Ansible only reads
|
||||
# `vault.yml` in this directory, never `vault.yml.example`). Reference scaffold +
|
||||
# OPERATOR HAND-OFF, same convention as platform/backup/minio-credentials.sops.yaml.example.
|
||||
#
|
||||
# WHAT THIS IS: the sudo (become) password for the `k3s` user on prod k3s nodes.
|
||||
# Prod deliberately does NOT use passwordless (NOPASSWD) sudo the way dev does —
|
||||
# operator decision, 2026-08-31 (DECISIONS-LOG.md "PHASE 7 OPENED", finding 3;
|
||||
# infra/PROD-PROVISIONING.md §5). Every play that becomes root on a k3s_prod host
|
||||
# (node-baseline.yml, k3s.yml, cilium.yml, kube-vip.yml, ...) needs this to escalate.
|
||||
#
|
||||
# HAND-OFF STEPS (operator):
|
||||
# (a) Create the vault passphrase file ONCE, outside the repo, if you don't already
|
||||
# have one from the `hypervisor`/`proxmox_hosts` vaults (it's the same file,
|
||||
# shared by every vaulted group in this repo):
|
||||
# mkdir -p ~/.config/home-k3s
|
||||
# install -m600 /dev/stdin ~/.config/home-k3s/ansible-vault-pass <<<'YOUR_VAULT_PASSPHRASE'
|
||||
# (b) Encrypt the REAL content into inventory/group_vars/k3s_prod/vault.yml — never
|
||||
# commit this .example file's content unencrypted as vault.yml itself:
|
||||
# cd infra/ansible
|
||||
# ansible-vault create inventory/group_vars/k3s_prod/vault.yml
|
||||
# # paste the single line below with the REAL password, save, exit
|
||||
# (c) Verify: ansible-vault view inventory/group_vars/k3s_prod/vault.yml
|
||||
# =============================================================================
|
||||
ansible_become_password: "REPLACE-ME"
|
||||
@@ -45,57 +45,3 @@ k3s-dev-3
|
||||
# `hypervisor` group above (that is `localhost` for host-prereqs.yml, a different machine).
|
||||
[proxmox_hosts]
|
||||
proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh_private_key_file=~/.ssh/home_network
|
||||
|
||||
# =============================================================================
|
||||
# PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1,
|
||||
# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). Node 1
|
||||
# (`k3s-prod-1`, HP EliteDesk 800 G5 Mini) is on the bench and SSH-reachable; nodes 2
|
||||
# and 3 have not arrived. This section is inventory/group_vars scaffolding only —
|
||||
# nothing has been run against node 1, and the operator has decided the CLUSTER stays
|
||||
# uninitialized (no k3s install, no --cluster-init) until all three nodes are present
|
||||
# (decision #2 in the DECISIONS-LOG entry above) — no single-server-etcd window, no
|
||||
# later member-add dance.
|
||||
#
|
||||
# KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT
|
||||
# hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x)
|
||||
# that port-forwards to the node's real 192.168.1.x address, which is why dev also
|
||||
# carries a separate `k3s_node_ip` per host. Prod needs no such split: `ansible_host`
|
||||
# below IS already the node's real, routable address, so `k3s_node_ip` is set to the
|
||||
# SAME value (kept explicit, not omitted, only so the k3s_server role's hostvar lookup
|
||||
# — see the GAP note below — has something correct to find once that gap is closed).
|
||||
# `ansible_port` is the node's own real sshd (22). See group_vars/k3s_prod/vars.yml for
|
||||
# the k3s_api_host override that reflects this directly-reachable topology.
|
||||
#
|
||||
# GROUP-NAME GAP: CLOSED 2026-08-31. The roles and playbooks used to hardcode the
|
||||
# LITERAL dev groups (`hosts: k3s_servers` / `k3s_primary` / `k3s_secondary`, and
|
||||
# `groups['k3s_primary']` inside roles/k3s_server + roles/cilium), so nothing could
|
||||
# select the k3s_prod_* groups below. Both halves are now variables, deliberately kept
|
||||
# separate because Ansible resolves them at different times:
|
||||
# * WHICH HOSTS A PLAY RUNS ON — `k3s_hosts` (whole-cluster plays), `k3s_primary_hosts`
|
||||
# / `k3s_secondary_hosts` (k3s.yml's split), each defaulting to the dev group.
|
||||
# EXTRA VARS ONLY: a play's `hosts:` is templated before any host exists, so
|
||||
# group_vars cannot be read there. Prod: `-e k3s_hosts=k3s_prod` etc.
|
||||
# * WHICH GROUP A ROLE MEANS BY "MY PRIMARY"/"MY SERVERS" — `k3s_primary_group` /
|
||||
# `k3s_servers_group`, set in group_vars/all.yml (dev) and group_vars/k3s_prod/vars.yml
|
||||
# (prod), so roles resolve their own environment with NO per-run flag — the same
|
||||
# mechanism k3s_api_host already used.
|
||||
# Running with no extra vars is byte-for-byte the dev behaviour it always had. See
|
||||
# infra/ansible/README.md "Targeting prod" and PROD-PROVISIONING.md §6.
|
||||
# =============================================================================
|
||||
|
||||
[k3s_prod_primary]
|
||||
k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
|
||||
|
||||
[k3s_prod_secondary]
|
||||
# Addresses for nodes 2/3 ARE decided (operator, 2026-08-31 — infra/PROD-PROVISIONING.md
|
||||
# §0), but the nodes themselves are NOT yet racked/imaged/reachable. Left commented so
|
||||
# no play can accidentally target a host that isn't there; uncomment each line only once
|
||||
# that node is actually on the bench and answering SSH.
|
||||
# k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
|
||||
# k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
|
||||
|
||||
# Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
|
||||
# pattern; named to match the group_vars directory so Ansible auto-loads it).
|
||||
[k3s_prod:children]
|
||||
k3s_prod_primary
|
||||
k3s_prod_secondary
|
||||
Loaded 100 of 175 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user